I’d try and go more casual with the jackets. Tweed in the fall and winter. Washed cotton or linen in the summer. You could also opt to wear cardigan (either fine merino knits or chunky shawl collared wool) or V-neck sweaters.
I think the key is to dress more on the casual/country style than formal/city style.