Well, it really depends on how you define formal. Formal to some means a full suit, conservative shoes and tie. Sounds to me like you want to do something less than that, perhaps more business casual.
If business casual “done right” is your goal, I think looking at this GQ feature will be up your alley. While I certainly wouldn’t go and necessarily copy everything verbatim (especially the brands they recommend), the images are a good inspirational starting point to pull ideas from.
Still though, I’d say your big hangup seems to be that you’re worried about sticking out among those around you. It seems like you’re afraid to “dress up” because others around you constantly dress down — strangers, coworkers, friends, etc. That’s something I can’t really help you with.
You have to have the self confidence to wear what you want to wear despite what the world will think of you. I don’t see anything wrong about being different — especially if it makes you feel good about yourself and is honest to who you are and want to be.
The one thing I’ve come to realize is that you cannot — and never will — completely control a person’s final opinion of you. You can’t change someone’s mind and make everyone happy by constantly doing things that make them feel comfortable and pleased. You can only control how you feel and what you do — and people can take it or leave it.
Whenever I go out, I almost universally “dress up” with a jacket, tie and pocket square. It makes almost no difference to me who I am with or where I’m going. I want to wear what makes me feel the most confident. When people ask why I’m “dressed up”, I just tell them that this is what I wear and that I like to look nice.
You won’t ever elevate your self-confidence if you’re constantly carrying the baggage of everyone’s opinions.