I guess it depends if you’re a fan of brass buttons (ie: like a traditional navy blazer would have), how much you’re willing to spend in total and what you define as casual.
Brooks Brothers’ “blazer” section has got a whole selection that’s pretty much what you’d expect. I think the most casual one is the Social Primer blazer, because of the patch pockets and the fun lining and contrast fabric under the collar.
But you could also maybe go with the Cambridge blazer, cut in a sack-fit, 3/2-roll and non-brass buttons. More of sport coat, in my opinion. No patch pockets though.
You could also wait a little bit longer and see what comes out for the full spring-summer line. You’ll likely find a few jackets in more casual fabrics (cotton, linen, etc.), although who knows if they’ll be more odd jackets than blazers.
Keep in mind that with all the blazers, you can always buy a button set and swap out the brass if you feel like they make the blazer too formal for you.