I think if you read the picking a necktie series I just wrapped up, then you’ll have a good idea. Adding a sweater vest is just another layer. I would seek to contrast hues and colors when adding the V-neck.
I don’t think you can go wrong with a burgundy or red V-neck sweater though for both navy and grey blazers. Neutrals in charcoal, grey and navy will also work, although I would avoid pairing those colors with similar colored jackets.