i think if you are happy to stop then do it, there's no point in worrying what other people think or want.
you could even go back to exclusive bf if you wanted to (i don't think you do!) but if you feel its time to go on to formula then go for it. i definitely agree its almost definitely to do with something you are eating which is upsetting him (more likely caffeine, chocolate, dairy, citrus or spicy food). maybe he is ready for a stage 2 teat?????!