No matter what, someone will always have something to say about your choices, so either way, walk your way and be happy...
If you are struggling with this, know one thing: "people have something nasty to say about everyone," including Mother Theresa. One of my mom's favorite sayings: la gente siempre mira la paja de ajena y no la propia. People will always look through someone else's garbage and not their own.
Several years ago, I came across this amazing passage by poet and historian Jennifer Michael Hecht: "Either life means something or it means nothing. Either way you need to relax to get through the panic and get in on the game. Most of all--if you can remind yourself--in stable and unstable times, that change is the nature of things you will rise and be able to help others. ( I add, help yourself!) It goes on: "Only don't think that if you are perfectly generous and kind and wise you will escape the derision of humanity. People have something nasty to say about everyone". Even Mother Theresa.
We owe it to ourselves to live life with grace, always remembering to honor our internal intuition.