Saying no is for a lot of us not so easy, and then not only for ourselves but also not to harm others or we are not sure how others would react if you say no.
No is a powerful and also a loving word, in times like this no has a negative attitude in my opinion but it can be out of love it depends how you say no.
What I mean by loving is that no can be used in self protection, warnings and to take care of others.

You don't need to protect yourself if you say no even if people start to ask questions. It is your right to do so. But saying yes and no should be in balance, saying to much yes causes that people ask you a lot of things wich can feel draining to you. Saying no too much causes that people take a step away.

But if you feel it is good to say no, say no without any doubt.