I think freedom is an important heuristic for happiness / well-being, but I don't think its absence makes all else meaningless.
If, for example, someone locked me in a room and tortured me, I think it would be meaningful to me if they stopped torturing me.
Freedom, therefore, is more of a means to an end. What's actually important is that we get what we want (i.e. find happiness / comfort).
But since, generally speaking, nobody else can know better what we want than we do, freedom is often the best guarantee that we won't be deprived of what we want when we most want it. It's always risky to give up our freedoms, because we often don't know in advance what options may be important to us at a later point in time.
But it's ultimately a balance.
For example, someone who lives in an apartment might be willing to give up some of their own freedom of doing jumping jacks at midnight, with the expectation that their own upstairs neighbor won't wake them up every night. But you wouldn't want to disallow all upstairs thumping, lest you be expected by your downstairs neighbor to tiptoe around all the time.
We can use similar logic for vaccines. We wouldn't want to force people to receive every vaccine that comes on the market. But for an especially dangerous disease, a vaccine mandate can be the most effective way to protect everyone - especially people who, because they're too young or immunocompromised, can't get the vaccine themselves.
"Live and let live" is certainly a good rule of thumb, and freedom is certainly a good strategy for enabling us to "get what we want". But "getting what we want" (which you might simply call "being happy") is still, I think, the only inherently meaningful end goal.