I had a religious upbringing, and used to be afraid of atheists - until I became one.

The reason why I was afraid of them was because I was afraid of the concept of atheism. The existence of atheists (or even the possibility of atheists) forced me to consider how tenuous my own belief in God was. And I was afraid to not believe in God because my whole family and social structure and outlook on life was based on a belief in God. And it's scary to imagine that a world without God will be okay if you've lived your life thinking otherwise.

It's not a fear of bodily harm. It's an existential crisis.