I think there are probably a few reasons for it, one is that people may feel that religion is going to get pushed on to them whether they like it or not, or they feel that the other person is too religious and they might feel that the "religious" person might be watching them all the time to wait for them to do something wrong so they can preach at them on how they should be living their life.
Another reason that people don't like about religious people is that they might be labelled the same, even though they may be an atheist, just by association.
It's all pretty sad really, as each person should be taken on their own merit regardless of race or religion. There are good and bad in all faiths, religions, and non believers.