@Joe,
Ideas aren't the problem.
It's the fact that we don't do a very good job distinguishing an idea in the abstract from people in the world.
For instance, we can make certain assumptions about a theist. Or an atheist. But whatever that assumption, the fact of the matter is that we NEVER really know what goes on inside people's heads, regardless of their professed beliefs. And in practice, plenty of acts of horror and plenty of acts of virtue have been committed by theists, atheists, fundamentalist Christians, fundamentalist Muslims, pagans, wiccans, Democrats, Republicans, Americans, etc.
Only in extremely rare cases should people LOSE the benefit of the doubt because of a group identity, and this happens when the group's identity and practice are sufficiently small and homogeneous that there's no escaping it. For instance, I believe it IS fair to ask questions about people like Martin Heiddeger and Richard Strauss who belonged to the Nazi party.
On the other hand, I don't think it's fair to ask such questions about people merely by virtue of atheism or theism -- the inherent ideas are just way too nonspecific.