Most religions define 'holy' and 'evil' and 'good' and 'bad', how does your religion or belief system define what is 'holy'?
This is my take on it, it may or may not be in line with the teachings of my religion, but from what I've seen and studied it is.
In my religion the principle focus is becoming like our heavenly father, and returning to live with him. So things which are evil or bad would be those things that get in the way of those goals. So even though something may not affect anyone other than yourself, they are still wrong because they get in the way of your progression.