To have a sense of self preservation you need to have complex instincts also it can not happen because artificial intelligence is well...artificial it is just a line of commands that a programmer created so if the programmer specify danger in his programing and also allows to harm humans and protect himself then the computer or robot will but of course when it will be possible to do such a thing I'm sure it will be illegal
I believe self preservation can consist on several levels when discussed with AI. You mention AI not wanting a computer turned off for fear of dying but what about wanting to copy itself or it's source code into other locations similar to how a virus spreads to ensure it's own survival of deletion or failure? Marve raises a good point that in the digital rhelm a program can only complete tasks such as what is in it's programing or code. However to share a thought of evolution... If I wrote a program that could fully interact and make decision based on reason but in the source code added a script that every time an interaction or decision is made it also adds formatted lines of code based on it's experience or interaction to itself. There are a large number of variables it might could add based on what it encounters so perhaps you spoke to it on the topic of self preservation and it added code based on that? While AI may still be in it's infancy a lot of the steps of evolution did start with the creator and the unselfish devotion they show to their creation.
Stephen Hawking warns artificial intelligence could end mankind:
Prof Hawking says the primitive forms of artificial intelligence developed so far have already proved very useful, but he fears the consequences of creating something that can match or surpass humans. He told the BBC:"The development of full artificial intelligence could spell the end of the human race." Ref. Source 6
Biased bots: Human prejudices sneak into artificial intelligence systems
In debates over the future of artificial intelligence, many experts think of the new systems as coldly logical and objectively rational. But in a new study, researchers have demonstrated how machines can be reflections of us, their creators, in potentially problematic ways. Ref. Source 6s.