If you are asking if the Lord punish us if we reject a calling, I don't think so. He is fully aware of our situation and circumstances and He judge us accordingly.
As long as your reason is good and you consulted with the Lord in prayer then I think you're fine. The problem starts when we make excuses that really aren't that important.