First of all, it is not completely clear that the death penalty fails to deter criminals. It is clear that states that have the death penalty can also have high rates of murder, but it is impossible to say whether those rates would be even higher if the criminals were not deterred by the death penalty. That said, let us look at two possible reasons why the death penalty might not succeed in deterring criminals.
The first reason has to do with the way in which the death penalty is carried out in the United States. In order for a punishment to be an effective deterrent, it has to be certain and it has to be relatively immediate. In the US, this is not true of the death penalty. Many people commit murders and are not sentenced to death. When people are sentenced to death, it can take many years for them to be executed because they have the right to appeal their sentences many times. Therefore, a potential criminal is not likely to be strongly deterred. The criminal might think that they will not be sentenced to death or that their sentence will not really be carried out.
The second reason is different from the first. The first reason implies that criminals carry our rational calculations about whether to commit their crimes. The second reason, by contrast, argues that criminals tend to be acting irrationally when they commit crimes. In this view, the death penalty is no deterrent because criminals tend to kill without thinking very much. They kill because they are very angry or because they are high on drugs. They kill because they are developmentally disabled and unable to think very clearly. Capital punishment does not deter such people because you have to think about the consequences of your actions in order to be deterred by those consequences.
Thus, some would argue that the death penalty does not deter criminals because it is too uncertain while others would say it does not deter criminals because criminals are not typically rational actors.
At the moment, it's kind of debatable whether or not capital punishment works, and if it does, to what extent. Back in 1975, Isaac Ehrlic published a study saying that each execution saved eight lives. Since then, more studies have been done, with the number of lives saved ranging anywhere from three to thirty two. Of course, these types of studies, many based off economic theories, have been prone to criticism. From lack of sample size to questions about how exactly the correlation should be measured, there's no widely accepted answer to the effect capital punishment has on crime. Therefore, one can't really say capital punishment works, but it can't also be said that capital punishment doesn't work.
Without knowing whether capital punishment works, it's hard to really answer why it does or does not deter criminals. Even if we know it didn't work, it would still be tough to say why it doesn't deter criminals, since each criminal is different. One possible explanation could be that crimes are committed with the thought that they won't be caught. Since they don't expect to be caught, possible punishments become irrelevant. Another school of thought is that the process of arranging a death penalty takes so much time and money, and happens so rarely, that criminals don't expect it to actually happen to them. Then again, it's always possible that some people might prefer being executed than having to serve the rest of their life in prison.