There is a biomechanical aspect to this question, and an evolutionary one.
The biomechanical answer is that the virus will fit with a certain type of cell, like a lock and key, an analogy which is often used for other forms of chemical specificity like enzymes and substrates. The virus is usually coated with a layer of proteins, called the capsid. Part of the function of the capsid is to simulate the signaling effect that the host cells are capable of recognizing. I usually use the analogy of a fake ID; if the bouncer sees an ID with the right information, he lets you in. Unfortunately the bouncer is more intelligent than the cell; he can make subjective judgments, but the cell can't. If it receives the right chemical signal, it lets the virus in, because it doesn't know any better.
Part of cell differentiation is the establishment of different proteins and different protein signals. Thus, if a virus has the correct signaling protein for cell type A, it probably won't be able to infect cell B.
The evolutionary aspect to this answer comes from the fact that infecting specific cells lends a competitive advantage. Note that viruses are capable of evolving, despite not being alive.
Assume we have three types of virus; one of which can infect cell A, one which can infect cell B, and one that can infect both A and B. The only way this is possible is by carrying the signaling proteins for both cells A and B. This could be accomplished by evenly splitting the proteins on the capsid between A-type and B-type, but that would make the virus only half as effective at infecting either cell compared to the viruses that only infect type A or B cells.
Alternately, the virus could be larger, to make more room in the capsid for a greater number of A and B signaling proteins, but this may be a disadvantage as well; being larger would make it easier to target, as well as causing fewer viruses to be produced per infection.