The answer would vary from reader to reader, apropos to historical period and religious and cultural sensibility (for example, the audience in Shakespeare's time would probably interpret Shylock's obstinacy to another cause than today's audience) but, on the most simple level, it is because Shylock is a member of an oppressed group. Even though he is a rich moneylender, he lives in a society controlled completely by Christians. And while Venice in the sixteenth century was somewhat more tolerant than other societies were toward Jews (and certainly more tolerant than 1930s Germany, for example, over three hundred years later in what was supposed to be a more enlightened time), the Venice of Shylock and Antonio tolerated rather than embraced members of Jewish faith and heritage. The truth is that Shylock is vengeful because all the people around him despise him. He is used for his moneylending (Jews were tolerated in Christian Europe because they, as non-Christians, were able to lend money at interest -- called "usury" by the Bible -- but for Christians like Antonio it was considered a sin to lend at interest like Shylock did -- see Act I) but he was spit upon and insulted by Christians. Even Antonio says that he is proud to insult a Jew; it was considered a good thing for a Christian to do. Shylock says:
[Shylock:] Signior Antonio, many a time and oft,(105)
In the Rialto you have rated me
About my moneys, and my usances:
Still have I borne it with a patient shrug,
For sufferance is the badge of all our tribe:
You call me,—misbeliever, cut-throat dog,(110)
And spet upon my Jewish gaberdine, (I.iii)
and Antonio does not deny it. Antonio, and many others, see nothing wrong with persecuting a Jew. Effectively, the Christian society in which Shylock lives dehumanizes him. This finally gets to Shylock, and makes him act in the vindictive way he does. He makes no excuses for his cruelty (just as Antonio makes no excuses for his own cruelty):
[Shylock:] You'll ask me, why I rather choose to have
A weight of carrion flesh, than to receive
Three thousand ducats: I'll not answer that:
But, say, it is my humour. is it answer'd?
What, if my house be troubled with a rat(45)
And I be pleas'd to give ten thousand ducats
To have it ban'd? What, are you answer'd yet?
Some men there are love not a gaping pig;
Some, that are mad if they behold a cat;
And others, when the bagpipe sings i' the nose,(50)
Cannot contain their urine: for affection,
Master of passion, sways it to the mood
Of what it likes, or loathes. Now, for your answer.
As there is no firm reason to be render'd,
Why he, cannot abide a gaping pig;(55)
Why he, a harmless necessary cat;
Why he, a woollen bagpipe,—but of force
Must yield to such inevitable shame,
As to offend himself, being offended;
So can I give no reason, nor I will not,(60)
More than a lodged hate, and a certain loathing,
I bear Antonio, that I follow thus
A losing suit against him. Are you answer'd? (IV.i)
Shakespeare was wont to make his villains sometimes inscrutable in their motives (such as Iago in Othello, or Don John in Much Ado About Nothing), and in doing this he allows the audience to attribute motives to the characters themselves. This makes a play have staying power. The anti-Semitic audience of the Globe in London in the 1590s would have thought it normal for a Jew to be evil; the modern audience can see how Shylock was made into a monster by his oppressors.