Each of them suffers from both pride and prejudice, of course. Elizabeth's pride is not necessarily just a consequence of her position in life, but that's certainly part of why she is so defensive about money and class. Her prejudice shows from the very beginning--and it's not just directed at Mr. Darcy. She has no more time for her mother's and sisters' empty-headed foolishness than she does for her pretentious rich neighbors.
Darcy's prejudice is not necessarily against those who lack wealth and social position--it's against a certain lack of class (the other kind of class) often, in his experience, found among the country poor. We understand the source of his feelings as Wickham's consistent abuse of their relationship (using Darcy for his money or position or both) is revealed throughout the course of the story.
Certainly there is pride and there is prejudice. Clearly, though, there is also misunderstanding and hasty misjudgments. And both of them are guilty.