Why do Mercutio and Benvolio think that Romeo avoids them after the party?
Romeo's two friends assume that Romeo wants to be alone. Mercutio states in Act 2, scene 1:
He is wise;
And, on my life, hath stol'n him home to bed.
Benvolio, though, believes that he has jumped an orchard wall. His assumption is correct for Romeo has, indeed, jumped the wall surrounding the Capulet orchard so that he may see Juliet, with whom he has become infatuated.
We learn, especially from what Mercutio says, that they think that Romeo's desire to avoid them is because he wishes to sulk about his unrequited love. Romeo has been desperately trying to get Rosaline's attention, but his attempts have been a miserable failure. He has consequently been avoiding all kinds of contact and has preferred to be by himself.
Benvolio has earlier, in Act 1, Scene 1, discussed Romeo's desire for isolation with Lady Montague, Romeo's mother. When she enquires after her son, he informs her:
Early walking did I see your son:
Towards him I made, but he was ware of me
And stole into the covert of the wood.
He tells her that he did not pursue Romeo because it was quite clear that the younger Montague wanted to be by himself. Lady Montague comments that Romeo has been behaving quite strangely and has kept to his room behind closed windows and curtains. She states that he prefers the darkness of his room. When Benvolio asks Lord Montague if he knows the reason for Romeo's condition, the latter replies that he has no idea. Benvolio then vows to get to the truth.
When Benvolio confronts Romeo, he learns about Rosaline and Romeo's failure at wooing her. Romeo is distraught and Benvolio tells him that he should let his eyes roam and rest on other beauties as well. It is quite ironic that Romeo tells his friend at the end of their talk that "thou canst not teach me to forget" because exactly the opposite happens at the Capulet ball.
Romeo becomes so overwhelmed by Juliet that Rosaline almost instantly becomes a distant memory. None of his friends, however, are aware of his new-found infatuation. By the time they look for him they are still under the impression that he is melancholic about Rosaline's rejection and that he wants to be by himself to wallow in grief and disappointment.
In Shakespeare's Romeo and Juliet, Benvolio and Mercutio think Romeo has gone off to be alone or sulk, because Rosaline does not love him.
They look for him and, at first, Mercutio says that Romeo "hath stolen him home to bed." He says he's gone home to bed.
Benvolio, however, points out that he saw Romeo leap over an orchard wall:
He ran this way, and leapt this orchard wall.
And Mercutio assumes he has gone to sulk. He makes fun of Romeo, pretending to conjure him up in the name of Rosaline:
I conjure thee by Rosaline's bright eyes,
By her high forehead and her scarlet lip.
By her fine foot, straight leg, and quivering thigh,
And the demesnes that there adjacent lie,
That in thy likeness thou appear to us.
And Benvolio, telling Mercutio that it's time to go, concludes that Romeo
...hath hid himself among these trees
To be consorted with the humorous night.
Blind is his love, and best befits the dark.
Of course, Romeo is now in love with Juliet, rather than with Rosaline.