I'm not actually sure if Chaucer disliked any of the pilgrims per se. One of Chaucer's main goals is to represent people from many different classes and social circles, and so he's trying to represent people (and society) as they really are. Inevitably, this process involves highlighting some unsavory characteristics. However, even if Chaucer writes about people with flaws, he doesn't necessarily dislike them; rather, he's merely trying to represent people as realistically as possible.

That said, if you absolutely have to choose a pilgrim that Chaucer dislikes, there are ways you can go about doing that. What I would do is look for descriptions of pilgrims that seem to highlight negative characteristics far more than positive ones. The Miller is one such pilgrim and, as an example, take a look at how Chaucer describes him (the excerpt is from eNotes' excellent online version of the text): 

Upon the coping of his nose he had
A wart, and thereon stood a tuft of hairs,
Red as the bristles in an old sow’s ears;
His nostrils they were black and very wide.
A sword and buckler bore he by his side.
His mouth was like a furnace door for size.
He was a jester and could poetize,
But mostly all of sin and ribaldries.
He could steal corn and full thrice charge his fees... (10-18)

Here, we can see that Chaucer characterizes the Miller as a somewhat ugly cheater who's also fond of telling dirty jokes. Based on this description, which seems to be mostly negative, we could reasonably assume that Chaucer dislikes this pilgrim. Likewise, if you check out Chaucer's description of the Pardoner, it would appear that Chaucer also dislikes him and views him as something of a scoundrel.