I think it's important to go directly to "The Prologue" to answer this question in order to not speak in generalities. In fact, it's only about 30 lines into the first section of The Canterbury Tales where we can find the answer to your question:
Ready to go on pilgrimage and start / For Canterbury, most devout at heart, / At night there came into that hostelry / Some nine and twenty in a company / Of sundry folk happening then to fall / In fellowship, and tehy were pilgrims all / That towards Canterbury meant to ride.
So here Chaucer writes of the "nine and twenty." Ah, but not so fast! Keep in mind that it is the narrator that joins the group. The narrator brings the number to 30, then. Also, there is the ethereal character called "the Host" who joins and brings the number to 31. Therefore, taking Chaucer's words as well as these two important additions into account, the number of people goint to Canterbury during The Canterbury Tales is exactly 31.
Please read the article by Caroline D. Eckhardt, "The Number of Chaucer's Pilgrims: A Review and Reappraisal," in the Yearbook of English Studies, Vol.5 (1975) Pages 1-18.
Chaucer himself states that the total number of pilgrims is twenty-nine, but the actual count is thirty.
Most people say thirty, but the narrator says some nine and twenty joined him (number 30) at the Tabard. Then the host joins them as they go to Canterbury, so 31.