Agnostics bother Pi and atheists do not simply because Pi doesn't like the apparent indecision on behalf of the agnostics. To him it seems it does not matter what you believe, so long as you believe in something. Atheists do not believe in God, but agnostics have a much more open belief stating they do not know for sure whether or not God exists.
In Chapter 7 he says about agnosticism:
"To choose doubt as a philosophy of life is akin to choosing immobility as a means of transportation.”
However, Pi's understanding of agnosticism is misplaced, agnosticism is not about not know for sure, but believing that as we can never know the answer it is naive to insist one has it.