Granger is talking about how insignificant every human actually is. Not a single one of us will live but more than a glimpse in the long scheme of time. What we do with that little moment is important, but even those who are well-recognized in their time, and even after, rarely last forever in the memories of those who follow. Granger followed the words you quoted with these:
But even when we had the books on hand, a long time ago, we didn't use what we got out of them. We went right on insulting the dead. We went right on spitting in the graves of all the poor ones who died before us.
His point is that we don't do much with what we have while we have it. So before we get conceited and think we have life and everything around us all figured out, it is important to recognize how small we are in the large scheme of things. On his own, Montag isn't much. But the power of the group could be tremendous if each one did their part.
In my mind this means two things. Yes, it does mean in the scheme of things they are small, and unimportant. The world is a huge place compared to this group of men. The second thing is that they have very much information between them all. Since nobody even knows about this information, if they were to die at that instant, nothing would be changed. The information is nothing unless they disclose it to the world, and try to convince people that it really does mean everything to making a succssessful society.