Whitman was using this experience as an example of the difference between abstract secondary epistemology (the organized lecture of the learned astronomer) and the primary experience of looking at the stars "in perfect silence." His point was that experiencing nature should be first-hand, not merely "logically" deducted. This short passage from Leaves of Grass holds Whitman's whole philosophy: the importance of preserving our "nature" in the face of pressure by Society to intellectualize and categorize and civilize, as though our whole identity as humans lay in our "toga" thinking, our ability to "think logically" at the cost of experiencing the world by observing and feeling nature. It is the essence of Romanticism.
There are all sorts of people in the world and there are all sorts of interests and vocations. Walt Whitman was a poet, a nature lover, a free spirit, a wanderer. What was good for him was not necessarily right for somebody else--including you. His poem may make a strong emotional impression, but he certainly wasn't suggesting that there should be no astronomers, no scientists, no scholars. Every person has to find where he or she belongs in life. This can only be done finally and fully through soul-searching. We need people like Walt Whitman to remind us of spiritual values, but we need a lot of other people to fill a lot of other slots in our complex society. If you look up in perfect silence at the stars, the stars may not tell you a thing except that they are very old and very far away.