In this poem, the narrator is advising his son on how to face life if he wants to be a "Man."
This involves, first, expecting disillusionment. The young men needs to hang on to his level-headed sense of self in both good times and bad, accepting from the start that other people will blame him, doubt him, lie to him, and hate him. The narrator also advises him to hang on to his dreams even when he faces adversity and to hold onto his money lightly, willingly to risk it all and yet remain stoic if he loses it. Finally, the narrator advises his son in the last stanza not to be bedazzled by rank or lose the common touch. He says:
If you can talk with crowds and keep your virtue,
Or walk with Kings—nor lose the common touch,
If neither foes nor loving friends can hurt you,
If all men count with you, but none too much . . .
In other words, the father is saying to the son that "if" he stays true to himself and his own integrity, he will be a man, a mature individual. The line "if all men count with you, but none too much" means you should respect other people (they should "count with you") but not depend on any one person too much: you should trust yourself first.