This line comes from the first stanza of this famous poem about the necessary qualities that are needed to be a man, and it interestingly explores the necessary relations that one must have with others in terms of relying on them and yet understanding the way in which confidence and doubt forms a natural part of human relations. Note the line in context:
If you can trust yourself when all men doubt you,
But make allowance for their doubting too...
The line therefore claims that part of being "a man" is being so sure of yourself and your abilities no matter what others think or may say about you. Even if everyone else doesn't believe you, being a man involves having enough self-belief to continue nonetheless and not be swayed by lack of support from others. However, this alone is not enough to be a "man," as it is also important to be able to respect doubts of others and to use them to reflect and evaluate your schemes and plans. It is not about going ahead with your own ideas and recklessly ignoring the advice and wisdom of others, but knowing when it is right to trust others and having the humility to take the advice and thoughts of others into consideration when appropriate. Ultimately, wisdom is needed when to go ahead with your own ideas and when to listen to others.