In the most literal sense, the speaker in the poem is talking to his son. The father is providing a set of moral guidelines or benchmarks that will, if adopted and practiced by the boy as he grows into adulthood, allow him to become an adult with personality and behavioral characteristics that the father deems as highly desirable. When the speaker, in the final line, says "you'll be a Man, my son!", he is implying that this would be the achievement of becoming a highly responsible and respected individual, not simply an adult.
On a deeper level, the speaker is addressing the advice to all readers of the poem. Anyone who takes the advice contained in the poem to heart and practices the attitudes and actions suggested therein will become a worthy person, ready and able to make positive contributions to the world.