In short, the answer to your question is no, he is not being racist. Longfellow does indeed mention women in "The Village Blacksmith." In the poem, he is writing about the village blacksmith, who is an upstanding member of society. The blacksmith owes nothing to anyone and works an honest job. He is looked up to in the community and is an honorable man. Longfellow goes on to tell us that the blacksmith is one of those men who is honest, hardworking, and respected.
"He goes on Sunday to the church,
And sits among his boys;
He hears the parson pray and preach,
He hears his daughter's voice,
Singing in the village choir,
And it makes his heart rejoice."
In this stanza from the poem, we see the blacksmith worshiping with his community and his children. He goes on to say how he loves to hear his daughter's voice in the choir and it does his heart good.
"It sounds to him like mother's voice,
Singing in Paradise!
He needs must think of her once more,
How in the grave she lies;
And with his hard, rough hand he wipes
A tear out of his eye."
We see that the blacksmith is just a good man and father. He is grieving the death of his wife and going on taking care of his children. There is nothing in the poem that would suggest a racial theme, and there are mentions of women in the poem. He talks about his daughter and the death of his wife, whom he loved very much. This is a beautiful poem about the character of a man who works hard everyday and takes care of his children while still mourning the loss of his wife.