Each stanza in Thomas Hardy's "The Voice" does, indeed, have a different rhythm, each reflecting the meaning or intent of the stanza.
In the first stanza, the speaker of the poem, probably Hardy himself, is hopeful that the woman's voice he hears calling him is that of his beloved, presumably Hardy's departed wife of 40 years, Emma. The lines are longer and move more trippingly over the tongue than the other stanzas, indicative of his hopefulness and eagerness to be reunited with the "woman much missed," "the one who was all" to him. It is a stanza of hopefulness, and that is reflected in the rhythm of the language and lines.
The second stanza is one of pausing and waiting, and the lines slow down and are a bit more clipped. Notice only two words in the entire stanza contain more than one syllable, as if speaking only short words would allow him more opportunity to be still and listen.
His hope has gone by the third stanza:
Or is it only the breeze in its listlessness
Travelling across the wet mead to me here,
You being ever dissolved to wan wistlessness,
Heard no more again far or near?
The lines in this stanza are even shorter, as if fading away into the reality of loneliness and despair once again. Finally, in the last stanza, the lines are the shortest yet--except for the third line. Here, the length of the line matches the imagery of the line:
"Wind oozing thin through the thorn from norward." The oozing extends the line, just as anything slow-moving and oozing would. The rest of the stanza displays a hopelessness which is to become a forever state for the speaker.
Then, he is back to his final line--"And the woman calling." Short, terse, as if there's nothing more to be said. And, of course, there isn't.