To answer this question you need to look at the poem as a whole, from start to finish. As you read the first stanza and then look at the last stanza and the way that the poem describes that the slave whose dream we have just been able to see has actually died from exhaustion and maltreatment. Note how this last stanza presents the slave:
He did not feel the driver's whip,
Nor the burning heat of day;
For Death had illumined the Land of Sleep,
And his lifeless body lay
A worn-out fetter, that the soul
Had broken and thrown away!
The last stanza thus reveals the reason why the rice lay ungathered, as indicated in the first stanza. The slave, worn down by days and weeks and months of slave labour in the "burning heat of day" and mistreatment, as indicated in the "driver's whip," has collapsed, exhausted, onto the floor, leaving the rice ungathered. He has entered a state of unconsciousness where he is able to "return" to his homeland, ironically just before dying and gaining the liberty that he so keenly desired.