It is not stated blatantly, in the poem "I Wandered Lonely as a Cloud", as to why the speaker is wandering about. Instead, the poem focuses upon what happened as the speaker wandered and what came out of his wandering.
This being said, it seems to be much more important as to the outcome of the wandering than the wandering itself. Therefore, as a reader, one could infer a reason as to why the speaker was out wandering. Given poems are subjective, and inference may be used in all types of texts, a reader could give their own interpretation as to why the speaker was out when he came upon the field of daffodils.
One could infer that the speaker was having a bad day (given after, when having a bad day, he is able to reflect upon the field and feel better) and simply out for a walk. This could be justified through using the first line of the poem.
The word 'lonely' could refer to the fact that the speaker did not feel as though he had anyone to talk with and this was why he was out walking. Another interpretation could be that the speaker was out simply alone (lonely) and happened upon the field.