The answer to your question can be found at the beginning of the story as we are introduced to Norman Gortsby and also, thanks to the third-person-limited point of view, given access to his thoughts and feelings. Dusk, to him, is a time that he likes, especially given his own "present mood." Let us look more closely at why he likes this time of day so much:
Dusk, to his mind, was the hour of the defeated. Men and women, who had fought and lost, who hid their fallen fortunes and dead hopes as far as possible from the scrutiny of the curious, came forth in this hour of gloaming, when their shabby clothes and bowed shoulders and unhappy eyes might pass unnoticed, or, at any rate, unrecognised.
To Gortsby's mind, then, dusk is "the hour of the defeated" because of the way in which men and women who are victims of life and have lost in some way are able to come out without fear of being spotted or recognised or having their failure commented upon by others. This is of course Gortsby's "imagination" at work, but also something else: he counts himself among the "defeated" because of some unspecified failure in a more "subtle ambition" and therefore this explains his state of mind.