The first part of this answer lies in the way that Gortsby imagines the people who come out at the hour of dusk, which, in his mind, is a time for those who have failed in life to emerge. Note how he describes such failures:
Men and women, who had fought and lost, who hid their fallen fortunes and dead hopes as far as possible from the scrutiny of the curious, came forth in this hour of gloaming, when their shabby clothes and bowed shoulders and unhappy eyes might pass unnoticed, or, at any rate, unrecognised.
Dusk then fits Gortsby's mood as he contemplates the way that others have failed in life and then moves on to reflect on his own failure in his "subtle ambition." Of course, there is an irony in this, as during the course of the story, Gortsby himself is shown to fail in beign taken in by the young man and his story, and giving him some money, when at the end the young man turns out to be a confidence trickster after all. It is therefore appropriate that Gortsby should feel such an affinity with dusk. He has failed before, and during the course of the story, he fails again.