Let us remember that this interpretation only is in the mind of the protagonist, Gortsby, who enjoys dusk because of the way that he associates it with "the hour of the defeated." Note what he says about dusk and how it frees people to go out who would normally stay in their homes for fear of being noticed:
Men and women, who had fought and lost, who hid their fallen fortunes and dead hopes as far as possible from the scrutiny of the curious, came forth in this hour of gloaming, when their shabby clothes and bowed shoulders and unhappy eyes might pass unnoticed, or, at any rate, unrecognised.
Note the way in which, in Gortsby's mind, who classes himself as one of these men and women, dusk allows those who have experienced failure to go out without attracting attention. He sits next to an old man who has a "drooping air of defiance" and likewise the young man has supposedly come across bad luck. All the characters, apparently, are shown to be "defeated" through their bearing, character and situation, but this belief of Gortsby's perhaps only makes him more vulnerable to the trick of the young man.