Let us remember that the description we receive of the people that move around at dusk comes from Norman Gortsby alone, and reflects his own "subtle failure" and his feeling of being unsuccessful and suppressed by the successful people around him. However, having said that, we need to answer this question by reference to the description of the people that Gortsby feels such solidarity with. Note how they are described:
Dusk, to his mind, was the hour of the defeated. Men and women, who had fought and lost, who hid their fallen fortunes and dead hopes as far as possible from the scrutiny of the curious, came forth in this hour of gloaming, when their shabby clothes and bowed shoulders and unhappy eyes might pass unnoticed, or, at any rate, unrecognised.
The unsuccessful people therefore feel suppressed by the successful people because of the way that they feel judged and criticised. Thus it is that they come out, "bat-like," only at night to hide their "fallen fortunes and dead hopes." By doing this they can walk around anonymously and not be recognised, avoiding the shame of being singled out as somebody who has failed in life.
the defeated men withdraw themselves as they find the harsh mocking and cynical vision of the successful men too much to bear. therefore they feel more comforted and less threatened in the pleasure grounds of successful people in their absence :D