The section where Gurov goes to visit Anna Sergeyevna comes in the third part of this excellent short story. One of the interesting descriptions that we are given is of the hotel room in which he stays:
He arrives at S. in the morning and engaged the best suite in the hotel, which had a carpet of grey military frieze and a dusty ink-pot on the table, surmounted by a headless rider, holding his hat in his raised hand.
It appears that the headless rider could be symbolic of the impulsive way in which Gurov has "lost his head" and acted impulsively, against his rational side that wanted the relationship with Anna Sergeyevna to be over and was glad when she left. Note the way that his intellect or capacity for reason is overwhelmed by his heart when he goes to the theatre and sees Anna Sergeyevna again:
Anna Sergeyevna came in, too. She seated herself in the third row of the stalls, and when Gurov's glance fell on her, his heart seemed to stop, and he knew in a flash that hte whole world contianed no one nearer or dearer to him, no one more important to his happiness.
Although Anna Sergeyevna is described as "little" and "in no way remarkable," Gurov's emotions are overwhelmed and he realises that she "filled his whole life, was his grief, his joy, all that he desired." Perhaps the symbol of the headless horseman thus captures the way in which Gurov as a character has "lost his head."