In the first part of the story, the banker's study provides the setting. The story is told in the form of recollection as the banker paces from corner to corner. The banker recalls the party where he made a bet of two million roubles with a young man, wagering that he couldn't survive fifteen years imprisoned in solitary confinement.
The place of solitary confinement is described as the "garden" wing of the banker's house. We never see it ourselves, but we learn the prisoner has access to books and music. We learn of his behaviors and desires through the notes he sends out.
In part two, it is the last day of the bet. It is cold, dark, and rainy, with a harsh wind. For the first time, the setting is described precisely, as the banker heads to the prisoner's wing:
The clock had just struck three. The banker was listening. In the house everyone was asleep, and one could hear only the frozen trees whining outside the windows. Trying to make no sound, he took out of his safe the key of the door which had not been opened for fifteen years, put on his overcoat, and went out of the house. The garden was dark and cold. It was raining. A keen damp wind hovered howling over all the garden and gave the trees no rest. Though he strained his eyes, the banker could see neither the ground, nor the white statues, nor the garden-wing, nor the trees. Approaching the place where the garden wing stood, he called the watchman twice. There was no answer. Evidently the watchman had taken shelter from the bad weather and was now asleep somewhere in the kitchen or the greenhouse.
Finally, the banker enters the prisoner's room. We see that it has a table, two chairs, and books strewn on the carpet. We discover that the prisoner has learned to have no interest in the things of the world, including the banker's money.