Mr. Hyde, the reader infers along with Mr. Utterson, lives in the plain gray building spoke about in chapter one. The building is describes as a tall gray structure with no windows. Mr Utterson has only seen Mr. Hyde enter the front door using a key very few times. Looking back over the details of the gray structure, the reader can infer a lot about the character of Mr. Hyde himself. The building is ominous and mysterious. It is alone. It is broken down. The building in its present state is not what it once was. Buildings are used as a motif all throughout "The Strange Case of Dr. Jekyll and Mr. Hyde".
It is Mr. Enfield who sees Mr. Hyde enter the building a few times; Mr. Utterson only sees Hyde use the key once, which is when he approaches him.