Scout never tells the reader specifically why Boo Radley's brother decided to cement the knothole of the tree. We can only speculate that Nathan Radley discovered that Boo was leaving items in the tree (or perhaps saw Jem and Scout retrieving them), and he decided to put an end to this form of communication. We know little to nothing about the relationship between the two Radley brothers, but it can be assumed that Nathan moved back into the family home in order to take care of Boo, and he probably followed his father's lead in requiring Boo to stay inside. Boo had been arrested twice before, and he no doubt suffered from some sort of mental illness, so Nathan (like his father) probably believed that Boo was safer--to both himself and the residents of Maycomb--shut away in the house. Since Boo had no social contact with any other people, Nathan probably worried about what would happen should Boo and the Finch children become friends. Atticus would probably not have minded, and we know that by the end of the story Boo only means to protect Jem and Scout. But Boo's history of violence and odd behavior left Nathan little choice but to continue his father's practice of house imprisonment.
Boo Radley used to hang out with the Cunnighams when he was a young kid and this fact shamed his family.
The Radleys do not want Boo to shame them again and hence, they disallow him to get out of the house. They are afraid that he will do something dumb again.