The secret garden may be very beautiful, but for Mr. Craven it only conjures up feelings of sorrow. The garden was his wife's pride and joy, and so he naturally associates it with her memory. When Mrs. Craven passed away, the garden effectively died with her. Her husband had the garden locked up and determined that no one must ever go there again. His reasoning appeared to be that if Mrs. Craven could no longer enjoy her special garden, then no one else could either.
Mr. Craven thinks that by locking up the garden he can banish painful memories of his wife's untimely death. Opening the garden would bring that pain rushing back. That being the case, if he ever caught Mary (or anyone else) in the garden, he would instantly feel angry and upset. All those unhappy memories would come flooding back.
Truth be told, Mr. Craven is so sad and bitter over his wife's death that he resents anything that lives, including himself. That category also includes the secret garden and the children who should be playing there. So long as Mr. Craven remains in this tortuous state of mind, it's a certainty that he will be extremely upset should anyone try to venture into the enchanted confines of the secret garden.
Further Reading