George explains that it isn't so funny because he feels a sense of responsibility for taking care of Lenny. Lenny's Aunt Clara, an acquaintence of George, raised Lenny. When she died, Lenny just started working with George all the time until they because so used to being together. Now, George feels the parental responsibility for Lenny's safekeeping.
This is odd for the other men at the ranch because of the nature of the cut-throat job market at the time; however, the other men at the ranch haven't had the family-like relationship that Lenny and George have been a part of which is why they can't understand how difficult it is for George to leave Lenny.
George deflects the insinuation of the comment by explaining how he feels responsible for Lenny, because of Lenny's mental limitations.
Apart from what everybody else said, here is an extra thing.
During this whole conversation, George seems very embarassed or scared about what happened. At first they are talking about Aunt Clara, while George is laying the cards out. He lets it slip about Weed and instantly stops laying his card (he freezes).
***Could George be coming up with a euphimism about what actually happened at Weed? George seems to try and force that Lennie just scared the girl, and that he's nice. Maybe, he did much worse than scared her, probably he killed her. ***