Tom breaks Myrtle's nose because he told her to stop talking about Daisy, his wife. When Myrtle continues to say her name, Tom backhands her, breaking her nose. This shows that Tom has a short temper and also that he basically has no real love for either woman, Daisy or Myrtle. Although he never hits Daisy in the book, one can assume that he is probably verbally abusive. The fact that he is having the affair with Myrtle shows his intense dishonesty and egotism.
In chapter two of The Great Gatsby, Tom takes Nick with him to the Valley of Ashes. There they run into Myrtle and Tom tells her to follow he and Nick. They go into New York City to the Mornigside Apartments. Tom had gotten an apartment there for his affair with Myrtle. They throw a little party and things quickly get out of hand. Myrtle starts mentioning Daisy's name, and Tom warns her to stop. Myrtle keeps on taunting Tom by chanting Daisy's name over and over, and Tom backhands her, breaking her nose.
Tom never apologizes to Myrtle or the guests at the party. The party soon breaks up after this event. We see, early on in the book, that Tom is quick tempered. Obviously he is not much of a man, if he hits Myrtle. We see more of his character coming out. It makes us wonder why Daisy stays married to him. Is it because of his wealth and name? Does she love him? Obviously Tom doesn't love Daisy, if he did he wouldn't be having an affair. If Tom can hit Myrtle, does he also abuse Daisy in some way?
In the events of Tom slapping and breaking Myrtle's nose, we are set up for the tragedies that are to follow. If Daisy had stoop up to Tom, maybe the whole thing would have ended another way. Tom's temper and his need to be in control, lead to tragic events, that ripple to many characters.
Tom, in my eyes, has always come off as this controlling guy without much hold on his anger. The reason Tom broke Myrtles nose is because she would not stop talking about about his wife even though he warned her to stop talking about her. I guess either he was feeling guilty and felt like the only way for her to officially shut up was to literally shut her up or he just didn't want the thought of his wife to pull his mood down. I am going to bank with the first explanation.