Drawing an inference can only provide an "educated guess" as to why certain things happen in a story—most especially in that the author leaves certain details unclear. The reader, then, must search for meaning "between the lines."
An inference is present when one draws...
...a reasonable conclusion from the information presented.
In F. Scott Fitzgerald's The Great Gatsby, based upon the kind of person we know Tom to be, we can infer that he does not want to leave the East coast, or return to Chicago, because he has been having numerous affairs while married to Daisy—allegedly a "bad" one in Chicago.
Early on in the story, we find that Daisy and Jordan are certain that Tom's most recent mistress calls him on the phone in the middle of the day. Daisy doesn't take this quietly, but at one point excuses herself from the room; soon after, voices of husband and wife can be heard arguing.
In Chapter Seven...
"The rumor is," whispered Jordan, "that that's Tom's girl on the telephone."
We were silent. The voice in the hall rose high with annoyance: "Very well, then, I won't sell you the car at all...I'm under no obligation to you at all...and as for your bothering me about it at lunch time, I won't stand that at all!
As the chapter progresses, Tom admits to his infidelities. (The reader is made aware of the most recent one when Tom stops the car he is driving at the gas station to buy gas, and the owner's wife, Myrtle—Tom's mistress—stares out the window, believing Jordan is Tom's wife.) Tom says:
Once in a while I go off on a spree and make a fool of myself, but I always come back...
Tom has fooled himself into believing that his "mistakes" are excusable. However, when Gatsby tells Tom that Gatsby and Daisy have been in love for five years (not having an affair), Tom dismisses it as foolishness.
Daisy, on the other hand, has (seemingly) come to the end of her rope with Tom.
"You're revolting"...her voice, dropping an octave lower, filled the room with thrilling scorn: "Do you know why we left Chicago? I'm surprised that they didn't treat you to the story of that little spree."
The very fact that Tom refers to his affairs as "sprees," and now Daisy uses the same word again, clearly infers that Tom and Daisy left Chicago because of some "situation" that occurred there. (It seems to refer to an accident Tom had—with a woman in the car other than Daisy.)
Tom is a scoundrel, caring little for Daisy. However, as soon as Gatsby expresses his love for Daisy, Tom is ready to make amends:
I'm going to take better care of you from now on.
Gatsby tells Tom that Daisy is leaving her husband, and Daisy says that it's true. With Tom's casual attitude about Myrtle and his disregard for Daisy's feelings, it seems that the difficulty in Chicago arose from his infidelity, which had not stopped even after leaving that city behind.
In The Great Gatsby, Daisy and Tom are married, but Jay Gatsby loves Daisy and forces her to admit that she loved him once, and might still. Tom admits that he cheats on her:
"Once in a while I go off on a spree and make a fool of myself, but I always come back..."
Daisy bursts out:
"Do you know why we left Chicago? I'm surprised that they didn’t treat you to the story of that little spree."
The details of this particular "spree" are never explicitly stated, but earlier, Jordan Baker tells Nick (the narrator) a story about Daisy's traumatic bridal shower:
She groped around in a waste-basket she had with her on the bed and pulled out the string of pearls. "Take 'em down-stairs and give 'em back to whoever they belong to. Tell 'em all Daisy's change' her mine. Say: 'Daisy's change' her mine!'."
She began to cry -- she cried and cried. I rushed out and found her mother’s maid, and we locked the door and got her into a cold bath.
(Quotes: Fitzgerald, The Great Gatsby, ebooks.adelaide.edu.au)
She also tells Nick about a car accident soon after the wedding in which the girl in the car was not Daisy, but a hotel chambermaid. Clearly, they left Chicago because Tom could not stop philandering, and his reputation brought them humiliation. Jordan claims that Daisy never drank or stepped out, so her more passive role would have been harder than Tom's constant indiscretions and distractions.