What has happened just before this is that Tom's girlfriend, Myrtle Wilson, has phoned him at home. Tom, Nick, Daisy and Jordan Baker are having dinner. Tom gets up to speak to Myrtle, and Jordan tells Nick that it is Tom's girlfriend calling and why can't she have the decency not to call him at home?
When Tom and Daisy come back into the room, the situation is very tense. Tom and Daisy have apparently had an argument over Myrtle's call, and when they return, they act like nothing has happened. Nick is extremely uncomfortable, and wonders if Tom and Daisy notice that he cannot even look them in the eye. His comment about calling the police indicates that he is so upset, he doesn't really know WHAT to do -- so his comment is really a sarcastic way of saying that he is totally freaked out.
In The Great Gatsby, Tom and Daisy's home and their lives are not what they might at first seem. Nick, in the scene you ask about, is exposed to their marital misery.
Nick is simply being funny or facetious, although his joke is not without seriousness.
The "fifth guest," Tom's woman in New York, calls for a second time, and Nick's words are his reaction to the second call. A few paragraphs later, when Daisy tells Nick that she's become cynical, Nick comments that "Evidently she had reason to be." This, too, refers to Tom's adultery, and possibly to Daisy's being married to Tom, period. Nick doesn't like Tom or think much of him.
Not only is Tom going out on Daisy, and not only does Tom's woman call twice during this little get together, but Jordan also contributes to Nick's feeling the need to call the police, to call for help. Jordan, "unashamed," leans forward in her chair and tells Tom to be quiet so she can hear what's going on with Tom and Daisy and the woman in New York. Things are just not right in this household. Instead of being discrete and ignoring the situation, Jordan unabashedly tries to listen in. She already appears a bit amoral.
Nick is overwhelmed and shocked by these people, and by Daisy's situation. Instead of being interested or stimulated by the possiblility of scandal, etc., he is a bit disgusted. His thought of calling the police is just his indirect way of revealing this.
The description of Tom as "hulking" and having a "cruel body" develop the idea that he is cruel and physically powerful. He also has very little respect for women, as he has a mistress, talks to her during a dinner party, and has had mistresses in the past.
Daisy is described as irritating and demanding of Tom. Their marriage is a facade as Fitzgerald describes the beautiful house but the people within are emotionally corrupt and bankrupt. Daisy is barely there for her daughter - not a great mother.
These two descriptions and the scene of the phone call heard through the thin wall makes for a potentially dangerous situation and Nick is afraid that Tom may hurt Daisy, who seems capable of provoking him. Of course as in most of the book they are also drinking consistently, adding to the problem of not reacting well in tense situations.