At the very beginning of this nested tale the first narrator gives away the meaning of the title. It all begins with the narrator (presumably Henry James, or another unnamed speaker) explaining that they are at an old house during the winter telling stories about ghosts and apparitions. Some stories are better than others, and those who are listening are waiting for an ultimate story that would beat all the others. Someone suggests,
"I quite agree—in regard to Griffin's ghost, or whatever it was—that its appearing first to the little boy, at so tender an age, adds a particular touch. [...] If the child gives the effect another turn of the screw, what do you say to TWO children—?"
"We say, of course," somebody exclaimed, "that they give two turns! Also that we want to hear about them."
The universal use of turning a screw is to make something tighter, or tense enough, to resist falling apart from movement. The idea that the story will be about two ghost children instead of one would "turn the screw" even tighter on the listeners, who will be more tense at the prospect of two haunted young spirits--what could have possibly happened? Why were these children victimized? The tragic idea a haunted child definitely thickens the plot and makes it more tense.
Another mention of the title appears in chapter XXII, with the Governess as the narrator, where she says
I could only get on at all by taking "nature" into my confidence and my account, by treating my monstrous ordeal as a push in a direction unusual[...] after all, for a fair front, only another turn of the screw of ordinary human virtue.
Here the term is used by her to describe how the situation in the home is really testing her mentally and emotionally. She takes on the challenge and tells herself that she will put up a strong front. She assumes that this is just another turn of the screw--another pull of her strength--to work with.