I think the effect created is the arousal of suspicion. What I think is one of the coolest aspects of The Great Gatsby is the ability of Fitzgerald to leave so many clues throughout the text that might lead a reader to an assumption or conclusion. This is exactly what we do when we hear that he might have murdered or was a spy.
A great effect to the reader is these abilities to practice guessing, jumping to conclusions and inferring what could be his identity based on what's heard about him. We are also forced as readers to consider what could be truth or what could be lies.
We can know (because he regularly tells us) that Nick tells the truth. So watch what Nick discovers and then reports. He will tell you the truth. He lives in this suspicion with the reader as the story progresses past that chapter 3 moment of rumor.
To me, the effect of these rumors is to make Gatsby more of a mysterious figure.
In the story, there are all these rumors about Gatsby and his past. Most of these are seen early in the story. As we read these rumors, we start to wonder about what kind of a person Gatsby really is.
By making Gatsby seem mysterious, the author is also showing us that the people who are around him do not really know him at all. This has the effect of showing us that he lives very much apart from his surroundings.