The reader realizes that the story is science fiction when Jonas sees an apple change.
There are plenty of hints that the community in the story is not like our own. Something spooky and disturbing happens in December, but there is not really a reason for that to require science fiction to be possible. The fear created by the jet flying overhead in the first chapter is also strange, but again that could happen in any world.
We start to wonder when we realize that everything is controlled in the community. Everyone looks alike, dresses alike, and acts alike. Rules are strictly enforced with a combination of lecturing and public shaming at the least, and harsh punishments at the most (which we later find out includes death). There are no animals, which is also suspect.
When Jonas has a strange experience with an apple, the reader finally realizes that this is a different world.
There was absolutely nothing remarkable about that apple. He had tossed it back and forth between his hands a few times, then thrown it again to Asher. And again—in the air, for an instant only—it had changed. (Ch. 3, p. 24)
For something to happen to the apple, for it to really change, there has to be science fiction involved. This leads to the knowledge that no one sees in color, and the transferring of memories seems more possible. All of this is clearly the realm of science fiction, and therefore requires a certain amount of suspension of disbelief from the reader. In other words, we just go with it, because it’s a good story.
Lowry, Lois (1993-04-26). The Giver (Newbery Medal Book). Houghton Mifflin Harcourt. Kindle Edition.