Odysseus is opposed to eating the lotus fruit because he sees the effects that it has on his crew members: it makes them forget how badly they want to return home to Ithaca and their families, and they only want to stay in the Land of the Lotus Eaters. Odysseus desperately wants to return home to his wife, Penelope, and his son, Telemachus, both of whom he left, at that point in the chronology, just over ten years or so ago in order to fight in the Trojan War. He tells Alcinous that the lotus eaters were peaceful but "whosoever of them ate the lotus' honeyed fruit wished to bring tidings back no more and never to leave the place." Odysseus actually had to use force to get the three men who had eaten the fruit back to the ship, and they resisted him and cried the whole way. He even had to tie them up so that they did not escape. He doesn't want to eat the lotus fruit because he wants to go home, and he doesn't want to forget this purpose.
Odysseus was opposed to eating Lotus because of the medicinal effects of the plant. The lotus tree's fruit can cause drowsiness and lethargy. Odysseus needed to be alert in order to complete his tasks. His refusal of the lotus fruit is due to his awareness of its negative effects. The plant is also addictive.
Interestingly, there was an island civilization near North Africa that consumed only the lotus fruit. They were called "lotophagi".
The lotus fruit affects the mind and memory. If they were to eat the lotus plant, they would forget their destination and might not want to return home.