When the Giver says that the community "went to Sameness," he means that society chose to make everything seem the same because people's perceived differences are what caused conflict. For example, he talks about a time "when flesh was many different colors," but now it all looks the same. Society gave up color because it was something they could not control. However, the Giver says that "We've never completely mastered Sameness." Jonas gets upset when he cannot see colors all the time. He feels that it is not fair that things lack color, saying, "If everything's the same, then there aren't any choices! I want to wake up in the morning and decide things! [. . .] But it's all the same, always."
However, Jonas can see color—as can the Giver—and so this means that differences still do exist, it is just that people are manipulated so that they no longer perceive those differences. Generally speaking, sameness is the absence of difference, yes; if things are the same, then they are identical and have no differences. However, sameness here really refers to the communal inability to perceive difference rather than a complete lack of difference itself.