The answer to this can be found on p. 381 in the paperback edition of the book. This about 6 pages into Chapter 19. You are right when you mention the people from Borneo. It is their presence on Madagascar that Diamond thinks is so astonishing.
Diamond points out that Madagascar is a mere 250 miles from Africa but is separated from Asia by the whole of the Indian Ocean. Borneo, for example, is 4,000 miles away. Given that, it is truly astonishing that there should be people on Madagascar speaking a language related to that of Borneo and looking like people from that island. He finds it astonishing that there should be these people from so far away who clearly got to Madagascar, as he says, "presumably voyaging in boats without maps or compasses."
Diamond refers to Madagascar as an “anomaly” due to its diversity of peoples and its unusual language. Madagascar is inhabited by two peoples – African blacks and Southeast Asians. What is significant is that Madagascar is only 250 miles of the African coast but completely across the Indian Ocean from Asia and Australia.
Similarly, an unusual language is spoken on the island. The language spoken by all the people of Madagascar is Austronesian and is very similar to that spoken on the island of Borneo. Borneo is over 4,000 miles across the Indian Ocean. Diamond states:
“No other people remotely resembling Borneans live within thousands of miles of Madagascar…. It’s as if Columbus, on reaching Cuba, had found it occupied by blue-eyed, blond-haired Scandinavians speaking a language close to Swedish, even though the nearby North American continent was inhabited by Native Americans and speaking Amerindian languages. ” p. 381
Archeologists have revealed that the early settlers on Madagascar had iron tools, livestock and crops. Based upon this Diamond believes the “…colonists were not just a small canoe load of fishermen blown off course; they formed a full-fledged expedition.” Somehow these prehistoric peoples sailed 4,000 miles without advanced tools or knowledge.