Forster repeatedly advised in his writings to "only connect" as the antidote to animosity and enmity, but colonialism showed him otherwise. And he learned that the hard way, for Aziz is modeled after a lover that meant a good deal to him in his life. Consider the end of the novel: Aziz says, "half kissing" Fielding, that after the English leave India, they will be friends. Fielding ressponds, holding Aziz "affectionately, 'Why can't we be friends now?'" because they both want that. "But the horses didn't want it--they swerved apart; the earth didn't want it....the temples, the tank, the jail, the palace, the birds, the carrion ... didn't want it, they said in their hundred voices, "'no, not yet,'" and the sky said, 'No not here.'" As all of these aspects of nature indicate (even the horses draw apart), friendship between colonizer and colonized goes against the order of colonialism for colonialism itself is unnatural.
Forster suggests at the end of the novel that a friendship between a British and an Indian citizen can't be endured. It can begin and develop to a certain point, but it can't last. Aziz tells Fielding on their last ride together that the British must give India its freedom, and Fielding disagrees with him. They respect each other as men, but they can't be friends until the country of one man (Great Britain) respects the country of the other man (India). No matter how hard they try to maintain their friendship, their differing political views will always be there to remind them of their differences.
Colonialism is total control of one country by another. The people from the governing country who live in the colony either don't respect the customs and traditions of the native people or don't bother to understand their culture. This is an affront to people like Aziz. The fact that Fielding diametrically opposes Aziz in his views on the rule of India shows a disrespect that can't be tolerated to Aziz and the Indian people. True friendship cannot withstand a lack of respect by one of the people involved.