The short answer is that the dwarf that lives in the Brobdingnagian court hates Gulliver because Gulliver has usurped his place as the shortest person in Brobdingnag. Gulliver says that the dwarf, upon meeting a creature that was even smaller than he, grew so "insolent [...], that he would always affect to swagger and look big as he passed by [Gulliver] in the queen's antechamber [...]." Gulliver would often stand on tables to speak with the courtiers, and the dwarf would strut by him, trying to look really big, and he would usually employ sarcasm or insults to deride Gulliver for his littleness. Gulliver really has no recourse other than to call the dwarf "brother" and try to avoid him whenever possible.
In his anger, the dwarf actually drops Gulliver into a bowl of cream for fun, and Gulliver nearly drowns. At another point, the dwarf jams Gulliver's lower half into a bone from the queen's table, tearing up his pants. For this, the dwarf was whipped (and I can't imagine that endeared Gulliver to him any more). Gulliver endures the dwarf's meanness many times before the queen actually gives the dwarf away, rendering Gulliver safe from any more harm at the dwarf's hands.