In 1984, the Brotherhood is the name of the secret organization whose role is to bring down the Party. In Part Two, Chapter Eight, when Winston and Julia are at O'Brien's apartment, the reader gets a sense of how the Brotherhood fulfills this role. Its members murder, cheat, forge, and betray their country. In order to keep its membership and activities secret, the Brotherhood keeps no records or membership lists.
In Part Three of 1984, however, it becomes clear that the Brotherhood (and its head, Emmanuel Goldstein) do not exist. This has an important effect on the Brotherhood's role: instead of being a protest group, the Brotherhood is used by the Party to highlight those who secretly wish to rebel. Once the Party knows the identities of these people, it imprisons and tortures them until they relent, as we see in the cases of Winston and Julia.
The Brotherhood is the secret, anti-government organization headed by Emmanuel Goldstein. Whether or not the organization really exists is not made 100% clear, but it is doubtful that it exists. More likely, it, as well as Goldstein, are made up by the government of Oceania as a means of seeking out revolutionaries and anarchists. Also, by letting people believe that this person and organization exists and then by showing people the terrible fate that befalls those who join or associate with this organization, the Party keeps people in line. People see or hear about the tortures carried out against anyone who tries to go against the Party and their fear of that happening to them keeps them from going against the Party. In the story, Winston tries to join the Brotherhood and that is how O'Brien traps Winston and Julia who eventually are tortured into loving Big Brother and the Party. Essentially then, the Brotherhood is merely bait dangled by the Party to catch those who might want to overthrow the government.
It essentially is the bait of the Party. It is logically concluded, though subject to further speculation, that the Brotherhood does not exist, but that it is a concept that people should not relate to, as it represents an enemy of the Party, and if they associate with the concept, they have committed a thoughtcrime. It is to test the citizen's loyalty to the Party potentially.