The Brotherhood is the Masons or Freemasons, a powerful secret brotherhood that still exists. Fortunado asks Fortunado if he is a Freemason. When asked if he is in the Masons, he replies with a gesture the narrator considers rude because he is not in the Masons and does not understand it. Although it does not specify what the grotesque movement is, it seems to be some kind of sign. Fortunado does not believe that the narrator is actually a Mason.
“You are not of the masons.”
“Yes, yes,” I said; “yes, yes.”
“You? Impossible! A mason?”
“A mason,” I replied.
“A sign,” he said, “a sign.”
“It is this,” I answered, producing from beneath the folds of my roquelaire a trowel.
Fortunado does not take the narrator seriously. They leave it at that, and continue. This is a sign that the narrator has some evil plans however. Why else would he bring a trowel? Fortunado does not realize the significance of the trowel. Its presences, and the discussion of the Masons, foreshadows Fortunado’s untimely end. The narrator uses the trowel to brick him up in the crypt.