The best way to remember this is to remember that the Federalists were the party of the elites. They were the first political party to hold power in our country and it makes sense, then, that they were the party of the people with the wealth and power. Once you remember that, it should be easy to remember what sorts of occupations and such each side had.
The Federalists were engaged in the sorts of occupations that made people wealthy. They were generally urban people. They were the merchants and the high-level artisans. When they were from rural areas, they were the large planters rather than the small farmers. In general, they were the ones with status and wealth. The Antifederalists, of course, were the opposite. They were more commonly found in the rural areas. They were not necessarily poor, but they were also not the elites. They were the small farmers and the workers, not the people with high status.