Which genres of popular literature are considered masculine?
This is a reasonably subjective question, which cannot have a definitive answer; your definition, or your teacher's definition, of terms involved will have influence.
Masculine literature as a genre has always been defined by its appeal to Men over Women, or to traditionally Male topics such as hunting and war. The Men's Magazines of the pulp era were uniformly sensationalistic, measuring their success by the number of stories involving exploration, nature, violence, and sex (male-dominated). From the pulp tradition came the two-fisted crime novels of the Fifties, and then pulp as a genre to itself faded with the seventies.
Today, "Masculine" popular literature is mostly confined to the thriller and mystery genres. Both involve male heroes confronting extraordinary (or mysterious) enemies, at great personal peril, and usually winning out in the end. Robert B. Parker was the Grandmaster of modern two-fisted mystery fiction until his death in 2011; he also wrote about themes of masculine/feminine interaction from a psychological perspective. Lee Child, in his Jack Reacher series, has exemplified the wandering "man with no name." Vince Flynn, Clive Cussler, and Douglas Preston/Lincoln Child all write in the thriller genre, involving larger-than-life heroes.
Ultimately, this will depend more on the definition of the term than on the actual literature. Generally speaking, if you have a story that is dominated by male characters, or traditionally male themes, it can be considered "masculine" literature.