While Victor Frankenstein does not have a traditional college major as one might have in today's colleges and universities, he studies a variety of relevant topics that lead him to creating his monster. The concise response is that Frankenstein studied chemistry.
In chapter 2, Frankenstein reflects upon his studies. He mentions being self-taught, having to seek out his favorite studies on his own time. He was fascinated by the concept of discovering the secret to eternal life, as many of his favorite authors discussed in their works. By the time he was fifteen, he decided that mathematics was the subject most worthy of his attention.
In chapter 3, he continues to discuss his academic development. He arrived at the University of Ingolstadt, where he studied with M. Krempe and M. Waldman. M. Krempe criticized Victor's background in philosophy and urged him to start his studies anew.
M. Waldman was a chemistry professor who advised Frankenstein. At the beginning of chapter 4, Victor says, "From this day natural philosophy, and particularly chemistry, in the most comprehensive sense of the term, became nearly my sole occupation." In the end, one could conclude that Frankenstein studied chemistry during his years in college.