Yes, it is subject to diminishing marginal return. For the first hour you study, you'll probably get a lot higher of a grade. But if you studied for 6 hours, the last couple hours probably wouldn't do you much good. You'd be too tired for one thing. For another, you probably won't understand something by studying 6 hours if you didn't understand it by studying for 5.
The variable input is the amount of studying. The fixed input I'd say would be your brain and your body and perhaps your talent for economics. More studying doesn't help because you can't (short-run) increase your brain capacity or your body's ability to concentrate for a long time.
I would like to answer this question rather careful as it may influence the behavior of some students in relation to their studies.
It should be noted that even as per economic principles, the law of diminishing marginal returns is not applicable to the entire range of possible production levels. In general, firms experience a situation of increasing marginal returns as they start from zero or very low production levels. As they increase production they are generally able to reduce the marginal cost. This continues upto a level of production, with decreasing rate of fall of cost, till it reaches a point of minimum marginal cost or maximum marginal return. This phenomenon of reduction in marginal cost is also known as economies of scale.
After the point of maximum marginal returns, which is same as point of minimum marginal cost the reverse phenomenon of dis-economies of scale is observed. In this stage, as the total production level is increased further, the phenomenon of diminishing marginal returns is observed. With each additional unit produced the marginal cost of production increase. Stated in the form of diminishing marginal returns, each additional unit of a factor of production gives progressively diminishing quantity of additional production.
It should be noted that the point at which the point of maximum marginal returns is reached and total relationship between production level and the marginal returns also depends on the nature of technology employed.
Now coming to the specific question of studying for exams, the theory of either diminishing or increasing marginal returns cannot apply, as in studies we are not talking about producing for selling. In studies the production is for self consumption. Therefore. what is relevant to studies is only utility to the student from the quantity produced and not just the quantity produced. However we can say that a phenomenon similar to the laws of increasing and diminishing marginal returns can be observed in studies also.
Let us also examine what are its implication for students. I believe the most important thing to note is that the stage of maximum marginal returns depends on the technology used. Thus it would be worthwhile for all students to learn and use appropriate methods of studies that give them maximum return for their efforts. Second important point to note that best level of studies appropriate for you may be at a point above the level of maximum marginal returns. It is important that even business firms in a competitive market find it better to go on producing and selling beyond this point of minimum marginal cost, till their marginal cost increases to become equal to the marginal revenue.