In Knowles' A Separate Peace, I don't think the reader is given any specific information about Brinker not being an athlete. It's possible that a line or two about the subject exists and I'm just not remembering it, but in general, Brinker's interests just lie in other areas. Not everyone that could be an athlete desires to specialize in athletics.
Brinker is an organizer and a leader, and he likes to be in control, or at least think he is. Remember his obsession with the hard cider he secures for their "Olympics." Also, he is the organizer and the driving force behind the mock trial that tries to establish what or who exactly caused Finny's accident. Brinker is busy dealing with normal teenage angst, as well as his impending enlistment into some form of the armed forces fighting WWII. His interests just don't lie in athletics.
Brinker is more of a boss and a controller than an athlete. It does not seem to be of great interest for him to go out for sports. The book states that Brinker looked athletic but was not. He was into other things that he liked, and they kept him busy. Brinker was into politics, arrangements (organization), and offices. Brinker is a popular guy who likes to be centralized with others.
Brinker is also a determined person. He is determined to try and monitor his cider at the sports events and later he is determined to get to the truth behind Finny accident. He is a friend and an adversary all rolled into one. Brinker is also a realist and says what he thinks which he directs at Finny in trying to make him face his own affliction openly.