The biggest argument would be that such a view takes an incredibly narrow view of what constitutes education. Surely education isn't just about academic learning, but also about other kinds of learning too. Sports in this sense are a vital part of a child's learning because they learn things through playing sports and participating in team activities that they are unable to learn through sitting at a desk in a classroom. Essential life skills such as team work, cooperation, working together, developing skills and strategies to win are developed through the playing of sports. To not have any extra-curricular sporting activities would therefore mean that the students would suffer.
Your argument needs to focus on the holistic nature of education and the way that it should involve the whole person. Education is never just about the brain and intellect of a student. Education should focus on not just the brain but also the body and the social element of who we are as humans. Sports in this sense are valuable and completely necessary when we think of education as being a holistic phenomenon.
The most commonly given reason that this would be a bad idea is because sports are said to keep many at-risk kids in school. The idea is that there are people who would quit school if it weren't for the fact that there are sports for them to play. If sports are cut, these students drop out.
Similarly, there are those who say that sports are good for all students who participate, whether they are at-risk or not. These people say that sports teach good time-management habits (because you are so much busier and have to manage your time), good teamwork skills, and all-around strength of character. In this view, having sports is part of the educational process.