The basic issue in these cases has to do the the establishment clause in the First Amendment. This is the part of the amendment that says that the government cannot make laws "respecting an establishment of religion."
At its most basic, this means that the government cannot set up an official state religion. The clause is also taken to mean more generally that the government cannot promote religion. The Supreme Court has set up a three part test known as the "Lemon Test" to determine when the government is doing too much to promote religion. As the link below says, the Lemon Test says that a governmental action is acceptable if it
(1) has a secular, or non-religious, purpose; (2) has a main effect that neither advances nor restricts religion; and (3) does not foster excessive entanglement, or mixing, between religion and government.
On issues of prayer in school, the courts have had to prevent violation of this test while also preventing the violation of student rights to free speech. The general rule has been that any prayers at school events have to truly be spontaneously done by students. The prayers cannot be approved of by the school or in any way promoted by the school.