There is one main way that the judicial branch can check the powers of the other two branches in the American governmental system. That is through the power of judicial review. Judicial review is the power to declare acts of the other two branches to be unconstitutional.
This is a very serious check. It is possible for the judicial branch to invalidate a law that has been passed by Congress and signed by the president. If the judiciary rules that this law (or an action of the president) violates the Constitution, the law is no longer valid. This gives the courts the power to, all on its own, throw out something the other two branches have done.
We may see this happen with the health care reform law passed this year. One judge, at least, has ruled that it is unconstitutional and that Congress and the president may not require people to have health insurance. If this happens, the judicial branch will be telling the other two branches that something they worked hard on and passed is illegal.
Another very important check or limit the Federal courts have over the Congress and the President is that they are appointed for life. Once they are nominated and approved by the President and the Senate, respectively, then they cannot be fired unless they commit a crime and are impeached.
In this way, they are an independent judiciary, and are immune to any pressure the Congress or the President might want to place on them to rule on cases in a certain, more favorable, way to them. This is critical because it allows them to stay true to the Constitution without fear of reprisal from either the public or the other two branches. Their jobs are secure regardless of their rulings.