How can the judicial branch check the powers of the other two branches?
In theory the judiciary can indeed check the powers of the other branches of government. In that sense, the power of the judiciary (which it arrogated to itself in Marbury v Madison) to strike down unconstitutional laws is indeed significant. However, in practice, there are a number of problems with what may be called legal, as opposed to political constitutionalism, as exemplified by the Federal court system.
The first problem is that of enforcement. The courts can strike down unconstitutional laws, but can in reality do little to enforce their own rulings. This can allow court decisions to be ignored, or more likely, creatively re-interpreted to challenge the relevant court's decision in a surreptitious manner. The court must rely on the Federal authorities for enforcement, the very same authorities who may very well be in breach of the Constitution themselves.
Federal courts are also not in permanent session. They cannot then maintain a persistent check upon the alleged abuses of the other branches. Indeed, to some extent the courts are playing catch-up, inevitably responding to unconstitutional acts in a reactive, rather than a proactive way. And even when the courts do eventually rule, their judgements relate to specific cases and cannot provide the kind of general oversight required of administrative decisions.
Those who advocate a more strictly constructionist method of constitutional interpretation use this example to explain why the judiciary has been somewhat remiss in recent decades in reining in the dramatic expansion of the Federal government and its powers, going way beyond those expressly authorized by the Constitution.
There is one main way that the judicial branch can check the powers of the other two branches in the American governmental system. That is through the power of judicial review. Judicial review is the power to declare acts of the other two branches to be unconstitutional.
This is a very serious check. It is possible for the judicial branch to invalidate a law that has been passed by Congress and signed by the president. If the judiciary rules that this law (or an action of the president) violates the Constitution, the law is no longer valid. This gives the courts the power to, all on its own, throw out something the other two branches have done.
We may see this happen with the health care reform law passed this year. One judge, at least, has ruled that it is unconstitutional and that Congress and the president may not require people to have health insurance. If this happens, the judicial branch will be telling the other two branches that something they worked hard on and passed is illegal.
Another very important check or limit the Federal courts have over the Congress and the President is that they are appointed for life. Once they are nominated and approved by the President and the Senate, respectively, then they cannot be fired unless they commit a crime and are impeached.
In this way, they are an independent judiciary, and are immune to any pressure the Congress or the President might want to place on them to rule on cases in a certain, more favorable, way to them. This is critical because it allows them to stay true to the Constitution without fear of reprisal from either the public or the other two branches. Their jobs are secure regardless of their rulings.