Atticus himself provides the answer to this question. When Scout asks what would become of him if he lost his appeal, Atticus answers that he would "go to the chair" unless "the Governor commutes his sentence" (219). Rape could be a capital crime in Alabama even for white offenders, and despite the fact that he is obviously innocent of the crime, Tom faces death in the electric chair, the preferred method of execution in most states in the 1930s. But Atticus seems relatively optimistic that he has a chance of at least getting a new trial after appeal, and he is especially aggrieved that Tom was convicted on circumstantial evidence alone. But Atticus, being a respected white attorney, is in a position to place faith in the justice system, even after having witnessed the travesty of Tom's conviction. Tom, as a black convicted rapist in Jim Crow Alabama, apparently does not share this conviction. Committed to jail and awaiting his sentence, Tom attempts to run away and is killed by the guards. Upon learning the news, Atticus concludes that Tom "was tired of taking white men's chances and preferred to take his own" (236). In his position, it was impossible to see how he might receive justice, no matter what Atticus thought. Facing execution, he fled in desperation, an act that brought about the same result as a failed appeal would have done.
The trial of Tom Robinson is modeled after the trial of the Scottsboro Boys in 1931. During this time any black man convicted of raping a white woman would have been put to death. So, if Tom were to lose his appeal, he would have been killed.
Throughout the story of the arrest and trial of Tom Robinson in To Kill a Mockingbird,there are several parallels to the events leading up to the Scottsboro trials, such as the lynch mob, the women who lied about being raped, and the men who were actually innocent. Their trials went to appeal, and then, finally, to the United States Supreme Court.
In the Jim Crow South, the lynching of black men accused of raping white women was common. Tom may be afraid this will happen to him. Also, the chances of his winning an appeal (even though there was no medical evidence of rape and Mayella clearly lied) are not likely, especially considering that the jury would most likely be all white.
The worst that would happen would be that Tom would have to serve his sentence. However, Tom's actions prevent Atticus from even filing an appeal, showing that Tom doesn't believe he has a chance to win the case in a higher court.