Comment on the answer given by pohnpei397:
It seems to me that the murderer is not the narrator, but Uncle Caesar.
The dead Caswell is clutching something in his hand. When he finally drops it, the narrator picks it up and pockets it. Later, when the narrator is riding away from Nashville, he examines the object. It is a "yellow horn overcoat button the size of a fifty-cent piece, with frayed ends of coarse twine hanging from it."
This is the unique button that the narrator noticed, near the beginning of the story, on Uncle Caesar's coat.
Uncle Caesar, who always tried to take care of Mrs. Caswell, has murdered Caswell, who always steals her money. The narrator, who sides with Uncle Caesar, takes the evidence (the button) and destroys it by throwing it out of the train window.