The only real reason to believe that this man is in fact Ichabod Crane comes from his physical description. We are told that he is tall and dry looking and that he does not seem very inclined to laugh. These things could well describe Ichabod Crane. However, they could just as well describe any number of other people and are not, therefore, good grounds for concluding that the listener is in fact Crane.
If we look at what the man says, there is little to make us think that he is Crane. He does not dispute the story or try to say anything that would make the listeners think better of Crane. Instead, he simply asks what the moral of the story is and then says he thinks the story is somewhat extravagant. This hardly seems like what Crane would say if he heard the story being told. One would think that he would be trying to defend himself and to make himself look less like an object of humor.