Matt is not sure why he finds it hard to trust Ben at first, but he seems to have a keen sense of who means him harm and who only wishes to help. He shows this later in the book when he realizes that although he does not like Attean because of the Indian boy's scornful, superior attitude towards him, he knows he can trust him. In the same way, on the surface, the things Ben says and does sound and appear to be benevolent, but somehow, Matt perceives that he is not a man to be trusted.
Ben has "an easy grin" and "seem(s) harmless", but while the things he says are essentially proper and positive, his actions belie a certain pushiness and presumptuousness that is not appropriate. Appealing to Matt's childlike sense of decorum, he pretty much invites himself in, invites himself to dinner, and invites himself to spend the night. He is nosy, and is not shy about examining everything about the cabin with his eyes. Matt correctly perceives that the man is covetous of his father's rifle hanging by the door.
Ben is the type of person who just talks too much, and he also lets on that he might be in trouble with the law. He tells Matt that he is "keeping as fur off from (the) river (as he) can, till things quiet down...warn't nothin' they could prove, but they sure had it in for (him)". Matt's "hunch" that Ben cannot be trusted is proved to be accurate when, in the morning, he discovers that his visitor is gone, along with his father's gun (Chapter 3).