When one asks whether statements in the form "he said X" are facts or inferences, one is actually asking about two different statements, one being a claim that someone said a certain thing and the other being the claim made in the person's statement.

Take the statement: "Bob said: 'The sun revolves around the earth.' "

The complete statement, "Bob said: 'The sun revolves around the earth.' ", makes a factual claim that Bob said a specific thing. It is true if Bob indeed said 'The sun revolves around the earth', and false if Bob did not say it. The speaker uttering the statement does not appear to be inferring anything, but simply recounting a fact.

Bob's statement also appears to be a simple assertion of a fact. It is unlikely that it could be arrived out without a chain of inferences, i.e. that the sun appears to move in the sky (information any of us can obtain through sense perception) and that this movement can be explained by inferring that the apparent motion is caused by the sun's actual motion. It should be noted that the statement "the earth revolves around the sun" is also based on inference.

Whether something is inferred has no bearing on whether it is or is not true. In order to determine truth of inferences, one must look at whether the premisses are correct and whether the conclusions follow logically from the premisses.