The main problem with trying to compare "fact" and "inference" is that it is an odd pairing. Logicians or communication theorists distinguish between "premises" and "inferences", where a premise is an initial proposition assumed and an inference something that may be derived by performing logical operations on one or more premises.
The term fact (normally used in a rather imprecise popular form) refers not to the logical status of a proposition as a premise or an inference, but rather to some element of correspondence with external reality. The distance from New York City to Los Angeles might be a fact. The time it takes to drive from one to the other can be arrived at empirically (by driving from one to the other to find out how long it takes) or by inference, based on calculations of distance and speed. Thus the claim that "it takes X hours to drive from NYC to LA" can be either a statement of empirical fact or an inference, depending on how you arrived at the claim.
Facts are statements based on data or previous experiences that can be proved true or false.
Inferences are statements based on assumptions i.e., when we assume something based on what we see, that can be right or wrong. In short the best guess or predictions we can make regarding any situation or personality of a person etc.
Suppose when an accident takes place and we just hear about it and were not present their at that time. The fact is that the accident took place. Inference is to assume whose fault can it be, did people live or anything that we assume that can be right or wrong.