When Rutherford did his experiments the generally held view of matter was that it contained a mixture of positive and negative particles but that it was solid, with no space between the individual particles. Physicists also knew that like charges would repel and opposite charges would attract and that the closer together the charges were the stronger the interaction between them.
Rutherford designed an elegant experiment in which he used alpha particles, given off in the decay of radioactive isotopes, as his bullets. The alpha particles were known to have a positive charge. He set up targets consisting of thin films of various metals, primarily gold. Around the film, opposite the "gun" he placed a flourescent screen that would glow if hit by the alpha particles.
When he ran his experiments he found that most of the time the alpha particles went right through the metal film, as if there was nothing being hit by the particles. Occasionally the particles would deflect (bend) a little bit, and very rarely the alpha particles would bounce off at a greater angle or even bounce back toward the gun. Rutherford described it as if a cannon ball hit a sail on a ship and bounced back at the cannon that had fired the ball.
After numerous experiments and observations he determined that the metal films were not really solid but mostly empty space. Because of the behavior of the alpha particles he also concluded that something inside the film had a strongly positive charge that was deflecting the alpha particles when they came close. Furthermore, that positive charge was strong enough to actually stop the flight of the alpha particle and reverse its direction.
Rutherford contributed greatly to understanding in greater detail the structure of the atom.