Soil retains water in the interstitial spaces between the soil particles; the amount of space in a soil sample is described as its porosity. Generally speaking, larger sized particles don't pack together well, so they tend to have bigger spaces between them which can pass a lot of water through quickly. Soil made from smaller particles will have less space between the particles, and so will be less porous.
A soil made up of mixed particle sizes has much less porosity because the particles tend to sort in a way that minimizes the interstitial space. Study this illustration and you'll see why: