The fundamental concept here is quite simple: Bigger things are heavier. The clast size tells us how large the individual pieces of rock are, and thus how heavy they will be.
Therefore, a sediment with large clast size (such as gravel) will require more energy to transport than a sediment with a small claster size (such as sand). Small clast sediments can often be carried simply by the wind (sand dunes are actually sediment being transported by wind). Large clast sediments are more likely to be spread by water.
Small clast sediments will generally be carried further than large clast sediments. This creates an effect called sorting, where long periods of erosion will tend to separate out sediments by their clast size, with larger clasts being dropped first or left behind, and progressively smaller clasts being dropped further away until the very smallest clasts are carried the furthest.
If this were the only process, clasts would all be separated by now. There are other processes which can mix them back together, including volcanoes, glaciers, earthquakes, and landslides. These occurrences are much rarer, but have large, catastrophic effects; erosion, by contrast, is very slow but very steady.