Soil is actually a combination of weathered rock and organic material. The rich topsoil, thus, has the greatest amount of nutrients that plants need to grow. When you remove topsoil, you are left with a layer that contains more of the weathered rock material and less of the nutrient rich organic material. Plants would not be able to thrive and grow without the topsoil. When you return the topsoil layer to a building site, you are able to plant and grow a greater variety of plants and grasses.
One reason to save the top soil and spread it back around the same area is so that you don't have to transport it elsewhere. From purely a financial standpoint, it saves the builder money if he/she doesn't have to move the topsoil long distances.
From an ecological standpoint, the topsoil is not just dirt/soil. The top soil contains a lot of biological organisms. Insects and worms for example. Another type of living organism in the top soil is the plant life, i.e. weeds and other seeds. If the topsoil is transported to a new site, there is a risk to the native species in that new location.
Topsoil contains a lot of nutrients as well, so moving it to a new location robs the newly constructed area's ability to grow the type of plant life that a developer wants in the area. Something as simple as a garden to as complex as natural park areas would be harmed by the removal of topsoil.