Electron/hole mobility is inversely proportional to the effective mass (m*) of the particle (u=qt/m). Therefore, heavier also means slower/less mobile. Thus the question boils down to, "why do holes tend to be heavier in semiconductor materials?"
The effective mass (m*) of a particle is defined as the second derivative of Energy with respect to momentum, inverted. Recall that the energy-momentum diagrams for a semiconductor resemble two parabolas stacked on top of one another, one for the valence band (holes) and one for the conduction band (electrons)--see the link in the sources. Invariably, the conduction parabola is narrower than the valence parabola.
The narrower the parabola, the larger the second derivative of Energy with respect to momentum. Since m* ~ 1/D(D(E)), a narrow parabola = smaller m*. Therefore m*,electron < m*,hole. So a hole is heavier because of differences in the energy bands for holes and electrons.