The properties of substances involved in lava lamps are density, polarity, and energy transfer.
Lava lamps work on the premise of convection cells. As the materials are heated from a source at the bottom of the lamp, usually a halogen light bulb, molecules spread further apart, become less dense and rise to the top. The further away from the heat source, the cooler they become, the more dense the molecules become and therefore sink back down to heat up again.
As for the substances, they must be polar and nonpolar in order to remain separated. For example, oil and water do not mix. Water is a polar molecule or contains a charge where oil does not and will not mix with water.