Sodium and potassium metals are kept in kerosene or in dry mineral oil. Both of these metals are in Group 1 on the periodic table. All metals in that group are very reactive with water, including moisture in the atmosphere. You may have seen videos where people have thrown chunks of sodium metal into bodies of water. Potassium reacts even more violently.
The reaction proceeds as follows (I've used sodium in the example, but it could be any of the Group 1 metals):
2Na(s) + 2H2O(l) --> 2NaOH(aq) + H2(g)
As you can see above, hydrogen gas is formed during the reaction. In more explosive reactions (potassium, rubidium, and caesium), enough heat may be created that the hydrogen gas catches fire. In fact, caesium explodes when exposed to water.
In summary, these metals are stored in kerosene because kerosene contains no water and protects these metals from being exposed to any moisture in the air that would cause these metals to react, potentially violently.