There are two possible reasons....
Leakage of gas through tiny holes(most plastics have tiny holes between the molecules), these holes are large enough for the gases to escape this process is called effusion. Effusion process can be studied in Graham's law of effusion which states that the rate of effusion is inversely propostional to Sq.root of molar mass of gas. Helium has lower molar mass and has small size so effusion of helium gas from balloon is higher than other gas molecules.
Charles law also explains that why the size of balloon decreases after few days, according to charles law the change in size of balloon is due to temperature difference. The temperature inside the balloon decreases which inturn decrease the volume of the balloon since density of gas increases due to which balloon size appears to be small.