The primary reason why Lennie has a mouse in his pocket is that he likes to stroke it and play with it. The reason he *constantly* puts mice in his pockets is that he has an unfortunate habit of killing them.
The mouse is there to help build his character, and also to function as a symbol.
The mouse in the pocket helps establish the character of Lennie as more child than man. He still has his child-like wonder and has not yet learned that mice are to be feared and hated. He is similarly naiive in matters of the world.
The mouse is also there to show us his innocence. He does not know his own strength and hurts that which he loves without meaning to. The mouse also foreshadows what lies ahead; we know Lennie has the potential to hurt an innocent creature accidentally; what we do not know is that he has the potential to hurt a human.