Carlson thinks that Lennie took it before he left, and the others seem to agree.
Carlson came running in. 'The bastard's stole my Luger,' he shouted.
Since they have been talking about going after Lennie, it's clear who Carlson means.
Later, Curley confirms this:
'Don't shoot 'im?' Curley cried. 'He got Carlson's Luger. 'Course we'll shoot 'im.'
Knowing that all the other men think Lennie took the gun, George uses that assumption for his own purposes at the end.
'Did he have my gun?' [Carlson asked]
'Yeah. He had your gun.' [George responded]