I suppose that we ignore friction and the weight of a string, and that the system starts from rest. I'll use the downward y-axis starting from the initial position of `m.`
Denote the speed of a mass `m` as `V.` Then the outer edge of a pulley will have the same linear speed. It is known that the kinetic energy of a linearly moving mass m is `(m V^2)/2` and the kinetic energy of a rotating pulley is `(M V^2)/4.` The potential energy of a mass m is -mgh (it moves down), and of course V(t)=h'(t) (the speed is the derivative of the displacement).
So we obtain a simple differential equation:
`mgh=(m (h')^2)/2+(M (h')^2)/4,` or
Integrating this we obtain `2sqrt(h)=sqrt(a) t+C,` and C is obviously zero.
The solution is `h(t)=a/4 t^2` (downwards), and so the acceleration is `a/4=(mg)/(2m+M).` This is the answer.