Work is defined as a force acting over a distance.
We know the distance (5m) but we don't know the exact force, because it acts at an angle. This means that the original force of 20N is essentially split up in two directions; the direction that the ball heads in along the axis of displacement, and a perpendicular direction that causes that displacement to occur at a 30 degree angle.
We can use trigonometry to determine the force acting in the direction of displacement. If we project the force as if it was traveling along a flat, horizontal plane, and the resultant displacement of the ball is along a line inclined by 30 degrees to that original plane, then the force acting in the direction of that displacement equals 20 x cos 30. This is equal to 17.32. This is the force acting in the direction of displacement.
W = Fd
W = 17.32 x 5
W = 86.6 Nm (or J)