Geography is, in a sense, a cross between a social science and a physical science. It might be more proper to say that there are two kinds of geography. One is physical geography, which is a physical science. The other is human geography, which is a social science.
In physical geography, the focus is on the natural world. Physical geography studies things like the environment. It looks at things like landforms and the climate and how they come to be the way they are. This is clearly not a social science.
However, human geography is a social science. In human geography, we study the ways in which human activities are arranged in space. For example, we study how and why cities arise in the places that they do. We study the forms of cities and ask why they come to be shaped as they are. As another example, we study how and why people move about across the face of the earth. All of these are subjects proper to a social science because they are mainly concerned with human society and human activity.
Thus, world geography is both a social science and a physical science.