An open social system is one which is aware of, and involved with, its environment. A closed social system has rigid boundaries, views the external world as static, and relatively unchanging. The closed system functions within and for itself. (A watch is an example - check the link. A watch is closed. It does not need input from the outside world. However, it is not completely closed. It relied on a watch maker and for its function to have any merit, it must have someone using it for it to reach its potential or to have any meaning outside of its own ticking.)
Theoretically, a completely closed system would cease to exist, while a completely open system would be so open and integrated that it would cease to be distinguishable from its environment - like a substance dissolving completely and colorlessly in water. In practical reality of social systems, there probably is no completely closed or completely open system.
There can be systems externally open and internally closed and vice-versa. Depending on the environment and historical context, one may be more beneficial (for growth and evolution) than the other.
Typical examples of open social systems are schools. The more open (integrative and interdependent with) the external world (factors such as culture, economy, public and political policy), the more likely the schools are to be progressive and evolve. This is the generally accepted theory and it makes sense. A school as a closed system might be said to be looking out only for itself, its students and teachers, but as soon as it closes its doors to the outside world, it will fail. Ignoring social class, economics, political policies, and shifting ideas in all disciplines, the school would simply fall behind. A closed system defined laconically as "you do your thing, I'll do mine" could function in a perfect, non-changing world (if it was perfect, why would it change). But since this is not a perfect world (hence the need for continuous change, evolution, and progress), a system more open than closed is preferable. The reality is that the world is becoming more integrative and open systems recognize this increasing interdependence.