This is going to range extremely wide in scope and sequence. The discussion will go into different arenas. I would suggest that one particular and relevant limitation on human freedom is the role of economics. Economics and material reality can play a profound role on the reality of individual freedom. This is a source that can constrain freedom because one is bound by their material state of being. While this can change, the freedom that one has can be seen as being directly linked to the state of one's economic being.
I would also suggest that in a larger sense, our relation to other human beings can be seen as a source of constraint to our freedom. It is impossible to have pure freedom when interacting with others. Whether one wants to see it as a form of "the social contract" or metaphysically recognizing that our freedom comes at some cost in our being with others, pure freedom is something that is limited. The notion of being in a pluralist society or interacting with other people is one in which absolute pure freedom is absent. I believe that this is where one can find another source of constraint on freedom.