I think that society should not have the right to tell you how to behave on your own time, in your own home, as long as what you are doing does not harm other people. This, however, is a very hard line to draw.
For example, let us look at the issue of adultery. Should society ban adultery and punish that crime? You can say no one is hurt. But your spouse will be hurt by your adultery. And your kids might be hurt by it if the adultery results in a divorce. If your kids are hurt by it, society may eventually be hurt by your private actions if your kids grow up out of control because of the impact of the divorce.
You can say the same thing about homosexual behavior. Some people would say this is completely private. But others would argue that this behavior hurts society by eroding the idea that heterosexual relationships are the norm and the basis of society.
So I would say that private behavior should not be regulated, but I have to admit it is hard to know for sure when private behavior affects other people enough to warrant government intervention.
All societies do limit and regulate behavior to some degree. That is the nature of societies. Behavior which is dangerous or damaging to the society as a whole is limited by laws, but those laws are passed because society has deemed them important.
For example, perhaps one believes that using drugs in your own home, on your own time does no harm to society, so why should it be restricted. On the other hand, the illegal trafficking of those drugs, and the violence that is associated with addicts and dealers does harm society.
Most libertarians believe you should have the right to do whatever you want as long as it doesn't take away others' rights or hurt society at large. They believe that is the very definition of freedom.
I think that society has an obligation to curtail individual rights of privacy if these actions would threaten others. The idea of individual rights of privacy being limited when it comes into conflict with the vision of others. This comes out of a social contract that individuals enter with others in a pluralistic setting. If individuals' private actions are ones that would cause danger and harm to others, then I believe that some curtailing should be evident. I am not sure that pure rights, entitlements without something finite, can exist in a heterogeneous setting where others are threatened. For example, if an individual is engaging in the manufacturing or distribution of drugs or participating in actions that have socially destructive ends, there should be a curtailing of these rights as they endanger others' freedoms.