The main reason why the right of the people to assemble in public is not absolute is because of time, place, and manner restrictions. These restrictions apply to all assemblies, regardless of what they are for.
If the right to assemble were absolute, it would mean that we could never prevent people from assembling for a legal purpose anywhere. For example, we could not prevent people from assembling in the middle of a busy street at rush hour. We could not prevent people from assembling in a residential area at midnight and making a lot of noise. We would not be able, in an extreme case, to prevent people from assembling on the floor of the House or Senate and disrupting legislators’ efforts to make laws.
These are clearly scenarios that we would not be able to accept. Therefore, we put limits on the freedom to assemble. We make sure that those limits are meant only to prevent people from disrupting the lives of others in excessive ways. We enforce those limits without regard to the point of view of the people who want to assemble.