In my opinion, there are two possible reasons.
First, the government probably does not think these guys are any kind of a threat to them. I mean, so what if there are some people out walking around when no one knows they're there? It's like the thing about the tree falling in the forest with no one around. If those guys are out there reciting their books and no one knows, are they really there? So practically speaking, these guys have no chance of affecting society so why worry.
Second, if the government goes after them, people might become aware of them. If people become aware of them, they might start wondering why those guys are out there. They might start actually thinking. That would not be good for the government.
So it's better to just leave them alone because they can't do any harm out there on their own.