We probably have to define "success" in more specific terms. From the wording of the question, it can be argued that few documentary films are "box office" successes because their primary intent is not to turn a profit. The documentary genre is not driven by material gain. It is not geared towards making money. The people who take documentary films have a message to deliver and have a story to tell. They are willing to forgo financial success in the name of this message. Documentaries do not seek to draw from "star power" to make money. They are not driven by appealing to mass audiences. The documentary style is usually driven by the need to uncover an aspect of reality and not largely patronized by individuals who seek escapism in film. It is in this where the focus of the documentary is entirely different. As a result, documentaries are not, in large part, financially successful ventures.
I think that a case can be made that ensuring that the documentary is made is a form of success. In a setting where so much of filmmaking and an cinema artist's voice is tied to profit and studio backing, the documentarian embodies a lost form of the art. The documentary's success might lie in the fact that it speaks, its voice is present. It does not tie itself to the fruits of cinema labor. It does not have to make a profit in order to be deemed as a success. In being able to articulate its story, success is evident. Modern cinema is a world in which so much of voice is materially driven, and where financial profit is critical to one's being. Pivoting from this condition, the documentary seeks only to tell a story. In being able to do so, one can argue that its success is evident.
I will assume that the premise of your question (documentaries have not been great successes) is true, as I don’t personally know the facts and statistics. As for why this is the case, I think a primary reason is because of how our culture sees “movies” and “documentaries”.