There are a couple of elements that are happening in the question. One reality is that Dr. King feels so convinced in the authenticity of his cause that he does not see the need to respond to criticism of it. Dr. King believes that "work" and "deeds" are more important than rhetorical response:
Seldom do I pause to answer criticism of my work and ideas. If I sought to answer all the criticisms that cross my desk, my secretaries would have little time for anything other than such correspondence in the course of the day, and I would have no time for constructive work.
In this phrasing, he is able to draw out an important Biblical parallel in how figures like Jesus focused more on work and actual deeds than mere words. "Constructive work" is the fundamental justification behind why he does not answer critics of what he does. Dr. King pulls from the Biblical idea that individuals must commit themselves to work and service of a higher ideal and not worry about the external justification that might result. Living for an ideal is far more important than capitulating to criticism of it.
In the setting of the letter, Dr. King is able to respond to the clergy who criticized him because the fellow clergy who criticized him "are men of genuine good will" and that their "criticisms are sincerely set forth." It is for this reason that Dr. King addresses their concern. Similar to Gandhi's own writing while in jail, Dr. King understands that his imprisonment offers him an opportunity to articulate why he does what he does. It gives him a chance to speak out for what he deems important. In doing so, he demonstrates a sincere respect for those who have criticized him and in recognition of their intrinsic worth as members of clergy, Dr. King offers a response from the insides of a jail that might house his body, but not his spirit of resistance.