I think that what Greene is referring to here is the impact that mistakes would have on the prestige and honor of the court system.
The phrase that you mention is given right at the start of the story. We are told that the jury is afraid of making mistakes because mistakes are like "domes of silence muting the court." Clearly, this is said metaphorically.
I think what this means is that the court loses its influence or trust with the people when it makes mistakes. If this happens, it is like the court is being silenced. It is no longer telling the people "don't worry, we are here to do justice." That message is muted when a mistake is made.
A dome is also used as an Informal term for a human head.Here ,in one the crucial stage of the story ,we can see that the ideal witness Mrs Salmon who at first stood as the eyewitness of the story is not able to identify the real murderer among the Adams.Maybe this silence of Mrs Salmon that muted the whole court is implied by the above words.
here in the story the court is in a real confusion as it is a case of 'identical twins' Sooo...the court loses its influence or trust with the people when it makes mistakes...as they fail in doing their job that is to give justice....hence
"domes of silence muting the court
means that if any mistaks made the court mutes...