Judge Danforth comes to Salem with an air of superiority. He automatically does not like Parris's fawning attitude. Danforth is observant, experienced, and intelligent enough to realize that Parris is so defensive of the accusers because two of them are his family members. If Parris, like a good Puritan minister, had had control over his household, the situation would not have escalated to such a point. Additionally, Parris continues to interfere in court procedures and this gives the impression that he has some authority in the court. Danforth, being the power hungry individual that he is, does not want his authority questioned or shared.
In Act 4, when Parris tells Danforth that Abigail has absconded with his money, Danforth is furious and blames Parris. Parris's foolishness could bring about Danforth's downfall; so Danforth deals harshly with him and wants no more association with him.