A theocracy is a society that is ruled by a church, or by religious leaders. This comes from the Greek "theos" or god and "kratos" or rule (from which we get the "cracy" of democracy, autocracy, etc).
Miller calls Salem a theocracy because it was ruled, in effect, by the Puritan church. The Puritans believed that God would punish their community if it did not enforce proper religious practices. Therefore, they imposed their religious beliefs on everyone, thus making the colony a theocracy. This was possible in colonial America, and it was partly in reaction to this that the First Amendment to the Constitution was written after the US became independent.
He means that the society, in this case in the play's setting, was run by a Puritan church.