Were all the townspeople devil worshippers as Young Goodman Brown thought?
This is not clear. We never know if Young Goodman Brown actually experienced all of this or if it was a dream. Hawthorne did this deliberately. The story is about the duality of humanity. We all, or all the townspeople, are capable of good and evil. This is not to point out that we all are hypocrites. The point is that good people are capable of bad deeds and vice versa.
The other point of the story is that a failure to accept this is a denial of the reality of the world. Goodman Brown believes he has lost his religious faith and his faith in humanity at the very thought that a seemingly pious person, like Goody Cloyse, could speak, let alone worship the devil. Brown thinks that if evil exists to that extent, then there is no goodness in the world powerful enough to combat it.
To rephrase your question in a metaphorical sense, I’d ask if all the townspeople had evil thoughts, were tempted by evil deeds or actually committed evil things. The answer is yes, but everyone to greater or lesser degrees. Every human faces this duality. Everyone in the town has “walked with the devil at night,” which is to say that even if someone chooses the righteous path, he/she must have also contemplated the evil path. This is just the logic of choice. You can’t make a choice unless you are presented with two or more options. This is the good/bad duality characteristic of human nature.
Even Goodman Brown was left wondering if it was a dream or not. In the end, it doesn't matter. Whether it was a dream or a real experience, Brown could not accept the thought of such evil.