The reason for this is that they were people who might be expected to be on the side of the Patriots yet who were not. No one would have expected a British soldier to be on the Patriots' side so they would not have been hated. But Patriots might well have expected that their own neighbors would be on their side. Therefore, when they were not it would have seemed like something of a betrayal. This would have led to the Tories being hated and abused more than might seem natural.
Because the Loyalists were seen as something of traitors to the American cause, they were hated much more than a simple enemy would have been.
Sun Tzu might say something along the lines of: In war, friends of your enemy should be treated as the enemy.