I think that the destruction of the windmill is very significant. It is significant because of the physical impact that it has on the residents of the farm (causing them to have to work harder and have a more difficult life), but it is more important because of the impact that it has on the society of the farm.
I think the biggest impact that it has is the fact that it leads to all these executions and all this distrust. It is when the windmill is destroyed that Napoleon first really turns on his fellow animals (other than Snowball). After the windmill is destroyed, the farm becomes something of a police state where animals have to watch what they say and do and where animals can be executed for seemingly no reason.
So the destruction of the windmill is a huge deal because it plunges Animal Farm into this horror of executions and suspicions.
It was very significant as Napoleon used the destruction of the windmill to his advantage, accusing Snowball of sabotaging the windmill, thus causing it to collapse, blaming all the bad things that had happened to the farm squarely on Snowball's shoulders. Those who were at first fiercely loyal to Snowball had to back down, in fear of further backlash. Also, the animals had to work even longer hours carrying rocks up the quarry to rebuilt the damaged windmill, so they had lesser time to rest.
Next, after the destruction of the windmill, came Napoleon's rule by brute force- rule by terror. He was very suspicious, and was very cold-blooded, killing all those conspirators who opposed his rule and his totalitarian control. By then, it can be seen that Napoleon had completely disregarded the Seven Commandments, one of them saying that animals cannot kill other animals. When the hens rioted, they were killed for their resistance. But, an exception is that some animals were coerced into confessions- crimes that they did not commit but was forced to admit, and later, the dogs slit open their throats and they died on the spot. This largely shows the dark nature of Napoleon's character and shows how power-hungry and ambitious Napoleon was, to remain a foothold in the farm and maintain order and security in it.