Dr. Roylott's plan backfires because he is unsuccessful in killing Helen in the same way that he killed her twin sister, Julia, by allowing a deadly snake to crawl into a vent, enter her room and bite her during the night. Julia was bitten as she slept in her bed which was nailed to the floor so that the snake would easily land on the bed as it crawled down the fake bell cord.
Julia's murder was only the first part of Dr. Roylott's plan, he killed her so that he did not have to give her the inheritance that her mother put aside for both her daughters which they were to receive when they each married. Julia was engaged to be married when she "died suddenly" of unknown causes.
Helen becomes very suspicious, after she becomes engaged, and her stepfather, Dr. Roylott asks that she begin sleeping in Julia's old bedroom.
"Their client, Miss Stoner, is in "a pitiable state of agitation, her face all drawn and gray, with restless, frightened eyes, like those of some hunted animal." In spite of her miserable appearance Holmes sees in her the courage to fulfill a daring plan to trap the villainous Roylott."
Once Helen begins sleeping in her dead sister's room, she starts to have strange experiences in this room.
"Because during the last few nights I have always, about three in the morning, heard a low, clear whistle. I am a light sleeper, and it has awakened me. I cannot tell where it came from-perhaps from the next room, perhaps from the lawn. I thought that I would just ask you whether you had heard it." (Conan Doyle)
Dr. Roylott does not get the opportunity to go through with his plan because Helen seeks out the help of Sherlock Holmes because she is afraid for her life. She wants Holmes help in solving the mystery of the death of her sister. Once Sherlock Holmes gets involved in the case, he wisely determines that there is a motive of money involved regarding Dr. Roylott and the twins, Julia, the dead sister and Helen. Holmes ends up figuring out exactly how Julia Stoner was killed because he and Dr. Watson spend the night in Julia's old room, sitting in the dark listening. Holmes uncovers the truth, and Dr. Roylott is actually killed by his own snake. Had Sherlock Holmes not gotten involved with the case, Helen Stoner would have died under mysterious circumstances just like Julia did two years earlier.
In this case, however the backfiring is the failure of the plan, so in that sense, there is not much of a distinction anyways.
His plan backfired because Dr Roylotts did not manage to kill his intended target, instead killing himself however.
Backfiring is thus a subset of failure, but this should not contribute to much difference.
While pmiranda2857 did a great job detailing Dr. Roylott's motive, I think it is important to distinguish "failing" and "backfiring." If a plan fails, it simply means the plan did not work, whereas if something backfires, an element of the plan worked in the opposite way of how the person who formed the plan expected it to work. In other words, not all plans that fail are a result of something backfiring. The reason I am taking the time to clarify this term is because when pmiranda2857 begins, "Dr. Roylott's plan backfires because he is unsuccessful in killing Helen in the same way that he killed her twin sister, Julia," she is essentially saying, "Dr. Roylott's plan backfired because it failed," which is pushing us into the realm of circular reasoning. The actual backfiring that occurs here is not the failure itself, but the way in which it failed. Dr. Roylott wanted to kill Helen by setting a venomous snake upon her while she slept. This worked just fine with Helen's sister Julia, getting her neatly out of the picture. However, because Helen involved Sherlock Holmes, things turn out differently for Roylott when he sends the snake down the rope to bite Helen. Instead of a vulnerable sleeping Helen, an alert Watson is in the bed, carefully watched by Sherlock Holmes. Holmes is able to chase the snake back up the rope where it proceeds to bite Dr. Roylott instead of Helen. That is why his plan backfires. He set out to make the snake bite Helen to kill her, and instead the snake bites him and his own life is ended. The plan and the result are in total opposition of each other, based on Holmes' involvement and the snake's unexpected change in direction.