Let us remember that in Chapter Eight the dwarves and Bilbo appear to be in a dire situation. They are walking through the middle of an enormous forest with no hope or sign of ever reaching the end of it. In addition, they have ran out of water and food, and are incredibly thirsty and hungry. Note how they are described by the author in this chapter:
There was nothing now to be done but to tighten the belts round their empty stomachs, and hoist their empty sacks and packs, and trudge along the track without any great hope of getting to the end before they lay down and died of starvation.
It is in this state therefore that they see twinkles in the dark off the path, that they think are evidence of other people and, more importantly, food and water. It is their hunger and starvation that tempt them off the path and make them ignore the warnings they had been given by Beorn and Gandalf. Of course, the magic of the elves they disturb leads them ever-further away from the path until they are hopelessly lost.