How do you know that it is a hound which is frightening the family in The Hound of the Baskervilles?
The first thing which might tip off a reader about the mysterious beast which is pursuing and killing members of the Baskerville family is the title of the story-- The Hound of the Baskervilles. Even if one did not know the title of the story, several characters are convinced of and mention the (ghostly) murderous dog.
Early on in the story, we read about the death of Sir Charles Baskerville, who was found surrounded by giant paw-prints in the grass. Dr. Mortimer, who shares the death of Sir Baskerville with us, suspects that his friend may have been killed by a curse on the Baskerville family. Centuries prior, Sir Hugo Baskerville was supposedly attacked and killed by a giant dog. Since then, there has been a folktale of a hell hound which haunts the moors. All these factors considered, in addition to recent reports of a great, glowing dog being spotted on the moors, naturally direct the characters as well as the reader to suspect there may really be a killer hound.
Unfortunately, Holmes and Watson do not uncover any hard evidence of the dog (such as fur) until they encounter it head-on. Nonetheless, Holmes has been able to work out the motive and means of the real culprit.
Although Dr. Mortimer believes Sir Charles may have been killed by the fantastical hellhound of the "Baskerville curse," Sherlock Holmes, being a scientific and rational investigator, is not sucked in by that supernatural explanation. He believes a physical hound is involved.
First of all, by Dr. Mortimer's own account, the footprints of a gigantic hound were visible "some little distance off" from Sir Charles's fallen body. Dr. Mortimer also reports that prior to Sir Charles's death, neighbors in the area said they had seen a "huge creature, luminous, ghastly, and spectral" that reminded them of the dog in the Baskerville legend.
Although readers may not pick up on the significance of the clue as soon as Holmes does, the fact that Sir Henry has a single boot stolen from two pairs of shoes also indicates the presence of a real, living dog and a human agent behind the creature. The brown boot was taken to give the scent of Sir Henry, the next victim, to the hound, but since it was new, it held no scent, so an old black boot was taken next.
Finally, Watson hears "a strange cry" on the moor that sounds like "the cry of a hound."