These two works can be generally discussed as hybrids, taking into account how they diverge from their novelistic templates. Hybrids are a confluence of different elements: in these cases, genres are combined and subverted.
Novels have been categorized into different, basic genres—for example, Adventure, Romance, Gothic, and so on. Another designation I’ve seen is Domestic (as in, a portrait of marital relationships). Stevenson, who is known for swashbuckling adventure, has, in this case, written a story with a wedding at its center. At a basic level, the author has mixed the Adventure and Domestic genres. (To go deeper into the plot, the hybrid theme can be seen in Stevenson’s treatment of the union of a colonialist and a native woman and, more generally, in the melting pot of foreigners living amongst the islanders, all of whom are equal in class, in contrast to the subordinate natives.)
In Conan Doyle’s novels and short stories, Dr. Watson is Holmes’s foil. The Sherlock Holmes stories are always told from Watson's point of view and thus include his own subjectivity and judgement. This already necessarily differs from Holmes’ own thought processes and behavior. What’s out of the ordinary about The Sign of the Four is that the narrator shifts to the point of view of another character for a lengthy story-within-a-story. This effectively changes the narrative from a representation of the protagonists’ reality to that of the antagonist, Jonathan Small.
According to tetheredbyletters.com, “To qualify as a hybrid form you need only to cross genres, or alter a genre, into something unlike the original.”