Because Jim decided to sneak away "towards my escapade" on his own without informing any of the other men, the others just assumed that Jim had deserted the group in order to join up with Long John Silver. Jim managed to escape during the heat of the mid-day sun, taking "French leave" of his friends in order to find Ben Gunn's little boat. From there, he would sail until he found the Hispaniola: He would recover the schooner and rescue his friends. When they finally met up again, the doctor explained.
“So, Jim,” said the doctor, sadly, “here you are. As you have brewed, so shall you drink, my boy. Heaven knows I can not find it in my heart to blame you; but this much I will say, be it kind or unkind, when Captain Smollett was well you dared not have gone off, and when he was ill, and couldn't help it, by George, it was downright cowardly!”
The doctor thought differently when they discovered their young friend's true goal, however.