He pretty much has to go with them.  At the time that they head south, Jeff is still not really well enough to go back to the army or to try to escape back to the Union army as he will later do.  Remember, the only reason that he is with the Jackmans in the first place is that he go so sick and needed to be put somewhere to recover.  So he hasn't recovered yet and has to stay with the family.

I believe that you can find this in Chapter 20.  His sickness is malaria and he has not recovered enough to stay behind.