The main difference between the two is that British literature is normally written by British citizens and American literature by United States citizens. Many Canadians would point out, though, that as Canada is also located in North America, using "American" to signify the United States is a misnomer; Canada certainly also has a quite important literary tradition.
When you refer to "British" as opposed to "English" literature, this encompasses a long time period and multiple languages. The beginnings of British literature would include oral epics in Celtic languages, Old English literature, and medieval Latin literature written in England as well as a long tradition of modern English and Gaelic literature.
Literature in North America also begins with the oral traditions of the Native Americans (or First Nations in Canada). In the United States, Colonial Literature encompasses the period from the 1620s-1776 and is then followed by United States literature; both periods were heavily influenced by British models. Canadian literature is comprised of both anglophone and francophone literature, and continues to be written in both Canadian national languages.
While there are specific linguistic differences such as spelling and idiomatic phrases, all three literatures are so vast and varied that one should not really generalize about them.