The term colonialism as used in literary studies on the most basic level means the same thing it does in standard English. It refers to one nation establishing colonies in a region outside its borders. This is different from the types of conquest that expand a nation's borders in that it may only involve establishing limited enclaves or bureaucracies.
Literary critics are interested in the way colonialism affects the literatures of both imperial and subjugated nations. One particular area of interest is hybridity, in which literary works take elements from both traditions. Another area of interest is how literature and the other arts can act as part of imperial ideology or as sites of resistance to imperialism.
Many literary critics also work in the field of postcolonialism, examining how the literatures of former colonies have been shaped by the experience of colonialism and how they struggle to reinvent themselves as culturally and politically independent.