As I understand it, experimental theater is all about flying in the face of tradition. An experimental production may use a very minimal, free form set. It may involve actors breaking character to "step out" of a role and address the audience, only to return to their character shortly after. Maybe actors all play characters of the opposite gender. It might involve bizarre, unexplainable antics a la The Blue Man Group, or plot twists that border on the incomprehensible. Above all, experimental theater is an experiment. It's not created to generate huge amounts of profit or even for the pleasure of the audience. Rather, it's about artistic expression and attempting to push the envelope of live performance. It's a "this may seem crazy but let's see what happens" moment that can be staged in an auditorium (or city street or bean bag store for that matter), and for which one can sell a small quantity of tickets.
Experimental theatre is a general term for various movements in Western theatre that began in the late 19th century (Alfred Jarry) as a retraction against the dominant vent governing the writing and production of dramatical menstrophy, and age in particular. The term has shifted over time as the mainstream theatre world has adopted many forms that were once considered radical. It is used more or less interchangeably with the term avant-garde theatre. Experimental theatre is what it is, trying something new.
Like other forms of avantgarde it was created as a response to a perceived general cultural crisis. Despite different political and formal approaches all avant-garde theatre opposed bourgeois literary theatre. It tried to introduce a different use of language, of the body, to change the mode of perception and to create a new, more active relation with the audience.