This is a difficult question to answer without more information. The more information you give, the greater the detail in the answers. However, in essence theoretical vocabulary is the language that a particular discipline uses to discuss hypotheses and theories. A few examples should help.
If you were dealing with the topic of science, you would come across almost a new vocabulary. You would have to know the area well to be conversant with what was going on. However, the important point to keep in mind is that even this area of study is highly theoretical. Consider the study of interstellar clouds. You would come across words like: stellar remnants, interstellar medium, diffuse nebulae, area NGC 1999, and V380 Orionis.
Here is another example. You do not even have to go to the area of science. Even if you open up a theology text book and read about the theories behind divine election (predestination), you would come across words like: infralapsarianism and supralapsarianism.
In a word, theoretical vocabulary is the vocabulary created to expound new theories.
Thank you! The question I was asking specifically refers to literature and theoretical vocabulary and theories.