Ut tamquam scopulum sic fugias insolens verbum.
(Avoid the unusual word as if it were a cliff.)
Big words can often create a bad impression both in conversation and in writing. They can make people think you are just showing off. You should try to choose the right word but not the most impressive or sesquipedalian word.
One good way to improve your vocabulary is to look up the etymological derivation of a word when you consult your dictionary. A good collegiate dictionary provides the derivations of many words, and big words are usually evolved from simple words with simple meanings. It is interesting to see how words evolve. Computers have given us new meanings for words like "menu," "document," "swipe," "memory," "notepad," and countless others.
You should also think about prefixes and suffixes. Knowing just one of them can help you deduce the meanings of numerous words. For example, "icide" obviously means "killing," and we have "homicide," "patricide," "matricide," "fratricide," "suicide," "insecticide," "genocide," and others. "Ology" obviously means "the study of," and we have "psychology," "zoology," "biology," "anthropology," and numerous other "ologies." The roots of words are usually shown in collegiate dictionaries. In order to understand a word and absorb it into your working vocabulary, it is good to know not only what it means but why it means what it means, and also probably how the present-day meaning has evolved.
"Phobic" and "phobia" are other suffixes that occur to me. There are dozens of words ending in "phobia," and people keep finding or inventing new ones all the time. "Homophobia" is a popular word these days.
As far as books that will throw a lot of new words at you, I would recommend the novels of Henry James. He not only used some tough English words, but he liked to throw in an occasional French word. But the words he used were always appropriate and correct, because he was a stickler for those things. One word that was new to me and still amuses me was "endimanche." "Dimanche" is the French word for Sunday, and "endimanche" means "dressed up in one's Sunday best," but there is a subtle implication that the person described is not used to being well dressed and looks stiff, uncomfortable, and ill at ease.
The easiest Henry James novel to start with, if you have never read him, is Washington Square. His best novel by his own estimation is The Ambassadors, but it is hard reading. I think you should push yourself a little when it comes to reading. You don't have to be afraid of novels like War and Peace or James Joyce's Ulysses. And you don't have to understand every single word.