We are not given a specific year Invisible Man is set in, and the story actually spans many years, from the narrator's high school graduation through college and into his adulthood in Harlem. However, the story is written retrospectively, with the narrator looking back over the course of his life.
We can use context clues to locate a probable date span for the novel. Ellison began to write it in 1945, and published the chapter "Battle Royal" in 1947, though the novel as a whole did not appear until 1952. As Ellison moved to Harlem himself in 1936 and the novel is loosely based on his own life, we can set the time of his writing in the late 1930s, with the narrator assessing his experiences sometime before the start of World War II.
The dating is significant because the narrator comes of age in a pre–Civil Rights era of virulent racism. Much of the racism that is more covert today was openly expressed in the 1920s and 1930s. Without many role models of positive Black male identity in his period, it was easy for the narrator to feel invisible or despised, never valued for who he was as an individual by the wider white world.