The nightmare refers to the racist element in Maycomb. Atticus agreed to take Tom's case and this is in a town where racial tension and segregation still exist. The nightmare is that, since Atticus has defied the social code of Maycomb by defending an African-American, he (and his family) will have to deal with the backlash from that racist element.
While Tom is in Maycomb jail, prior to his trial, Mr. Tate and others come to Atticus' front lawn because they're concerned a mob might go after Tom. There is also concern, particularly from Jem (in Chapter 15) that Atticus himself could be in danger. These concerns are confirmed when a mob does indeed show up at the jail while Atticus is standing guard. Scout and Jem interrupt the mob. Jem is defiant and will not leave, despite Atticus' demands to do so. Scout directly addresses Walter Cunningham, Sr., and through her innocence and kind words about Walter, Jr., Walter Sr. convinces the mob to disperse. Ironically, Scout had thought that this was the same group who voiced concern on Atticus' front lawn. "There was a smell of stale whiskey and pigpen about, and when I glanced around I discovered that these men were strangers." (Chapter 15)
The nightmare temporarily goes away but returns when Tom is wrongfully convicted and when Bob Ewell continues to harass Atticus, Helen, and eventually the children. In general, the nightmare is this racist element and it also becomes Bob Ewell's personal vendetta against Atticus and what he represents.