You are correct in that the plot of this story does move in a fairly linear fashion in terms of its chronology. Goodman Brown leaves home, goes into the woods, meets with the Devil, attends a witches' sabbath, implores his wife Faith to resist, returns home the next morning a very changed man, and dies in gloom.

While he is in the forest with the Devil, however, he spots Goody Cloyse, an older woman who was responsible for his religious education when he was a boy. He also speaks briefly with the Devil about the lives and natures of his father and grandfather, men the Devil claims to have had some relationship with. However, these remembrances are all fairly brief and come up in conversation between Goodman Brown and the Devil. A proper flashback is something that takes somewhat longer and feels like an interruption in the chronology of the story's plot. It would likely also take place within the character's own head or be narrated by the speaker in a sort of interlude, but the references to Goodman Brown's memories here really occur as a result of the conversation and do not flash back, so to speak, to specific memories or sequences of moments. Because his memories do not interrupt the story's chronology, they cannot be counted as flashbacks.