In the first five lines of the poem, "The Road Not Taken," where does the speaker remember being?
It may be useful to answer this question from another angle. If we attempt to understand this poem symbolically, then the first five lines of the poem will reveal a lot more than is seen on the surface. "Two roads diverged in a yellow wood" could mean that the speaker is at the crossroads in his life. Two roads can symbolize two life choices which are mutually exclusive. "Yellow wood" means it is autumn, so this may be symbolic of a specific life period. Namely, the speaker may be in his autumn years (later years of one's life). Taking all of this into consideration, we can assume that the speaker is an older man who has to deal with making an important life choice. He attempts to take a good look at one of the choices ("and looked down one as far as I could"), but he cannot predict the consequences of that potential choice or any other choice. These two roads in "a yellow wood" symbolize the future which is utterly unpredictable. So, the speaker will have to rely on his intuition, which he does, as the rest of the poem reveals.
Frost is referring to a "yellow" wood. This would be the woods of the Fall season. He is literally standing in a fall wood with a diverging path in front of him. This is the simple, literal answer. It is also possible that Frost is speaking in a non-literal manner and is recalling a moment in time where he was faced with a choice. A option in which he made a decision to take a certain path over another. This would refer back to his statement of "sorry I could not travel both."