It appears from the text itself that there is nothing to suggest that the ghosts of Miss Jessel and of Peter Quint are linked to any specific location. Rather, what draws them to where they haunt is the presence of Miles for Peter Quint and Flora for Miss Jessel. In the imaginings of the governess who tells us this tale, for example, Miss Jessel, on her first appearance in Chapter Six by the lake, "fixes" only Flora with "such awful eyes." Likewise, Peter Quint, on his first appearance, is seen peering in through the window, staring only at Miles and not paying attention to the governess. This of course gives the governess the idea that somehow these ghosts want to possess the souls of Miles and Flora. Note what she concludes at the end of Chapter Seven as she despairs at the situation she faces:
I don't save or shield them! It's far worse than I dreamed--they're lost!
Thus we can argue that the governess feels that the children are haunted or possessed by these spirits. It is the children that draws them to haunt Bly, not specific locations.