This is a question to which the answer is illusive. An analysis of grammatical construction and for contextual logic may reveal an answer. Grammatically, Miles is addressing the governess who had sprung upon him to shield his sight from the apparition at the window. He doesn't know, when he utters the words "Peter Flint -- you devil!" that the governess sees Flint at that very moment at the window; as far as Miles knows, it is he and the governess alone together. Despite James' propensity for clouding a sentence with interjected prepositional and other phrases ("was all for his judge, his executioner; yet it made him avert himself again, and that movement made me, with a single bound ...), there is no way to understand the grammatical construction comprising a noun followed by an em dash followed by an address of non-endearment as anything other than an address to the individual with whom the speaker is conversing, in this case, the governess. The em dash has a number of uses, one of which is to interrupt the sentence for the interjection of closely related but explanatory or emphatic information, such as "-- you devil!"
An examination of the logic of the story produces few clues to back up the grammar analysis though, which is why Miles' outburst confuses readers. Yet there are two clues to go by that do confirm the above analysis. First, though, it's helpful to examine why the logic of the story is misleading. The main reason is that Miles isn't one to get flustered, let alone utter demonstrative exclamations. His conversations with the governess are always conducted in a cool, little gentleman-like manner. Therefore his abrupt change of tone and address in shouting "-- you devil!" at the governess seems contradictory and inexplicable, especially when Flint is witnessed (by the governess) right at the window.
Now, back to the clues that lead away from the contradiction. Bear in mind that the governess has just lunged at Miles and pressed his face to her body to shield his eyes from the horrible specter now visible at the window. This is alarming for Miles, to say the least. The first clue follows after this and occurs when Miles deduces what is happening and asks if it is "she?" Miles' composure is broken through and he emphatically screams, "Miss Jessel, Miss Jessel!" teaching us the depth of his emotion and the heightened state of his terror. This clue leads to the surprise ending, which depends on intensely heightened emotion.
The second clue, ironically, is that after exclaiming "-- you devil!" to the governess, Miles falls in her arms and dies. This is delayed confirmation of the fact that he was indeed in such a state of horror and fear as to incongruously yell "-- you devil!" at the governess. James shows after-the-fact that either this was Miles pent up feeling for the governess all along or that in his terror he spoke wrongly from a heightened sense of fear. This is also James' way of demonstrating that the reader, though dubious, should take his story as truth for the characters in the story, not as their delusion, and that under the composed exterior of good breeding there may beat the heart of terror and intense passion.