Louis MacNeice's "Conversation" refers to that part of us which does not fit within the parameters defined by society as our "vagrancy." This vagrancy, a tendency to wander within our own minds and depart from what is expected of us, is present in everyone, rendering "ordinary" people "peculiar"—there is no such thing, MacNeice says in the first stanza, as a truly ordinary person, because we all have this tendency. While speaking to someone, we might see the vagrancy become apparent in the shift of their eyes, making it evident that they are thinking about something else, in search of "un-, or other, realities."
Sometimes, however, this wayward vagrancy can manifest in a different way. Rather than attempting to conceal itself from the conversationalist, the vagrancy might instead assault them: as you are "mistaken" for another time, or another place, the strange or inappropriate thought leaps from the other person to you, making itself known. This is something that is not supposed to happen, a fault in the fabric of social norms—a "dropped stitch."
Most people will recognize when they have made such an error and will apologize and tell themselves it will not happen again, but sometimes the attempt to conceal the error continues to make the error obvious. The nervous conversationalist might drop "swear-words like roses in their talk" in an attempt to divert attention away from this part of themselves we all pretend does not exist.