In Act 1, Scene 1, Robert and Catherine are celebrating Catherine's birthday. It's late into the night, and Robert brings up the fact that Catherine should be celebrating with her friends. When her father asks her why she isn't out with her friends, Catherine responds, "Because in order for your friends to take you out you generally have to have friends." Not only is Catherine friendless, but she's talking to a ghost. At the end of their conversation, Robert admits, "I'm also dead." Catherine is utterly alone, her only confidant a mere figment of her imagination.
Robert's mental illness is isolating in itself, given that he cannot communicate and relate to others. Catherine tells Hal that "[I] tried to listen when he talked. Talked to people who weren't there..." The inability to relate to the real world alienates Robert and divorces him from the people around him, including his own daughter.
There is a constant question of whether Robert's old notebooks have anything useful in them. Mostly they are the writing of a madman who cannot communicate effectively and translate the ideas in his head to the outside world. Hal and Catherine often talk about whether there is anything inside the notebooks. Hal admits with disappointment in Act I, Scene III: "The point is, that book - I'm starting to think it's the only lucid one, really. And there's no math in it."
Isolation is also developed through the lack of trust among the characters. Claire doesn't believe Catherine when Catherine claims that she completed the proof in her dad's notebook. Claire says in Act II, Scene II: "Catherine, I'm sorry but I just find this very hard to believe." Hal and Claire both doubt Catherine's mathematical genius until the very end of the play, throwing her statements into question and assuming she is lying or mentally unstable.