Once Doodle has begun to walk, the narrator/brother tells the reader, 

I began to believe in my own infallibility, and I prepared a terrific development program for him.

Clearly, the brother intends to make Doodle "normal" as he wishes to teach him to run, fight, swim, and to climb trees. So, once the summer vacation arrives, the brother takes Doodle to Horsehead Landing, where he puts his brother into the water and gives him swimming lessons or has him learn to row a boat. However, as the opening of school approaches, Doodle lags behind in his accomplishments. For, his swimming ability is poor, he has not the strength to climb ropes. Selfishly, the brother puts more pressure upon Doodle, but the little boy cannot meet the challenges; instead, he sleeps poorly and appears feverish. But, the brother feels that

It was too late to turn back, for we had both wandered too far into a net of expectations and left no crumbs behind. 

This last figure of speech alludes to the fairy tale of Hansel and Gretel, the children who find their way home from the forest into which the witch has taken them because Gretel has left crumbs of her bread, and they escape the evil intentions of their captor. 

So, despite the iron will of the brother, poor Doodle and the brother's plans come to nought. Worse than this, the brother is the unwitting agent of the demise of his weaker brother as he forces Doodle out into the boat and they later must fight against a storm, a storm that claims "the scarlet ibis" of a brother.