How did the dog, Montmorency, make a fool of himself?
I believe you are referring to the dog Montmorency in Three Men in a Boat (To Say Nothing of the Dog).
In the story, the author describes Montmorency as a cat-hating dog. Furthermore, he explains that it is in the nature of fox terriers such as Montmorency to provoke fights and to rejoice in the execution of relative pandemonium whenever the opportunity presents itself. The author emphasizes the supposedly blood-thirsty nature of fox terriers by recalling a story about how a lone fox-terrier starts a fight in the lobby of the Haymarket stores one day.
Presumably, a lovely young lady chains her fox-terrier in the lobby, among all the other dogs who are waiting for their owners. It isn't long before the little dog bites and attacks the other bigger dogs. Before long, a vicious fight commences among the dogs, with the bigger dogs fighting among themselves and the little dogs engaging in their own bruising operation.
Thus, the author uses this little story to highlight the unpredictable and combative nature of fox-terriers. He then relates how Montmorency apparently makes a fool of himself one morning when he spies a great, big Tom cat crossing the street. On immediate recognition of a cat nemesis, Montmorency lets out the 'cry of a stern warrior' and immediately charges at the cat 'at the rate of twenty miles an hour.'
However, the cat reacts unpredictably himself: he displays no fear at what he sees and exhibits nonchalance at Montmorency's fierce charge. This stance renders Montmorency so discomposed that he finds himself at a loss; to him, this is entirely a strange state of affairs when an enemy refuses to engage in a battle for supremacy. In the end, Montmorency slinks back home, ashamed of his own foolish behavior. The author states that, from then on, Montmorency always exhibits a piteous look whenever the subject of cats is brought up.