As the previous educator notes, "that mighty heart" is a metaphor for London, which normally "beats" with life and vigor. The exclamation at the end of the poem is the narrator's expression of wonder at the tranquility that settles on the city in "[t]he beauty of the morning." 

All of the places in which there is usually human activity and bustle—"ships, towers, domes, theatres, and temples"—"lie / Open to the fields, and to the sky" as though they, too, are reclining in repose. The only movement in the poem is that of the river, which "glideth at his own sweet will." The narrator, too, is still, for he has the good sense not to "pass by / A sight so touching in its majesty."

Wordsworth personifies the river ("he"), as he does the other inanimate objects in the poem: the City wears the morning "like a garment," and "the very houses seem asleep." He animates them and gives them more realness (emphasized by the adverb "very"), while taking the live, animate image of a heart, which is invisible to us, and making it still. 

The purpose of the exclamation, in my view, is to emphasize that London's beauty is apparent, even without its usual bustle. Wordsworth, as a Romantic poet, revels in the solitary pleasure of witnessing the city's natural and man-made beauty without the presence of others. The beauty of this stillness is both a revelation and an exclamation of the delight brought about by indulging in this solitary pleasure.