As the previous educator notes, "that mighty heart" is a metaphor for London, which normally "beats" with life and vigor. The exclamation at the end of the poem is the narrator's expression of wonder at the tranquility that settles on the city in "[t]he beauty of the morning." 

All of the places in which there is usually human activity and bustle—"ships, towers, domes, theatres, and temples"—"lie / Open to the fields, and to the sky" as though they, too, are reclining in repose. The only movement in the poem is that of the river, which "glideth at his own sweet will." The narrator, too, is still, for he has the good sense not to "pass by / A sight so touching in its majesty."

Wordsworth personifies the river ("he"), as he does the other inanimate objects in the poem: the City wears the morning "like a garment," and "the very houses seem asleep." He animates them and gives them more realness (emphasized by the adverb "very"), while taking the live, animate image of a heart, which is invisible to us, and making it still. 

The purpose of the exclamation, in my view, is to emphasize that London's beauty is apparent, even without its usual bustle. Wordsworth, as a Romantic poet, revels in the solitary pleasure of witnessing the city's natural and man-made beauty without the presence of others. The beauty of this stillness is both a revelation and an exclamation of the delight brought about by indulging in this solitary pleasure. 

Certainly the ending of the poem is remarkable for the exclamation that seems to escape the mouth of the speaker involuntarily:

Dear God! the very houses seem asleep;

And all that mighty heart is lying still!

Note how these final lines are significant because they emphasise what seems to impress the speaker most about this view of London: the city's calmness and tranquillity. In these lines, Wordsworth uses personification to compare the houses to sleeping creatures, emphasising the lack of movement and peacefulness of the view. Let us not ignore either that the poem ends on a paradox. A heart can't be both alive and still at the same time, and yet the "mighty heart" of London, that throbs with such action and movement at the best of times, from the speaker's vantage point appears to be "still," emphasising the tranquil and peaceful mood that dominates the poem.