Since you understand what "fifth business" is, think of how it applies to Dunstan. In what ways was he "someone who is neither a hero nor a villain", and how is he on the "sidelines" of life?
There is more than one way to think about Dunstan as "fifth business", I believe, but let's try this one. If the other "four" actors in the plot are Boy Staunton, Leola Cruikshank, Leisl, and Paul Dempster (Eisengrim), how is he neither a hero nor villain to them? How is he on the sidelines, neither an actor as the sweetheart, hero, villain, or confidante character (as those roles are filled by the other four)? I don't believe that these four characters fit pat into these roles, but it could be argued that Boy is the hero (although in the drama of Paul's mother, Boy is the villain), Leola is the sweetheart, Paul Dempster (as Eisengrim, much later) is the villain, and Leisl is the confidante (of the villain this time, not the sweetheart). To this quad of characters, it can be argued that Dunstan is very much the "fifth", in that he is present and necessary for much of the story of these actors, but he is not really an actor for himself in any of their stories.
As far as the development of Dunstan, I think that it could be argued that he develops after his encounter with Leisl in Mexico, when she tells him to stop being the "fifth business" character in the over-the-top opera plots of other people's lives. This is while Dunstan is in his fifties, living his strange celibate scholar/teacher lifestyle. After this conversation, which happens after his first sexual encounter with Leisl, Dunstan seems to live his life somewhat differently. Especially, he seems to allow himself more latitude to do what he pleases, especially when he retires and goes and lives in Switzerland with Leisl and Eisengrim. He seems to have fewer worries about "paying" for his saved life by self-denial and writing hagiography.
I also think that it could be argued that this identification with "fifth" business has a direct bearing on what Dunstan does in later books. It's hard to show the full arc of development without including the other two books of the trilogy, The Manticore and World of Wonders, but you could show that Dunstan becomes more reconciled with life after he finally has a name for what he has lived most of his life as.
There are other interpretations of the "fifth business" arrangement -- the ones including himself as the hero, Boy as the villain, Mary Dempster as the sweetheart, and the young Paul Dempster as the confidante (when Dunstan takes pity on the boy and teaches him conjuring tricks.) This can be flipped, and Dunstan can even be cast as the villain, for it is he who is the target for the snowball with the rock in it thrown by Boy, which hits Mary Dempster and causes Paul's premature birth and his mother's madness. Also, Dunstan teaches Paul the tricks which get him into the World of Wonders, and leads him into a life of virtual slavery and sexual exploitation, and separates mother from son forever. The operatic roles only fit under certain circumstances, and from certain points of view. Dunstan can be seen as both the "fifth" and, at various times, one of the main actors in his own story, and in others'. That is part of the point of the book -- that at any one time we can be any one of the four roles or, as is most often, the "inevitable fifth".