Check out the link below to the Enotes material on Fifth Business, the summary alone should provide you with a pretty clear idea of where to look for your references.
Dunstan's guilt is intricately tied with his identity as a character and the plot of the story. We might simplify things to say that his guilt stems from, and is defined by, the stone-throwing incident in his youth. Avoiding the stone indirectly caused a chain of subsequent events, culminating in Staunton's suicide, but also leaving a trail of damages in its wake. Dunstan, as implied by the title of the book, is not really a participant or instigator of these events (such as Mrs. Dempster having adulterous sex) but he can't really get away from the sense that these things were still caused by his actions. This leads to his caretaking for Mrs. Dempster, when by all accounts it isn't really his responsibility to do so.
Dunstan was also taught to be suspicious of anything that seemed tempting or pleasurable; thus he was predisposed to view life from the perspective that guilt and suffering were his lot, and that avoiding or justifying himself away from these things was in fact a form of sin. I would suggest that you define the stone-throwing as an event which exposes a hole in the rationale by which Dunstan was raised; it puts him in a Hamlet-style position of "damned if you do, damned if you don't", leading to his guilt and obsessions.