In this sentence, "That" is a demonstrative pronoun in the Subject slot. The apostrophe s is a contracted form of "that is" in which "is" is the be verb filling the Verb slot in the matrix sentence. The adverb "exactly" is the adverbial complement to the be verb (is).
The matrix Object slot is filled by the wh-clause "what he said...." In the wh-clause, "he" is the Subject of the subordinate clause embedded in the matrix clause and "said" is the Verb. The prepositional phrase "about you/me/yours" fills the wh-clause Object slot as an Indirect Object.
Starting with "yours": "Yours" is the possessive form of second person pronoun "you" and as such can be used in a Subject or Object slot. So if you wished, you could say "...he said about yours," with "about" being the preposition in a Prepositional Phrase filling the Object slot as an Indirect Object.
Contrast this to first person "I" and "me" in which "I" is the Subject form and must be used in the Subject slot, while "me" is the Object form and must be used in the Object slot. Since "me" is the first person Object form, it can certainly be used in an Object slot in an Indirect Object: "...he said about me." Finally, "you" is the same in Subject and Object use, so saying "...he said about you," with "you" as part of the Indirect Object in the Object slot is also correct.
The only reason one might prefer "you" over "me" or "yours" is that in previous usage, which is no longer popular, it was thought impolite to talk about yourself, hence the origin of excusing phrases like "if I do say so myself...." Similarly, it was once considered impolite to talk about someone else's possession without identifying it as in answer to "my what...?" Otherwise, aside from these old-fashioned polite rules of usage, as you have seen, there is no reason to consider "...he said of me" and "...he said of yours" as grammatically incorrect.