How is Ophelia's madness in Hamlet metatheatrical?
The answer to this question will depend a little bit upon how you define meta-theatrical. Usually meta-theatricality is when the drama comments on itself or on drama or acting in general. Hamlet is a great play in which to explore this concept because there are several places to evaluate what Shakespeare is doing in regards to being meta-theatrical. You are asking about Ophelia's madness, so consider it from this perspective. Is is "acting" mad or is she really mad? How is her behavior like or different from Hamlet whom we KNOW is only acting mad. Shakespeare intentionally gives us this juxtaposition in the play of real and acted behavior so that we can judge for ourselves what is going on with these characters. Another aspect of the meta-critical is when the audience questions how they are to respond to a character. Ophelia is crazy and that is a sad sight, but her suggestive songs are kind of funny and the audience can't help but laugh at her craziness. Is that what Shakespeare intended? For what purpose? When you think about those questions, you are thinking about the play from a meta-theatrical point of view.
Here are a couple of other things to look at in this play:
1. Hamlet's first conversation with the players about the group and why they are here at Elsinore. Especially look at Hamlet's speech as he launches into the speech of the play about Priam. Hamlet is talking about the dramatic tastes of audiences. Is that Shakespeare making a commenting about the audiences of his day?
2. Hamlet's advice about good acting that he gives the players before the play-within-a play.
3. The whole actual play-within-a-play.
4. Hamlet's scene of pretend madness at the court at the start of Act 2 and the start of Act 4.
Each of these places gives you a chance to glimpse into what Shakespeare might have been thinking about acting and story-telling. That intense look at the craft from within the craft is very interesting.