In "The Raven," what does the speaker mean by "Take thy beak from out of my heart"?
Since we know that the raven did not actually put its beak in or on the heart of the main character, it is clear that the expression was figurative, a metaphor used to let the audience understand the author's state of mind using something comparative, but without actually stating that he is comparing. Instead, he asserts that this is what the raven is doing.
This being said, the narrator at this point in the poem is asking the raven to go away. The presence of the raven, to the main character, alludes the memory of his late, beloved Lenore. Therefore, the pain of remembering her is proportional to the bird physically attacking the heart of the suffering man.
He asks that the raven leaves his abode and to never come back.
This shows that the main character is truly invested in his interaction with the raven and, as he imagines that the raven is there for Lenore, he also shows that it is hard for him to let go of the memory. So, he transfers the blame to the raven, whom he concludes is answering him back saying "Nevermore", which means that he will never leave.
Remember that at the end of the poem, the bird "still is sitting on the placid bust of Pallas". From what can be inferred, the bird (his pain) will never cease.
To understand such quotes it is important to examine them in context of the entire poem and not just look at them in isolation. It is clear that at this stage of the poem the speaker is becoming wilder and more demented, sure that the raven is some kind of messenger from hell sent to torment him about the death of his beloved, Lenore, and the impossibility of any hope of their reunion. Each time, the raven responds with "Nevermore" and the speaker becomes more and more frenzied in his reactions. In the section you have identified he bids the bird leave him:
"Leave my loneliness unbroken! - quit the bust above my door!
Take thy beak from out my heart, and take thy form from off my door!"
The speaker is therefore speaking metaphorically - "take thy beak from out my heart" is an indication of how the raven's presence and words are wounding him physically and psychologically, attacking his very core and centre as an individual.