When Yeats chooses to use “in the deep heart’s core” instead of “in the core of my heart,” the speaker is distancing the “peace” he feels from himself. This reinforces the idea that the peace found in the simple life of nature is universal.
And I shall have some peace there, for peace
comes dropping slow,
By referring to the heart with “the” instead of “my” he also produces a double metaphor for the heart of the island and the speaker’s heart. In other words, by retreating to the peace of the island he is entering the “heart” of nature and his own heart.
Of course, “core” does rhyme with “shore” in the third line of the last stanza, so arranging the words this way does maintain the abab rhyme scheme and thus both the structure and internal logic of the poem.
It's called a transferred epithet, but why does he use it? Why doesn't he say "my heart's deep core"? It fits the rhythm -- maybe a little more cumbersome, but not much.