In "A Wizard of Earthsea" in chapter three, why does the author use the unique writing device of having her characters answer in riddles?
Well, first, I have to disagree with part of your question: the writing device is not unique. For hundreds of years, riddles have been part of fantasy and fairy tale literature. This ranges from the myth of Oedipus to Tolkien's Lord of the Rings series. Now, as far as why LeGuin does it, she does it to create an air of mystery about Roke. This is for the readers—she's making it seem strange—but also for Ged. These wizards deal in esoteric knowledge. They study, and deal with, the hidden elements of life and the world. Riddles are a verbal way of showing this, and force the speaker and listener to demonstrate their mutual fitness to engage in magic.