What is the correct literary term for a book that begins at the end?
There are many novelists which apply the technique of beginning the text at the end of the story. Mary Shelley uses it in Frankenstein; Daphne du Maurier uses it in Rebecca; Edith Wharton uses it in Ethan Frome. This technique is referred to in a couple different ways.
First, some refer to this technique as a frame story. A frame story begins at the end, moves into a flashback to tell how the characters came to be where they are, and (eventually) joins back up where the story moved into the past. This technique is seen in Shelley's Frankenstein and Wharton's Ethan Frome. This has also been referred to as a story within a story.
Another name for this technique is a prelude. This serves as an introduction to a work. This technique is used in du Maurier's Rebecca. The use of this technique allows the author to foreshadow events that the readers will come in contact with during later reading.
It sounds like what you may be referring to is, as the other instructor has pointed out, a frame story. Such stories start at the ending and then flash back in time to demonstrate how this pivotal moment in the story came to be. This means that the story is beginning with a prolepsis, or a "flash-forward" in time. This kind of style operates as a postmodern narrative device and has come to be fairly popular in contemporary literature. One current novel that uses this technique would be Fight Club, which starts at the end with Tyler's final showdown with the unnamed narrator. You may also be thinking of a story that begins "in media res," or "in the middle of things." Although this is the literal Latin translation of the term, such stories can also begin at the end; this term is simply referencing the non-chronological nature of the storytelling.