First of all, please remember that this is not the original title of the book. The original title was (of course) in Japanese and contained an important pun that could not be translated. Therefore, Mishima was given a list of possible titles in English and this is the one that he chose.
The title refers to what happens to Ryuji over the course of the story. Ryuji had all these dreams of glory (glory was part of the punning Japanese title -- it could be read as "Glory in the Afternoon") but he abandoned them. He gave in to the mundane world and married Fusako instead of trying to make his life into something glorious. He falls from grace because he abandons his dreams and settles for a banal existence.