What Wilson means by a four-letter man is a college athlete who goes out for many different sports. At the upper-class Ivy League colleges like Harvard and Yale, it was common for athletes to wear varsity sweaters with symbols sewn on them to show which sports they participated in. To have four letters on a varsity sweater would be unusual and distinctive. Varsity-level college athletes were often described as “lettermen,” and varsity sweaters and jackets were often called “lettermen sweaters” and “lettermen jackets.” The sports would probably be football, tennis, and perhaps swimming, golf, track, boxing, and almost certainly rowing. The varsity sweaters and jackets would be in the school colors. There are still plenty of them sold, as can be seen on Google. Hemingway never went to college and was probably both envious and contemptuous of men who did. Since Robert Wilson is an Englishman, he must have known about these varsity sweaters and jackets because they were imitations of the same sorts of garments worn by athletes in the upper-class English schools like Oxford and Cambridge, where cricket would have been included.

Well, it is an intentionally obscure reference, but the Dictionary of Sex says that it is a discreet but insulting way to refer to a male homosexual (h-o-m-o are the four letters).

 One academic argues that it is "the c word" (female genitals--trying not to be crude here). Either would fit, given Hemingway's hypermasculinity, but I prefer the first answer.