You are correct, the relatively lower diversity of alleles on the X chromosome is indeed a result of natural selection. The X chromosome and its analog, the Y chromosome, determine sex in humans and other mammals. If you have two copies of the X chromosome (one from each parent), you are female. If you have one X (from the mother) and one Y (from the father), you are male.
Because the Y chromosome is truncated, it is smaller than the X chromosome and does not have as many genes. (See this link for a photo of the two chromosomes side by side.) This means that a male has only one copy of a number of genes, specifically the ones that are located on the lower region of the X chromosome. In females two copies of each gene are present, but one of a female's X chromosomes is inactivated during fetal development, presumably so both genders will have equal gene activity.
Since only one set of genes is active in this particular chromosome pair, natural selection has eliminated bad mutations on this chromosome more efficiently. In the case of any other chromosome, a mutation might persist because the other chromosome of the pair serves as a "backup", but that is not true in the case of X and Y. As a result, X chromosomes change more slowly. This has been a useful tool for genetics researchers, as they can compare changes in the X chromosome of a population to changes in the other chromosomes and calculate how long the population in question has been genetically isolated from other groups.