At fertilization, an egg which has an X chromosome is fertilized by a sperm which can either have an X chromosome(which will result in a female child) or a Y chromosome(which will result in a male child). Therefore, chromosomal sex is established at fertilization. However, gonadal sex occurs later and determines whether testes or ovaries will form. A gene called SRY will be expressed if a Y chromosome is present and this causes the gonads to become testes. These cells will produce testosterone which is a male androgen and the internal and external development of male sex organs will occur. However, if no Y chromosome is present, the default pathway of development is female. The gonads become ovaries, the duct system becomes fallopian tubes and the uterus and vagina develop. The male duct system degenerates. One's phenotypic sex is seen by the sex organs that develop externally which resemble either male or female reproductive organs.
The basic answer is that chromosomes determine and govern the sex traits and the growth and development of individuals, with every individual, in normal circumstances, having a pair of X Y chromosomes even though the female contributes two X and the male contributes an X and a Y. Early in embyronic development, the double X chromosomes dominate in female development while the extra X chromosome is deactivated in X-inactivation leaving the final pair of XY chromosomes for a male. However specialized X chromosomes in the male, in specialized locations, do not deactivate because they play a role in normal growth and development (Genetics Home Reference, National Institute of Healt, NIH.gov).