Primogeniture is characteristic of the English throne. The Celts in Britain, though followed the practice of tanistry well into the 15th century; kings in Scotland, Wales, and Ireland were elected. A similar practice existed in Scandinavia, and the Danes, in fact, elected their king.
Hamlet would certainly lose to Claudius in an election. Brothers often succeeded brothers, so Claudius would be an anticipated choice. Also Hamlet is too young, and he's away at college. It would seem a mark of real carelessness for a country under siege to hand the throne to a young, untried, and absent monarch.
Claudius has popular support and the approval of Gertrude, the supposed devoted wife of the late king and the "imperial jointress of this war-like state." The play argues that the choice of Claudius shows some wisdom. In his first speech he boldly attacks head-on the question of his brother's death and his own marriage to old Hamlet's widow. He sidesteps nothing but goes straight for what is probably the subject of a good deal of whispering in and out of the court. He then moves with consummate skill through the international problem, speeding Cornelius and Voltimand off to Norway --another direct attack. He continues, skillfully focusing slowly on the nation, then the members of the court, and finally on his own family -- and Hamlet. He's brave, bold, resolute, and eloquent. Hamlet is withdrawn, subdued, hesitant, and snide.
In a system where the throne is passed on through primogeniture, Hamlet would have become king automatically, due to primogeniture. However, that's not the only system of inheritance, and there are other systems, like this one, in which the brother inherits the throne. In agnatic seniority, the eldest male in the lineage inherits, which in this case would be the king's brother. Besides…it makes a much better play for Hamlet to still be prince than for him to be king, no? One would assume that Shakespeare blended history with dramatic potential to make this choice.