With regard to the setting of Shakespeare's Hamlet, I am not sure if there is a way to be completely sure of Shakespeare's choice of Denmark.
As with many of his other plays, Hamlet did not come up with the idea for Hamlet originally. There are several earlier versions. Just to mention two of many possibilities...
Saxo Grammaticus told a similar tale in his Historia Danica(c. 1200). About 15 years before Shakespeare’s version, François de Belleforest adopted the essential story in his Histoires Tragiques (1576), a popular collection of tales in French.
It is believed, because of specific details included in Hamlet that were not in earlier forms of the story, that the version to most influence Shakespeare's play, was something called Ur-Hamlet, probably written by Thomas Kyd, or possibly by Shakespeare himself.
Shakespeare's choice Denmark may be based on simple structural elements in developing the plot. For example, Denmark vs. Norway provides...
the subplot of impending war taking place whenthe Ghost first appears (as the guards are dressed for battle). This sense of a threat from another country heightens the tension present in the first few scenes of the play, which sets the mood.
Fortinbras is a foil for Hamlet, and though he did not actually exist, the relationship between the countries of Sweden, Norway and Denmark might have been of interest to Shakespeare, as they were all governed by the same ruler in Shakespeare's time. This would refer to the Kalmar Union. And while ruled by the same monarch, there were:
diverging interests (especially the Swedish nobility's dissatisfaction over the dominant role played by Denmark...)
This element existing among the three countries may have inspired Shakespeare to use Denmark because of that nation's strife with Norway. This would have allowed him to provide the potential-of-war subplot, as mentioned, and a neighboring heir to his throne, with a great deal in common with Hamlet, ostensibly heir to Denmark's throne, also a secondary element that supported Shakespeare's primary plot.
The fact that Fortinbras enters in the final scene to take the throne of Denmark—as Hamlet and his family are all dead at the end—might allude to the unity of Denmark, Norway and Sweden, allowing "Hamlet" to leave Denmark in more capable hands, while returning this once autonomous nation to the collective Kalmar Union.*
As I mentioned before, Shakespeare's choice of Denmark as a setting might simply have provided essential supporting details; these details may not be driving forces in the play, but elements that fluidly aided the movement of the main plot—with regard to what was happening with Claudius/Denmark's nobility, as well as Hamlet's psychological distress.
Denmark, and all that came with this setting, cleverly allowed the core of Hamlet's story to dominate the overall direction of the play, while providing supporting details that "fleshed out" the underlying conflicts of the play's dominant themes.