In II.i, when Portia and Morocco first meet on stage, Portia says to him
You must take your chance,
And either not attempt to choose at all
Or swear before you choose, if you choose wrong
Never to speak to lady afterward
In way of marriage: therefore be advised.
In other words, "either leave without trying to marry me, or gamble. If you win, you win me. If you lose, you can never marry anyone."
Right after that, Portia tells Morocco that he should go pray at the temple before dinner, and then after dinner, he can try to guess the casket and win her hand (II.i.43-4). From that, we learn that there is a kind of structure to this game and to hosting these suitors.
Later, in the scene where he gambles to try to win Portia (II.vii), we learn more about the game: there are three caskets. Each suitor, starting with Morocco, has to guess which casket is the winner. When he opens the casket, he will know if he won because the correct casket will have a portrait of Portia in it (II.vii.11-12 and II.vii.61-2).
So in summary, before he is allowed to guess, Morocco has to swear he will a) guess which casket has Portia's portrait and b) leave Belmont and never propose marriage to another woman if he loses.