The reference is taken from Act 2, scene 2 (lines 381-392) in which Hamlet alludes to the biblical Jephthah.
O Jephthah, judge of Israel, what a treasure hadst thou!
What a treasure had he, my lord?
'One fair daughter and no more,
The which he loved passing well.'
[Aside] Still on my daughter.
Am I not i' the right, old Jephthah?
If you call me Jephthah, my lord, I have a daughter
that I love passing well.
Nay, that follows not.
What follows, then, my lord?
'As by lot, God wot,'
and then, you know,
'It came to pass, as most like it was,'--
the first row of the pious chanson will show you
more; for look, where my abridgement comes.
The allusion is important on different levels. Firstly, Polonius has already reported to Claudius and Lady Gertrude that he believes that Hamlet's supposedly irrational behaviour stems from a lovesick obsession with his daughter, Ophelia. To him, Hamlet's repeated references to her confirm this suspicion.
Secondly, the biblical Jephthah had asked God to bestow upon him victory over the Ammonites and that he would either sacrifice or give to Him the first person who came to greet him after his victory. When he victoriously returned from said battle, his daughter was the first to meet him and he was true to his promise. Some scholars (even before Shakespeare's time) are of the opinion that Jephthah did not sacrifice her but that she, in respect of her father's sacred vow, offered to live a life of piety and virtue in God's service. She, therefore, spent her life as a virgin, thus leaving him no heirs.
Although Polonius does not at first grasp Hamlet's reference, he quickly understands that the prince is speaking about Ophelia. She had also, out of duty to her father, obeyed his instructions to not encourage Hamlet and thus remain chaste, as it were. Also important is the fact that Hamlet later also asks Ophelia to become a nun (Act 3, scene 1) and, similarly, undertake a vow of chastity. He had also alluded to this in his earlier conversation with Polonius when he said:
Let her not walk i' the sun: conception is a
blessing: but not as your daughter may conceive.
Friend, look to 't.
Thirdly, Hamlet's equivocal statements here are a continuation of his verbal duel with Polonius. He believes that Polonius is an old fool and plays him as such. It further supports his attempt to appear mad, an act which quite convinces Polonius that he has, indeed, lost his sanity.
Furthermore, the allusion, which involves sacrifice, also foreshadows Ophelia's apparent suicide later. She is so overwhelmingly distraught by her father's death at Hamlet's hand, and his confusing attitude toward her, that she loses her mind and starts behaving in an abstract and irrational manner. She later accidentally falls into a river and makes no attempt to save herself and drowns. She dies with her chastity intact.
In the end, though, Hamlet's trickery does not entirely work and results in not only his unfortunate demise but also that of other innocent victims who, unfortunately, were drawn into not only his plot, but also that of the malevolent Claudius.
The reference to Jephthah in "Hamlet" is by Hamlet to Polonius. In this scene hamlet is outwitting Polonius using many literary illusions, this one being biblical.
In the bible Jephthah prays for the help of god to win in battle, and in return, he offers the first person to walk through his door as a sacrifice. He believed that most likely it would be a servant... However, it turned out to be his virginal daughter. Jephthah's daughter sacrifices herself, telling her father that she will honour his promise to god, and dedicates her life to work in the temple.
Jephthah did not want to sacrifice something of actual value, like his daughter. Polonius is self seeking at best, wanting his daughter for what he can use her for. He was happy to have her engaged to Prince hamlet, but happier still to have her used to test whether or not he was mad.
Ophelia ends up making the sacrifice of her own love by testing Hamlet, in the "get thee to a nunnery" scene, causing her to go mad, and ultimately kill herself.
To call Polonius Jephthah is to insult his integrity, as well as foreshadowing his own lack of heirs, as Ophelia dies childless, and Laertes also in the final scene.
The reference or literary allusion to Jephthah is a Biblical one. Hamlet is comparing Polonius to Jephthah.
Jephthah (the story is from Judges 11 in the Bible) promises God a burnt sacrifice of the first person who comes to his door if he can claim victory over the Ammonites. The Lord grants his wish, but the first person at Jephthah's door is his one and only virgin daughter who he must now offer as a sacrifice to God.
Hamlet is scolding Polonius for sacrificing his own virgin daughter barring her marriage, children, and ultimately her life. He is also insulting Polonius, while at the same time kind of insulting himself when he mentions that Jephthah was "the son of a harlot" (referring to his own mother, while also slinging an insult at Polonius).