As you know, one definition of irony is the use words to truly mean their exact opposite. That's what's going on here in Swift's use of the word modest. Although he says that his proposal is modest (meaning not anything terribly special or different) it really is very outrageous.
After all, what Swift is saying is that Irish people should start selling their children to be used as food. That is about as far from modest as you can get. So, by calling it a modest proposal, he is being ironic -- trying to act like he thinks he's being reasonable when he knows he isn't.
The word "modest" is ironic because the narrator's proposal in this essay is anything but modest. Most would deem it outrageous: the narrator proposes that poor women should raise their babies like livestock, fattening them up so they can sell them when they are a year old to rich people as a delicacy for the dining table. Swift expects the reader to be shocked at this "modest" proposal.
But there is another layer of irony at play. Swift and the reader might be shocked at this proposal, but the narrator evidently does think his proposal is "modest," a simple, rational, common-sense plan to help end hunger and poverty in Ireland. Not only is the narrator's proposal outrageous, so is his callous but clueless disregard for Ireland's poor as anything other than economic widgets. Swift is critiquing not just the proposal but a rationalism divorced from humanity that could devise such a proposal and think it modest.