The quote you mention actually reads "where ignorance is bliss, 'tis folly to be wise," and it is perhaps one of the most misunderstood quotes in literature.
The quote is the last two lines of a poem written by Thomas Grey called "Ode on a Distant Prospect of Eton College." This is a nostalgic poem in which Grey longs for the days when he still lived in "ignorant bliss," not knowing about the hardships and suffering which would inevitably come.
The primary theme of the work is that suffering, unhappiness, misery, and death are inevitable. He is looking down at the boys at Eton College and wishing he could somehow spare them the knowledge of the pain which is inevitable in their adult lives. The last stanza reads this way:
To each his sufferings: all are men,
Condemn'd alike to groan—
The tender for another's pain,
Th' unfeeling for his own.
Yet, ah! why should they know their fate,
Since sorrow never comes too late,
And happiness too swiftly flies?
Thought would destroy their Paradise.
No more;—where ignorance is bliss,
'Tis folly to be wise.
Grey is not, as is commonly believed, supporting the idea that ignorance leads to happiness. Instead he wants young people to maintain their blissful innocence (ignorance) as long as possible before the difficulties of life consume them.