Formalist criticism is one way that a reader can approach his understanding of a text. When a reader looks at a poem, play, story or novel from a formalist perspective, he is looking solely at the work as something to be dissected, so he looks for all of the literary techniques and devices that an author uses to create the text and its meaning. He does NOT look at the author's life, he does NOT consider the text from a historical or psychological perspective; he does NOT consider how this text is like other texts -- those are all other modes of literary criticism.
Think of "Twinkle Twinkle" as an example. With formalist criticism the reader would notice the repetition of the word twinkle and consider connotation and denotation of the word. It would notice the first person speaker of the poem. He would note the use of simile in the 4th line (like a diamond in the sky). He would note the refrain of the first two lines in lines 5 and 6, and he would mark the meter and the ryhme scheme. Once the poem was literarily dissected, then the reader can consider how those elements work together to create the meaning of the poem as a whole.
You can read more about this mode of criticism at the site listed below.