Olivia admits how much she loves Cesario in "Twelfth Night," and Viola/Cesario says "I pity you." Why does this encourage Olivia?
Viola pities Olivia because she does not realise that Viola is not really a man, but a woman (compare Viola's soliloquy 'I left no ring with her', where she calls Olivia a 'poor lady', and suggests that she were 'better love a dream').
The simple answer to your question can be found in the lines below: 'I pity you', Viola says, and Olivia responds
That's a degree to love.
No, not a grize; for 'tis a vulgar proof
That very oft we pity enemies.
Olivia seems encouraged by Viola's pity because she believes that pitying her is, at least, one step towards loving her (a 'degree' to loving). Viola replies, rather persuasively (although Olivia doesn't seem to hear!), that pity isn't anything to do with love - because we pity enemies and we don't love them!
To feel pity, is to feel emotion; therefore, Olivia is hopeful Viola's/Cesario's feelings will grow. Feeling pity for someone is to feel compassion or sympathy. Typically, "pity" has a negative connotation; however, Olivia sees it as something positive: a glimpse of deeper affection.
*I did refer to the Shakespeare Lexicon and Quotation Dictionary by Alexander Schmidt.