Richard's worst crime in one sense was being Richard. Think about it: the famous murder of the Princes in the Tower was never proved, and a powerful but upstart royal house - the Tudors - had dispossessed the Plantagenet kings once and for all by killing Richard on the battlefield of Bosworth. The Tudors had a vested interest in portraying Richard as a vicious, ambitious, murdering villain, not fit to be king, and moreover a vile-looking cripple - one whose evil was 'stamped on his twisted body', to quote Stephen Greenblatt. He refers to the 'Tudor Myth' - the necessary smear-campaign of their immediate predecessor, in order to bolster the notion that the divinely appointed Tudors brought peace, prosperity and goodness to England out of the darkness of Plantaganet rule.
Richard III was an evil legend before he became the hero/villain of Shakespeare's play, but Shakespeare's 'Richard' is a highly complex figure, and we cannot assume that Shakespeare was merely subscribing to the 'Myth'. The dramatist creates a villain who, despite the constant references to his twisted appearance, is also highly intelligent and charismatic, even sexy. (He successfully woos Lady Anne - he doesn't rape her, she willingly surrenders - 1,ii) and he constantly turns to us, the audience, to brag about his treachery and make fun of his enemies, and we are to an extent 'wooed' too. Villain he may be, but he is supremely entertaining to watch, and the play becomes something quite other than a contest between 'good' and 'evil' once the audience identifies with the evil genius as 'hero'.
Of the many assassinations that Richard perpetrates (including Lady Anne's) the killing of his two young wards probably constitutes the turning point in our sympathies - dramatically orchestrated by Tyrrell's speech (3,iv) poignantly describing their deaths. Buckingham's revulsion at the deed probably mirrors ours, and after this point, Richard addresses us rather less, and the play's playful tone alters to something altogether darker.