Lennie and Curley are in an adversarial relationship, with Curley wanting to hurt Lennie simply because Lennie is a large man and Curley needs to assert himself and make himself look good, mostly at the expense of others. Lennie clearly does not want to fight, but Curley pushes him until George and the others in the bunkhouse can't tolerate it any longer. Curley knows that if the bigger man beats him, he'll get sympathy from others because of his smaller stature, and if Curley beats Lennie, Curley will get praise and admiration from others. Curley is a bully and Lennie is his target.
As for Lennie and Curley's wife, Lennie is spellbound by her. She looks beautiful, and smells nice, and talks softly to him. He knows he's to stay away from her, but he is unable to - and she encourages him. She recognizes him as a softhearted man, and one to whom she can talk and perhaps get sympathy. All of the other men think that if she talks to them, she's intentionally flirting, but Lennie has no awareness of intent, so he listens. He's also fascinated by all things soft (like the soft red dress of the woman he scared, and the scrap of velvet given to him by his aunt) and this is part of Curley's wife's appeal. Lennie is like a snake charmer and his snake - Lennie is unable to look away.