I think the answer to your question can be found in Sam's own words in the last chapter of the novel. He gives Brigid seven reasons why he is "sending her over" to San Quentin rather than letting her go. Then he says:
"Now on the other side we've got what? All we've got is the fact that maybe you love me and maybe I love you."
"You know," she whispered, "whether you do or not."
"I don't. It's easy enough to be nuts about you." He looked hungrily from her hair to her feet and up to her eyes again. "But I don't know what that amounts to. Does anybody ever?"
Sam Spade is a realist. He has had years of experience as a cop and as a private detective. He has seen the worst of humanity, and it has affected his character. He believes in hard facts and doesn't have much faith in emotions. He knows he would be crazy to trust Brigid O'Shaughnessy because he has seen how treacherous she can be. The fact that he turns her over to the police is pretty good evidence that he does not really love her. Does anybody ever know what love amounts to?