I would say, though, that the attraction is sexual, but not necessarily only sexual. I'd go with your second suggestion: that it is partially sexual.
The evidence for a sexual component you've clearly already seen. Look at lines like this: " Miss Fulton did not look at her; but then she seldom did look at people directly. Her heavy eyelids lay upon her eyes and the strange half-smile came and went upon her lips as though she lived by listening rather than seeing. But Bertha knew, suddenly, as if the longest, most intimate look had passed between them–as if they had said to each other: "You too?"–that Pearl Fulton, stirring the beautiful red soup in the grey plate, was feeling just what she was feeling."
That's not just sexual desire; that's connection. But I would say there is more there too. There's a kind of avid ownership, a way she makes Pearl and her other "finds" into something special for her, like adventures. This is sex mixed with a particular kind of power and perspective.