Pip recalls that when Mrs. Joe was in a bad mood, Joe would cross his two forefingers together when she wasn't looking as a sign to beware of her:
[Joe] crossed his two forefingers, and exhibited them to me, as our token that Mrs. Joe was in a cross temper.
This signal is part of the deep bond Joe and Pip share. They are drawn together by having to deal with Mrs. Joe, an often bad-tempered and hard-hearted woman. In contrast, Joe is softhearted and mild-mannered, which allows Mrs. Joe to bully him, while Pip is a child without much choice but to put up with her temper.
Mrs. Joe is also described as a good housekeeper but not the kind of person who makes a house comfortable for those who live there. She often feels she is put upon, with too much work to do, and takes this out on Joe and Pip. It being Christmas when Joe sends the secret signal about Mrs. Joe's bad mood, the two males are banished from the house to go to church while Mrs. Joe readies it for guests who are to arrive later in the day.
Dickens makes it clear that without Joe, Pip's childhood would have been a misery with Mrs. Joe, and as it is, Pip doesn't have the easiest time of it.