When Montag goes to meet Faber at his home, he (Faber) appears old, feeble, and pale. It is this paleness that is associated with whiteness. Apparently, Faber had not been out of his house in a very long time.
The old man looked as if he had not been out of the house in years. He and the white plaster walls inside were much the same. There was white in the flesh of his mouth and his cheeks and his hair was white and his eyes had faded, with white in the vague blueness there. Then his eyes touched on the book under Montag's arm and he did not look so old any more and not quite as fragile.
Faber is/was an English professor; therefore books were his life. With books being banned, a significant part of his life is missing. Without books as his inspiration and as a means to teach others, his life is drained of color and of variety. In other words, Faber is pale because he stays in his house all of the time, but his life is pale too. With no books, there is no variety in his life. His wall are bare because he loathes the parlour shows. But even the parlour shows lack any color (in the sense of being vivid, intellectual, genuine, or thought-provoking). Faber is old, with white hair, pale features, and colorless walls. These descriptions illustrate his physical and mental disposition. Without books, his life is boring, colorless.
Even the first time they met, there was a description of paleness. Just before going to Faber's home, Montag recalls meeting him at a park. Stretching the theme even more to sound, Faber's voice sounded "pale"; again, no color in his voice. The paleness shows Faber's fear as well as the emptiness he feels.