The pilgrims are concerned with seeking something externally, namely ivory and therefore they are safe from succumbing to the inner darkness. In order to succumb to inner darkness you must be able to look inside yourself. The pilgrims do not do this. In a similar way, the other characters in the book, the Manager, the Accountant and the Brickmaker also do not fall prey to the madness that occurs when one falls into the heart of darkness. They are not looking introspectively.
Marlow identifies with Kurtz and recognises what might happen were one to look too deeply. Marlow even says at the end that he has been to the edge but did not succumb like Kurtz did partially because he witnesses Kurtz's descent. At the end of the novel we return to Marlow on the Nellie retelling his story. He says he is apart from other men because he has gotten close to the darkness and this makes him different.
To sum up, the Pilgrims cannot fall prey to the inner darkness because they keep busy externally searching for material wealth. This search keeps them from looking inside themselves.