The two are very closely linked in so many ways. After all, it is ivory that the wealth within Africa is based on, and it is ivory that gives Kurtz his success and his reputation. As Marlow discovers as he reaches the heart of darkness, what Kurtz did was to dominate one tribe of Africans and use them as an army to conquer other African tribes nearby and take their ivory. This ivory was so highly prized and valued that it gave him immense importance and power, attracting the attention of his superiors. Ivory is therefore a symbol both of death--as it necessarily involves the death of the animals who yield the ivory--but also of wealth, and wealth in particular that fuelled other civilisations and countries, such as was the case through colonialism. Note how this symbolism is hinted at through the description of the accountant, who is playing dominoes when he is introduced at the beginning of the novella:
The Accountant had brought out already a box of dominoes, and was toying architecturally with the bones.
The dominoes would have been made out of ivory, and it is no accident that another nickname for dominoes was "bones." What is suggested by the Accountant (the capital letter suggesting he, just like the Lawyer, is a type of person meant to represent the powers of civilisation) "toying architecturally" with these dominoes is that Western civilisation is built on the exploitation and appropriation of ivory. In other words, our wealth and prosperity has a foundation of death and unjust seizing of resources that do not belong to us.