What an interesting question! There are certainly a few different angles you could take on this question.
First, the man who has no inner life can be thought of as a man who has no soul. Thus, if he has no soul, no desires and no ideas, he will become a slave to what is around him, doing only what other do and what is available in his environment. For example, if he lived in city, he would be surrounded by many people and lots of 'hustle and bustle' and be forced to keep up with them both literally, while walking, or riding his bike and figuratively for example moving upwards in his job. If he had an inner life, or inner desires, he may choose to live in the peaceful city.
Another way to think about this is a man who has no inner life has no close friends or family. He goes out into the world to work and live but when he comes home is alone and without hobbies or interests. He would be a slave to his job and society since he does not seek out other interests rather just goes with whatever is around him.
Hope this is helpful!