There are many factors that can have a negative impact on the productivity of workers.
If we are talking about workers in general, rather than the workers at a specific firm, we can say that lower educational levels and lower skill levels will make them less productive. That is, we can expect that workers with more education and skills will be more productive than those with fewer.
This question might be asking, however, about factors that would reduce the productivity of workers who are already working in a particular job for a particular firm. One thing that could reduce their productivity would be a decrease in morale. If they are less satisfied with their working conditions, they will not be as productive. Another could be the quality of management. For example, managers might not put together the optimal teams for certain jobs. This could make the workers work less efficiently.