The main reason for this is the different number of firms that are in each of these market structures.
In both perfect competition and monopolistic competition, there are large numbers of firms selling the same (in perfect competition) or similar (in monopolistic competition) product. By contrast, in an oligopoly there are very few firms selling the product.
Because there are only a few firms in an oligopoly, the firms are really in very direct competition with one another. Think about McDonalds and Burger King -- what any one of these does affects the other. But then think about all the Chinese restaurants in New York City. What one of these does can have very little impact on the others.