Hummingbirds are the only birds that can fly upside down. They do so because of their angled wing structure. However, a hummingbird can fly upside down only for a short period of time. A hummingbird can also fly backward; it does this when removing its bill from a tube flower.
Source: Kuttner, Paul. Science's Trickiest Questions, p. 128.