A refrigerator works by removing heat from the refrigerator section and releases this heat into the surroundings. Let us assume that the room that has the refrigerator is insulated from surroundings, that is no heat is added to or removed from the room (by external sources/sinks). When the door is left open, the entire room becomes part of the refrigerator section, and the refrigerator will now try to remove the heat from this new section (entire room plus the original refrigerator section), and all this removed heat will be dissipated back to the room. In essence the room will lose less heat and gain more heat (remember heat is removed from both the room and refrigerator section). Effectively, the average temperature of the room will rise.
If you leave the door open temporarily, the heat from the room will rush in and compressor starts working again. This causes the cooling system to absorb some heat from the room and release it back to the room (through coils on the back or underneath the refrigerator), thereby causing a small change in room's temperature.