The guy who spoke at Solarfest told us about some actual research that had been done: Same-model refrigerators were loaded with the same amount of food. A "control" fridge was left closed for several days, while the "experimental" fridge had its door opened far more frequently than even the most scatterbrained person would do in normal life, and held open for longer than people usually hold the door open. At the end, with the refrigerators cooled down to their set temperature, they compared the amount of electricity used by both (they'd had meters attached), and found the difference to be insignificant.
It makes sense if you think about it. Refrigerators cool the air inside, but cooling air isn't the point; the point is to transfer some heat out of the stuff that's in the fridge, and keep excess heat from getting back into the stuff. Keeping air cool is a means to an end.
Air itself has very little heat content, because it's so thin. Most of the heat a fridge is concerned with is in the much denser solid/liquid food. If you put warm food in, some of its heat will transfer into the cooler air, and the fridge will have to continuously cool that air until all of the excess heat has come out of the food, which happens slowly.
When you open the door, all you do is cycle a bit of cool air out, forcing the fridge to re-cool some air, but that takes very little energy. Since heat transfers much more slowly into solids, very little heat gets into your food while you have the door open. After you close the door, the fridge quickly re-cools some air, expending very little energy. There's hardly any extra heat to transfer out of the solid stuff, so that air stays cold. To really "waste" energy, you would need to do more than cycle a bit of air out. You'd need to take the cold stuff out of the fridge, and replace it with warmer stuff. But that's the reason you have a refrigerator in the first place.
If you want use your refrigerator more efficiently, don't worry about how often you open the door. Instead, if you have food items out that you plan to put back into the fridge, and that you don't need to warm up before you use them, put them back into the fridge before they have a chance to sit out for a while and get warm. Stuff matters much more than air.
Edit: Simplified way of thinking about it: What matters is the transfer of heat (not temperature) into the fridge; that's what it uses energy to get rid of. You can't easily move much heat into the fridge with air; you mostly do it with solids and liquids.