I think you, Simon and BEB have provided two distinct and consistent explanations of this.
i would now like to ask why, with conventional centre-hinged ailerons (I'm excluding balanced ailerons and Frise ailerons for the moment) you necessarily get adverse yaw.
I have been assured that this is simply the consequence of drag being proportional to lift. The lowered aileron results in more lift and, hence, more drag while the raised aileron reduces lift and drag.
Is this a correct and complete explanation?