In order to group something with a mean, + and – opperations need to have mathematical sense so that the same geometrical distance anywhere in the color space reflects the same amount of perceived color difference. For example, mean value in RGB always tends to generate a brownish color (like when you are painting and you mix lots of colors without any criteria and it ends up in some brownish nonsense color)