According to the legitreviews article, when the GTX 580 was downclocked to match stock GTX 480 speeds the performance difference was only 3.5%. So the extra shaders and such don't really give as much a boost as we thought, it was mostly the increased clock speed??
http://www.legitreviews.com/news/9429/
http://www.legitreviews.com/news/9429/