Well I just built the computer, so maybe I should have gone ahead and dropped the big dough on an uber processor. Or, maybe I'll just read a book while encoding video, since I need this computer to be quiet as well.
Primarily CPU, possibly limited by the speed of the ram, or if you're really running low on ram available the quantity could be an issue.
Depending on which formats you are coming from and going to, as well as which program(s) you're using, dual-core CPU's could potentially make a big difference or no difference at all.
Also, ATI has a new software thing coming out soon called Avivo (or something like that, I might be thinking of the name of another ATI tech) that uses your GPU to help with rendering and whatnot, from the demos I have seen the speed boosts you get are incredible.