Benchmarking practialities

[HHunt]

As I said over in the "Benchmarking a myth"-thread, I've been planning to benchmark 2003 against XP for gaming and other frivolous uses. I've now got everything up, and I've got some results.

The hardware:
Asus PP-DLW
Intel P4 Xeon 2.8 (HT enabled)
1Gb PC2700 DDR Ram
Radeon 9800 Pro , 128Mb VRAM
Promise TX2+ SATA-controller
2x 250Gb Maxtor MaxLine SATA HDs

The software:
Windows 2003 Server, Standard (SP1 and all updates)
Windows XP Professional (SP2 and all updates)
The same drivers, as far as possible
CS : Source

The method:
I installed each windows on it's own HD, in a 100Gb partition.
In XP, I disabled the theme, but otherwise left it alone.
In 2003 I changed the performance settings to favor foreground applications, and set Hardware Acceleration to max in the graphics settings.
After installing everything I rebooted, ran three video stress tests (at the default settings), rebooted, repeated, rebooted, repeated, giving a total of 9 runs for each OS. The results were fairly consistent through all the tests on both OSes.

The results:
This is the average over all nine runs.
XP: 90.07 fps
2003:: 88.27 fps

The conclusion:
XP is the faster OS for this particular benchmark, as expected. However, the difference is not by any means big. From what data I've got so far, it looks like 2003 will be a perfectly good OS for gaming if one for some reason wants to use it for that.

As for the practicalities: 2003 took a bit longer to set up, mainly because SP1 is huge. The two minimal tweaks I did took less time than setting a sensible default directory view in XP, so my hands-on time before I considered the install finished was roughly identical. I didn't do anything else to 2003 that I didn't also do to XP.
Also, a random piece of advice: Don't install windows with a CF reader in an USB port. I have a 7in1-reader and an install of Windows 2003 Server that resides on I:\ . (I know, it should have been J:\. Wonder where the last letter went.)

What now?
The disks are both going to be moved to the BSD-box and reformatted as UFS2 in a few days, but until then, I'd like to run a few more benchmarks. Suggestions are welcome, especially ones that would be easy to aquire and run.

 

[Carnival Forces]

howabout disk read/write benchies? i don't know of any, but the Disk Storage Forum should have some...

also, i wonder, after being fully tweaked out, which would be faster? I don't know of any tweaks for 2003, though, and since you're not planning to keep the OS's i wouldn't bother tweaking them...

 

[HHunt]

I agree, it'd be interesting to see ho much I could get out of 2003, but ... not this time. I'll see if I can find any disk benchmarks.

 

[S1nF1xx]

There's little doubt S2K3 could be used to game. The question is why?

Why would someone want to game on a server OS other than to extend his E-penis?

- Server costs 5X as much as XP Pro. (Which doesn't matter since anyone who is gaming on 2K3 is using a pirated version.)

- XP is what a lot of drivers are written for. Although XP and 2K3 are very similar, the drivers aren't optimized for it.


It's nice to see some benchmarks finally, so thanks to the OP for that. Now if someone can just come up with a convincing reason to use a server OS to game on.

 

[HHunt]

There's little doubt S2K3 could be used to game. The question is why?
Why would someone want to game on a server OS other than to extend his E-penis?
- Server costs 5X as much as XP Pro. (Which doesn't matter since anyone who is gaming on 2K3 is using a pirated version.)
- XP is what a lot of drivers are written for. Although XP and 2K3 are very similar, the drivers aren't optimized for it.
It's nice to see some benchmarks finally, so thanks to the OP for that. Now if someone can just come up with a convincing reason to use a server OS to game on.

Reasons? I don't need your stinking reasons.
Seriously, though: They're both free for me [1], so the price doesn't matter. It doesn't seem to be significantly slower, so that's not an issue. So far I haven't had any issues with drivers, nor indeed anything else.

What I'm left with is that from my POV they're functionally very much like each other. The only real difference is that 2003 has slightly different defaults, and if I want to, there's some interesting services to play with. [2]

I have to ask: Why not?

[1] And don't generalize like that. There's bound to be more people than me who can legally get free 2003 licenses from school/work/whatever.
[2] It's always nice to have another potential fallback server, though I guess I'm more likely to use the FreeBSD install if it comes to that.

 

[Phoenix86]

Pfft, I could have told you this, you could have just asked.

^.^

Good work. What level/test did you use in CS:S?

 

[HHunt]

Just the video stress test, so far. (It's by far the easiest. )
Any better ideas?

I won't be doing anything on this today, btw. Project handin tomorrow, and there's still pieces missing. Yay, etc.

 

[S1nF1xx]

I have to ask: Why not?

Because everything is optimized for XP, not 2K3.

Ball's in your court buddy.


I'm not trying to disprove your benchmarks or call them pointless. I would like to know though, what reasons people think of that makes them want to game on S2K3. I would guess they think it's more stable, which is kind of negated once you start using unsupported drivers/software.

Seriously though, why do people want to game on S2K3?

 

[djnes]

It's never been a question about performance. It's been proven over and over again that 2003 can be XP-ized and can run the latest games, etc.

However, anytime this is debated, one fundamental, logical fact is ignored. 99.9% of the people out there would have no way of getting 2003 for free, or even at a discount to make it comparable to XP in price. So, for 99.9% of the people out there, this debate is useless. You can get XP OEM for around $100 at the right places. Unless you can legally buy Server 2003 for around the same price, the entire point is moot.

I myself have legal corporate copies of both XP Pro and Server 2003. I still would never run 2003 on my main system because it doesn't function as a server. I do run 2003 on my server however, because that's what it's intended for. I could put 2003 on my aming system, but by the time I tweak it and customize it to do so, it would be very close to what XP gives me out of the box.

 

[djnes]

Seriously though, why do people want to game on S2K3?

Because it's one of those things people will do because it makes them look more knowledgeable in front of their friends....without ever stepping back and asking if it makes sense or not. The entire point of running 2003 is to use it as a server...and to have a client machine to interact with it. Running 2003 on a standalone machine absolutely defeats the purpose of it being a server OS anyway.

 

[Phoenix86]

Seriously though, why do people want to game on S2K3?

It's because what I call the core+ model (I made that term up just now) they started using in 2K.

Consider W2K the core.
W2K + code=W2K Server.
W2K Serve + coder=W2k Adv. Server.
W2k Adv. Server + code=W2k Data Center.

This is why those DVDs *ahem* floating around with multiple OS installs work and use so little space. They reuse the core files for each install, so you only have one copy of the core on the disk.

With XP we had a new core, and with this core we have new improvements. New improvements=better OS. So as a generalization we have XP > 2K with regards to performance (yes, there are cases where 2K performs better, but it's the exception, not the rule).

With S2K3 we again have a new core. Logic says the new core will be better (drivers cause this to lag a bit, at the initial release 2K was better than XP), and it probably is.

However, there is a monkey wrench thrown at us since we see the server part before the desktop. So people want to use the new core even if it's a server OS. Another twist is XP64, which is based off of... Dum-da-dum, S2K3, and *not* XP as the name implies.

Why? The newer core is better. Why release an old core with XP64 when the new one was already ready for S2K3?

 

[S1nF1xx]

^ I would believe that coming from someone like you, it does make sense. But when someone comes on here and goes "How do I install Server 2003, I want to game on it?", I doubt they thought it out that well, and I'm inclined to go the idea that they just want to look cool.

 

[djnes]

^ I would believe that coming from someone like you, it does make sense. But when someone comes on here and goes "How do I install Server 2003, I want to game on it?", I doubt they thought it out that well, and I'm inclined to go the idea that they just want to look cool.

Nor do they come up with such catchy phrases as "core+model".

 

[Phoenix86]

^ I would believe that coming from someone like you, it does make sense. But when someone comes on here and goes "How do I install Server 2003, I want to game on it?", I doubt they thought it out that well, and I'm inclined to go the idea that they just want to look cool.

^ I would believe that coming from someone like you, it does make sense. But when someone comes on here and goes "How do I install Server 2003, I want to game on it?", I doubt they thought it out that well, and I'm inclined to go the idea that they just want to look cool.

I'm inclined to believe you.

However, that doesn't mean the argument is w/o merit... It's a worthy test.

Nor do they come up with such catchy phrases as "core+model".

 

[S1nF1xx]

However, that doesn't mean the argument is w/o merit... It's a worthy test.

It is a decent debate. This is the first time I've seen anyone present the argument in a way other than how I stated earlier. And like I said, using your logic, it would seem worth the time to try and see how it works, though I see most people using it as enzyte for their E-penis.

Nor do they come up with such catchy phrases as "core+model".

Hah, no they don't.

 

[HHunt]

It's never been a question about performance. It's been proven over and over again that 2003 can be XP-ized and can run the latest games, etc.

However, anytime this is debated, one fundamental, logical fact is ignored. 99.9% of the people out there would have no way of getting 2003 for free, or even at a discount to make it comparable to XP in price. So, for 99.9% of the people out there, this debate is useless. You can get XP OEM for around $100 at the right places. Unless you can legally buy Server 2003 for around the same price, the entire point is moot.

I myself have legal corporate copies of both XP Pro and Server 2003. I still would never run 2003 on my main system because it doesn't function as a server. I do run 2003 on my server however, because that's what it's intended for. I could put 2003 on my gaming system, but by the time I tweak it and customize it to do so, it would be very close to what XP gives me out of the box.

I'm doing it for the remaining 0.1%
(It's not a moot point for me, and if anyone else finds it useful, even better. That a majority of the population can not get it for free is irrelevant.)

Also, the out of box performance of 2003 is already very close to XP. That was kind of my point. I have not "tweaked" 2003. Changing the wallpaper takes about as much time as changing the three settings I touched.

Agreed on the "core"-argument. It could be interesting to benchmark 2000 Pro vs 2000 Server after setting them up to be as identical as possible, just to see if there's any noticeable differences. If there isn't, that would suggest some interesting things. Mainly, that the changes done to a server-OS are "only" in the services, managment tools and settings, which are all reachable for a normal user. If so (and that's a big "if"), I expect a slightly tweaked 2003 will outperform XP when the drivers catch up.

 

[djnes]

I should have clarified that I didn't think your testing was useless. It's always good to test...even just out of curiosity. You did a good job with the testing.

 

[HHunt]

I should have clarified that I didn't think your testing was useless. It's always good to test...even just out of curiosity. You did a good job with the testing.

Thanks. Curiosity was indeed the main motivation.

 

[S1nF1xx]

Even though I've been the cynic (as usual), you did do a good job, and it would be interesting seeing more real-world benchmarks.

It would be cool to see video ripping/encoding benchmarks too. *hint*

 

[HHunt]

Hmm. I've always wanted to try DVD -> DivX .
Software ideas?

(I'd normally use mencoder, but ...)

 

[S1nF1xx]

DVD Shrink comes to mind.

 

[HHunt]

I'll see what I can do.

 

[Phoenix86]

Well if you wanted to do some more generic benchmarking look at something like Sandra that'll test a few generic markers like memory bandwidth, HD Tach for HDD testing, etc. Not sure about timing encoding runs, but I know there's stuff out there.

 

[HHunt]

Well if you wanted to do some more generic benchmarking look at something like Sandra that'll test a few generic markers like memory bandwidth, HD Tach for HDD testing, etc. Not sure about timing encoding runs, but I know there's stuff out there.

I expect HD Tach to give almost identical results, so ... *adds it to growing list*.
(After all, that only makes it more interesting if the results are different.)
The same goes for some of Sandra's benchmarks.

I've got a stopwatch and another computer to distract me, so I can do timing by hand, if needs be.

 

[Phoenix86]

I expect HD Tach to give almost identical results, so ... *adds it to growing list*.
(After all, that only makes it more interesting if the results are different.)
The same goes for some of Sandra's benchmarks.

You'd think that, but you need to remember how important drivers can be, esp chipset drivers... There could be a difference, but I wouldn't be surprised to see similar results. Remember, PIO/DMA modes affect transfer rates a lot, and that's driver dependant. Of course if there are no real updates to that section of the OS, and the drivers are not different, yeah should be identical.

I've got a stopwatch and another computer to distract me, so I can do timing by hand, if needs be.

Take the stop watch examples with a grain of salt, but if something's that noticable...

 

[HHunt]

You'd think that, but you need to remember how important drivers can be, esp chipset drivers... There could be a difference, but I wouldn't be surprised to see similar results. Remember, PIO/DMA modes affect transfer rates a lot, and that's driver dependant. Of course if there are no real updates to that section of the OS, and the drivers are not different, yeah should be identical.


Take the stop watch examples with a grain of salt, but if something's that noticable...

Well, AFAIK I'm using the same drivers from Promise with the same settings. Enabling TCQ took XP from somewhere around 83 fps up to around 90 in CS:S, btw. The first run of XP tests were without it, but when I saw the difference I dropped it from the final result. At least I think that was what caused it. Something was definitely wrong, and that was all I changed.

As for stopwatching, I expect reencoding a full DVD to reasonable DivX to take enough time that a few seconds off will be within the margin of error. If they're so close that it would have mattered, I'll write it off as "almost identical".