![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Does anyone else see this?
Atto giving different results at QDepth > 1 on NUMA based systems.
Driver, firmware, or just numa?
Tested: LSI based Dell sas 5/e, Dell perc 6/e 512mb, HP P411 512MB
More pronounced on HP card, but present on LSI cards too.
Base tests on a Dell vostro 220, with raid card in a (claimed) pci-e 2.0 slot.
Results: consistent results.
Test results on Dell R905, with raid card in pci-e 1.1 slot x8, 4x quad core, 128 GB ram, W2K8x64.
Results: inconsistent results.
R905 bios memory config: All tried.
Disk type: 300GB 15k sas
Background:
While testing an HP P411 with various stripe sizes and raid types on a Dell R905, with Atto set at a QDepth of 10, I was getting inconsistent results. Results that didn't make any sense.
A 4 disk raid 0 with any stripe size vs a single disk, could score much higher at some random block size, then be crushed at the next block size, by the single disk; or the logical disk could scale great up until some block size, then it would fall down and start scaling again at the next block size.
Upgrades to the HP drivers and firmware have made the scaling more linear, but I noticed something during the tests.
While I was testing the controller, I was frustrated at the inconsistent results. I was tired of creating various raid arrays and stripe sizes. At some point I created a 1 disk raid 0, a 2 disk raid 0, and a 4 disk raid 0, that all existed at the same time. Then I opened 3 atto windows.
Atto1= disk one, Atto2= disk two, Atto3= disk three.
No matter what the stripe/allocation size per logical disk, I would always get the same results. Every time. But, the the results per atto/disk window were wildly different, and not where they should be for diskX config vs diskY config.
Anyway, at some point I made a mistake and selected the same disk in two of the atto windows. The interesting thing is: The two atto windows gave wildly different results, for the same logical disk. I ran each window again, identical results.
Then I compared disk X to disk Y in each window. The numbers were consistent within each window. But window A was wildly different from window B for each disk. While the numbers were different from disk A to disk B in the same window, the graphs in each window remained remained consistent. Whether it was disk X or disk Y, the graphs had the same shape.
Task Manager:
While running these tests, task manager shows that the three atto windows are hitting different CPU's. (If all three are started at the same time, they all hit the same cpu. If a new atto is started while another atto is under load, it may land on another cpu.) This is where you can see the difference.
From my testing, I have found:
Atto results vary by 5% to 13% on multi cpu or numa systems, from one cpu to another, depending on the cpu that has been hit.
This affects both HP/PMC cards, and LSI 1078 based cards, but doesn't affect LSI 1078 cards as much.
Questions:
Can anyone reproduce or confirm these results?
Is this the result of a bad driver, poor firmware, or is it just the way that numa systems work?
Is there anything that can be done within the Windows registry or driver settings to combat/smooth this inconsistency from one cpu to another?
Atto giving different results at QDepth > 1 on NUMA based systems.
Driver, firmware, or just numa?
Tested: LSI based Dell sas 5/e, Dell perc 6/e 512mb, HP P411 512MB
More pronounced on HP card, but present on LSI cards too.
Base tests on a Dell vostro 220, with raid card in a (claimed) pci-e 2.0 slot.
Results: consistent results.
Test results on Dell R905, with raid card in pci-e 1.1 slot x8, 4x quad core, 128 GB ram, W2K8x64.
Results: inconsistent results.
R905 bios memory config: All tried.
Disk type: 300GB 15k sas
Background:
While testing an HP P411 with various stripe sizes and raid types on a Dell R905, with Atto set at a QDepth of 10, I was getting inconsistent results. Results that didn't make any sense.
A 4 disk raid 0 with any stripe size vs a single disk, could score much higher at some random block size, then be crushed at the next block size, by the single disk; or the logical disk could scale great up until some block size, then it would fall down and start scaling again at the next block size.
Upgrades to the HP drivers and firmware have made the scaling more linear, but I noticed something during the tests.
While I was testing the controller, I was frustrated at the inconsistent results. I was tired of creating various raid arrays and stripe sizes. At some point I created a 1 disk raid 0, a 2 disk raid 0, and a 4 disk raid 0, that all existed at the same time. Then I opened 3 atto windows.
Atto1= disk one, Atto2= disk two, Atto3= disk three.
No matter what the stripe/allocation size per logical disk, I would always get the same results. Every time. But, the the results per atto/disk window were wildly different, and not where they should be for diskX config vs diskY config.
Anyway, at some point I made a mistake and selected the same disk in two of the atto windows. The interesting thing is: The two atto windows gave wildly different results, for the same logical disk. I ran each window again, identical results.
Then I compared disk X to disk Y in each window. The numbers were consistent within each window. But window A was wildly different from window B for each disk. While the numbers were different from disk A to disk B in the same window, the graphs in each window remained remained consistent. Whether it was disk X or disk Y, the graphs had the same shape.
Task Manager:
While running these tests, task manager shows that the three atto windows are hitting different CPU's. (If all three are started at the same time, they all hit the same cpu. If a new atto is started while another atto is under load, it may land on another cpu.) This is where you can see the difference.
From my testing, I have found:
Atto results vary by 5% to 13% on multi cpu or numa systems, from one cpu to another, depending on the cpu that has been hit.
This affects both HP/PMC cards, and LSI 1078 based cards, but doesn't affect LSI 1078 cards as much.
Questions:
Can anyone reproduce or confirm these results?
Is this the result of a bad driver, poor firmware, or is it just the way that numa systems work?
Is there anything that can be done within the Windows registry or driver settings to combat/smooth this inconsistency from one cpu to another?