10GBe Mellanox issues?

LOCO LAPTOP

LOCO LAPTOP

[H]F Junkie
Joined
May 4, 2006
Messages
12,667
Since 10Gig SFP cards are so cheap on eBay I figured I would finally hookup my server and my desktop. However I have a few issues.

I currently have a HP 1410-24G switch, until my HP 1420-24G-2SFP+ 10G gets in. When hooking up a HP/Mellanox ConnectX-2 using a Cisco SFP-H10GB-CU5M or SFP-H10GB-CU3M to the HP or my spare Cisco switch, I get Cable Unplugged in both machines. However Cisco to HP switch using these cables, not a problem. Not sure if it's me who picked the wrong cables or they don't like 1Gb switches?

Now I figured lets connect them to together, took the CU5M cable from my pc to server hook it up, works... for a little while. Swapped to the CU3M, same thing. What happens is, I start transferring a file at 360-385MB/s About 30 to 90 seconds in (random), the speed drops to 25-30MB/s and then completely stops a few seconds later, starts working again and stops. Same happens when watching a video from the server, starts fine than skips and stops. During that whole time I can ping the that nic, and I loose access to remote desktop using the 10G network card, on board works fine.

My server is running Server 2012R2, my pc is running Windows 10, both have Asus boards and using the latest drivers from Mellanox website.

Is there an issue with the cards, picked the wrong cables, or am I missing something? I'm new to all this 10G and SFP stuff.
 
What switches?

1GbE switches will not use/accept SFP+/10GbE adapters.

I use ConnectX-2s extensively.. can you check for packet errors? Are they running hot? Is there anything in the event log?
 
Switches: HP 1410-24G and Cisco SG300-20. Both will talk to each other just fine with the 10G cables, at 1G. It's the switches to the desktop that does not work. I'm not worried about this as a I have a 10G switch coming anyway.

I will check for packet errors, The cards are not running hot, I did have to put a fan on the on in my desktop last night as it was a bit hotter but was not near it's limits.

I have 2 errors in the event log: (Firmware version on both cards FW: 2.9.1000, PEX 3.3.0500)

The File Transfer (SMB) performance may be affected as Network Direct functionality is not supported in ConnectX-2 firmware version.

Mellanox ConnectX-2 Ethernet Adapter device reports that the "QOS (ETS) capability is missing". The current firmware does not support the QOS (ETS) capability. Please burn the latest firmware and restart your machine. (The issue is reported in Function SetHardwareAssistAttributes).

Oddly enough i'm 500GB doing a transfer from my deskop to the server and it's been 15 minutes, no slow down's and no errors. I have made no changes at all today.
 
Okay got the slow downs (under 2MB/s) again near the end. Nothing new in the event logs, No errors, cards are not hot. Disk I/O on both machines next to none.

T3r2uAs.png


EDIT: I just noticed the server is running at PCIe x4. Think that has to do with it?
 
Last edited:
10GbE fiber is no different, though you should really use OM3 (Aqua) over OM2 (Orange). You can't connect 10GbE to a 1GbE switch.

Just connect the machines to each other.
 
FrankD400 said:
10GbE fiber is no different, though you should really use OM3 (Aqua) over OM2 (Orange). You can't connect 10GbE to a 1GbE switch.

Just connect the machines to each other.
Click to expand...
The cables I have are 10G SFP+ Twinax cable, passive, I really don't fiber as I know I will likely break the fiber cable as I'm not too careful when moving stuff,

That's how it's hooked up right now and giving me issues, I'm just lost on where the issue is. bad card, motherboard compatibility, user error.
 
PCIE x4 is not an issue. If you're getting slowdowns it's probably due to the OS. Try using iperf to test speeds instead of SMB.

Also, your firmware is old, update it to the newest or second newest one.
 
After updating it to 2.9.1200 from Mellanox site, I get network cable unplugged now, after doing the server card. I did my desktop card first and it worked until the server got updated, now it no longer works. I still have to play with it and see if I can make it work.

I found 2.9.8350 on their site as well but has a older date (2011) and bricked one of the cards, I had to reflash to 2.9.1200 to bring it back.
 
Updated both to 2.9.1200 and working correctly now. I also got my 10G HP switch installed. Firmware 2.9.8350 will (for me) brick both cards. Flashing back to 2.9.1200 and everything is normal again.

After hours of testing it comes down no issues unless it's SMB to my Raid 10 network drives. Which kills the point of me getting this as it's all I use it for.
 
You must log in or register to reply here.
Back
Top