Poor NFS read performance

[Fryguy8]

So, I'm trying to increase my NFS read performance, as I feel I should be able to consistently get into the 90s or 100s:

First

1976844+0 records in
1976843+0 records out
1012143616 bytes (1.0 GB) copied, 14.3723 s, 70.4 MB/s

That's my current speed reading some random file (not cached).

Here is the speed reading a file direct from the disk, proving that the disk isn't the bottleneck:

8963+1 records in
8963+1 records out
9398410906 bytes transferred in 41.523973 secs (226336986 bytes/sec)

Server is:
FreeBSD 8-STABLE
ZFS raidz of 5 samsung 750GB
82572EI PRO/1000 PT Desktop Adapter (Copper)
16GB ram, 6-core AMD phenom

Client is:
Archlinux, 2.6.39-ARCH kernel
Intel Corporation PRO/1000 GT Desktop Adapter
4GB ram, Q6600

Switch is a 5-port netgear prosafe, all cat6 cables.

Using 9k jumbo frames on both machines. NFS mount is:

type nfs (rw,noatime,vers=3,rsize=32768,wsize=32768,tcp,addr=192.168.10.1)

Autonegotiation is turned off for both machines, both hardcoded to 1000BaseT with full-duplex.

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
	options=1db<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,POLLING,VLAN_HWCSUM,TSO4>
	ether 00:1b:21:29:b0:ab
	inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255
	inet6 fe80::21b:21ff:fe29:b0ab%em0 prefixlen 64 scopeid 0x1 
	inet 192.168.10.200 netmask 0xffffffff broadcast 192.168.10.200
	inet 192.168.10.201 netmask 0xffffffff broadcast 192.168.10.201
	inet 192.168.10.202 netmask 0xffffffff broadcast 192.168.10.202
	inet 192.168.10.203 netmask 0xffffffff broadcast 192.168.10.203
	inet 192.168.10.204 netmask 0xffffffff broadcast 192.168.10.204
	inet 192.168.10.205 netmask 0xffffffff broadcast 192.168.10.205
	inet 192.168.10.206 netmask 0xffffffff broadcast 192.168.10.206
	inet6 2001:470:1f07:d6c::fe00 prefixlen 64 
	inet6 2001:470:1f07:d6c::fe01 prefixlen 64 
	inet6 2001:470:1f07:d6c::fe02 prefixlen 64 
	inet6 2001:470:1f07:d6c::fe03 prefixlen 64 
	inet6 2001:470:1f07:d6c::fe04 prefixlen 64 
	inet6 2001:470:1f07:d6c::fe05 prefixlen 64 
	inet6 2001:470:1f07:d6c::fe06 prefixlen 64 
	inet6 2001:470:1f07:d6c::1 prefixlen 64 
	nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
	media: Ethernet 1000baseT <full-duplex>
	status: active

eth0      Link encap:Ethernet  HWaddr 00:0E:0C:D9:71:53  
          inet addr:192.168.10.111  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: 2001:470:1f07:d6c:20e:cff:fed9:7153/64 Scope:Global
          inet6 addr: fe80::20e:cff:fed9:7153/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:34916928 errors:0 dropped:25397 overruns:0 frame:0
          TX packets:19890332 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:50720483706 (48370.8 Mb)  TX bytes:5057575977 (4823.2 Mb)

What else should I be looking at to get speed to 90+ like I"ve seen in other setups?

 

[lightp2]

Dear OP,
1. I am not a NFS/FreeBSD user.
2. Just an inquiry --> Can you try FreeBSD-NFS-Server to FreeBSD-NFS-client NFS test?

There are some FreeBSD users in this forum. some may be able to suggest. This is especially for FreeBSD/ZFS combination seems like there are certain kernel tuning also recommended. However, the immediate is to do item 2 to get baseline performance understanding.

 

[packetboy]

Check your Ethernet flow control settings...I've found that FC sucks wind, especially when using consumer grade switches.

See my post here on how to use ethtool to review and fix. I'm using managed switches, so I'm able to manually disable FC on the switch ports too...no sure what will happen if using unmanaged switches.

http://hardforum.com/showthread.php?t=1601960

While you're there, you might as well check your Ethernet duplex settings as this is the number 1 cause of Ethernet performance problems.

Also, what CPUs are you using?

 

[Fryguy8]

Flow control is off (I'd already read your post).

CPUs are mentioned in the initial post, a Q6600 and a phenom x2 3.2ghz