10Gb network

[melloa]

Folks,

As a newbie I only can post here, so apologies if not posted under Network.

I'm upgrading my network to 10Gb and trying to get hand full boxes on those ports. So you better understand, I'm using two SFP+ Aruba S2500 switches. That net six 10Gb ports for my servers/workstations, as two are used to uplink the switches.

As all new implementation the learning curve is tough and this one is not an exception...

Currently I have:

- FreeBSD 11.2 server/Chelsio T320 - working
- FreeNAS 11.1 serverChelsio T320 - working
- Windows 10 workstation/Mellanox - working
- ESXi 6.5 T320 - No Go (waiting on a T5 to re-try)

and the most important to me:

- Linux Mint 19 workstation/Chelsio T320

Did a fresh install of LM 19 to remove any of previous tests/changes I'd done. Card shows as expected:

Code: Select all

mello@olinda:~$ dmesg | grep -i Chelsio
[    0.927537] cxgb3: Chelsio T3 Network Driver - version 1.1.5-ko
[    1.269054] cxgb3 0000:06:00.0 eth0: Chelsio T320 10GBASE-R NIC (rev 4) PCI Express x1 MSI-X
[    1.269056] cxgb3 0000:06:00.0 eth1: Chelsio T320 10GBASE-R NIC (rev 4) PCI Express x1 MSI-X

Driver is loaded as it also shows here:

Code: Select all

mello@olinda:~$ ifconfig rename2
rename2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.10.60  netmask 255.255.255.0  broadcast 10.10.10.255
        inet6 fe80::207:43ff:fe06:b659  prefixlen 64  scopeid 0x20<link>
        ether 00:07:43:06:b6:59  txqueuelen 1000  (Ethernet)
        RX packets 6333088  bytes 4122670256 (4.1 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8833892  bytes 10859459898 (10.8 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16  memory 0xf7881000-f7881fff  

When testing with iperf, using the default 1500 MTU, I get:

Code: Select all

root@bhyve:~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.152 port 5001 connected with 10.10.10.60 port 56146
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.89 GBytes  1.62 Gbits/sec

If I turn jumbo frames on, through put goes down to a craw:

Code: Select all

mello@olinda:~$ sudo ifconfig rename2 mtu 9000
mello@olinda:~$ iperf -c bhyve
------------------------------------------------------------
Client connecting to bhyve, TCP port 5001
TCP window size:  325 KByte (default)
------------------------------------------------------------
[  3] local 10.10.10.60 port 56712 connected with 10.10.10.152 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.2 sec   326 KBytes   261 Kbits/sec

Interesting that iperf to/from the Mint box is good, but not the same 95.4Gb I get between the servers:

Code: Select all

mello@olinda:~$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.60 port 5001 connected with 10.10.10.60 port 46248
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  28.1 GBytes  24.1 Gbits/sec

I'll free-up a Mellanox MNPA19-XTR 10G SFP Optical Network Card from the ESXi in a couple weeks, but if anyone has any experience with those T320 that would like to share, so I won't toss the card in my recycle bin, I'd appreciate.

Thanks,
Al

[melloa]

I guess that no one has used Chelsio on Mint before, as I didn't see any replies.

Today I started playing with the Mellanox MT26448. Good thing is that fresh install found the card from the get go, but speeds are still slow. ~1Gb/s tx and ~980Kb/s rx.

More to come.

[catweazel]

speeds are still slow. ~1Gb/s tx and ~980Kb/s rx.

Please post the output of ethtool <name-of-interface>.