10Gb network

All Gurus once were Newbies
Forum rules
There are no such things as "stupid" questions. However if you think your question is a bit stupid, then this is the right place for you to post it. Please stick to easy to-the-point questions that you feel people can answer fast. For long and complicated questions prefer the other forums within the support section.
Before you post please read how to get help
Post Reply
melloa
Level 1
Level 1
Posts: 18
Joined: Mon Apr 04, 2016 10:26 pm

10Gb network

Post by melloa » Fri Oct 19, 2018 2:59 pm

Folks,

As a newbie I only can post here, so apologies if not posted under Network.

I'm upgrading my network to 10Gb and trying to get hand full boxes on those ports. So you better understand, I'm using two SFP+ Aruba S2500 switches. That net six 10Gb ports for my servers/workstations, as two are used to uplink the switches.

As all new implementation the learning curve is tough and this one is not an exception...

Currently I have:

- FreeBSD 11.2 server/Chelsio T320 - working
- FreeNAS 11.1 serverChelsio T320 - working
- Windows 10 workstation/Mellanox - working
- ESXi 6.5 T320 - No Go (waiting on a T5 to re-try)

and the most important to me:

- Linux Mint 19 workstation/Chelsio T320

Did a fresh install of LM 19 to remove any of previous tests/changes I'd done. Card shows as expected:

Code: Select all

mello@olinda:~$ dmesg | grep -i Chelsio
[    0.927537] cxgb3: Chelsio T3 Network Driver - version 1.1.5-ko
[    1.269054] cxgb3 0000:06:00.0 eth0: Chelsio T320 10GBASE-R NIC (rev 4) PCI Express x1 MSI-X
[    1.269056] cxgb3 0000:06:00.0 eth1: Chelsio T320 10GBASE-R NIC (rev 4) PCI Express x1 MSI-X
Driver is loaded as it also shows here:

Code: Select all

mello@olinda:~$ ifconfig rename2
rename2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.10.60  netmask 255.255.255.0  broadcast 10.10.10.255
        inet6 fe80::207:43ff:fe06:b659  prefixlen 64  scopeid 0x20<link>
        ether 00:07:43:06:b6:59  txqueuelen 1000  (Ethernet)
        RX packets 6333088  bytes 4122670256 (4.1 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8833892  bytes 10859459898 (10.8 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16  memory 0xf7881000-f7881fff  
When testing with iperf, using the default 1500 MTU, I get:

Code: Select all

root@bhyve:~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.152 port 5001 connected with 10.10.10.60 port 56146
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.89 GBytes  1.62 Gbits/sec
If I turn jumbo frames on, through put goes down to a craw:

Code: Select all

mello@olinda:~$ sudo ifconfig rename2 mtu 9000
mello@olinda:~$ iperf -c bhyve
------------------------------------------------------------
Client connecting to bhyve, TCP port 5001
TCP window size:  325 KByte (default)
------------------------------------------------------------
[  3] local 10.10.10.60 port 56712 connected with 10.10.10.152 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.2 sec   326 KBytes   261 Kbits/sec
Interesting that iperf to/from the Mint box is good, but not the same 95.4Gb I get between the servers:

Code: Select all

mello@olinda:~$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.60 port 5001 connected with 10.10.10.60 port 46248
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  28.1 GBytes  24.1 Gbits/sec
I'll free-up a Mellanox MNPA19-XTR 10G SFP Optical Network Card from the ESXi in a couple weeks, but if anyone has any experience with those T320 that would like to share, so I won't toss the card in my recycle bin, I'd appreciate.

Thanks,
Al

melloa
Level 1
Level 1
Posts: 18
Joined: Mon Apr 04, 2016 10:26 pm

Re: 10Gb network

Post by melloa » Mon Oct 22, 2018 12:21 am

I guess that no one has used Chelsio on Mint before, as I didn't see any replies.

Today I started playing with the Mellanox MT26448. Good thing is that fresh install found the card from the get go, but speeds are still slow. ~1Gb/s tx and ~980Kb/s rx.

More to come.

User avatar
catweazel
Level 17
Level 17
Posts: 7756
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: 10Gb network

Post by catweazel » Mon Oct 22, 2018 12:36 am

melloa wrote:
Mon Oct 22, 2018 12:21 am
speeds are still slow. ~1Gb/s tx and ~980Kb/s rx.
Please post the output of ethtool <name-of-interface>.
¡uʍop ǝpısdn sı buıɥʇʎɹǝʌǝ os ɐıןɐɹʇsnɐ ɯoɹɟ ɯ,ı

Post Reply

Return to “Newbie Questions”