Quantcast

1gbit LAN/NIC performance, queue speed bug?

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

1gbit LAN/NIC performance, queue speed bug?

Robert Lewandowski
Hello,

PROBLEM: transfer speed is ONLY HALF if queue is defined in pf.conf
although queue is 950Mbit (1000Mbit-5%)
pf disabled: 768 Mbits/sec
pf enabled, queue 950Mbit: 337 Mbits/sec

ANALYSIS:

- OpenBSD 4.8 default intallation.
- Test made between OpenBSD 4.8 and Debian Linux.
(between two Debian systems speed is more than 900Mbit/s)

*********************************************************
LAN interface: Intel PRO/1000 PT Desktop Adapter (PCIe, model:
EXPI9300PTBLK)
DMESG: em0 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82572EI)" rev
0x06: apic 1 int 16 (irq 5), address 00:1b:21:05:1f:39
*********************************************************
Default settings of TCP window size:
net.inet.tcp.recvspace=16384
net.inet.tcp.sendspace=16384
*********************************************************

1a) pf disabled

root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
------------------------------------------------------------
Client connecting to 10.0.0.6, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.8 port 27600 connected with 10.0.0.6 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  54.7 MBytes    459 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  1.0- 2.0 sec  54.7 MBytes    458 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  2.0- 3.0 sec  54.7 MBytes    459 Mbits/sec

1b) pf enabled, no queue

root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
------------------------------------------------------------
Client connecting to 10.0.0.6, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.8 port 46912 connected with 10.0.0.6 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  53.9 MBytes    452 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  1.0- 2.0 sec  52.6 MBytes    441 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  2.0- 3.0 sec  54.1 MBytes    454 Mbits/sec

1c) pf enabled, added queue to default pf.conf:

altq on em0 cbq bandwidth 1Gb queue { q_lan }
queue q_lan bandwidth 950Mb cbq (default)

root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
------------------------------------------------------------
Client connecting to 10.0.0.6, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.8 port 38266 connected with 10.0.0.6 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  33.9 MBytes    284 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  1.0- 2.0 sec  35.0 MBytes    294 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  2.0- 3.0 sec  35.8 MBytes    300 Mbits/sec


*********************************************************
TCP window size changed to 131072.
net.inet.tcp.recvspace: 16384 -> 131072
net.inet.tcp.sendspace: 16384 -> 131072
*********************************************************

1a) pf disabled

root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
------------------------------------------------------------
Client connecting to 10.0.0.6, TCP port 5001
TCP window size:   128 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.8 port 32680 connected with 10.0.0.6 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  91.5 MBytes    768 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  1.0- 2.0 sec  92.1 MBytes    773 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  2.0- 3.0 sec  91.2 MBytes    765 Mbits/sec

1b) pf enabled, no queue

root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
------------------------------------------------------------
Client connecting to 10.0.0.6, TCP port 5001
TCP window size:   128 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.8 port 41092 connected with 10.0.0.6 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  80.5 MBytes    675 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  1.0- 2.0 sec  80.1 MBytes    672 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  2.0- 3.0 sec  80.2 MBytes    673 Mbits/sec

1c) pf enabled, added queue to default pf.conf:

altq on em0 cbq bandwidth 1Gb queue { q_lan }
queue q_lan bandwidth 950Mb cbq (default)

root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
------------------------------------------------------------
Client connecting to 10.0.0.6, TCP port 5001
TCP window size:   128 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.8 port 12499 connected with 10.0.0.6 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  40.1 MBytes    337 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  1.0- 2.0 sec  40.1 MBytes    336 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  2.0- 3.0 sec  40.0 MBytes    335 Mbits/sec

*********************************************************

any ideas, suggestions about this situation?

--

best regards,
Robert Lewandowski

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

Daniel Gracia
dmesg missing!

Your computer horsepower will definitely affect the maximum bandwith pf
will be able to manage.

El 16/11/2010 12:52, Robert Lewandowski escribiC3:

> Hello,
>
> PROBLEM: transfer speed is ONLY HALF if queue is defined in pf.conf
> although queue is 950Mbit (1000Mbit-5%)
> pf disabled: 768 Mbits/sec
> pf enabled, queue 950Mbit: 337 Mbits/sec
>
> ANALYSIS:
>
> - OpenBSD 4.8 default intallation.
> - Test made between OpenBSD 4.8 and Debian Linux.
> (between two Debian systems speed is more than 900Mbit/s)
>
> *********************************************************
> LAN interface: Intel PRO/1000 PT Desktop Adapter (PCIe, model:
> EXPI9300PTBLK)
> DMESG: em0 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82572EI)" rev
> 0x06: apic 1 int 16 (irq 5), address 00:1b:21:05:1f:39
> *********************************************************
> Default settings of TCP window size:
> net.inet.tcp.recvspace=16384
> net.inet.tcp.sendspace=16384
> *********************************************************
>
> 1a) pf disabled
>
> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 3] local 10.0.0.8 port 27600 connected with 10.0.0.6 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0- 1.0 sec 54.7 MBytes 459 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 1.0- 2.0 sec 54.7 MBytes 458 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 2.0- 3.0 sec 54.7 MBytes 459 Mbits/sec
>
> 1b) pf enabled, no queue
>
> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 3] local 10.0.0.8 port 46912 connected with 10.0.0.6 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0- 1.0 sec 53.9 MBytes 452 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 1.0- 2.0 sec 52.6 MBytes 441 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 2.0- 3.0 sec 54.1 MBytes 454 Mbits/sec
>
> 1c) pf enabled, added queue to default pf.conf:
>
> altq on em0 cbq bandwidth 1Gb queue { q_lan }
> queue q_lan bandwidth 950Mb cbq (default)
>
> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 3] local 10.0.0.8 port 38266 connected with 10.0.0.6 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0- 1.0 sec 33.9 MBytes 284 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 1.0- 2.0 sec 35.0 MBytes 294 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 2.0- 3.0 sec 35.8 MBytes 300 Mbits/sec
>
>
> *********************************************************
> TCP window size changed to 131072.
> net.inet.tcp.recvspace: 16384 -> 131072
> net.inet.tcp.sendspace: 16384 -> 131072
> *********************************************************
>
> 1a) pf disabled
>
> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size: 128 KByte (default)
> ------------------------------------------------------------
> [ 3] local 10.0.0.8 port 32680 connected with 10.0.0.6 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0- 1.0 sec 91.5 MBytes 768 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 1.0- 2.0 sec 92.1 MBytes 773 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 2.0- 3.0 sec 91.2 MBytes 765 Mbits/sec
>
> 1b) pf enabled, no queue
>
> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size: 128 KByte (default)
> ------------------------------------------------------------
> [ 3] local 10.0.0.8 port 41092 connected with 10.0.0.6 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0- 1.0 sec 80.5 MBytes 675 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 1.0- 2.0 sec 80.1 MBytes 672 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 2.0- 3.0 sec 80.2 MBytes 673 Mbits/sec
>
> 1c) pf enabled, added queue to default pf.conf:
>
> altq on em0 cbq bandwidth 1Gb queue { q_lan }
> queue q_lan bandwidth 950Mb cbq (default)
>
> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size: 128 KByte (default)
> ------------------------------------------------------------
> [ 3] local 10.0.0.8 port 12499 connected with 10.0.0.6 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0- 1.0 sec 40.1 MBytes 337 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 1.0- 2.0 sec 40.1 MBytes 336 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 2.0- 3.0 sec 40.0 MBytes 335 Mbits/sec
>
> *********************************************************
>
> any ideas, suggestions about this situation?

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

Joel Sing-3
In reply to this post by Robert Lewandowski
On Tuesday 16 November 2010, Robert Lewandowski wrote:

> Hello,
>
> PROBLEM: transfer speed is ONLY HALF if queue is defined in pf.conf
> although queue is 950Mbit (1000Mbit-5%)
> pf disabled: 768 Mbits/sec
> pf enabled, queue 950Mbit: 337 Mbits/sec
>
> ANALYSIS:
>
> - OpenBSD 4.8 default intallation.
> - Test made between OpenBSD 4.8 and Debian Linux.
> (between two Debian systems speed is more than 900Mbit/s)
>
> *********************************************************
> LAN interface: Intel PRO/1000 PT Desktop Adapter (PCIe, model:
> EXPI9300PTBLK)
> DMESG: em0 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82572EI)" rev
> 0x06: apic 1 int 16 (irq 5), address 00:1b:21:05:1f:39
> *********************************************************
> Default settings of TCP window size:
> net.inet.tcp.recvspace=16384
> net.inet.tcp.sendspace=16384
> *********************************************************
>
> 1a) pf disabled
>
> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  3] local 10.0.0.8 port 27600 connected with 10.0.0.6 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0- 1.0 sec  54.7 MBytes    459 Mbits/sec
> [ ID] Interval       Transfer     Bandwidth
> [  3]  1.0- 2.0 sec  54.7 MBytes    458 Mbits/sec
> [ ID] Interval       Transfer     Bandwidth
> [  3]  2.0- 3.0 sec  54.7 MBytes    459 Mbits/sec
>
> 1b) pf enabled, no queue
>
> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  3] local 10.0.0.8 port 46912 connected with 10.0.0.6 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0- 1.0 sec  53.9 MBytes    452 Mbits/sec
> [ ID] Interval       Transfer     Bandwidth
> [  3]  1.0- 2.0 sec  52.6 MBytes    441 Mbits/sec
> [ ID] Interval       Transfer     Bandwidth
> [  3]  2.0- 3.0 sec  54.1 MBytes    454 Mbits/sec
>
> 1c) pf enabled, added queue to default pf.conf:
>
> altq on em0 cbq bandwidth 1Gb queue { q_lan }
> queue q_lan bandwidth 950Mb cbq (default)
>
> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  3] local 10.0.0.8 port 38266 connected with 10.0.0.6 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0- 1.0 sec  33.9 MBytes    284 Mbits/sec
> [ ID] Interval       Transfer     Bandwidth
> [  3]  1.0- 2.0 sec  35.0 MBytes    294 Mbits/sec
> [ ID] Interval       Transfer     Bandwidth
> [  3]  2.0- 3.0 sec  35.8 MBytes    300 Mbits/sec
>
>
> *********************************************************
> TCP window size changed to 131072.
> net.inet.tcp.recvspace: 16384 -> 131072
> net.inet.tcp.sendspace: 16384 -> 131072
> *********************************************************
>
> 1a) pf disabled
>
> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size:   128 KByte (default)
> ------------------------------------------------------------
> [  3] local 10.0.0.8 port 32680 connected with 10.0.0.6 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0- 1.0 sec  91.5 MBytes    768 Mbits/sec
> [ ID] Interval       Transfer     Bandwidth
> [  3]  1.0- 2.0 sec  92.1 MBytes    773 Mbits/sec
> [ ID] Interval       Transfer     Bandwidth
> [  3]  2.0- 3.0 sec  91.2 MBytes    765 Mbits/sec
>
> 1b) pf enabled, no queue
>
> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size:   128 KByte (default)
> ------------------------------------------------------------
> [  3] local 10.0.0.8 port 41092 connected with 10.0.0.6 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0- 1.0 sec  80.5 MBytes    675 Mbits/sec
> [ ID] Interval       Transfer     Bandwidth
> [  3]  1.0- 2.0 sec  80.1 MBytes    672 Mbits/sec
> [ ID] Interval       Transfer     Bandwidth
> [  3]  2.0- 3.0 sec  80.2 MBytes    673 Mbits/sec
>
> 1c) pf enabled, added queue to default pf.conf:
>
> altq on em0 cbq bandwidth 1Gb queue { q_lan }
> queue q_lan bandwidth 950Mb cbq (default)
>
> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size:   128 KByte (default)
> ------------------------------------------------------------
> [  3] local 10.0.0.8 port 12499 connected with 10.0.0.6 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0- 1.0 sec  40.1 MBytes    337 Mbits/sec
> [ ID] Interval       Transfer     Bandwidth
> [  3]  1.0- 2.0 sec  40.1 MBytes    336 Mbits/sec
> [ ID] Interval       Transfer     Bandwidth
> [  3]  2.0- 3.0 sec  40.0 MBytes    335 Mbits/sec
>
> *********************************************************
>
> any ideas, suggestions about this situation?

The default length for a queue is 50 packets - this only allows you to queue
around 75,000 bytes and the burstiness of TCP slow-start is likely to well
exceed this in your configuration (due to the BDP). I'd suggest increasing
the queue length - also run 'pfctl -vvs queue' or 'systat queue' and see
what's happening with regards to packets drops.
--

   "Stop assuming that systems are secure unless demonstrated insecure;
    start assuming that systems are insecure unless designed securely."
          - Bruce Schneier

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

RLW-2
W dniu 2010-11-16 16:14, Joel Sing pisze:

>
> The default length for a queue is 50 packets - this only allows you to queue
> around 75,000 bytes and the burstiness of TCP slow-start is likely to well
> exceed this in your configuration (due to the BDP). I'd suggest increasing
> the queue length - also run 'pfctl -vvs queue' or 'systat queue' and see
> what's happening with regards to packets drops.


root@router-test (/root)# systat queue

QUEUE                             BW SCH  PRIO     PKTS    BYTES
DROP_P   DROP_B QLEN BORROW SUSPEN     P/S     B/S
root_em0                       1000M cbq     0  1947967 2879364K
0        0    0      0      0   29412  44525K
  q_lan                          950M cbq        1947967 2879364K
  0        0    0      0      0   29412  44525K



root@router-test (/root)# pfctl -vvs queue

queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {q_lan}
   [ pkts:    4793481  bytes: 7256036778  dropped pkts:      0 bytes:
    0 ]
   [ qlength:   0/ 50  borrows:      0  suspends:      0 ]
   [ measured: 29385.4 packets/s, 355.86Mb/s ]
queue  q_lan on em0 bandwidth 950Mb cbq( default )
   [ pkts:    4793481  bytes: 7256036778  dropped pkts:      0 bytes:
    0 ]
   [ qlength:   0/ 50  borrows:      0  suspends:      0 ]
   [ measured: 29385.4 packets/s, 355.86Mb/s ]


best regards,
Robert Lewandowski

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

RLW-2
W dniu 2010-11-16 17:04, RLW pisze:

> W dniu 2010-11-16 16:14, Joel Sing pisze:
>
>>
>> The default length for a queue is 50 packets - this only allows you to
>> queue
>> around 75,000 bytes and the burstiness of TCP slow-start is likely to
>> well
>> exceed this in your configuration (due to the BDP). I'd suggest
>> increasing
>> the queue length - also run 'pfctl -vvs queue' or 'systat queue' and see
>> what's happening with regards to packets drops.
>
>
> root@router-test (/root)# systat queue
>
> QUEUE BW SCH PRIO PKTS BYTES DROP_P DROP_B QLEN BORROW SUSPEN P/S B/S
> root_em0 1000M cbq 0 1947967 2879364K 0 0 0 0 0 29412 44525K
> q_lan 950M cbq 1947967 2879364K 0 0 0 0 0 29412 44525K
>
>
>
> root@router-test (/root)# pfctl -vvs queue
>
> queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {q_lan}
> [ pkts: 4793481 bytes: 7256036778 dropped pkts: 0 bytes: 0 ]
> [ qlength: 0/ 50 borrows: 0 suspends: 0 ]
> [ measured: 29385.4 packets/s, 355.86Mb/s ]
> queue q_lan on em0 bandwidth 950Mb cbq( default )
> [ pkts: 4793481 bytes: 7256036778 dropped pkts: 0 bytes: 0 ]
> [ qlength: 0/ 50 borrows: 0 suspends: 0 ]
> [ measured: 29385.4 packets/s, 355.86Mb/s ]
>
>
> best regards,
> Robert Lewandowski
>
>

If I am reading it wright, no packets are droped.

Changing values like:
kern.somaxconn
net.inet.ip.maxqueue
net.bpf.bufsize
net.bpf.maxbufsize
net.inet.ipcomp.enable
net.inet.tcp.ackonpush
net.inet.tcp.ecn

does not help either. It only has some influence on network speed with
PF disabled. With PF enabled speed is alwasy around 350mbit/s :((

So any new ideas about debuging the problem or possible solution?


best regards,
Robert Lewandowski

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

RLW-2
In reply to this post by Joel Sing-3
W dniu 2010-11-16 16:14, Joel Sing pisze:

> On Tuesday 16 November 2010, Robert Lewandowski wrote:
>> Hello,
>>
>> PROBLEM: transfer speed is ONLY HALF if queue is defined in pf.conf
>> although queue is 950Mbit (1000Mbit-5%)
>> pf disabled: 768 Mbits/sec
>> pf enabled, queue 950Mbit: 337 Mbits/sec
>>
>> ANALYSIS:
>>
>> - OpenBSD 4.8 default intallation.
>> - Test made between OpenBSD 4.8 and Debian Linux.
>> (between two Debian systems speed is more than 900Mbit/s)
>>
>> *********************************************************
>> LAN interface: Intel PRO/1000 PT Desktop Adapter (PCIe, model:
>> EXPI9300PTBLK)
>> DMESG: em0 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82572EI)" rev
>> 0x06: apic 1 int 16 (irq 5), address 00:1b:21:05:1f:39
>> *********************************************************
>> Default settings of TCP window size:
>> net.inet.tcp.recvspace=16384
>> net.inet.tcp.sendspace=16384
>> *********************************************************
>>
>> 1a) pf disabled
>>
>> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
>> ------------------------------------------------------------
>> Client connecting to 10.0.0.6, TCP port 5001
>> TCP window size: 16.0 KByte (default)
>> ------------------------------------------------------------
>> [  3] local 10.0.0.8 port 27600 connected with 10.0.0.6 port 5001
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  0.0- 1.0 sec  54.7 MBytes    459 Mbits/sec
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  1.0- 2.0 sec  54.7 MBytes    458 Mbits/sec
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  2.0- 3.0 sec  54.7 MBytes    459 Mbits/sec
>>
>> 1b) pf enabled, no queue
>>
>> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
>> ------------------------------------------------------------
>> Client connecting to 10.0.0.6, TCP port 5001
>> TCP window size: 16.0 KByte (default)
>> ------------------------------------------------------------
>> [  3] local 10.0.0.8 port 46912 connected with 10.0.0.6 port 5001
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  0.0- 1.0 sec  53.9 MBytes    452 Mbits/sec
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  1.0- 2.0 sec  52.6 MBytes    441 Mbits/sec
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  2.0- 3.0 sec  54.1 MBytes    454 Mbits/sec
>>
>> 1c) pf enabled, added queue to default pf.conf:
>>
>> altq on em0 cbq bandwidth 1Gb queue { q_lan }
>> queue q_lan bandwidth 950Mb cbq (default)
>>
>> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
>> ------------------------------------------------------------
>> Client connecting to 10.0.0.6, TCP port 5001
>> TCP window size: 16.0 KByte (default)
>> ------------------------------------------------------------
>> [  3] local 10.0.0.8 port 38266 connected with 10.0.0.6 port 5001
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  0.0- 1.0 sec  33.9 MBytes    284 Mbits/sec
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  1.0- 2.0 sec  35.0 MBytes    294 Mbits/sec
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  2.0- 3.0 sec  35.8 MBytes    300 Mbits/sec
>>
>>
>> *********************************************************
>> TCP window size changed to 131072.
>> net.inet.tcp.recvspace: 16384 ->  131072
>> net.inet.tcp.sendspace: 16384 ->  131072
>> *********************************************************
>>
>> 1a) pf disabled
>>
>> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
>> ------------------------------------------------------------
>> Client connecting to 10.0.0.6, TCP port 5001
>> TCP window size:   128 KByte (default)
>> ------------------------------------------------------------
>> [  3] local 10.0.0.8 port 32680 connected with 10.0.0.6 port 5001
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  0.0- 1.0 sec  91.5 MBytes    768 Mbits/sec
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  1.0- 2.0 sec  92.1 MBytes    773 Mbits/sec
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  2.0- 3.0 sec  91.2 MBytes    765 Mbits/sec
>>
>> 1b) pf enabled, no queue
>>
>> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
>> ------------------------------------------------------------
>> Client connecting to 10.0.0.6, TCP port 5001
>> TCP window size:   128 KByte (default)
>> ------------------------------------------------------------
>> [  3] local 10.0.0.8 port 41092 connected with 10.0.0.6 port 5001
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  0.0- 1.0 sec  80.5 MBytes    675 Mbits/sec
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  1.0- 2.0 sec  80.1 MBytes    672 Mbits/sec
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  2.0- 3.0 sec  80.2 MBytes    673 Mbits/sec
>>
>> 1c) pf enabled, added queue to default pf.conf:
>>
>> altq on em0 cbq bandwidth 1Gb queue { q_lan }
>> queue q_lan bandwidth 950Mb cbq (default)
>>
>> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
>> ------------------------------------------------------------
>> Client connecting to 10.0.0.6, TCP port 5001
>> TCP window size:   128 KByte (default)
>> ------------------------------------------------------------
>> [  3] local 10.0.0.8 port 12499 connected with 10.0.0.6 port 5001
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  0.0- 1.0 sec  40.1 MBytes    337 Mbits/sec
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  1.0- 2.0 sec  40.1 MBytes    336 Mbits/sec
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  2.0- 3.0 sec  40.0 MBytes    335 Mbits/sec
>>
>> *********************************************************
>>
>> any ideas, suggestions about this situation?
>
> The default length for a queue is 50 packets - this only allows you to queue
> around 75,000 bytes and the burstiness of TCP slow-start is likely to well
> exceed this in your configuration (due to the BDP). I'd suggest increasing
> the queue length - also run 'pfctl -vvs queue' or 'systat queue' and see
> what's happening with regards to packets drops.

ok, a set qlimit to 200 and then 500, no difference

queue root_em0 on em0 bandwidth 1Gb priority 0 qlimit 500 cbq( wrr root
) {q_lan}
   [ pkts:     858820  bytes: 1300055838  dropped pkts:      0 bytes:
    0 ]
   [ qlength:   0/500  borrows:      0  suspends:      0 ]
   [ measured: 29222.9 packets/s, 353.90Mb/s ]
queue  q_lan on em0 bandwidth 950Mb qlimit 500 cbq( borrow default )
   [ pkts:     858820  bytes: 1300055838  dropped pkts:      0 bytes:
    0 ]
   [ qlength:   0/500  borrows:      0  suspends:      0 ]
   [ measured: 29222.9 packets/s, 353.90Mb/s ]


best regards,
RLW

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

James Records
What does CPU usage look like when this is happening?  is there any other
resources that appear to be constrained?

J

On Wed, Nov 17, 2010 at 3:21 AM, RLW <[hidden email]> wrote:

> W dniu 2010-11-16 16:14, Joel Sing pisze:
>
>> On Tuesday 16 November 2010, Robert Lewandowski wrote:
>>
>>  Hello,
>>>
>>> PROBLEM: transfer speed is ONLY HALF if queue is defined in pf.conf
>>> although queue is 950Mbit (1000Mbit-5%)
>>> pf disabled: 768 Mbits/sec
>>> pf enabled, queue 950Mbit: 337 Mbits/sec
>>>
>>> ANALYSIS:
>>>
>>> - OpenBSD 4.8 default intallation.
>>> - Test made between OpenBSD 4.8 and Debian Linux.
>>> (between two Debian systems speed is more than 900Mbit/s)
>>>
>>> *********************************************************
>>> LAN interface: Intel PRO/1000 PT Desktop Adapter (PCIe, model:
>>> EXPI9300PTBLK)
>>> DMESG: em0 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82572EI)" rev
>>> 0x06: apic 1 int 16 (irq 5), address 00:1b:21:05:1f:39
>>> *********************************************************
>>> Default settings of TCP window size:
>>> net.inet.tcp.recvspace=16384
>>> net.inet.tcp.sendspace=16384
>>> *********************************************************
>>>
>>> 1a) pf disabled
>>>
>>> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
>>> ------------------------------------------------------------
>>> Client connecting to 10.0.0.6, TCP port 5001
>>> TCP window size: 16.0 KByte (default)
>>> ------------------------------------------------------------
>>> [  3] local 10.0.0.8 port 27600 connected with 10.0.0.6 port 5001
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  0.0- 1.0 sec  54.7 MBytes    459 Mbits/sec
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  1.0- 2.0 sec  54.7 MBytes    458 Mbits/sec
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  2.0- 3.0 sec  54.7 MBytes    459 Mbits/sec
>>>
>>> 1b) pf enabled, no queue
>>>
>>> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
>>> ------------------------------------------------------------
>>> Client connecting to 10.0.0.6, TCP port 5001
>>> TCP window size: 16.0 KByte (default)
>>> ------------------------------------------------------------
>>> [  3] local 10.0.0.8 port 46912 connected with 10.0.0.6 port 5001
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  0.0- 1.0 sec  53.9 MBytes    452 Mbits/sec
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  1.0- 2.0 sec  52.6 MBytes    441 Mbits/sec
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  2.0- 3.0 sec  54.1 MBytes    454 Mbits/sec
>>>
>>> 1c) pf enabled, added queue to default pf.conf:
>>>
>>> altq on em0 cbq bandwidth 1Gb queue { q_lan }
>>> queue q_lan bandwidth 950Mb cbq (default)
>>>
>>> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
>>> ------------------------------------------------------------
>>> Client connecting to 10.0.0.6, TCP port 5001
>>> TCP window size: 16.0 KByte (default)
>>> ------------------------------------------------------------
>>> [  3] local 10.0.0.8 port 38266 connected with 10.0.0.6 port 5001
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  0.0- 1.0 sec  33.9 MBytes    284 Mbits/sec
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  1.0- 2.0 sec  35.0 MBytes    294 Mbits/sec
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  2.0- 3.0 sec  35.8 MBytes    300 Mbits/sec
>>>
>>>
>>> *********************************************************
>>> TCP window size changed to 131072.
>>> net.inet.tcp.recvspace: 16384 ->  131072
>>> net.inet.tcp.sendspace: 16384 ->  131072
>>> *********************************************************
>>>
>>> 1a) pf disabled
>>>
>>> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
>>> ------------------------------------------------------------
>>> Client connecting to 10.0.0.6, TCP port 5001
>>> TCP window size:   128 KByte (default)
>>> ------------------------------------------------------------
>>> [  3] local 10.0.0.8 port 32680 connected with 10.0.0.6 port 5001
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  0.0- 1.0 sec  91.5 MBytes    768 Mbits/sec
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  1.0- 2.0 sec  92.1 MBytes    773 Mbits/sec
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  2.0- 3.0 sec  91.2 MBytes    765 Mbits/sec
>>>
>>> 1b) pf enabled, no queue
>>>
>>> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
>>> ------------------------------------------------------------
>>> Client connecting to 10.0.0.6, TCP port 5001
>>> TCP window size:   128 KByte (default)
>>> ------------------------------------------------------------
>>> [  3] local 10.0.0.8 port 41092 connected with 10.0.0.6 port 5001
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  0.0- 1.0 sec  80.5 MBytes    675 Mbits/sec
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  1.0- 2.0 sec  80.1 MBytes    672 Mbits/sec
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  2.0- 3.0 sec  80.2 MBytes    673 Mbits/sec
>>>
>>> 1c) pf enabled, added queue to default pf.conf:
>>>
>>> altq on em0 cbq bandwidth 1Gb queue { q_lan }
>>> queue q_lan bandwidth 950Mb cbq (default)
>>>
>>> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
>>> ------------------------------------------------------------
>>> Client connecting to 10.0.0.6, TCP port 5001
>>> TCP window size:   128 KByte (default)
>>> ------------------------------------------------------------
>>> [  3] local 10.0.0.8 port 12499 connected with 10.0.0.6 port 5001
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  0.0- 1.0 sec  40.1 MBytes    337 Mbits/sec
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  1.0- 2.0 sec  40.1 MBytes    336 Mbits/sec
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  2.0- 3.0 sec  40.0 MBytes    335 Mbits/sec
>>>
>>> *********************************************************
>>>
>>> any ideas, suggestions about this situation?
>>>
>>
>> The default length for a queue is 50 packets - this only allows you to
>> queue
>> around 75,000 bytes and the burstiness of TCP slow-start is likely to well
>> exceed this in your configuration (due to the BDP). I'd suggest increasing
>> the queue length - also run 'pfctl -vvs queue' or 'systat queue' and see
>> what's happening with regards to packets drops.
>>
>
> ok, a set qlimit to 200 and then 500, no difference
>
> queue root_em0 on em0 bandwidth 1Gb priority 0 qlimit 500 cbq( wrr root )
> {q_lan}
>  [ pkts:     858820  bytes: 1300055838  dropped pkts:      0 bytes:    0 ]
>  [ qlength:   0/500  borrows:      0  suspends:      0 ]
>  [ measured: 29222.9 packets/s, 353.90Mb/s ]
> queue  q_lan on em0 bandwidth 950Mb qlimit 500 cbq( borrow default )
>  [ pkts:     858820  bytes: 1300055838  dropped pkts:      0 bytes:    0 ]
>  [ qlength:   0/500  borrows:      0  suspends:      0 ]
>  [ measured: 29222.9 packets/s, 353.90Mb/s ]
>
>
> best regards,
> RLW

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

RLW-2
Thanks for all the answers, but the problem still exists.

To sum up:

OpenBSD 4.8 default install

cpu0: Intel(R) Celeron(R) CPU 2.80GHz ("GenuineIntel" 686-class) 2.80 GHz
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
em0 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82572EI)" rev 0x06:
apic 1 int 16 (irq 5), address XX:XX:XX:XX:XX:XX

1.
- tcp send and receive window @ default
- pf disabled
- transfer speed on em0 tested by iperf: 458 Mbits/sec
- top shows:

load averages:  0.90,  0.42,  0.18
25 processes:  1 running, 23 idle, 1 on processor
CPU states:  0.6% user,  0.0% nice, 51.5% system, 33.5% interrupt, 14.4%
idle
Memory: Real: 9164K/43M act/tot  Free: 442M  Swap: 0K/759M used/tot


2.
- tcp send and receive window: 131072
- pf disabled
- transfer speed on em0 tested by iperf: 767 Mbits/sec
- top shows:

load averages:  0.85,  0.56,  0.29
25 processes:  1 running, 23 idle, 1 on processor
CPU states:  1.6% user,  0.0% nice, 71.5% system, 26.9% interrupt,  0.0%
idle
Memory: Real: 9172K/43M act/tot  Free: 442M  Swap: 0K/759M used/tot


3.
- tcp send and receive window: 131072
- pf enabled, no queue
- transfer speed on em0 tested by iperf: 677 Mbits/sec
- top shows:

load averages:  0.84,  0.58,  0.38
25 processes:  1 running, 23 idle, 1 on processor
CPU states:  1.4% user,  0.0% nice, 70.1% system, 28.5% interrupt,  0.0%
idle
Memory: Real: 9184K/44M act/tot  Free: 441M  Swap: 0K/759M used/tot


4.
- tcp send and receive window: 131072
- pf enabled
- to default pf.conf added (as Joel Sing suggested qlimit changed from
50 to 500):
altq on em0 cbq bandwidth 1Gb qlimit 500 queue { q_lan }
queue q_lan bandwidth 950Mb qlimit 500 cbq (default)
- transfer speed on em0 tested by iperf: 337 Mbits/sec
- top shows:

load averages:  0.94,  0.68,  0.45
25 processes:  1 running, 23 idle, 1 on processor
CPU states:  0.6% user,  0.0% nice, 79.4% system, 20.0% interrupt,  0.0%
idle
Memory: Real: 9184K/44M act/tot  Free: 441M  Swap: 0K/759M used/tot


root@router-test (/root)# systat queue
2 users    Load 0.69 0.58 0.48                      Wed Nov 17 21:34:38 2010

QUEUE               BW SCH  PR  PKTS BYTES DROP_P DROP_B QLEN BORR SUSP
P/S  B/S
root_em0         1000M cbq   0 1803K 2666M      0      0    0    0    0
293  44M
  q_lan            950M cbq     1803K 2666M      0      0    0    0    0
293  44M


root@router-test (/root)# pfctl -vvs queue (transfer starts slowly, then
at best time transfer is 322.11Mb/s than drops - i dont know is it a
proper behaviour)

queue root_em0 on em0 bandwidth 1Gb priority 0 qlimit 500 cbq( wrr root
) {q_lan}
   [ pkts:    4372637  bytes: 6619141152  dropped pkts:      0 bytes:
    0 ]
   [ qlength:   0/500  borrows:      0  suspends:      0 ]
   [ measured: 26598.4 packets/s, 322.11Mb/s ]
queue  q_lan on em0 bandwidth 950Mb qlimit 500 cbq( default )
   [ pkts:    4372637  bytes: 6619141152  dropped pkts:      0 bytes:
    0 ]
   [ qlength:   0/500  borrows:      0  suspends:      0 ]
   [ measured: 26598.4 packets/s, 322.11Mb/s ]



best regards,
RLW

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

RLW-2
W dniu 2010-11-17 22:11, Kenneth Gober pisze:

> this is very illuminating.  it says that initially, tcp send/receive
> window is your bottleneck.  when you increase them to 131072, the cpu
> becomes your bottleneck and remains that way no matter what else you
> tweak (idle drops to 0.0% and stays there).
>
> the next question is, of course, why is your cpu spending all its time
> in "system" and "interrupt"?  is it doing unnecessary work, or is the
> work necessary and your cpu just isn't fast enough?  I don't have ready
> answers to these questions.
>
> -ken
>

Hello,

I see, that while I am testing network speed by iperf, 100% CPU is being
used, but is that normal for default install of OpenBSD 4.8 with default
pf.conf??

I have second computer exactly like that one (IBM ThinkCentre A51P), on
which i am running this tests but with P4 3Ghz CPU 2mb cache (not
celeron 2.8) and the same is happening (100% CPU).

LAN interface is Intel PRO/1000 PT Desktop Adapter (PCIe, model:
EXPI9300PTBLK) and this is the only pcie adapter in computer (maybe
broadcom integrated nic is also pcie but is not used)

So the conclusion might be:
- there is problem with my Intel NIC model/cheapset
- there is problem with em driver
- there is problem with my hardware (I need serwer motherboard with pcie
and pci 64bit 66mhz)
- I need faster CPU than P4 3GHz ??


------------
best regards,
RLW

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

RLW-2
W dniu 2010-11-18 06:27, Tomas Bodzar pisze:
>
> How about output of 'vmstat -i' and "screen" of default systat (systat vmstat)
>

root@router-test (/root)# vmstat -i
interrupt                       total     rate
irq0/clock                    4836013      100
irq83/em0                     2771337       57
irq83/bge0                          1        0
irq81/pciide0                    4790        0
irq85/ichiic0                  172044        3
Total                         7784185      160


http://erydium.pl/upload/vmstat.gif

http://erydium.pl/upload/systat.gif


----
best regards,
RLW

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

Schöberle Dániel-6
In reply to this post by RLW-2
> > this is very illuminating.  it says that initially, tcp send/receive
> > window is your bottleneck.  when you increase them to 131072, the cpu
> > becomes your bottleneck and remains that way no matter what else you
> > tweak (idle drops to 0.0% and stays there).
> >
> > the next question is, of course, why is your cpu spending all its time
> > in "system" and "interrupt"?  is it doing unnecessary work, or is the
> > work necessary and your cpu just isn't fast enough?  I don't have ready
> > answers to these questions.
> >
> > -ken
> >
>
> Hello,
>
> I see, that while I am testing network speed by iperf, 100% CPU is being
> used, but is that normal for default install of OpenBSD 4.8 with default
> pf.conf??
>
> I have second computer exactly like that one (IBM ThinkCentre A51P), on
> which i am running this tests but with P4 3Ghz CPU 2mb cache (not
> celeron 2.8) and the same is happening (100% CPU).
>
> LAN interface is Intel PRO/1000 PT Desktop Adapter (PCIe, model:
> EXPI9300PTBLK) and this is the only pcie adapter in computer (maybe
> broadcom integrated nic is also pcie but is not used)
>
> So the conclusion might be:
> - there is problem with my Intel NIC model/cheapset
> - there is problem with em driver
> - there is problem with my hardware (I need serwer motherboard with pcie
> and pci 64bit 66mhz)
> - I need faster CPU than P4 3GHz ??

  or

- there is a problem with iperf  :)

How about measuring with something else? Did you try tcpbench? Or something
even simpler, like scp-ing from /dev/null to /dev/null? With pf and queues
enabled you can monitor the B/S rate.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

RLW-2
W dniu 2010-11-18 13:42, Schvberle Daniel pisze:

>> So the conclusion might be:
>> - there is problem with my Intel NIC model/cheapset
>> - there is problem with em driver
>> - there is problem with my hardware (I need serwer motherboard with pcie
>> and pci 64bit 66mhz)
>> - I need faster CPU than P4 3GHz ??
>
>    or
>
> - there is a problem with iperf  :)
>
> How about measuring with something else? Did you try tcpbench? Or something
> even simpler, like scp-ing from /dev/null to /dev/null? With pf and queues
> enabled you can monitor the B/S rate.
>
>

there is no tcpbench in packages for 4.8 and for debian linux

1.
transferring file by scp from router-test to linux machine:
transfer speed: 16.1MB/s ~ 128.8 Mbits/s

root@router-test (/root)# top

load averages:  1.59,  1.05,  0.67
28 processes:  2 running, 25 idle, 1 on processor
CPU states: 33.3% user,  0.0% nice, 47.9% system, 14.2% interrupt,  4.6%
idle
Memory: Real: 11M/80M act/tot  Free: 405M  Swap: 0K/759M used/tot


root@router-test (/root)# systat queue

2 users    Load 1.49 0.99 0.74                      Thu Nov 18 16:27:25 2010

QUEUE               BW SCH  PR  PKTS BYTES DROP_P DROP_B QLEN BORR SUSP
P/S  B/S
root_em0         1000M cbq   0   10M   14G      0      0    0    0    0
122  17M
  q_lan            950M cbq       10M   14G      0      0    0    0    0
122  17M


2.
transferring file back from linux machine to router-test:
transfer speed:  19.9MB/s ~ 159.2 Mbits/s

root@router-test (/root)# top

load averages:  1.13,  0.95,  0.69
25 processes:  1 running, 23 idle, 1 on processor
CPU states: 40.1% user,  0.0% nice, 33.5% system, 26.3% interrupt,  0.0%
idle
Memory: Real: 11M/80M act/tot  Free: 405M  Swap: 0K/759M used/tot


3.
as comparison transfer speed between two debian boxes:
- tested by iperf: 940 Mbits/sec
- transfering file by scp: 42.6MB/s ~ 340.8 Mbits/s

top:

Tasks:  81 total,   1 running,  80 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.8%us, 13.8%sy,  0.0%ni, 79.0%id,  0.0%wa,  1.0%hi,  4.3%si,
0.0%st
Mem:   1028836k total,   924868k used,   103968k free,    23800k buffers
Swap:  1951856k total,    51524k used,  1900332k free,   617788k cached


----
best regards
RLW

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

David Coppa
2010/11/18 RLW <[hidden email]>:

> there is no tcpbench in packages for 4.8 and for debian linux

Because it's in base: /usr/bin/tcpbench

ciao,
david

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

RLW-2
W dniu 2010-11-18 16:52, David Coppa pisze:
> 2010/11/18 RLW<[hidden email]>:
>
>> there is no tcpbench in packages for 4.8 and for debian linux
>
> Because it's in base: /usr/bin/tcpbench
>
> ciao,
> david
>


I removed Intel NIC and run test on broadcom integrated Gbit NIC to see
if there is problem with em driver

bge0 at pci1 dev 11 function 0 "Broadcom BCM5705K" rev 0x03, BCM5705 A3
(0x3003): apic 1 int 16 (irq 5), address XX:XX:XX:XX:XX:XX
brgphy0 at bge0 phy 1: BCM5705 10/100/1000baseT PHY, rev. 2


1.
pf enabled, queue 950mbit, qlimit 500
iperf test: 410 Mbits/sec

root@router-test (/root)# top

load averages:  0.95,  0.53,  0.26
23 processes:  1 running, 21 idle, 1 on processor
CPU states:  1.2% user,  0.0% nice, 84.4% system, 14.4% interrupt,  0.0%
idle
Memory: Real: 8972K/42M act/tot  Free: 443M  Swap: 0K/759M used/tot


2. test made between two OpenBSD 4.8 boxes (there is no tcpbench for debian)


transfers by tcpbench:

Conn:   1 Mbps:      399.972 Peak Mbps:      406.093 Avg Mbps:      399.972
       133996       45932008      370.419  100.00%
Conn:   1 Mbps:      370.419 Peak Mbps:      406.093 Avg Mbps:      370.419
       134999       46833528      373.920  100.00%
Conn:   1 Mbps:      373.920 Peak Mbps:      406.093 Avg Mbps:      373.920
       136074       43531224      323.953  100.00%
Conn:   1 Mbps:      323.953 Peak Mbps:      406.093 Avg Mbps:      323.953
       137002       41013960      353.950  100.00%
Conn:   1 Mbps:      353.950 Peak Mbps:      406.093 Avg Mbps:      353.950
       137996       50500448      406.442  100.00%
Conn:   1 Mbps:      406.442 Peak Mbps:      406.442 Avg Mbps:      406.442


root@router-test (/root)# top (while running tcpbench)

load averages:  1.26,  0.80,  0.49
22 processes:  1 running, 20 idle, 1 on processor
CPU states:  0.0% user,  0.0% nice, 77.2% system, 15.6% interrupt,  7.2%
idle
Memory: Real: 8752K/43M act/tot  Free: 442M  Swap: 0K/759M used/tot


root@router-test (/root)# systat queue (while running tcpbench)

2 users    Load 0.82 0.69 0.51                      Thu Nov 18 17:13:10 2010

QUEUE               BW SCH  PR  PKTS BYTES DROP_P DROP_B QLEN BORR SUSP
P/S  B/S
root_bge0        1000M cbq   0 7300K   10G      0      0    0    0    0
314  47M
  q_lan            950M cbq     7300K   10G      0      0    0    0    0
314  47M

----

Now back on Intel NIC

1.
pf enabled, queue 950mbit, qlimit 500
iperf test: 347 Mbits/sec


2.
transfers by tcpbench:

Conn:   1 Mbps:      328.701 Peak Mbps:      336.374 Avg Mbps:      328.701
        29002       41936224      335.490  100.00%
Conn:   1 Mbps:      335.490 Peak Mbps:      336.374 Avg Mbps:      335.490
        30001       41394096      331.484  100.00%
Conn:   1 Mbps:      331.484 Peak Mbps:      336.374 Avg Mbps:      331.484
        30999       39930144      320.402  100.00%
Conn:   1 Mbps:      320.402 Peak Mbps:      336.374 Avg Mbps:      320.402
        32003       42171560      336.363  100.00%
Conn:   1 Mbps:      336.363 Peak Mbps:      336.374 Avg Mbps:      336.363
        33001       41970888      336.440  100.00%
Conn:   1 Mbps:      336.440 Peak Mbps:      336.440 Avg Mbps:      336.440
        34002       38258208      305.760  100.00%
Conn:   1 Mbps:      305.760 Peak Mbps:      336.440 Avg Mbps:      305.760


root@router-test (/root)# top (whiel runing tcpbench)

load averages:  1.20,  0.59,  0.25
24 processes:  1 running, 22 idle, 1 on processor
CPU states:  0.2% user,  0.0% nice, 75.6% system, 21.2% interrupt,  3.0%
idle
Memory: Real: 8904K/43M act/tot  Free: 442M  Swap: 0K/759M used/tot


root@router-test (/root)# systat queue  (while running tcpbench)
2 users    Load 0.57 0.54 0.28                      Thu Nov 18 17:25:26 2010

QUEUE               BW SCH  PR  PKTS BYTES DROP_P DROP_B QLEN BORR SUSP
P/S  B/S
root_em0         1000M cbq   0 2963K 4381M      0      0    0    0    0
279  42M
  q_lan            950M cbq     2963K 4381M      0      0    0    0    0
279  42M

----

so... the same machine, different NIC, different testing programs, same
behavior (~50% of defined queue speed, very high CPU usage)

the only thing i can try now is moving intel nic and hdd to other
computer with pcie slot and run test on it to see is it hardware
(motherboard) problem.


----
best regards,
RLW

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

Claudio Jeker
On Thu, Nov 18, 2010 at 05:29:41PM +0100, RLW wrote:

> W dniu 2010-11-18 16:52, David Coppa pisze:
> >2010/11/18 RLW<[hidden email]>:
> >
> >>there is no tcpbench in packages for 4.8 and for debian linux
> >
> >Because it's in base: /usr/bin/tcpbench
> >
> >ciao,
> >david
> >
>
>
> I removed Intel NIC and run test on broadcom integrated Gbit NIC to
> see if there is problem with em driver
>
> bge0 at pci1 dev 11 function 0 "Broadcom BCM5705K" rev 0x03, BCM5705
> A3 (0x3003): apic 1 int 16 (irq 5), address XX:XX:XX:XX:XX:XX
> brgphy0 at bge0 phy 1: BCM5705 10/100/1000baseT PHY, rev. 2
>
>
> 1.
> pf enabled, queue 950mbit, qlimit 500
> iperf test: 410 Mbits/sec
>
> root@router-test (/root)# top
>
> load averages:  0.95,  0.53,  0.26
> 23 processes:  1 running, 21 idle, 1 on processor
> CPU states:  1.2% user,  0.0% nice, 84.4% system, 14.4% interrupt,
> 0.0% idle
> Memory: Real: 8972K/42M act/tot  Free: 443M  Swap: 0K/759M used/tot
>
>
> 2. test made between two OpenBSD 4.8 boxes (there is no tcpbench for debian)
>
>
> transfers by tcpbench:
>
> Conn:   1 Mbps:      399.972 Peak Mbps:      406.093 Avg Mbps:      399.972
>       133996       45932008      370.419  100.00%
> Conn:   1 Mbps:      370.419 Peak Mbps:      406.093 Avg Mbps:      370.419
>       134999       46833528      373.920  100.00%
> Conn:   1 Mbps:      373.920 Peak Mbps:      406.093 Avg Mbps:      373.920
>       136074       43531224      323.953  100.00%
> Conn:   1 Mbps:      323.953 Peak Mbps:      406.093 Avg Mbps:      323.953
>       137002       41013960      353.950  100.00%
> Conn:   1 Mbps:      353.950 Peak Mbps:      406.093 Avg Mbps:      353.950
>       137996       50500448      406.442  100.00%
> Conn:   1 Mbps:      406.442 Peak Mbps:      406.442 Avg Mbps:      406.442
>
>
> root@router-test (/root)# top (while running tcpbench)
>
> load averages:  1.26,  0.80,  0.49
> 22 processes:  1 running, 20 idle, 1 on processor
> CPU states:  0.0% user,  0.0% nice, 77.2% system, 15.6% interrupt,
> 7.2% idle
> Memory: Real: 8752K/43M act/tot  Free: 442M  Swap: 0K/759M used/tot
>
>
> root@router-test (/root)# systat queue (while running tcpbench)
>
> 2 users    Load 0.82 0.69 0.51                      Thu Nov 18 17:13:10 2010
>
> QUEUE               BW SCH  PR  PKTS BYTES DROP_P DROP_B QLEN BORR
> SUSP P/S  B/S
> root_bge0        1000M cbq   0 7300K   10G      0      0    0    0
> 0 314  47M
>  q_lan            950M cbq     7300K   10G      0      0    0    0
> 0 314  47M
>
> ----
>
> Now back on Intel NIC
>
> 1.
> pf enabled, queue 950mbit, qlimit 500
> iperf test: 347 Mbits/sec
>
>
> 2.
> transfers by tcpbench:
>
> Conn:   1 Mbps:      328.701 Peak Mbps:      336.374 Avg Mbps:      328.701
>        29002       41936224      335.490  100.00%
> Conn:   1 Mbps:      335.490 Peak Mbps:      336.374 Avg Mbps:      335.490
>        30001       41394096      331.484  100.00%
> Conn:   1 Mbps:      331.484 Peak Mbps:      336.374 Avg Mbps:      331.484
>        30999       39930144      320.402  100.00%
> Conn:   1 Mbps:      320.402 Peak Mbps:      336.374 Avg Mbps:      320.402
>        32003       42171560      336.363  100.00%
> Conn:   1 Mbps:      336.363 Peak Mbps:      336.374 Avg Mbps:      336.363
>        33001       41970888      336.440  100.00%
> Conn:   1 Mbps:      336.440 Peak Mbps:      336.440 Avg Mbps:      336.440
>        34002       38258208      305.760  100.00%
> Conn:   1 Mbps:      305.760 Peak Mbps:      336.440 Avg Mbps:      305.760
>
>
> root@router-test (/root)# top (whiel runing tcpbench)
>
> load averages:  1.20,  0.59,  0.25
> 24 processes:  1 running, 22 idle, 1 on processor
> CPU states:  0.2% user,  0.0% nice, 75.6% system, 21.2% interrupt,
> 3.0% idle
> Memory: Real: 8904K/43M act/tot  Free: 442M  Swap: 0K/759M used/tot
>
>
> root@router-test (/root)# systat queue  (while running tcpbench)
> 2 users    Load 0.57 0.54 0.28                      Thu Nov 18 17:25:26 2010
>
> QUEUE               BW SCH  PR  PKTS BYTES DROP_P DROP_B QLEN BORR
> SUSP P/S  B/S
> root_em0         1000M cbq   0 2963K 4381M      0      0    0    0
> 0 279  42M
>  q_lan            950M cbq     2963K 4381M      0      0    0    0
> 0 279  42M
>
> ----
>
> so... the same machine, different NIC, different testing programs,
> same behavior (~50% of defined queue speed, very high CPU usage)
>
> the only thing i can try now is moving intel nic and hdd to other
> computer with pcie slot and run test on it to see is it hardware
> (motherboard) problem.
>

No the problem is altq. Altq(4) was written when 100Mbps was common and
people shaped traffic in the low megabit range. It seems to hit a wall when
doing hundreds of megabits. Guess someone needs to run a profiling kernel
and see where all that time is spent and then optimize altq(4).

--
:wq Claudio

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

RLW-2
W dniu 2010-11-18 17:41, Claudio Jeker pisze:
>>
>
> No the problem is altq. Altq(4) was written when 100Mbps was common and
> people shaped traffic in the low megabit range. It seems to hit a wall when
> doing hundreds of megabits. Guess someone needs to run a profiling kernel
> and see where all that time is spent and then optimize altq(4).
>

Its nice to hear from OpenBSD developer on this matter.

I am wondering who is gonna be that "someone"? ;) and when it could happen?

Claudio can you add this problem as a bug to fix maybe in next release?


----
best regards,
RLW

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

Stuart Henderson
On 2010-11-18, RLW <[hidden email]> wrote:

> W dniu 2010-11-18 17:41, Claudio Jeker pisze:
>>>
>>
>> No the problem is altq. Altq(4) was written when 100Mbps was common and
>> people shaped traffic in the low megabit range. It seems to hit a wall when
>> doing hundreds of megabits. Guess someone needs to run a profiling kernel
>> and see where all that time is spent and then optimize altq(4).
>>
>
> Its nice to hear from OpenBSD developer on this matter.
>
> I am wondering who is gonna be that "someone"? ;) and when it could happen?

The "someone" running a profiling kernel to identify the hot spots could be you.

cd /sys/arch/<arch>/config
config -p <kernelname>
build a kernel from the ../compile/<kernelname>.PROF directory in the usual way
kgmon -b to start profiling
(generate some traffic)
kgmon -h to stop profiling
kgmon -p to dump stats
gprof /bsd gmon.out to read stats...

Assuming you're interested in routed traffic (rather than queuing traffic
generated on a box itself), make sure you run the traffic source and sink
on other machines routing through the altq box, don't source/sink traffic
on the altq box itself.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

RLW-2
W dniu 2010-11-19 00:24, Stuart Henderson pisze:

> The "someone" running a profiling kernel to identify the hot spots could be you.
>
> cd /sys/arch/<arch>/config
> config -p<kernelname>
> build a kernel from the ../compile/<kernelname>.PROF directory in the usual way
> kgmon -b to start profiling
> (generate some traffic)
> kgmon -h to stop profiling
> kgmon -p to dump stats
> gprof /bsd gmon.out to read stats...
>
> Assuming you're interested in routed traffic (rather than queuing traffic
> generated on a box itself), make sure you run the traffic source and sink
> on other machines routing through the altq box, don't source/sink traffic
> on the altq box itself.
>
>

Sorry to others developers for not recognizing them.

I am interested in making network traffic on gbit lan around
800-900mbit/s not 350mbit/s with altq so its rather the option mentioned
by you in brackets ;)

I am rather standard OpenBSD user and to be honest I think it would be
faster and simpler if I just give some OpenBSD developer access to this box.

----
best regards,
RLW

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

Amit Kulkarni-5
In reply to this post by Stuart Henderson
Hi,

Could somebody include this in the FAQ? I found Daniel Hartmeier
personal page which shows how to get stack trace and line numbers. I
know that the stacktrace info is included somewhere on openbsd.org.
But the way Daniel presented was much better. That particular
presentation or a link should be in the FAQ too. They would be really
useful.

Thanks

> The "someone" running a profiling kernel to identify the hot spots could be you.
>
> cd /sys/arch/<arch>/config
> config -p <kernelname>
> build a kernel from the ../compile/<kernelname>.PROF directory in the usual way
> kgmon -b to start profiling
> (generate some traffic)
> kgmon -h to stop profiling
> kgmon -p to dump stats
> gprof /bsd gmon.out to read stats...
>
> Assuming you're interested in routed traffic (rather than queuing traffic
> generated on a box itself), make sure you run the traffic source and sink
> on other machines routing through the altq box, don't source/sink traffic
> on the altq box itself.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 1gbit LAN/NIC performance, queue speed bug?

Tomas Bodzar-4
On Mon, Nov 22, 2010 at 5:34 AM, Amit Kulkarni <[hidden email]> wrote:
> Hi,
>
> Could somebody include this in the FAQ? I found Daniel Hartmeier
> personal page which shows how to get stack trace and line numbers. I
> know that the stacktrace info is included somewhere on openbsd.org.
> But the way Daniel presented was much better. That particular
> presentation or a link should be in the FAQ too. They would be really
> useful.

Like this one http://www.openbsd.org/faq/faq2.html#Bugs ? Example on the end
looks exactly like his job.


>
> Thanks
>
>> The "someone" running a profiling kernel to identify the hot spots could be you.
>>
>> cd /sys/arch/<arch>/config
>> config -p <kernelname>
>> build a kernel from the ../compile/<kernelname>.PROF directory in the usual way
>> kgmon -b to start profiling
>> (generate some traffic)
>> kgmon -h to stop profiling
>> kgmon -p to dump stats
>> gprof /bsd gmon.out to read stats...
>>
>> Assuming you're interested in routed traffic (rather than queuing traffic
>> generated on a box itself), make sure you run the traffic source and sink
>> on other machines routing through the altq box, don't source/sink traffic
>> on the altq box itself.

12
Loading...