首页 > 代码库 > [development][dpdk] 网卡收包性能测试

[development][dpdk] 网卡收包性能测试

 

一: 多队列及中断信息收集

[root@T185 ~]# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             512
RX Mini:        0
RX Jumbo:       0
TX:             512
[root@T185 ~]# ls /sys/class/net/eth0/queues/
rx-0  rx-10  rx-12  rx-14  rx-16  rx-18  rx-2   rx-21  rx-23  rx-4  rx-6  rx-8  tx-0  tx-10  tx-12  tx-14  tx-16  tx-18  tx-2   tx-21  tx-23  tx-4  tx-6  tx-8
rx-1  rx-11  rx-13  rx-15  rx-17  rx-19  rx-20  rx-22  rx-3   rx-5  rx-7  rx-9  tx-1  tx-11  tx-13  tx-15  tx-17  tx-19  tx-20  tx-22  tx-3   tx-5  tx-7  tx-9
技术分享
[root@T185 ~]# cat /proc/interrupts |less -S
[root@T185 ~]# cat /proc/interrupts
            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12      CPU13      CPU14      CPU15      CPU16      CPU17      CPU18      CPU19      CPU20      CPU21      CPU22      CPU23      
   0:        186          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-edge      timer
   1:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-edge      i8042
   8:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-edge      rtc0
   9:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-fasteoi   acpi
  12:          4          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-edge      i8042
  16:        256          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-fasteoi   ehci_hcd:usb1
  23:        298          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-fasteoi   ehci_hcd:usb2
  88:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  DMAR_MSI-edge      dmar0
  89:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  DMAR_MSI-edge      dmar1
  90:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      aerdrv
  91:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      aerdrv
  92:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      aerdrv
  93:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      aerdrv
  94:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      aerdrv
  95:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      aerdrv
  96:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      aerdrv
  97:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      aerdrv
  98:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      aerdrv
  99:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      aerdrv
 100:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      aerdrv
 101:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      aerdrv
 102:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      aerdrv
 103:      24847          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      ahci
 104:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      isci-msix
 105:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      isci-msix
 106:    3319852          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-0
 107:     981541          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-1
 108:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-2
 109:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-3
 110:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-4
 111:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-5
 112:     669706          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-6
 113:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-7
 114:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-8
 115:      28009          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-9
 116:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-10
 117:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-11
 118:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-12
 119:      28069          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-13
 120:     338786          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-14
 121:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-15
 122:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-16
 123:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-17
 124:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-18
 125:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-19
 126:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-20
 127:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-21
 128:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-22
 129:      28004          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-TxRx-23
 130:         59          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0
 131:      85702          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-0
 132:      84484          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-1
 133:      89210          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-2
 134:      86210          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-3
 135:      86086          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-4
 136:      89464          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-5
 137:      94737          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-6
 138:      86082          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-7
 139:      84812          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-8
 140:      86437          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-9
 141:      83901          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-10
 142:      90394          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-11
 143:      86578          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-12
 144:      89534          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-13
 145:      92897          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-14
 146:      97958          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-15
 147:      26913          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-16
 148:      26913          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-17
 149:      26913          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-18
 150:      26913          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-19
 151:      26913          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-20
 152:      26913          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-21
 153:      26913          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-22
 154:      26913          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-TxRx-23
 155:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1
 157:      97476          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth3
 NMI:        736         48          9          4          2          1        133          2         14          4         13         12          6        102         67          2          1          1          1        163          3        640        550       2718   Non-maskable interrupts
 LOC:     892399      69189      85565      60959      35207      24688     159437      24227     114514      64519      50015      24473      19392     198772     108184      34002      21177      18652      18684     250274      43307   42755165   42680063    2737380   Local timer interrupts
 SPU:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Spurious interrupts
 PMI:        736         48          9          4          2          1        133          2         14          4         13         12          6        102         67          2          1          1          1        163          3        640        550       2718   Performance monitoring interrupts
 IWI:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IRQ work interrupts
 RTR:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   APIC ICR read retries
 RES:    2281297     404101       4908        813        277        323        594        864       4512       1682        768        369        109       3271        222        508         86         81        145       7654        369        964        460        202   Rescheduling interrupts
 CAL:        325        389        365        373        385        385        741        491       1434        921        652        488        384        357        387        385        383        384        395       3966        679        586       1914        427   Function call interrupts
 TLB:         19          5         18        157          3          6         11          7         12         78          2          3         12        781        637         30         25         12          7        807        686       7816       4244      50309   TLB shootdowns
 TRM:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Machine check exceptions
 MCP:        217        217        217        217        217        217        217        217        217        217        217        217        217        217        217        217        217        217        217        217        217        217        217        217   Machine check polls
 ERR:          0
 MIS:          0
[root@T185 ~]# 
interrupts

 

二: 设置CPU隔离

在grub.conf 中,增加isolcpus参数。

 

三: 修改收发包队列数

技术分享
[root@T185 tong]# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             64
RX Mini:        0
RX Jumbo:       0
TX:             512

[root@T185 tong]# ethtool -G eth0 rx 1  tx 1
[root@T185 tong]# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             64
RX Mini:        0
RX Jumbo:       0
TX:             64
ethtool

不能设成1, 最小64.所以,对于中断并没有什么影响.这好像不对,不知道这是什么鬼。

下载最新的驱动,https://downloadcenter.intel.com/download/14687 并查看README

技术分享
Command Line Parameters                                                         
-----------------------                                                         
If the driver is built as a module, the following optional parameters are used  
by entering them on the command line with the modprobe command using this       
syntax:                                                                         
modprobe ixgbe [<option>=<VAL1>,<VAL2>,...]                                     
                                                                                
There needs to be a <VAL#> for each network port in the system supported by     
this driver. The values will be applied to each instance, in function order.    
For example:                                                                    
modprobe ixgbe InterruptThrottleRate=16000,16000                                
                                                       
。。。 。。。

Multiqueue                                                                      
----------                                                                      
Valid Range:                                                                    
0, 1                                                                            
0 = Disables Multiple Queue support                                             
1 = Enabled Multiple Queue support (a prerequisite for RSS)                     
                                      
README

README里给的参数竟然不对,尼玛

[root@T185 tong]# modprobe -r ixgbe
[root@T185 tong]# modprobe ixgbe MQ=1,0
[root@T185 tong]# ls /sys/class/net/eth0/queues/
rx-0  rx-10  rx-12  rx-14  rx-16  rx-18  rx-2   rx-21  rx-23  rx-4  rx-6  rx-8  tx-0  tx-10  tx-12  tx-14  tx-16  tx-18  tx-2   tx-21  tx-23  tx-4  tx-6  tx-8
rx-1  rx-11  rx-13  rx-15  rx-17  rx-19  rx-20  rx-22  rx-3   rx-5  rx-7  rx-9  tx-1  tx-11  tx-13  tx-15  tx-17  tx-19  tx-20  tx-22  tx-3   tx-5  tx-7  tx-9
[root@T185 tong]# ls /sys/class/net/eth1/queues/
rx-0  tx-0
[root@T185 tong]#

 

四:中断绑定:

cpu结构:

[root@T185 tong]# ./cpu_layout.py 
============================================================
Core and Socket Information (as reported by /proc/cpuinfo)
============================================================

cores =  [0, 1, 2, 3, 4, 5]
sockets =  [0, 1]

       Socket 0        Socket 1        
       --------        --------        
Core 0 [0, 12]         [6, 18]         
Core 1 [1, 13]         [7, 19]         
Core 2 [2, 14]         [8, 20]         
Core 3 [3, 15]         [9, 21]         
Core 4 [4, 16]         [10, 22]        
Core 5 [5, 17]         [11, 23]        
[root@T185 tong]# 

 

另外两个工具:lstopo,和hwloc。

中断绑定:

#! /bin/bash

function bind() # $lcpus, $irq_list, $set
{
        for i in $2
        do
                echo -n $i ": "
                if [ $3 -eq 1 ]
                then
                        echo $1 > /proc/irq/$i/smp_affinity_list
                        echo "Done!"
                else
                        cat /proc/irq/$i/smp_affinity_list
                fi
        done
}

function usage()
{
        echo "Usage:"
        echo -e "\tirq_bind.sh 0(print)/1(set) "
        echo
}

if [ $# -ne 1 ]
then
        usage
        exit
fi

if [ \( "$1" != "1" \) -a \( "$1" != "0" \) ]
then
        usage
        exit
fi

irq_list=`cat /proc/interrupts |grep eth0 |awk -F: {print $1}`
bind "1,13" "$irq_list" $1

irq_list=`cat /proc/interrupts |grep eth1 |awk -F: {print $1}`
bind "2,14" "$irq_list" $1

 

注意: 网卡驱动重新加载过了之后,需要重新对中断进行绑定。

 

五: 发包:

tcpreplay -i eth0 -l0 -L10000000 -p400000 -K syy/50000.pcap 

 

[development][dpdk] 网卡收包性能测试