首页 > 代码库 > [development][dpdk] 网卡收包性能测试
[development][dpdk] 网卡收包性能测试
一： 多队列及中断信息收集
[root@T185 ~]# ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 512 RX Mini: 0 RX Jumbo: 0 TX: 512
[root@T185 ~]# ls /sys/class/net/eth0/queues/ rx-0 rx-10 rx-12 rx-14 rx-16 rx-18 rx-2 rx-21 rx-23 rx-4 rx-6 rx-8 tx-0 tx-10 tx-12 tx-14 tx-16 tx-18 tx-2 tx-21 tx-23 tx-4 tx-6 tx-8 rx-1 rx-11 rx-13 rx-15 rx-17 rx-19 rx-20 rx-22 rx-3 rx-5 rx-7 rx-9 tx-1 tx-11 tx-13 tx-15 tx-17 tx-19 tx-20 tx-22 tx-3 tx-5 tx-7 tx-9
[root@T185 ~]# cat /proc/interrupts |less -S [root@T185 ~]# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 0: 186 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge timer 1: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042 8: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge rtc0 9: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi acpi 12: 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042 16: 256 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi ehci_hcd:usb1 23: 298 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi ehci_hcd:usb2 88: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0 89: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1 90: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 91: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 92: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 93: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 94: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 95: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 96: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 97: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 98: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 99: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 100: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 101: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 102: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 103: 24847 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge ahci 104: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge isci-msix 105: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge isci-msix 106: 3319852 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-0 107: 981541 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-1 108: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-2 109: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-3 110: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-4 111: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-5 112: 669706 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-6 113: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-7 114: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-8 115: 28009 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-9 116: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-10 117: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-11 118: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-12 119: 28069 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-13 120: 338786 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-14 121: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-15 122: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-16 123: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-17 124: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-18 125: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-19 126: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-20 127: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-21 128: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-22 129: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-23 130: 59 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0 131: 85702 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-0 132: 84484 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-1 133: 89210 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-2 134: 86210 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-3 135: 86086 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-4 136: 89464 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-5 137: 94737 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-6 138: 86082 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-7 139: 84812 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-8 140: 86437 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-9 141: 83901 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-10 142: 90394 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-11 143: 86578 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-12 144: 89534 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-13 145: 92897 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-14 146: 97958 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-15 147: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-16 148: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-17 149: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-18 150: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-19 151: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-20 152: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-21 153: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-22 154: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-23 155: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1 157: 97476 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth3 NMI: 736 48 9 4 2 1 133 2 14 4 13 12 6 102 67 2 1 1 1 163 3 640 550 2718 Non-maskable interrupts LOC: 892399 69189 85565 60959 35207 24688 159437 24227 114514 64519 50015 24473 19392 198772 108184 34002 21177 18652 18684 250274 43307 42755165 42680063 2737380 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 736 48 9 4 2 1 133 2 14 4 13 12 6 102 67 2 1 1 1 163 3 640 550 2718 Performance monitoring interrupts IWI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 APIC ICR read retries RES: 2281297 404101 4908 813 277 323 594 864 4512 1682 768 369 109 3271 222 508 86 81 145 7654 369 964 460 202 Rescheduling interrupts CAL: 325 389 365 373 385 385 741 491 1434 921 652 488 384 357 387 385 383 384 395 3966 679 586 1914 427 Function call interrupts TLB: 19 5 18 157 3 6 11 7 12 78 2 3 12 781 637 30 25 12 7 807 686 7816 4244 50309 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 Machine check polls ERR: 0 MIS: 0 [root@T185 ~]#
二： 设置CPU隔离
在grub.conf 中，增加isolcpus参数。
三： 修改收发包队列数
[root@T185 tong]# ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 64 RX Mini: 0 RX Jumbo: 0 TX: 512 [root@T185 tong]# ethtool -G eth0 rx 1 tx 1 [root@T185 tong]# ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 64 RX Mini: 0 RX Jumbo: 0 TX: 64
不能设成1, 最小64．所以，对于中断并没有什么影响．这好像不对，不知道这是什么鬼。
下载最新的驱动，https://downloadcenter.intel.com/download/14687 并查看README
Command Line Parameters ----------------------- If the driver is built as a module, the following optional parameters are used by entering them on the command line with the modprobe command using this syntax: modprobe ixgbe [<option>=<VAL1>,<VAL2>,...] There needs to be a <VAL#> for each network port in the system supported by this driver. The values will be applied to each instance, in function order. For example: modprobe ixgbe InterruptThrottleRate=16000,16000 。。。 。。。 Multiqueue ---------- Valid Range: 0, 1 0 = Disables Multiple Queue support 1 = Enabled Multiple Queue support (a prerequisite for RSS)
README里给的参数竟然不对，尼玛
[root@T185 tong]# modprobe -r ixgbe [root@T185 tong]# modprobe ixgbe MQ=1,0 [root@T185 tong]# ls /sys/class/net/eth0/queues/ rx-0 rx-10 rx-12 rx-14 rx-16 rx-18 rx-2 rx-21 rx-23 rx-4 rx-6 rx-8 tx-0 tx-10 tx-12 tx-14 tx-16 tx-18 tx-2 tx-21 tx-23 tx-4 tx-6 tx-8 rx-1 rx-11 rx-13 rx-15 rx-17 rx-19 rx-20 rx-22 rx-3 rx-5 rx-7 rx-9 tx-1 tx-11 tx-13 tx-15 tx-17 tx-19 tx-20 tx-22 tx-3 tx-5 tx-7 tx-9 [root@T185 tong]# ls /sys/class/net/eth1/queues/ rx-0 tx-0 [root@T185 tong]#
四：中断绑定：
ｃｐｕ结构：
[root@T185 tong]# ./cpu_layout.py ============================================================ Core and Socket Information (as reported by ‘/proc/cpuinfo‘) ============================================================ cores = [0, 1, 2, 3, 4, 5] sockets = [0, 1] Socket 0 Socket 1 -------- -------- Core 0 [0, 12] [6, 18] Core 1 [1, 13] [7, 19] Core 2 [2, 14] [8, 20] Core 3 [3, 15] [9, 21] Core 4 [4, 16] [10, 22] Core 5 [5, 17] [11, 23] [root@T185 tong]#
另外两个工具：lstopo，和hwloc。
中断绑定：
#! /bin/bash function bind() # $lcpus, $irq_list, $set { for i in $2 do echo -n $i ": " if [ $3 -eq 1 ] then echo $1 > /proc/irq/$i/smp_affinity_list echo "Done!" else cat /proc/irq/$i/smp_affinity_list fi done } function usage() { echo "Usage:" echo -e "\tirq_bind.sh 0(print)/1(set) " echo } if [ $# -ne 1 ] then usage exit fi if [ \( "$1" != "1" \) -a \( "$1" != "0" \) ] then usage exit fi irq_list=`cat /proc/interrupts |grep eth0 |awk -F: ‘{print $1}‘` bind "1,13" "$irq_list" $1 irq_list=`cat /proc/interrupts |grep eth1 |awk -F: ‘{print $1}‘` bind "2,14" "$irq_list" $1
注意： 网卡驱动重新加载过了之后，需要重新对中断进行绑定。
五： 发包：
tcpreplay -i eth0 -l0 -L10000000 -p400000 -K syy/50000.pcap
[development][dpdk] 网卡收包性能测试
声明：以上内容来自用户投稿及互联网公开渠道收集整理发布，本网站不拥有所有权，未作人工编辑处理，也不承担相关法律责任，若内容有误或涉及侵权可进行投诉： 投诉/举报 工作人员会在5个工作日内联系你，一经查实，本站将立刻删除涉嫌侵权内容。