首页 > 代码库 > 基于业务封装API进行Redis服务性能测试记录

基于业务封装API进行Redis服务性能测试记录

背景

         开发方面给予redis开源客户端做了二次封装,且做了reids集群部署;ld要求对redis服务性能做一次摸底测试;

测试需求

  1. 单实例的读写压力极限
  2. 单机的读写压力极限(可能瓶颈在网卡)
  3. proxy单实例的压力极限
  4. proxy单机的压力极限
  5. 主备的切换的可靠性测试 ------------ 本次未做
  6. 平滑迁移的有效性  ------------ 本次未做
  7. 迁移对访问性能的影响(可选) ------------ 本次未做

 

从上述开发提出的需求看出,本次是以性能验证为主要目的的一次测试;运行场景为通过封装的API,及不同实例配置来验证处理能力;

 

测试准备工作

1、业务分析;

  • 1个proxy实例可能对应多个redis实例;
  • 实际业务目前只有一个主实例;
  • 理想情况:每个redis实例分别有主、备2个实例;
  • 业务调用都是通过redis_access.h文件中的方法来使用的
  • redis_access.h中封装有8个公共方法作为api;

----------- 其实都是拼装命令、参数后使用RAcess::Query来进行执行;

----------- 另外封装有序列化,以及从CA中查询IP端口等方法;读:Get,写:SetNX/Set,删除:Del;其中用的最多的是Get和SetNX;

----------测试时采用Set;

区别:

SetNX对key有判断,如果key存在则不写入;

Set:无论是否存在都进行写入动作;

  •  数据长度,无法确定具体业务长度可能是多少;此处采取两种业务长度

 

    A、1024字节;---- 测试单实例、单机极限;

    B、114字节;----- 测试实际业务及各配置实例数的表现;

 

  • 业务模型:

a、 Redis实例进行RDB备份;AOF备份;

RDB:时间间隔内生成数据集的时间点快照;

AOF:实时记录所有的写操作命令和数据; ----- 这个文件会很大,目测是有单读的脚本进行备份等处理;

b、主实例写入后同步给备份实例;主实例挂掉后,JCA切换配置配置中心服务为备份实例的IP端口;

 

 

2、用例设计(此处与业务数据一起贴出

a)         采用并发测试,峰值模型;

代理机实例redis实例单条数据长度(B)clientiTotalReqiReqSucciReqFailed失败占比iAvgReq(/s)dAvgConTime(s)dMaxConTime(s)dMinConTime(s)iReqLess25ms<=25ms占比iReqLess50ms(25ms,50ms]iReqLess100msiReqLess200msiReqLess300msiReqLess400msiReqLess500msiReqLarger500ms
111024005050677400677345550.01%677390.00147050.0037580.0003767731499.99%00.00%0000031
111024001001009073579072191380.02%907350.00219450.004650.00041590718899.98%00.00%0000031
11102400200200103732910369723570.03%1037320.00384050.0071810.000303103695099.96%00.00%0000022
111024003003009440949435665280.06%472040.00633350.0095480.00036994356699.94%00.00%000000
111024505000595208595142660.01%595200.0016510.0270650.00040559506099.98%750.01%000007
111024100100008780128678461660.02%868000.0022760.046610.00056586660298.70%12290.14%0000015
111024200200009033989030833150.03%903390.00439550.0510960.00042689928399.54%35410.39%25200007
111024300300008397428392385040.06%839730.0071150.0632320.00036883148299.02%71440.85%60000000

 

PS: 单实例600时失败数量已经0.03%,继续加大只会增加失败率;

代理机实例redis实例单条数据长度(B)client(保持读写4:1)iTotalReqiReqSucciReqFailed失败占比iAvgReq(/s)dAvgConTime(s)dMaxConTime(s)dMinConTime(s)iReqLess25ms<=25ms占比iReqLess50ms(25ms,50ms]iReqLess100msiReqLess200msiReqLess300msiReqLess400msiReqLess500msiReqLarger500ms
111024读:160645701645617840.01%645700.0024590.0056150.00052364561799.99%00.00%000000
写:40189338189300380.02%189330.0020840.0074530.00048318930099.98%00.00%000000
汇总数据:8350398349171220.01%835030.00227150.0074530.00048383491799.99%00.00%000000
111024读:3207051767048822940.04%705170.0045180.0096820.0002770488299.96%00.00%000000
写:80216001215927740.03%216000.0036670.0095110.00057621592799.97%00.00%000000
汇总数据:9211779208093680.04%921170.00409250.0096820.0002792080999.96%00.00%000000
111024读:2404435974433672300.05%443590.0053970.0106590.00034544336599.95%00.00%000002
读:2404061494059262230.05%406140.0058930.0107370.00222340592699.95%00.00%000000
写:120225609225534750.03%225600.0052830.0107550.00076322553199.97%00.00%000003
汇总数据:107535510748275280.05%1075330.0055243330.0107550.000345107482299.95%00.00%000005
1010114读:16018261121826049630.00%1826110.0008660.0509230.000195182431199.90%16880.09%4800002
写:40468944468924200.00%468940.0008330.0506110.0002246846299.90%4530.10%900000
汇总数据:22950562294973830.00%2295050.00084950.0509230.000195229277399.90%21410.09%5700002
1010114读:160129446012943481120.01%1294460.0012240.0527410.000204129263099.86%16240.13%9400000
读:16012405251240463620.00%1240520.0012790.0527160.000202123873399.86%16470.13%8300000
写:80632294632256380.01%632290.0012420.0526770.00022563132599.85%8710.14%6000000
汇总数据:316727931670672120.01%3167270.0012483330.0527410.000202316268899.86%41420.13%23700000
1010114读:240150090715006862210.01%1500900.0015910.0649880.00021149898399.87%15270.10%161000015
读:240142133314211681650.01%1421330.0016770.0649670.000207141920699.85%17910.13%16400007
写:120733923733901220.00%733920.0016140.0649540.00021373291399.86%8930.12%80000015
汇总数据:365616336557554080.01%3656150.0016273330.0649880.000207365110299.86%42110.12%405000037
1010114读:320154955415489236310.04%1549550.0020320.0759630.000218154463399.68%36010.23%68900000
读:320145678414561696150.04%1456780.0021650.0759570.000211145192699.67%35510.24%69200000
写:1606106316104052260.04%610630.0019240.0757230.00022660888599.71%12610.21%25900000
汇总数据:3616969361549714720.04%3616960.0020403330.0021650.075723360544499.68%84130.23%164000000
010114读:3206100206097063140.05%610020.0052530.6277380.00009960767599.62%13650.22%1090553022
读:3205952285951071210.02%595220.0053960.6291120.0001159378299.76%5430.09%28704850010
写:1603453033451541490.04%345300.0046280.2170360.00010534438099.73%3030.09%2430228000
汇总数据:155055115499675840.04%1550540.0050923330.0053960.217036154583799.70%22110.14%639012660212
05114读:3208405498402443050.04%840540.0038150.2071150.00009883355599.17%63600.76%32801000
读:3208142798139773020.04%814270.0039460.2073290.00009580664999.06%70010.86%32403000
写:1604349064347741320.03%434900.0036760.07040.00010743102199.11%35940.83%15900000
汇总数据:208973420889957390.04%2089710.0038123330.0039460.0704207122599.11%169550.81%81104000
105114读:320135490713546532540.02%1354900.0023480.0724270.000185135300299.86%14350.11%21000006
读:320131769713173903070.02%1317690.0024170.0724070.000203131566599.85%15090.11%206000010
写:160715005714928770.01%715000.0022210.072170.00022271410999.87%7150.10%9700007
汇总数据:338760933869716380.02%3387590.0023286670.0024170.07217338277699.86%36590.11%513000023
01114读:3204385364382382980.07%438530.0072790.0145270.00180943823899.93%00.00%000000
读:3204175194172073120.07%417510.007640.0166280.00280341720799.93%00.00%000000
写:1602426642425091550.06%242660.0065650.0138320.00015124250999.94%00.00%000000
汇总数据:109871910979547650.07%1098700.0071613330.007640.013832109795499.93%00.00%000000
101114读:3209337929335862060.02%933790.0034090.0236990.0002793358599.98%00.00%000001
读:3208841708840321380.02%884170.00360.0201150.00034588403299.98%00.00%000000
写:160469366469311550.01%469360.0033950.0140320.00034646929999.99%00.00%0000012
汇总数据:228732822869293990.02%2287320.0034680.00360.014032228691699.98%00.00%0000013
11114读:3204251744248613130.07%425170.0075030.0135770.00013242486199.93%00.00%000000
读:3203931933928933000.08%393190.0081150.0171250.00337439289399.92%00.00%000000
写:1602052252050761490.07%205220.0077750.0159420.00067220507699.93%00.00%000000
汇总数据:102359210228307620.07%1023580.0077976670.0081150.013577102283099.93%00.00%000000
105114读:320137700513767532521377000.0023120.0652740.0002021375096127737700003
读:320134004913397842651340040.0023760.0889870.00020513380421019703000020
写:160714969714849120714960.0022150.088990.00020671397651036300000
汇总数据:343202334313866373432000.0023010.0023760.065274342711428061443000023
108114读:320120853312082482851208530.0026360.0761970.0002091198618889074000000
读:320118527311849683051185270.0026850.0762250.000211175468878271800000
写:160651023650887136651020.0024360.0760970.00023646235428736300002
汇总数据:304482930441037263044820.0025856670.0026850.076097302032121959182100002
1010114读:320134893913486702691348930.0023580.0783530.00020713430004604106500001
读:320132842013281322881328420.0023950.0782620.00020913224754579107800000
写:160713346713206140713340.002220.0781770.000238710428224952900000
汇总数据:339070533900086973390690.0023243330.0023950.078177337590311432267200001
05114读:32077899777892077778990.0040950.0659850.0001037500552828857700000
读:320708764708471293708760.0045030.0641150.0000956788852910648000000
写:160394627394498129394620.0040360.0653870.0001093798191441826100000
汇总数据:188238818818894991882370.0042113330.0045030.064115180875971812131800000
08114读:320910940910678262910940.0035281.004780.000190164583344050288024
读:320806025805857168806020.0039590.2165910.00013479970650719520128000
写:160483931483798133483930.0032910.2165770.0001374806232558545072000
汇总数据:220089622003335632200890.0035926670.0039590.21657721819741596319020488024
010114读:320770839770580259770830.0042211.0170510.0000527671172531156074601020
读:320603203603055148603200.0052971.0126240.000134600003249317203750210
写:160374955374820135374950.004251.0130570.00016337335512097401690211
汇总数据:174899717484555421748980.0045893330.0052971.012624174047562334020129001441

 

 

环境准备工作

  •  Redis集群配置:由redis集群平台配置(BID,实例,设备等),同时需要部署CA,注册机器设备;
  •  服务器配置

redis:   

proxy

cpu cores : 6

cpu cores : 8

processor:24  

processer:32

 mem:128G  

mem:32G

 disk:600G 

disk:600G 
  网卡:1000Mb/s网卡:1000Mb/s
  • 服务部署:IP略;RPM:redis服务,redis_stencel,tweprox,共2redis服务(1主1备),6加压机,2proxy机器;

脚本开发工作

 

  • 压力工具:test_client框架(C++编写),修改框架代码分段传参;编写业务脚本,采用中奖结构体,2个字段;长度控制:重复填充对象;
  • 服务器性能数据收集:sar/vmstat/mpstat/top等工具手工收集;

 

测试执行和数据分析

  1. 执行:手工执行;
  2. 服务器数据简介:
  • Proxy:
  • 114B时10实例,800并发,总cpu低于20%,,部分内核吃满;开启irqbalance后依然如此;磁盘写入很低,内存/网卡等占用正常;
  • 1024时,单实例600写并发;网卡200MB+;假设继续加实例和并发,则网卡很容易就被吃满;
  •  Redis:114B时,800并发,redis总CPU30%左右;部分内核吃满,修改部分参数,开启irqbalance后,分布平均;磁盘写入正常,并不大;网卡在15~20MB左右;

开启irqbalance前,所有软中断全部压在cpu0

开启irqbalance后,CPU分布(同一个用例的效果未抓图,此处切另一个用例的图):

 

通过观察/proc/softirq文件,可以看到被沾满的CPU大部分是在等待网卡;判断增加网卡可能会提升数据表现;

  • l数据观察原则:
      • 数据观察点:由CPU/内存/网卡/磁盘等大项目进行观察;有异常的地方重复执行用例,观察具体实例数据;
      • 每次均应收集服务器性能指标数据,以备对比;
      • 数据采样:单个用例多次执行,如结果浮动不大,则采取中间值;如波动较大,找出原因后,给出区间;

测试结果

1、114字节的数据,读写比例4:1下性能数据:  ------源自开发估算

直连redis,单实例       10w/s
直连redis,单机(10实例)       35w/s
1接入机+1 redis机       40w/s
2接入机+1 redis机       62w/s

2、理论值114字节数据可以达到100W/S,实际只有62W左右; ------ 根据执行结果估算;

3、瓶颈在于两点:

  • Proxy的内核负载不平均,软中断过高;已开启irqbalance,但还是不平均;
  • Proxy的网卡流量限制;------ 暂无法解决;

 

其他说明:

l  每次修改实例数后,需要重新执行写入动作,让数据平均分布到各实例,否则在读取的时候不准确;部分读命令无法找到数据;

l  本次测试场景考虑比较简单,属于验证能力类的测试;比如未考虑网络影响(实际上主备实例是跨机房的,这里是同机房不同机器)

另外,发现直连比代理方式的bo/bi更多,未能找到多写入了什么东东。暂时未解决此处;

 

基于业务封装API进行Redis服务性能测试记录