首页 > 代码库 > 压测:celey backend为rabbitmq pk redis

压测:celey backend为rabbitmq pk redis

使用celery的backend异步获取结果,本文使用rabbitmq 和 redis分别作为backend,代码对比如下

from celery import Celery, platformsimport timeimport osfrom datetime import datetimeapp = Celery(proj,             broker=amqp://admin:admin@ip:5672,     //rabbitmq             backend=amqp://admin:admin@ip:5672,    //redis             //backend=‘redis://ip:10013/0‘,             include=[tasks]             )app.conf.update(    CELERYD_PREFETCH_MULTIPLIER=1,    CELERY_ACKS_LAT = True,)@app.taskdef fun_1(n):    return {"task_id": "0a14c9ac91de419880c4499a8f57418e",            "status": 0,            "desc": str(n) }@app.taskdef chk_total_callback(tasks_rets):    end_time = datetime.now()    print "end_time", end_time    return 0if __name__ == "__main__":    app.start()

发送消息

from tasks import *                                                    from celery import group, chain                                        from datetime import datetime                                          import time                                                                                                                                                                                                          #job_group = group([fun_1.s(i) for i in range(30000)])                 beg_time = datetime.now()                                              for j in range(8000):    job_group = group([fun_1.s(i) for i in range(20)])                     chain(job_group, chk_total_callback.s())()                             #time.sleep(0.25)                                                                                                                         print "beg_time:", beg_time

压测场景一:

注:1000*20:一次发送1000次,分片20

力度

backend

耗时

内存峰值

均值

1000*20

 

rabbitmq

18:40:36-18:41:21

0.504G

45ms

redis

18:48:37-18:51:21

11.84M

82ms

2000*20

rabbitmq

21:57:06 -21:58:41

1.008G

47.5ms

redis

18:58:52-19:04:10

20.26M

159ms

3000*20

rabbitmq

21:51:40-21:54:03

1.512G

47.6ms

redis

19:43:31-19:51:41

28.18M

163ms

4000*20

rabbitmq

20:59:12-21:02:28

2.268G

49ms

redis

20:00:00-20:10:52

37.10M

163ms

压测场景二:

500*20: 500次,每次间隔1s,分片20

力度

backend

耗时

内存峰值

耗时

500*20

 

rabbitmq

22:34:28-22:43:01

0.504G

513s

redis

22:49:23-22:57:56

7.63M

513s

压测场景三:

8000 * 0.25 = 2000s, 任务数:8000   8000 * 20 + 8000 = 8000 * 21=168000 * 2=336000

力度

backend

耗时

内存峰值

耗时

8000 * 0.25 = 2000s

*20

 

rabbitmq

10:51:38-11:28:06

1.26G

2188s

redis

11:56:17-12:33:00

65.96M

2203s

16000 * 0.25 = 4000s

*20

 

rabbitmq

18:26:39-19:39:37

4.28G

73*60=4380

redis

20:13:57-22:49:33

130.69M

9360s

压测:celey backend为rabbitmq pk redis