首页 > 代码库 > 压测:celey backend为rabbitmq pk redis
压测:celey backend为rabbitmq pk redis
使用celery的backend异步获取结果,本文使用rabbitmq 和 redis分别作为backend,代码对比如下
from celery import Celery, platformsimport timeimport osfrom datetime import datetimeapp = Celery(‘proj‘, broker=‘amqp://admin:admin@ip:5672‘, //rabbitmq backend=‘amqp://admin:admin@ip:5672‘, //redis //backend=‘redis://ip:10013/0‘, include=[‘tasks‘] )app.conf.update( CELERYD_PREFETCH_MULTIPLIER=1, CELERY_ACKS_LAT = True,)@app.taskdef fun_1(n): return {"task_id": "0a14c9ac91de419880c4499a8f57418e", "status": 0, "desc": str(n) }@app.taskdef chk_total_callback(tasks_rets): end_time = datetime.now() print "end_time", end_time return 0if __name__ == "__main__": app.start()
发送消息
from tasks import * from celery import group, chain from datetime import datetime import time #job_group = group([fun_1.s(i) for i in range(30000)]) beg_time = datetime.now() for j in range(8000): job_group = group([fun_1.s(i) for i in range(20)]) chain(job_group, chk_total_callback.s())() #time.sleep(0.25) print "beg_time:", beg_time
压测场景一:
注:1000*20:一次发送1000次,分片20
力度 | backend | 耗时 | 内存峰值 | 均值 |
1000*20
| rabbitmq | 18:40:36-18:41:21 | 0.504G | 45ms |
redis | 18:48:37-18:51:21 | 11.84M | 82ms | |
2000*20 | rabbitmq | 21:57:06 -21:58:41 | 1.008G | 47.5ms |
redis | 18:58:52-19:04:10 | 20.26M | 159ms | |
3000*20 | rabbitmq | 21:51:40-21:54:03 | 1.512G | 47.6ms |
redis | 19:43:31-19:51:41 | 28.18M | 163ms | |
4000*20 | rabbitmq | 20:59:12-21:02:28 | 2.268G | 49ms |
redis | 20:00:00-20:10:52 | 37.10M | 163ms |
压测场景二:
500*20: 500次,每次间隔1s,分片20
力度 | backend | 耗时 | 内存峰值 | 耗时 |
500*20
| rabbitmq | 22:34:28-22:43:01 | 0.504G | 513s |
redis | 22:49:23-22:57:56 | 7.63M | 513s |
压测场景三:
8000 * 0.25 = 2000s, 任务数:8000 8000 * 20 + 8000 = 8000 * 21=168000 * 2=336000
力度 | backend | 耗时 | 内存峰值 | 耗时 |
8000 * 0.25 = 2000s *20
| rabbitmq | 10:51:38-11:28:06 | 1.26G | 2188s |
redis | 11:56:17-12:33:00 | 65.96M | 2203s | |
16000 * 0.25 = 4000s *20
| rabbitmq | 18:26:39-19:39:37 | 4.28G | 73*60=4380 |
redis | 20:13:57-22:49:33 | 130.69M | 9360s |
压测:celey backend为rabbitmq pk redis