标签云

微信群

扫码加入我们

WeChat QR Code

I have a system set up currently that is using celery with a redis backend to do a bunch of asynchronous tasks such as sending emails, pulling social data, crawling,etc. Everything is working great, but I am having group figuring out how to monitor the system (aka the number of queue up messages). I started looking through the celery source but I figured I would post my questions in here: First off, here are my configurations: BROKER_BACKEND= "redis" BROKER_HOST = "localhost" BROKER_PORT = 6379 BROKER_VHOST= "1" REDIS_CONNECT_RETRY = True REDIS_HOST= "localhost" REDIS_PORT= 6379 REDIS_DB= "0" CELERY_SEND_EVENTS= True CELERYD_LOG_LEVEL = 'INFO' CELERY_RESULT_BACKEND = "redis" CELERY_TASK_RESULT_EXPIRES= 25 CELERYD_CONCURRENCY = 8 CELERYD_MAX_TASKS_PER_CHILD = 10 CELERY_ALWAYS_EAGER =TrueThe first thing I am trying to do is monitor how many messages are in my queue. I assume, behind the scenes, the redis backend is just pushing/popping from a list, although I cannot seem to find that in the code. So I mock up a simulation where I start about 100 tasks and am trying to find them in redis: My celeryd is running like this: python manage.py celeryd -c 4 --loglevel=DEBUG -n XXXXX --logfile=logs/ celery.log So I should only have 4 concurrent workers at once ..... Two thing I do not understand: Problem 1: After I have queued up 100 task, and look for them on redis, I only see the following: $ redis-cli redis 127.0.0.1:6379> keys * 1) "_kombu.binding.celery" redis 127.0.0.1:6379> select 1 OK redis 127.0.0.1:6379[1]> keys * 1) "_kombu.binding.celery" 2) "_kombu.binding.celeryd.pidbox" redis 127.0.0.1:6379[1]>I cannot seem to find the tasks to get a number of how many are queued (technically, 96 should be since I only support 4 concurrent tasks) Problem 2 $ ps aux | grep celeryd | cut -c 13-12041258 0.20.22526232 9440 s004S+2:27PM 0:07.35 python manage.py celeryd -c 4 --loglevel=DEBU41261 0.00.12458320 2468 s004S+2:27PM 0:00.09 python manage.py celeryd -c 4 --loglevel=DEBU38457 0.00.8255984834672 s004T12:34PM 0:18.59 python manage.py celeryd -c 4 --loglevel=INFO38449 0.00.9251724436752 s004T12:34PM 0:35.72 python manage.py celeryd -c 4 --loglevel=INFO38443 0.00.22524136 6456 s004T12:34PM 0:10.15 python manage.py celeryd -c 4 --loglevel=INFO84542 0.00.024601124 s000T27Jan12 0:00.74 python manage.py celeryd -c 4 --loglevel=INFO84536 0.00.025067284 s000T27Jan12 0:00.51 python manage.py celeryd -c 4 --loglevel=INFO41485 0.00.02435120564 s000S+2:54PM 0:00.00 grep celeryd41264 0.00.12458320 2480 s004S+2:27PM 0:00.09 python manage.py celeryd -c 4 --loglevel=DEBU41263 0.00.12458320 2480 s004S+2:27PM 0:00.09 python manage.py celeryd -c 4 --loglevel=DEBU41262 0.00.12458320 2480 s004S+2:27PM 0:00.09 python manage.py celeryd -c 4 --loglevel=DEBU If anyone could explain this for me, it would be great.