Imagine that you already have a Celery cluster deployed in production. Now you want to monitor it. What's the best way to do it?
Despite Celery being one of the most popular way to distribute work load across multiple processes and servers in Python, its ecosystem lacks monitoring tools. Flower is all we have. Well, let's deploy it!
Dokku is my favourite way to deploy my apps to production. It is an open-source PaaS which acts like a Heroku (but without a nice GUI).
I personally use Dokku for all my Python projects. You can check one of my setups here: https://github.com/ffmemes/ff-backend
Make sure your have Dokku installed on your server.
Create Dokku app with envs
Just read the snippet below - it should be crystal clear what is happening:
dokku apps:create flower
dokku domains:set flower flower.yourdomain.com
dokku config:set flower FLOWER_BASIC_AUTH=admin:danokhlopkov
If you set a domain for your flower, make sure that you also set the proper DNS record in your DNS provider: create A-record with the subdomain you want pointing to the IP address of your server.
Flower also supports different kinds of authentication methods. You can check all available options here: https://flower.readthedocs.io/en/latest/auth.html.
Set Broker URL
Unfortunately, Flower doesn't respect the
BROKER_URL environment variables, so we need to modify the Flower's Dockerfile CMD arguments and copy-paste there your
REDIS_URL. Dokku allows to do that by setting up the env variable called
dokku config:set flower DOKKU_DOCKERFILE_START_CMD='celery --broker=<COPY_PASTE_YOUR_REDIS_OR_RABBITMQ_URL> flower'
Make sure this broker url is accessible from the machine where you want to deploy your flower. If you deployed Redis / RabbitMQ with Dokku, don't forget to use
dokku redis:expose command to expose a port of a db and make it accessable outside of your local Docker network.
Expose proper ports
Unfortunately (again), Flower doesn't respect
$PORT env and its Dockerfile always exposes 5555 port. Let's tell Dokku and its Nginx to proxy it to
80 (we don't want to have ports in our URL, it's disgusting):
dokku ports:set flower http:80:5555
Deploy Flower from docker image
dokku git:from-image flower mher/flower:latest
And now your Flower should be deployed! Click on a link from concole output to visit your new & almost useless Celery monitoring tool.
You can also add https if you want: just use dokku letsencrypt plugin:
dokku letsencrypt:enable flower
Flower environment variables: https://flower.readthedocs.io/en/latest/config.html#environment-variables
If you miss the old Flower's realtime dashboard (its tab was called "monitor"), you can integrate Flower data to your Prometheus: https://github.com/mher/flower/blob/master/docs/prometheus-integration.rst