Crawler application, Celery and Redis with docker-compose

Dockerize crawler application, Celery and Redis with docker-compose

Build & Launch

docker-compose up -d --build

Flask application's endpoints on port 5001, the code in api/app.py is for endpoints

To shut down all container:

docker-compose down

All of tasks is defined in queue/tasks.py

Scale up worker

To add more workers:

docker-compose up -d --scale worker=5 --no-recreate

Flower server for monitoring workers on port 5555

Dispatch the fetch task to worker

To tar a folder with your base images

curl -X POST -F '[email protected]" http://localhost:5001/fetch/<task_name>