We migrated from Celery to Prefect a couple of years back and have been very happy. But ours is a small op which handles tasks in 1000s and not millions. It’s been night and day in terms of visibility and tracking. I would definitely recommend it.
It’s a heavy weight that covers a lot of use cases. But we just run simple ProcessWorkers for our regular needs and ECS worker for heavier ML tasks.
It’s a heavy weight that covers a lot of use cases. But we just run simple ProcessWorkers for our regular needs and ECS worker for heavier ML tasks.