[Feature Request]: Task Executor integrate any distributed queue library

[devMls]

Is there an existing issue for the same feature request?

• I have checked the existing issues.

Is your feature request related to a problem?

Maybe is not a priority but when the rag growth , It is interesing.

Separate the ingest process from the user enviroment.

Many projects can be used as base, for example celery+redis, are highly related with this arquitecture (ragflow alredy uses redis and dokcer)

https://docs.celeryq.dev/en/stable/index.html

Describe the feature you'd like

I want distribute the ingest process of task executors

Describe implementation you've considered

use celery+redis

Documentation, adoption, use case

https://docs.celeryq.dev/en/stable/index.html

Additional information

No response

[KevinHuSh]

As I heard, Celery is so hard to debug. There's no clue when task goes wrong.

[yingfeng]

Celery is not a good choice in terms of simplicity, so we choose to use redis to serve as the task queue instead.
Regarding to larger throughput, it's not a key issue, because the bottleneck lies in the computation instead of task queue itself. Regarding to reliability, both celery and redis are not good choices. We may consider such choice as redpanda in future if it is a real issue.