[Question]: Data source synchronization failed

[wulonga]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (Language Policy).
• Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
• Please do not modify this template :) and fill in all the required fields.

Describe your problem

I used the data source to synchronize the data of the self-built system's Minio storage bucket. However, I encountered an error. I checked the historical records and modified the files in the Docker, but the error still persisted.
The project I'm using is started by Docker Compose, with the release version of 0.23.1.

Please quickly answer this question. I'm in a hurry. Thank you.@Magicbook1108

[dosubot]

The sync failure is caused by a Python TypeError: the Minio/S3 connector returns a regular (synchronous) generator, but the sync logic tries to use it with async for, which only works with async generators. This is a known issue in RAGFlow v0.23.1 and affects Minio, S3, and some other connectors. Other connectors (like Confluence and GitHub) avoid this by wrapping their synchronous generators in an async wrapper, but the Minio/S3 connector does not, which leads to this error.
Details and workaround discussion

There isn't a configuration-only fix for this in the official Docker image. The recommended workaround is to use the nightly Docker image, which often includes this fix before the next stable release. If you need to stay on the stable release, you'll have to wait for an updated image or patch the connector code to wrap the generator in an async wrapper (see the linked issue for code examples).

Recent Minio-related pull requests (like #12446) improved health checks but do not address this async iteration bug.

If you want to patch it yourself, the fix is to wrap the generator in an async function before returning it, like this:

async def async_wrapper(sync_gen):
    for batch in sync_gen:
        yield batch
# Then return async_wrapper(document_batch_generator)

This will make the connector compatible with async for and resolve the error.
See more details and examples

Let me know if you need help with the patch or want more details on the workaround.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[wulonga]

The sync failure is caused by a Python TypeError: the Minio/S3 connector returns a regular (synchronous) generator, but the sync logic tries to use it with async for, which only works with async generators. This is a known issue in RAGFlow v0.23.1 and affects Minio, S3, and some other connectors. Other connectors (like Confluence and GitHub) avoid this by wrapping their synchronous generators in an async wrapper, but the Minio/S3 connector does not, which leads to this error. Details and workaround discussion

There isn't a configuration-only fix for this in the official Docker image. The recommended workaround is to use the nightly Docker image, which often includes this fix before the next stable release. If you need to stay on the stable release, you'll have to wait for an updated image or patch the connector code to wrap the generator in an async wrapper (see the linked issue for code examples).

Recent Minio-related pull requests (like #12446) improved health checks but do not address this async iteration bug.

If you want to patch it yourself, the fix is to wrap the generator in an async function before returning it, like this:

async def async_wrapper(sync_gen):
for batch in sync_gen:
yield batch

Then return async_wrapper(document_batch_generator)

This will make the connector compatible with async for and resolve the error. See more details and examples

Let me know if you need help with the patch or want more details on the workaround.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

   

===============

Could it be the env file in the 23 version source code? Was the modification made here in the nightly version?

[Magicbook1108]

#12405

If you are running ragflow from source development, pull the latest code.
If you are running ragflow from docker, you should either edit the code in docker container or pull the nightly image.

To edit the code inside the Docker container (ask AI for detailed guidance if you are unfamiliar with Docker or vim):

1. docker exec -it docker-ragflow-cpu-1 /bin/bash
2. Use vim to open rag/svr/sync_data_source.py and apply the changes from PR Fix: remvoe async wrappers  #12405.
3. Save the file and exit vim.
4. docker restart docker-ragflow-cpu-1

Note: I haven’t personally tried pulling the nightly image. You may need to explore that option on your own if you choose that approach.

[LiruiYu33]

If you want to upgrade to the nightly mirror, pls
docker compose -f docker/docker-compose.yml down
git pull
RAGFLOW_IMAGE=infiniflow/ragflow:nightly
docker compose -f docker/docker-compose.yml pull
docker compose -f docker/docker-compose.yml up -d

[wulonga]

@Magicbook1108 @LiruiYu33
Thank you both sincerely. It worked. On my side, I was missing the last step - not restarting the container. But after restarting, it was all good.