How can I implement a single commit handler when writing to a dataset using both Rust and Python simultaneously?

[hushengquan]

My application scenario involves writing datasets using Python and Rust. I implemented an External Manifest Store in Rust, which provides atomic transaction commits. However, this solution cannot be used with Python, and the commit handler has requirements for the writers, specifically that only one commit handler can be active at a time.Are there any solutions to this problem?

[westonpace]

The commit handler is essentially the "compare and swap" primitive at the heart of Lance's concurrency model. If you want two different commit handler approaches to run at the same time they would have to have some way of interacting to maintain atomicity.

The default commit handler relies on object storage to provide the atomicity (either through a POSIX compliant rename-or-fail for disks or through write-if-not-exist for cloud storage) The advantage is that it doesn't matter what language or library the client is using. Why does the default commit handler not work?

Is it possible for your custom rust commit handler to have python bindings? Even so, if your commit handler is in-process then either all your writers have to be in the same process or your commit handlers need some way of communicating like TCP, etc.

[hushengquan]

The commit handler is essentially the "compare and swap" primitive at the heart of Lance's concurrency model. If you want two different commit handler approaches to run at the same time they would have to have some way of interacting to maintain atomicity.

The default commit handler relies on object storage to provide the atomicity (either through a POSIX compliant rename-or-fail for disks or through write-if-not-exist for cloud storage) The advantage is that it doesn't matter what language or library the client is using. Why does the default commit handler not work?

Is it possible for your custom rust commit handler to have python bindings? Even so, if your commit handler is in-process then either all your writers have to be in the same process or your commit handlers need some way of communicating like TCP, etc.

Some object storage systems do not support the put-if-not-exist semantic, making it impossible to guarantee transactional atomicity.Is there a way to ensure atomicity guarantees for transaction commits across APIs written in different languages?

[westonpace]

The dynamodb commit handler is an example of how you might do this. An external service (dynamodb) is used to provide the atomicity. You could use any other external database in the same manner or you could build your own external service.

[hushengquan]

The dynamodb commit handler is an example of how you might do this. An external service (dynamodb) is used to provide the atomicity. You could use any other external database in the same manner or you could build your own external service.

I have implemented the external service(mysql) in rust, but it is currently not usable in the Python API.

[hushengquan]

The external manifest store depends on ExternalCommitHandler, but Python cannot directly use this commit handler.The Python API ensures atomicity through CommitLock.

[westonpace]

Ah, so if there was some kind of PyCommitHandler that exposed the commit handler API up to python (so you could implement the commit handler interface in python against your mysql service) then would that would work?

[hushengquan]

Ah, so if there was some kind of PyCommitHandler that exposed the commit handler API up to python (so you could implement the commit handler interface in python against your mysql service) then would that would work?

Yes, I have the same idea. But if we want both Rust and Python to use the same Commit Handler, are there any good ways to achieve this?

[hushengquan]

@westonpace Hello, could you please take another look at this?