Catalog-Aware Table Commits

[jackye1995]

There is one thing that is unique about Iceberg compared to all the other table formats - Delta, Lance, Paimon, Hudi, that is a commit in Iceberg MUST go through a catalog.

This brought convenience that a catalog has full visibility of activities in a table, regardless of how the table is used. There would not be a case that the table is updated by some process, but the catalog is not aware of that happening because the commit directly goes through the storage.

When designing Lance Namespace, we put an experimental idea of storage-managed vs implementation-managed tables based on the Iceberg design: storage-managed Lance table is directly committed against a storage, whereas a implementation-managed Lance table acts similar to an Iceberg table - the namespace implementation designs how to commit and get the true source of truth of which version of the Lance table is the latest, and that can vary across implementations (e.g. by storing a property of latest_manifest_location). By doing that, similar to Iceberg, a table has to go through UpdateTable to commit any change, giving much more control to the data infra team that typically is the table admin and owner.

Over the past few months of bouncing the idea with a few groups, I think I get 2 general feedbacks:

1. people like that there is a centralized endpoint for commit that can enforce stronger access control and able to more easily trigger operations downstream.
2. people don't like that now the latest version of the Lance dataset has to be resolved using the namespace implementation but no longer the storage. It also does not work well with branching.

So here is the refreshed design I am proposing:

1. we no longer need the concept of storage-managed vs impl-managed, because we will NOT maintain a latest metadata pointer like Iceberg does. The table's latest version must be always resolved using the Lance manifest naming scheme: https://lance.org/format/table/transaction/#manifest-naming-schemes
2. we will add a new operation called CommitTable, which will accept a Lance transaction to be committed against the underlying table. The operation should be able to accept all the transaction types we support: https://lance.org/format/table/transaction/#transaction-types. The expectation is that the transaction is sent to the operation to be committed, and the impl will commit following the Lance dataset storage commit protocol https://lance.org/format/table/transaction/#commit-protocol such that the new version is still numerically ordered in the _versions directory.
3. for REST namespace, because an independent server will do the commit rather than client side Lance SDK, it becomes important that we define the expected behavior correctly. I think we should turn our current conflict_resolver.rs logic into the format specification, so we can precisely define when the server sees a pair of conflicting commit, what is the expected behavior. The server can choose to not follow exactly (e.g. some pairs of conflict are not handled and just fail), but if the server chooses to handle it, it should do it according to the spec to avoid wrong commit outcome.

Now to go back to answer the question of how to ensure ALL commits go through a namespace implementation, I think the implementation will be able to do that through access control.

During credentials vending phase, the access of a writer to the Lance table should NOT allow access to write to the _versions directories in both the main root location and any branch dataset location that should be controlled. This ensure that no one can just randomly commit to the table without going through the centralized CommitTable endpoint. and CommitTable will be the one that does the actual commit.