Unable to write custom metadata via configuration since version 0.9.0

[torshind]

Environment

Delta-rs version: 0.9.0

Binding: python

Environment:

• OS: Ubuntu 22.04.2 WSL2
• Other: Python 3.10.6

---

Bug

What happened: with version 0.9.0 a custom field isn't written anymore in metadata

What you expected to happen: read a custom metadata field without errors

How to reproduce it: write and read a table using configuration={"configTest": "foobar"} like in the unit tests of the previous version

[roeap]

Thanks for reporting @torshind.

Had a look and this is in fact a regression caused by us switching to a new implementation for the create command, which only honors "official" delta configuration. After reading the delta protocol a bit more is seems that it is completely feasible to write custom data into the table config, even if readers / writers do not know what to do with this.

That said, a more common pattern - at least from what I have seen - is to include custom metadata with the commit information an not with the global table metadata. As it used to be the custom configuration would also only be included if a new table is created, and not if passed in subsequent writes to the table.

I guess my questions are:

• would you expect to be able to update custom config on the table metadata also after creating the table (it is possible now on the rust side, but not python)
• would adding metadata to the commit info on individual commits also serve your use-case?

[torshind]

Hello, thanks for you answer.

• Yes, I think updating the metadata after creating the table is a useful thing;
  one use case I can think of is a tag that depends on the current state of the table data.
• if you're referring to the userMetadata field I found in the standard, it's less useful for us at the moment, but I can't tell now if we might need it later, we're still in the design phase.