[Python] Promote usage of the Arrow PyCapsule Protocol (for the C Data Inteface)

[jorisvandenbossche]

#37797 added a Python layer for working with the C Data Interface through capsules and defined dunder methods, described at https://arrow.apache.org/docs/dev/format/CDataInterface/PyCapsuleInterface.html.

Currently, libraries are using pyarrow's _export_to_c/_import_from_c semi-private methods to get the C Data Interface structs / construct pyarrow object from the structs.
With the formalized PyCapsule protocol, the idea is that this communication of the structs now happens with defined dunder methods __arrow_c_schema__ / __arrow_c_array__ / __arrow_c_stream__ to get the data as a C Data Interface struct wrapped into a capsule instead as raw integer (export). In addition, the pyarrow constructors (pa.array(..), pa.table(..), ..) will now also accept any object implementing the relevant dunder method (import).

This new Python-level interface has the benefits to be 1) more robust (not passing around pointers as integers, the capsule will ensure the data is released when an error would happen in the middle), and 2) not tied to pyarrow the library. This means that other libraries can also start accepting any Arrow-compatible data structure, instead of harcoding support for pyarrow and using pyarrow's _export_to_c.

We want to promote other libraries to start supporting this protocol as well, which can mean:

• Add the dunders on their own objects where appropriate (so that other libraries will recognize your data structures as holding Arrow-compatible data)
• Where ingesting data (eg where there might be specific pyarrow support right now), recognize objects that implement the dunder methods

Linking some issues here: pola-rs/polars#12530, data-apis/dataframe-api#279, pandas-dev/pandas#56587, OSGeo/gdal#9043 and OSGeo/gdal#9132

In addition, there are also some steps we can take within the Arrow project itself to further promote this:

• ADBC support: feat(python/adbc_driver_manager): export handles and ingest data through python Arrow PyCapsule interface arrow-adbc#1346
• nanoarrow support: feat(python/adbc_driver_manager): export handles and ingest data through python Arrow PyCapsule interface arrow-adbc#1346
• [Python][Docs] Document the PyCapsule protocol in the "extending pyarrow" section of the Python docs #39196
• [Python][Java][Docs] Update the "Integrating PyArrow with Java" documentation section to not use _import_from_c/_export_to_c #39197
• [Python][R][Docs] Update the "Integrating PyArrow with R" documentation section to not use _import_from_c/_export_to_c #39198

There are also some other follow-up discussions about certain aspects of the protocol or APIs we provide in pyarrow:

• [Python] RecordBatchReader constructor from stream object implementing the PyCapsule Protocol #39217
• [Python] Public API to consume objects supporting the PyCapsule Arrow C Data Interface #38010
• [Python] Arrow PyCapsule Protocol: standard way to get the schema of a "data" (array of stream) object? #39689

[pitrou]

cc @wjones127

[kylebarron]

I made a feature request for duckdb here: duckdb/duckdb#10716

[kylebarron]

I've been working a bit to promote the protocol; here's a running tally:

implemented: (at least partially, return objects with pycapsule protocol and/or check for existence of protocol in constructors)

• pandas: ENH: support the Arrow PyCapsule Interface on pandas.DataFrame (export) pandas-dev/pandas#56587, ENH: support the Arrow PyCapsule Interface on pandas.Series (export) pandas-dev/pandas#59518, ENH: support Arrow PyCapsule Interface on Series for export pandas-dev/pandas#59587, ENH: add basic DataFrame.from_arrow class method for importing through Arrow PyCapsule interface pandas-dev/pandas#59696
• ibis: feat(pyarrow): support Arrow PyCapsule interface on ibis.Table objects ibis-project/ibis#9143, feat(pyarrow): support __arrow_c_schema__ on ibis.Schema objects ibis-project/ibis#9665
• pyarrow
• nanoarrow
• arro3
• lonboard
• pyogrio
• gdal Python bindings: add a ogr.Layer.WriteArrow() method consuming __arrow_c_stream__ or __arrow_c_array__ interfaces OSGeo/gdal#9133, Python bindings: implement __arrow_c_stream__() interface for ogr.Layer OSGeo/gdal#9043
• polars: feat(python): Implement Arrow PyCapsule Interface for Series/DataFrame export pola-rs/polars#17676 and feat(python): Support PyCapsule Interface in DataFrame & Series constructors pola-rs/polars#17693
• narwhals: feat: Support Arrow PyCapsule Interface for export narwhals-dev/narwhals#786
• quak: feat: Support for Arrow PyCapsule interface manzt/quak#23
• datafusion: Add PyCapsule support for Arrow import and export datafusion-python#825, Implement Arrow PyCapsule Interface datafusion-python#752
• duckdb: Feature Request: Python: Support Arrow PyCapsule Interface & remove `pyarrow` dependency duckdb/duckdb#10716, [Arrow] Support consuming an "arrow_array_stream" PyCapsule duckdb/duckdb#13386, [Arrow] Accept objects that provide the __arrow_c_stream__ producer method duckdb/duckdb#13425, [Arrow] Support producing an "arrow_array_stream" PyCapsule duckdb/duckdb#13418 (comment), to be released in duckdb 1.1
• geopandas: ENH: Add export to GeoArrow geopandas/geopandas#3219, ENH: GeoArrow import geopandas/geopandas#3301

in-progress:

• vegafusion: Read Arrow C Stream from Arrow PyCapsule Interface vega/vegafusion#501

issues:

• fastexcel: Implement Arrow PyCapsule Interface & make pyarrow optional dependency ToucanToco/fastexcel#268
• daft: Support for Arrow PyCapsule Interface Eventual-Inc/Daft#2504
• lance: Support for Arrow PyCapsule Interface lancedb/lance#2630
• lightgbm: [python-package] Adding support for polars for input data  microsoft/LightGBM#6204 (comment)
• pillow: Arrow Support python-pillow/Pillow#8329
• altair: Support Arrow PyCapsule Interface vega/altair#3568
• h3ronpy: Adopt Arrow PyCapsule Interface and remove required pyarrow dependency nmandery/h3ronpy#55