Proposed curio api

[hannahhoward]

Proposes an HTTP API to curio that could be used by either Storacha or another hot storage market provider for PDP proofs

Preview

[magik6k]

Might require a UCAN header for auth

[magik6k]

Also for scalability it would be a good idea to have the main endpoint tell you where to go with the data.

Unfortunately redirecting PUT/POST with 3xx is really wonky (pretty much broken with the Go http.Client), so it might just require some two-endpoint setup.

[hannahhoward]

ah ok. I get it -- our main endpoint is like this too! :)

[magik6k]

Maybe link to some spec which lays out how those proof-sets look like / what they are

[hannahhoward]

Truth be told I'm just betting based off the PDP service contract doc

[magik6k]

IPNI is pretty much implemented now, with the following schema that coordinates it on the curio side:

-- Table for storing IPNI ads
CREATE TABLE ipni (
    order_number BIGSERIAL PRIMARY KEY, -- Unique increasing order number
    ad_cid TEXT NOT NULL,
    context_id BYTEA NOT NULL, -- abi.PieceInfo in Curio
    -- metadata column in not required as Curio only supports one type of metadata(HTTP)
    is_rm BOOLEAN NOT NULL,

    previous TEXT, -- previous ad will only be null for first ad in chain

    provider TEXT NOT NULL, -- peerID from libp2p, this is main identifier on IPNI side
    addresses TEXT NOT NULL, -- HTTP retrieval server addresses

    signature BYTEA NOT NULL,
    entries TEXT NOT NULL, -- CID of first link in entry chain

    unique (ad_cid)
);

CREATE TABLE ipni_head (
    provider TEXT NOT NULL PRIMARY KEY, -- PeerID from libp2p, this is the main identifier
    head TEXT NOT NULL, -- ad_cid from the ipni table, representing the head of the ad chain

    FOREIGN KEY (head) REFERENCES ipni(ad_cid) ON DELETE RESTRICT -- Prevents deletion if it's referenced
);

-- This table stores metadata for ipni ad entry chunks. This metadata is used to reconstruct the original ad entry from
-- on-disk .car block headers or from data in the piece index database.
CREATE TABLE ipni_chunks (
    cid TEXT PRIMARY KEY, -- CID of the chunk
    piece_cid TEXT NOT NULL, -- Related Piece CID
    chunk_num INTEGER NOT NULL, -- Chunk number within the piece. Chunk 0 has no "next" link.
    first_cid TEXT, -- In case of db-based chunks, the CID of the first cid in the chunk
    start_offset BIGINT, -- In case of .car-based chunks, the offset in the .car file where the chunk starts
    num_blocks BIGINT NOT NULL, -- Number of blocks in the chunk
    from_car BOOLEAN NOT NULL, -- Whether the chunk is from a .car file or from the database
    CHECK (
        (from_car = FALSE AND first_cid IS NOT NULL AND start_offset IS NULL) OR
        (from_car = TRUE AND first_cid IS NULL AND start_offset IS NOT NULL)
    ),

    UNIQUE (piece_cid, chunk_num)
);

Now, IPNI likes larger ads, so ideall storacha would create aggregate ads for multiple pieces; we can extend ipni_chunk to support reading from storacha-stored pieces (though really technically just the piececid works fine there)

[magik6k]

We definitely need to define how authorization works on this endpoint. This can't just be entirely open.

Also should define the lifecycle of the uploaded data somehow:

• How long is it expected to stick around in storage after upload before being included in a proof-set? When should the data be removed if not added to a proof set?
  • Signalling for expected indexing with IPNI / ipfs-type (trustless gateway/bitswap) retrievals, and who can retrieve the piece?
• What is the contract for retrieval - is it retrievable atomically when the notify hook is called? After inclusion in a proof set?

[magik6k]

Also what are the size bounds for pieces that you expect curio to support? We can support even very large pieces (100G+), but I don't think a client-push model is a good idea above 1GB, where managing short-term buffers becomes a real concern, and download retry becomes non-optional