-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ipfs files concat [ <local paths> | <cids> ] #9177
Comments
My take is: hard-error on directories, support only files and pipes. Just like |
I put together a test repo using js-unixfs to show how concat could work under the hood with building up nodes from several sub nodes. https:/RangerMauve/js-ipfs-stitch-test/ Agreed that directories should be an error. I don't think we can |
Another high-level API, which would be super useful, and essentially becomes easy to support, given the core It could be a subcommand: Maybe there's two subcommands: This just adds a common first step that would often be needed before using |
@ikreymer too complex. You'd simply:
|
yeah, I guess could live with that, was just thinking the separate split file makes for an easier user API, especially if to be supported in libraries as well as CLI, and maybe dealing with hundreds of split points.. |
I've implemented a small library in JS that includes |
Wrote something in go: https:/anjor/unixfs-cat/blob/main/unixfs_cat.go Happy to work more on it if it's useful/along the lines of the thinking here. |
We are missing a high level API for concatenating existing UnixFS files into bigger ones.
Having it would allow for improved deduplication in scenarios when bigger archives in formats like WARC (https://webrecorder.net) consist in big part of smaller files that are already on IPFS, allowing for CID/DAG reuse.
Use cases
ipfs object patch append-data
Proposed design
Add
concat
command toipfs files
that accepts two or more UnixFS-compatible DAGs and returns a CID that is a logical concatenation of all DAGs.FAQ / Open questions
We need to agree how to handle edge cases, below are my initial ideas,
feedback on ergonomics and potential implementation caveats is appreciated
What happens when passed DAGs are all files?
Should this support directories? It opens additional questions:
The text was updated successfully, but these errors were encountered: