-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Road to a unixfs-like in golang with ADLs #258
Comments
in terms of getting to a working implementation, we can defer the two 'bigbytes' items, because the existing https:/ipfs/go-ipfs-chunker interface provides
|
We get a lot of value out of having just the directory/map stuff done. I would think it's good to focus fire on that first. Yes. There's probably no harm in writing the file chunking stuff into the current I think we'll need some "big bytes" interface work (e.g. |
Yep, i agree that we do eventually need a cleaner interaction with The main thing I was pointing out is that initially it'll probably be desirable to use the existing set of chunker implementations, and since those are not fully streaming, it probably is another nudge that we can push off figuring out the final streaming designs since we'll end up with that crutch anyway for a while. |
@guseggert: you had an issue/comment where you detailed some of the problems with "huge directories" right? I'd like to link that to this as I believe it will be alleviated. I couldn't find it in my 5 minutes of searching, so maybe I'm imagining things. |
Yes the issue is here: ipfs/kubo#8455 |
There's a https:/ipfs/go-unixfsnode/ repo nowaday, which I believe has even more of these features. |
This issue is a short checklist/roadmap for the major areas of work needed in order to make something "unixfs-like" happen in golang and be implemented almost entirely in ADLs.
What?
"unixfs-like"
By "unixfs-like", I mean both of: we can reimplement a feature-parity unixfsv1 (like what go-ipfs does) this way; and we would also reuse the majority of these code and components in building a "unixfsv2"; and most importantly, someone else could grab most of the components we'll need here and build their own custom evolution of what they consider a "unixfs-like" with minimal hurdles.
"implemented as ADLs"
By "implemented almost entirely in ADLs" I mean: there should be a few pieces of code which implement key functionality by making complex structures act "like a
map
-kind node" and "like abytes
-kind node". So: directories? Map. Files? Bytes. Even though either of them can be sharded? Yep. And that should, when composed, make a unixfs-like system almost trivial to assemble.The desirable outcomes of this are two (or three): one, it means things like pathing and traversals and selectors over them are already defined (because it's just "working over a map", etc); two, it's a very clear abstraction which allows for reuse when someone wants to make a new system. (Also, three, in the longer run -- we hope to make ADLs into a language-agnostic plugin system. But that is not today :) Today we are happy to implement it in one language, get something working, and plan to move on later.)
other sells
It's been suggested that we could significantly improve UX in other systems like the IPFS Gateways and their rendering of data by having this work done -- and make it so much easier to do that it becomes more likely for those improvements to actually get done. When unixfs systems (including unixfsv1 as a retrofit) are available as ADLs, it means we can build more consistent and reusable rendering technologies with a lot less code, less special cases, less complexity to explain, etc.
Roadmap
Without further ado, some checkboxes...
NodeBuilder
/NodeAssembler
)Node
)Not all of this work has to take place in this repo, and several parts of it can be started without blocking on other parts. It also doesn't all have to be done by the same people. This is just a tracking and roadmap-sharing issue.
The text was updated successfully, but these errors were encountered: