IPLD Importers - endeavor notes

[jbenet]

The "IPLD Data Importers" endeavor is the effort to create IPLD (or even unixfs) data structures out of various kinds of data.

Motivation:

The idea is that chunking POSIX files based on things like Rabin Fingerprinting is good but not great, and that many times we can be much smarter about the files, chunking them by taking into account the file's internal structure. Doing this for a few of the most important file formats may see huge dedup winnings (tar archives, VM images, containers, ...).

Examples:

• If importing a .tar archive into ipfs, import it s.t. (a) each tar header is an object, and (b) each file is fed back into the importer(s).
• If importing a video, maybe we can chunk intelligently around (a) container formats, (b) key frames, etc.
• If importing a VM image, maybe we can chunk s.t. we import the filesystem itself correctly.

Notes:

• IPLD Data Importing - Set of Importers js-ipfs#41
• https:/ipfs/js-ipfs-unixfs-engine
• go-ipfs
  • https:/ipfs/go-ipfs/tree/master/importer
  • https:/ipfs/go-ipfs/tree/master/tar
  • https:/ipfs/go-ipfs/tree/master/unixfs

Diagram:

# meta importer - uses the other importers

 +------+           +----------+           +------------+
 |      |           |          |           |            |
 |      |           | Pick an  |           | IPLD graph |
 | file | --------> | Importer | --------> | for file.  |
 |      |           | for file |           |            |
 |      |           |          |           |            |
 +------+           +----------+           +------------+

# file type specific importers

 +------+           +----------+           +------------+
 |      |           |          |           |            |
 |      |           | TAR IPLD |           | IPLD graph |
 | .tar | --------> | IMPORTER | --------> | rep a .tar |
 |      |           |          |           | (smartly)  |
 |      |           |          |           |            |
 +------+           +----------+           +------------+


 +------+           +----------+           +------------+
 |      |           |          |           |            |
 |      |           | MP4 IPLD |           | IPLD graph |
 | .mp4 | --------> | IMPORTER | --------> | rep a .mp4 |
 |      |           |          |           | (smartly)  |
 |      |           |          |           |            |
 +------+           +----------+           +------------+

# Info N/A. Use "good enough" importer

 +------+           +----------+           +------------+
 |      |           |          |           |            |
 | unkn |           | RABIN FP |           | IPLD graph |
 | file | --------> | CHUNKING | --------> | for file w |
 |      |           | IMPORTER |           | rabin chnk |
 |      |           |          |           |            |
 +------+           +----------+           +------------+