Lotus Chain Snapshots - Full Chain Snapshots

[travisperson]

We are starting some work to improve the way snapshots are create and managed to improve SLA and we want to gather some feedback from the community.

There are currently two snapshot types provided:

Name	End height	Message start height	State start height
Lightweight	Current block	Current block - 2000 blocks	Current block - 2000 blocks
Full chain	Current block	Genesis block	Current block - 2000 blocks

https://lotus.filecoin.io/lotus/manage/chain-management/#syncing

Background

The Full chain snapshots are currently created in a very crude manner. They are done using the same chain export that is used to create the Lightweight snapshots. Because the Full chain snapshot contains all messages, this results in 99% of duplicate data exported when compared to a previous export (eg all messages from genesis to current block are exported, with only 24 hours of new messages added from the previous export).

The primary purpose of the "Full chain" snapshot is to provide all messages. This allows a user to then recompute all state if they need.

We are looking into changing the way we create these Full chain snapshots to make for a more efficient process. There are a few ways this can be done, but most likely this will result in the Full chain snapshot loosing the 2000 blocks of state data, and instead being only messages (and maybe block headers).

The lightweight snapshot already contains all blockheaders, plus the 2000 latest state-roots and the companying messages.

To replace the functionality of Full chain snapshots, we want to provide a way for users to inflate / backfill a Lightweight snapshot with the missing messages.

For example; this might look like

• All Filecoin messages being provided over IPFS. A user could then iterate over all block headers (which are included in lightweight snapshots), and fetching all messages.
• A single car of just all messages (similar to Full chain, but no state roots)
• A bunch of car files, one for each tipset, or maybe every 24h

---

If you are using the Full chain snapshot we would like to hear from you to understand your use-case, and the impact these changes might have on you. Please respond to this discussion post.

[rickiey]

I hope to keep state roots, it's useful. The way to provide a variety of build full chain node is always good.
An import method I envision:
Every certain level is made into a separate snapshot (eg, 10000 height, including state roots), if need the last 50000 height, I import the most recent 5 snapshots (10,000 height per snapshot), I need more time to import more snapshots.

[travisperson]

We are not removing state root data completely. We are going to continue to produce the lightweight snapshot, which is all the data required to bootstrap a node into the network.

We plan to continue to produce these snapshots every 2 hours. The snapshots contain ~16 hours of the most recent state-roots in their entirety.

[travisperson]

Every certain level is made into a separate snapshot (eg, 10000 height, including state roots), if need the last 50000 height, I import the most recent 5 snapshots (10,000 height per snapshot), I need more time to import more snapshots.

This is something i've wanted to look into doing as well, but it's currently not on our immediate roadmap. Right now our primary focus to to ensure the continue existence of snapshots to allow people to bootstrap into the network. We may later provide access to past snapshots that will provide a way for people to selectively fetch different parts of the chain history.

[mur-me]

@travisperson, speaking about the way how to do this by N blocks or day, glif.io team have seen an example from the Sentinel team and their lilium project.
They store info on s3 bucket separated by days, link - https://lilium.sh/data/dumps/
Probably, it can be useful idea.

[strahe]

We have implemented the first two internally, but we use s3 and minio.

Is there a way to export only the newly added data, not the full state tree？

[travisperson]

There is no supported way to export only newly created data right now. It might be possible to implement something like this by recording all write to the datastore during state computation. However, I think this might be a bit tricky do to the way lotus handles it's execution. I haven't really dug into it much though.

[mur-me]

From my point of view, processing of full chain snapshot already takes around several days for big instance and 2000 or little bit more latest blocks are okay to be processed by node itself.
But it will be cool if we can somehow provide car files to lotus for the latest N days.

[travisperson]

From my point of view, processing of full chain snapshot already takes around several days for big instance and 2000 or little bit more latest blocks are okay to be processed by node itself.

I don't understand this statement, maybe my discussion topic wasn't very clear.

A snapshot is required to join the Filecoin network in less than 24 hours. It is not possible to sync the entire chain (takes months). The lightweight snapshot (better described as a minimal snapshot) provides this functionality. We plan to keep this snapshot and produce it every 2 hours. The snapshot takes about 1 hours to produce, and some additional time to upload and then download for users. This means there might be a 4 hours window of new tipsets to process, which is about 480. Tipsets take ~3 seconds to process.

There is another snapshot that is produced: The full chain snapshot, which is exactly like the lightweight snapshot, but contains ALL messages back to genesis; see the table. It does not contain all state despite its name, it's not a "full" snapshot. We want to stop producing this snapshot, and find another way to provide messages to people who want to download them.

This mean there will no longer be a single snapshot that can give you all the chain messages, plus the required data to join the network. These will instead be two different things.

[LexLuthr]

I am archiving this discussion as there has been no new input from the author for more than 7 days.