You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In #8103 we saw users attempting to pin "large" data sets (in this case, 90GiB) reach a hangup during the pinning process.
If the size of the dataset you're attempting to pin is larger then GCThreshold, the chunks will be immediately garbage collected when ctl+c'ing or terminating the pin. This defeats the purpose of IPFS as users should be able to "resume" downloading chunks from where they left off if pinning is interrupted.
Instead of letting users pin potentially massive amounts of data which will unknowingly be destroyed, maybe a warning when this condition exists in ipfs pin would be beneficial?
ipfs pin /ipns/blah
warning: pinned dataset is larger than your GCThreshold, data will need to be re-downloaded if interrupted!
It would be better to guide users to increase their GCThreshold instead of silently letting them believe that IPFS can't resume pinning.
#8412 is another alternative strategy to fix this (Chunk garbage collection minimum time) #3121 seems to be another alternative strategy for this (best effort pins)
The text was updated successfully, but these errors were encountered:
I don't believe this is feasible: it is not possible to tell the size of various non-UnixFS DAGs without fetching them. DAG-CBOR won't have size in the root node, so this warning would only work for UnixFS data, and the behavior of ipfs pin should be codec-agnostic as much as possible.
iiuc there is a way of doing best-effort pins with MFS – see #3121 (comment)
Closing this and suggest continuing in #3121 / #8412 where we discuss approaches that are codec-agnostic.
Checklist
Description
In #8103 we saw users attempting to pin "large" data sets (in this case, 90GiB) reach a hangup during the pinning process.
If the size of the dataset you're attempting to pin is larger then GCThreshold, the chunks will be immediately garbage collected when ctl+c'ing or terminating the pin. This defeats the purpose of IPFS as users should be able to "resume" downloading chunks from where they left off if pinning is interrupted.
Instead of letting users pin potentially massive amounts of data which will unknowingly be destroyed, maybe a warning when this condition exists in
ipfs pin
would be beneficial?It would be better to guide users to increase their GCThreshold instead of silently letting them believe that IPFS can't resume pinning.
#8412 is another alternative strategy to fix this (Chunk garbage collection minimum time)
#3121 seems to be another alternative strategy for this (best effort pins)
The text was updated successfully, but these errors were encountered: