GC improvements

[ghost]

From ipfs/infra@63887a6#commitcomment-14734054 in the context of ipfs/infra#125

@jbenet added a note a day ago

do we need the period? isnt the watermark enough?

@lgierth commented on 63887a6 a day ago

This is an old commit -- it ended up as 20m in master: e177f10

Periodic GC is run regardless of the GCPeriod option being present, from what I understand: https:/rht/go-ipfs/blob/48a33ffb673fb3eaf6d20cd7ca6bed04426a2abb/cmd/ipfs/daemon.go#L492-> L507

We need periodic GC because for now only ipfs cat maybe-runs GC.

@rht commented on 63887a6 a day ago

The default value for GCPeriod is set here.

We need periodic GC because for now only ipfs cat maybe-runs GC.

The latter because surveying the entire repo for each core.Resolve is too expensive.

@jbenet commented on 63887a6 12 hours ago

The latter because surveying the entire repo for each core.Resolve is too expensive.

It should definitely happen on writes.

Also, we should have a way of not running GC again in the case that:

• above gc watermark
• runs gc once
• gc does not drop storage below watermark
• another write happens (no pinset removes)
• running gc again will waste a ton of read time
• another write should not trigger gc

MaybeGC could track when new unpinned things could happen:

• set a dirty bit when things removed from pinset (things become cached)
• set a dirty bit when things new writes without adding pins (new things cached)
• maybeGC skips GC if dirty bit not set

[whyrusleeping]

@lgierth update here?