fs: split out include/exclude into new FilterFS

[jedevc]

This patch creates a new FilterFS to pair with a new FilterOpt, which contains the old WalkOpt parameters.

Essentially, this splits the functionality of the old FS into a new FS and the FilterFS. The new FS implemenation simply performs operations over a specified local filesystem, while the FilterFS wraps an existing FS implementation to apply filtering patterns.

This allows the same logic for filtering to apply to any underlying filesystem, for example, the StaticFS and MergeFS implementations in BuildKit (see moby/buildkit#4094).

To do this, we also make some reasonably substantial changes to the developer-facing API. We need to use fs.WalkDirFunc instead of the old filepath.WalkFunc, which is required to preserve the lazy stat() semantics from b9e22fc.

Existing implementations and callers of FS should fairly easily be able to update to support the new API, which just requires updating to support a new "path" parameter, and modify the signature to be fs.WalkDirFunc (which may actually improve performance depending on the exact usage).

[tonistiigi]

This Walk() method here seems out of place. The signature for walk should be Walk(fs, .., cb). FS should implement something that make this possible, eg. ReadDir(name) and f.ReadDir()

[jedevc]

The current semantics are difficult to express if we change to that model.

Imagine the following ExcludePatterns: (/foo, !/foo/x).

Currently, if we Walk this filesystem, the callback will only be called for /foo if /foo/x exists, otherwise it gets skipped.

If we try and express ReadDir like this, then ReadDir("/") will need to check for the existence of /foo/x to know if /foo should be returned. This would be quite a major rewrite of the include/exclude patterns logic, not sure if you think this would be worth it?

[tonistiigi]

Maybe the logic then needs to be that filter walks whole baseFS internally either on constructor/function-call or on the first use.

The current semantics are difficult to express if we change to that model.

I see, but I don't see the benefit then of defining a FilterFS if it doesn't actually satisfy the FS interface.

[jedevc]

fsutil/fs.go

Lines 17 to 20 in 36ef4d8

	type FS interface {
	Walk(context.Context, filepath.WalkFunc) error
	Open(string) (io.ReadCloser, error)
	}

It does though? fsutil.FS only defines Open and Walk for now.

I think switching to be more in match with io/fs is a good idea, but I'd propose that's out of scope for this PR - unless we want to make the argument to do a large PR to completely rework the internals of this package.

[tonistiigi]

I'm still not quite sure why implement statFile with Walk instead of defining Stat() (and then maybe removing path from Walk) but that discussion can be left for some follow-up.

[jedevc]

@tonistiigi is there anything blocking this? Happy to take another pass (or prep a follow-up) if there's anything needed here.