v0.24.0 Space Migration doesn't work with custom data directory

[jameskerr]

To reproduce, open version 24, change the data directory in the preferences, import a file, then checkout the main branch and start the app. You'll see these error messages.

And in the console:

Is it possible to migrate the data when some spaces are in one data directory, while others are elsewhere? I did an experiment where one space was in the default data dir while another space was in a custom data dir. The first space got migrated to a pool, while the second one threw the error above.

[philrz]

In preparation to verify the fix, here's a detailed repro. As shown in the attached video, I start from a v0.24.0 environment in its default config and import one pcap, which on macOS populates a directory below ~/Library/Application Support/Brim/data/spaces. I then went to Preferences and set my Data Directory setting to ~/datadir and imported yet another pcap, which populates a directory below that destination.

Import.mp4

As noted above, at this point if I upgrade to a release that lacks the Brimcap with the fix in the linked PR (such as Brim commit 7b0b6df, using artifact https://storage.googleapis.com/brimsec-releases/brim/v0.25.0-prerelease-7b0b6df7.0/macos/Brim-0.25.0-prerelease-7b0b6df7.0.dmg) and allow the Space migration to proceed, we see the expected "Some spaces were not migrated" error. The data that I'd imported to the default Data Directory got migrated ok, but for the data I'd imported after changing to the custom Data Directory remained behind in that directory. A Pool is seen as created in anticipation of its migrated data, but it's empty.

Repro.mp4

[philrz]

Verified with a test artifact https://storage.googleapis.com/brimsec-releases/brim/v0.25.0-prerelease-24bfd33b.0/macos/Brim-0.25.0-prerelease-24bfd33b.0.dmg which is based on Brim commit b174140 but pointed at Brimcap commit 4f871b4 that has the benefit of the fix in the linked PR.

As shown in the attached video, having started from the same v0.24.0 state as in the repro above, now when upgrading to the newer release and proceeding with Space migration, the contents of both Spaces are successfully migrated to Pools. The directory and its contents for the Space below the custom Data Directory is deleted as part of the migration process, just as we see when the Space is coming from the default Data Directory.

Verify.mp4

While verifying this, I realized that the Data Directory setting may no longer have a purpose, so I've opened separate issue #1788 to pursue that.

Thanks @mattnibs!