zqd: Support pcap ingest for archive stores #1450

mattnibs · 2020-10-09T16:06:56Z

Because of differences in how an archive ingests data implement
new ingest.PcapOp (ingest.archivePcapOp) for archive-backed spaces.

In order to take advantage of the streaming nature of ingest, abandon
the brute force snapshort approach taken with file stores. Instead,
temporary log files created by each pcap analyzer is are tailed and
written to the archive store as new data arrives (see fs.TFile and
Fs.DirWatcher). As a result for pcap ingest on archive stores, new
data should constantly arriving.

In order to continue pcap ingest support for soon-to-be deprecated
filestore spaces, keep around existing ingest.PcapOp (renamed to
ingest.legacyPcapOp).

closes #1132

pkg/fs/tail.go

pkg/fs/tail_test.go

zqd/ingest/legacypcap.go

Because of differences in how an archive ingests data implement new ingest.PcapOp (ingest.archivePcapOp) for archive-backed spaces. In order to take advantage of the streaming nature of ingest, abandon the brute force snapshort approach taken with file stores. Instead, temporary log files created by each pcap analyzer is are tailed and written to the archive store as new data arrives (see fs.TFile and Fs.DirWatcher). As a result for pcap ingest on archive stores, new data should constantly arriving. In order to continue pcap ingest support for soon-to-be deprecated filestore spaces, keep around existing ingest.PcapOp (renamed to ingest.legacyPcapOp). closes #1132

mattnibs · 2020-10-12T16:45:02Z

I'm noticing a race with fs.DirWatcher that appears to show up in linux tests not mac: DirWatcher.Stop() can be called when fsnotify has not yet got the event for a created file. I'm going to add an additional poll call once Stop() is called to scan and emit any existing files that were not caught.

driver/driver.go

pkg/fs/tail.go

alfred-landrum

Before this gets merged, I'd like to try this out with Brim, by temporarily making archive store the default.

zqd/ingest/archivepcap.go

alfred-landrum · 2020-10-13T15:57:05Z

zqd/ingest/archivepcap.go

+ return atomic.LoadInt64(&r.recordsRead)
+}
+
+// Snap for archivePcapOp is functionally useless. It is only here to satisfy


When we can run Brim against this branch, I'm curious how this impacts the live updating. I thought Brim triggered updating its view during pcap by looking for new snapshots, but I haven't looked at that code in while.

As discussed offline, it is. Without an incrementing snapshot brim will not know new data is available to query. As discussed, this should really be keeping track of record count (and bytes) written to the archive, return it in the status updates, and Brim should use this to establish if new data is available. Will file an issue.

alfred-landrum · 2020-10-13T15:57:54Z

zqd/ingest/logtailer_test.go

+ dr *logTailer
+}
+
+func TestLogTailer(t *testing.T) {


Just want to register that I'm +1 on using testify/suite.

It is definitely very useful for certain circumstances.

Co-authored-by: Alfred Landrum <[email protected]>

pkg/fs/tail.go

zqd/ingest/logtailer.go

Co-authored-by: Noah Treuhaft <[email protected]>

alfred-landrum

I pulled this branch, merged to master, altered zqd to choose archive store by default, and then ran Brim against it. I then uploaded the 'all.pcap' file & watched the archive tsdir's get created during import. (as noted, the live updates in Brim aren't working yet, but that will happen soon via #1488 ). I was able to verify that the resulting archive had the same count as the doing the process with the filestore. So 👍 from me.

mattnibs · 2020-10-14T14:41:25Z

Thanks for the thorough review @alfred-landrum

(as noted, the live updates in Brim aren't working yet, but that will happen soon via #1448 ).

You mean #1488, correct?

alfred-landrum · 2020-10-14T14:42:10Z

Thanks for the thorough review @alfred-landrum

(as noted, the live updates in Brim aren't working yet, but that will happen soon via #1448 ).

You mean #1488, correct?

oops, yes, fixed the comment!

mattnibs force-pushed the archive-pcap-ingest branch 4 times, most recently from 5f8a9c2 to 661152b Compare October 9, 2020 16:13

mattnibs requested a review from a team October 9, 2020 16:14

alfred-landrum reviewed Oct 9, 2020

View reviewed changes

mattnibs force-pushed the archive-pcap-ingest branch from 661152b to 45f000e Compare October 11, 2020 18:46

mattnibs force-pushed the archive-pcap-ingest branch from 45f000e to 55095e1 Compare October 11, 2020 18:56

pr feedback

6869f9d

mattnibs force-pushed the archive-pcap-ingest branch from 3a2c8f7 to 6869f9d Compare October 12, 2020 15:00

mattnibs force-pushed the archive-pcap-ingest branch from 1802110 to e2142bd Compare October 12, 2020 21:02

fix fs.DirWatcher race

82eafe7

mattnibs force-pushed the archive-pcap-ingest branch from e2142bd to 82eafe7 Compare October 12, 2020 21:07

alfred-landrum reviewed Oct 13, 2020

View reviewed changes

driver/driver.go Outdated Show resolved Hide resolved

pkg/fs/tail.go Show resolved Hide resolved

pkg/fs/tail.go Outdated Show resolved Hide resolved

pr feedback

420db3b

mattnibs force-pushed the archive-pcap-ingest branch from d8ce701 to 420db3b Compare October 13, 2020 13:28

mattnibs requested a review from alfred-landrum October 13, 2020 13:28

alfred-landrum reviewed Oct 13, 2020

View reviewed changes

mattnibs and others added 4 commits October 13, 2020 12:13

Update zqd/ingest/archivepcap.go

bb76eb3

Co-authored-by: Alfred Landrum <[email protected]>

pcap counter

c13a10e

pr feedback

a14a875

fix tests

61c96de

nwt approved these changes Oct 13, 2020

View reviewed changes

pkg/fs/tail.go Outdated Show resolved Hide resolved

zqd/ingest/logtailer.go Outdated Show resolved Hide resolved

mattnibs and others added 2 commits October 13, 2020 17:45

Update zqd/ingest/logtailer.go

9f142cd

Co-authored-by: Noah Treuhaft <[email protected]>

Update pkg/fs/tail.go

6f1f4d8

Co-authored-by: Noah Treuhaft <[email protected]>

alfred-landrum approved these changes Oct 14, 2020

View reviewed changes

mattnibs merged commit 5c0f1f5 into master Oct 14, 2020

mattnibs deleted the archive-pcap-ingest branch October 14, 2020 20:53

philrz mentioned this pull request Oct 15, 2020

zqd: Support archive space pcap ingest support #1132

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

zqd: Support pcap ingest for archive stores #1450

zqd: Support pcap ingest for archive stores #1450

mattnibs commented Oct 9, 2020

mattnibs commented Oct 12, 2020

alfred-landrum left a comment

alfred-landrum Oct 13, 2020

mattnibs Oct 13, 2020 •

edited

Loading

mattnibs Oct 14, 2020

alfred-landrum Oct 13, 2020

mattnibs Oct 13, 2020

alfred-landrum left a comment •

edited

Loading

mattnibs commented Oct 14, 2020

alfred-landrum commented Oct 14, 2020

zqd: Support pcap ingest for archive stores #1450

zqd: Support pcap ingest for archive stores #1450

Conversation

mattnibs commented Oct 9, 2020

mattnibs commented Oct 12, 2020

alfred-landrum left a comment

Choose a reason for hiding this comment

alfred-landrum Oct 13, 2020

Choose a reason for hiding this comment

mattnibs Oct 13, 2020 • edited Loading

Choose a reason for hiding this comment

mattnibs Oct 14, 2020

Choose a reason for hiding this comment

alfred-landrum Oct 13, 2020

Choose a reason for hiding this comment

mattnibs Oct 13, 2020

Choose a reason for hiding this comment

alfred-landrum left a comment • edited Loading

Choose a reason for hiding this comment

mattnibs commented Oct 14, 2020

alfred-landrum commented Oct 14, 2020

mattnibs Oct 13, 2020 •

edited

Loading

alfred-landrum left a comment •

edited

Loading