Brim ZJSON Update

[jameskerr]

Closes #1110

History

The zealot client has always simply passed through the raw json objects from zqd to Brim. This made the Brim code responsible for making objects to access and present the data.

In the beginning we used models/Log.js to represent a "record" of data. That class became outdated as we discovered the potential of our zng data model. We partially migrated to brim/record.js and brim/field.js to represent these new concepts, but all these classes were created without enough maturity to properly design them. Steve's recent changes to the zjson format and our migration to typescript have presented us an opportunity to redesign.

Also, Flow and the old zjson did not allow us to properly type annotate the expected shape of the payloads.

ZQD and ZJSON Type Annotations

The zjson format and the zqd payloads are now fully typed in zealot and available in the Brim source. Access all the types like this:

import {zjson, zqd} from "zealot"

function doSomething(p: zqd.SearchRecords) {
  // ...
}

function munge(items: zjson.Items) {
// ...
}

zng Classes

In this PR and a new set of classes have been written to represent the zng data types in JavaScript. They are in zealot/zng/types and consist of:

• Array
• Primitive
• Record
• Set
• Union

The previous two iterations (Log and brim.record) of representing the data from zqd have been removed. Zealot also constructs these objects before returning them to it's caller (Brim).

They all have a common ZngClass interface that looks something like this:

export interface ZngClass<ValueType> {
  toString(): string
  getType(): string
  getValue(): ValueType
  isSet(): boolean
  serialize(): SerializedData
}

The Record type has more methods for accessing the data within it.

import {zng} from "zealot"

// Construct a record like so
const columns = [{name: "ts", type: "time"}, {name: "_path", type: "string"}]
const values = ["32412553.2332", "conn"]

const record = new zng.Record(columns, values)

record.get("ts") // => new zng.Primitive("time", "32412553.2332")
record.get("missing-column") // => !! throws error
record.try("missing-column") // => null
record.at(0) => // the data in the first column

Nested Records can be accessed with the get or try methods using the familiar dot syntax in the column name.

record.get("id.resp_h") // => zng.Primitive("addr", "198.162.0.1") 
record.get("id").get("resp_h") // => same as above

In the app now, we do not flatten the records we receive from zealot to display the main viewer. We keep them in their nested format and access them with the dot syntax. However, records do have a .flatten() method and it is only called in the log detail pane when displaying the table of fields.

Schema
To represent a "log type" or the "schema" key in the zjson, we now have the Schema class. It's simply a wrapper around the columns array of a record. It has a .flatten() method. That is called to save column show/hide preferences.

Field
This class is a little different from the previous brim.field() class. It simply attaches a name to the data.

const name = "delicious_fruit"
const data = new zng.Set("string", ["apple", "orange", "pear"])
const field = new zng.Field(name, data)

You can get at fields from a record object like so:

record.getField("uid") // => new zng.Field("uid", new zng.Primitive("string", "a1c2f..."))
record.tryField("nowhere") // => null
record.getFields() // => zng.Field[]
record.flatten().getFields() // => collapses all the nested records and makes an array of fields

Serialization

Since we often need to pass these objects over ipc, Field, Record, and all the other zng Classes have a serializable interface.

const data = record.serialize()
const rec = zng.Record.deserialize(data)
// bam

const data = field.serialize()
const field = zng.Field.deserialize(data)

Deno Tests

Zealot used to be a Deno project and used the Deno version of TypeScript. I learned early in this process that it's very hard to share code and types between a Deno project and a Node project. As a result:

• The zealot folder is a normal Node TypeScript project now
• Within zealot, the test_api/ folder is where the integration tests with zqd are run using deno test on the bundled, es module version of the zealot code located in zealot/dist/index.es.js. (the js module ecosystem...🤯)
• To avoid a bunch of duplicate dependencies, zealot depends on some of brim's deps like typescript and jest. It uses rollup to bundle the commonjs and es module versions of the package.

P.S. Sorry Mason for such a big diff...again

[jameskerr]

I bet the tests are failing because #1109 went into the master zqd branch, but is not yet in the zjson zq branch.

[mason-fish]

Overall this is fantastic!! I left a few questions/comments that I don't consider to be blocking but am curious of your feedback.

[mason-fish]

Should we put PrimitiveField in here as well? It stands out to me to be defined within Brim instead

[mason-fish]

do you mind renaming the parameter to cell as well?

[mason-fish]

similar comment to before, mind matching the prop name to the type? log -> record

[mason-fish]

Nice, this new serialize/deserialize model is also what I had in mind and am using in the electron update PR 👍

[mason-fish]

I DO like the try/get vs get/mustGet interface. Though I think I'd prefer we just use all of one or the other instead of both

[mason-fish]

👍

[jameskerr]

After @mason-fish and I just spoke, we decided to:

• Make it an ongoing effort to replace local variables named "log" with "record" and "field" with "cell" (when appropriate).
• Update the Current selectors to use "get" and "try" instead of "get" and "mustGet", inverting the meaning of "get".