Shareable embeddables as output of "run query in background" (Make it slow)

[stacey-gammon]

I've been starting to wonder if we can combine this effort with some of the make it slow efforts to solve some of those issues.

The goal for the Make it slow project is to have embeddables that are taking a really long time requesting data, to allow the user to make this request in the background. Once the results are ready the embeddables need to know to not do the live query again but to switch to the saved queries. Having this logic everywhere, especially in the interpreter (live query vs search id), could be complicated. But if we already have this concept of a shareable embeddable that never hits Kibana or ES servers, then we could use that.

We would need this shareable generation to be available in the background though, so it'd have to be server side, and the results would be stored in a saved object for later retrieval (like reporting).

The UI side of things though, would solve all the same issues we have with the make it slow project. We don't want the user to be able to generate another live query by doing things like using an input control or changing the time range. So if we had an internal page that was a shareable "viewer", this could be the same UI we use when user's background queries are complete.

So here is what I'm thinking the flow could be something like:

• Embeddables that hit server side will expose an async method that is something like createShareable. The result of that function should be a new saved object that has the original embeddable's id, the new embeddable type, and the new embeddable's input parameters. This new embeddable will be "shareable" so for example, instead of having input that is { timeRange, filters }, it'll be { data } (might actually still need time and filters for UI purposes but changing it wouldn't affect the data that is shown).

** Actually... this might not work because of security. We can't store one user's search results in the .kibana index that another user could potentially see.

• On the client side, if an embeddable detects a really slow query, it can ask the user if they want to create a snapshot of the data in the background for later retrieval. It has a link to it's parent embeddable so we can use this to somehow to 1) make sure there is a single notification per dashboard, not one notification for every slow panel, and 2) have the output be the dashboard not the individual panel.
• If the user says "yes", create a snapshot, each Embeddable runs this createSnapshot functionality. Parent container will have to wait till all of its children are done creating their snapshots, then use that output to create its final shareable version of input.
• Creating this shareable will have to be done server side so the user can shut down the browser window, and come back later to retrieve it (since for "make it slow" we are assuming it could take days to get all the data needed for this snapshot). The Embeddable could register a task to do this. The task would run the query as a background search and keep checking periodically for it to be done. Once it's done, it would generate the shareable version, store it as a saved object with the data as output.

Anyway, still need to think through this one more, but I think this might help us move forward faster with both features.

cc @lukasolson @streamich

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[lukasolson]

The current line of thinking is that the data powering the "static" (or "shareable") version of the embeddable will be tied to a specific space, not to a specific user (which would require object-level security).

Other than that, this is actually very crystallizing to me. My initial thought is that this makes a lot of sense to implement this way. Obviously there are still a lot of details we will need to hammer out.

Other thoughts: Since a dashboard itself is an embeddable, its createShareable would simply loop over its children and invoke their createShareable... Is that correct?

[stacey-gammon]

The current line of thinking is that the data powering the "static" (or "shareable") version of the embeddable will be tied to a specific space, not to a specific user (which would require object-level security).

Whaaa, really? When was this discussed? Isn't it a security issue since two users in the same space may have access to different data indices? I thought that's why ES was handling all the complicated security for background search api, making sure one user can't use another users search id to get their data.

Since a dashboard itself is an embeddable, its createShareable would simply loop over its children and invoke their createShareable... Is that correct?

Yep, although with the make it slow, I think the initiator would be the child, that then calls createShareable on the parent, which yea, would then loop though it's children. So the final dashboard shareable data would be like:

{ 
  type: SHAREABLE_DASHBOARD,
   input: {
    panels: {
      '1': {
        type: SHAREABLE_VISUALIZE,
         explicitInput: {
           data: {...}
        }
      },
      '2': {
        type: SHAREABLE_SEARCH,
         explicitInput: {
           data: {...}
        }
      },
      '3': {
        type: MARKDOWN,  // any embeds that are already static can just use the same type.
         explicitInput: {
           data: {...}
        }
      }
   }
}

Not sure we'd actually need a special SHAREABLE_DASHBOARD type, but we might want it so it's clear not to have inputs as filter and time range since that wouldn't change the data.

[stacey-gammon]

... on the other hand, maybe having this data be per space would actually be more usable. After all, multiple people would probably want to view the results if the query took a really long time to complete. Maybe we just need to make it clear that the results will be visible to everyone in that space (maybe we even need a feature control for this so only some people can create shareable snapshots that make aggregated data visible to everyone in a space).

[lukasolson]

How would this work for an "in-progress" (or partial) dashboard? It was my assumption that when a user sends to background, they may still want to return to that specific snapshot before it's completed and see the progress.

[stacey-gammon]

Good question. If we want to support that then we probably can't reuse the shareable snapshot approach where there is no concept of viewing a snapshot that is in the process of being created.

Maybe we do need embeddables with three types of inputs:

1. Live query version: { query, filter, timeRange }
2. Shareable version: { data }
3. In progress, background search version: { searchId }

3. though is going to be tough to get right. Search id wouldn't actually work with the interpreter because you'd need multiple search ids and some how be able to re-construct the expression passing in search ids instead of how the expression is built now off of data from 1. That's kind of why I was thinking if we already want to solve 2. maybe we can just use that version first even though it's not as flexible. I don't know yet of any proposals for how 3. would work for all our different kinds of embeddables, especially those based off a dynamic expression.

[stacey-gammon]

Closing this, we will move forward with #53335