Corrupt saved object passes import validation, breaks UI, and blocks upgrades

[VimCommando]

Kibana version: 7.9.3 upgrading to 7.13.4

Elasticsearch version: 7.9.3 upgrading to 7.13.4

Server OS version: Elastic Cloud

Browser version: N/A

Browser OS version: N/A

Original install method (e.g. download page, yum, from source, etc.): Elastic Cloud

Describe the bug: When upgrading from 7.9.3 to 7.13.2, invalid saved object will completely block the upgrade. These are logged as failed saved objects migrations.

Steps to reproduce:

1. Create a 7.9.3 deployment
2. Save this into an import.ndjson file and import into Kibana:

   {"id":"45e5f440-4657-11ec-ab60-cf77590a89d1","migrationVersion":{"dashboard":"7.9.3"},"references":[],"type":"dashboard","updated_at":"2021-11-15T21:01:59.043Z","version": "WzM3LDFd"}
   {"exportedCount":1,"missingRefCount":0,"missingReferences":[]}
   

3. Kibana will happily accept this corrupt dashboard; even though it is missing the entire attributes block.

Expected behavior:

1. Kibana does not import objects that will break UI and upgrade features
2. Saved objects migration does not fail entire migration due to one bad object

Screenshots (if relevant):

The corrupt dashboard breaks the entire Saved Objects page:

It blocks the dashboard list, the UI never renders the page:

AND it entirely blocks an upgrade in Elastic Cloud:

Errors in browser console (if relevant):
TypeError: record.title is undefined
(See screenshot above)

Provide logs and/or server output (if relevant):

From the kibana.log we see this same dashboard blocks the upgrade:

{
    "@timestamp": "2021-11-15T21:29:29.000Z",
    "tags": [
      "error",
      "savedobjects-service",
      "kibana",
      "logging"
    ],
    "source.keyword": [".../instance-0000000000/logs/kibana.log"],
        "message": [
      "Error: Failed to transform document 45e5f440-4657-11ec-ab60-cf77590a89d1. Transform: dashboard:7.11.0",
      "Doc: {\"type\":\"dashboard\",\"id\":\"45e5f440-4657-11ec-ab60-cf77590a89d1\",\"references\":[],\"migrationVersion\":{\"dashboard\":\"7.9.3\"},\"updated_at\":\"2021-11-15T21:06:01.904Z\",\"version\":\"WzMzLDFd\"}",
          "at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:562:13)",
          "at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:631:22)",
          "at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:350:20)",
          "at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:290:22)",
          "at /usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:98:16",
          "at Immediate.<anonymous> (/usr/share/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:88:17)",
          "at processImmediate (internal/timers.js:464:21)"
    ]
}

Any additional context:

At first this may seem like the answer is "well, don't add a corrupt dashboard" but the example here came from a genuine user experience, they hit this exact scenario attempting a cloud-managed upgrade.

So my main question is: why can't we just flag bad objects in the saved object migration for review after the upgrade, instead of abandoning the entire process?

[VimCommando]

I tried importing that same bad dashboard into a 7.11 instance and it did block it, so at least that part of the saved object import validation is already fixed.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

Manually tempering with export files is not supported and can lead, one way or the other, to corrupted state and undesired behavior after import.

We're aware of these problems, and are actively working on improving the validation and data-consistency checks of our saved objects API. #104088 is the first step, that will add per-type validation schema, to add a layer of protection against tempered or invalid data during both create and import operations.

[VimCommando]

The only reason I manually tampered with the file was to exactly recreate the issue that naturally occurred in a live cluster. I'm not sure how the original dashboard ended up in that state.

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[rudolf]

@elastic/kibana-presentation We don't know the root cause of this corrupt data, but using #104088 for your saved objects will prevent this from happening in the future.

[ThomThomson]

Closing this in favour of #139684, as adding schema validations (as described in that issue) will fix this one.