Searches against a large number of unavailable shards result in very large responses

[original-brownbear]

Searching a large number of unavailable shards through e.g. the * pattern while a large cluster is recovering from a full restart or so, leads to extremely large responses containing an exception for each shard.

This example shows a 375M on heap response for ~25k unavailable shards being returned. Over the wire, this serialises with a similar size and we'd have 700M+ peak heap usage for a search request when the valid search response in a green cluster might be much smaller than this.
It seems to me that we could mostly resolve this by not returning the stack trace in this specific case of the shard not available exception (it doesn't seem valuable for users and we made a similar fix around unavailable shards in snapshots state responses)?

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[benwtrent]

@original-brownbear

It seems originally we did not record these failures at all but behavior was changed in: #64337

So, it seems a good middle ground is as you said, let's not return the trace in these class of failures.

Sounds fair?

[benwtrent]

Well, including unavailable shards as shard failures, but adjusting their exception serialization is a bit more complicated.

Will need to discuss the best way to approach this.

Some silly ideas:

• simply don't return them as shard failures (effectively revert: Do not skip not available shard exception in search response #64337)
• Adjust ShardSearchFailure to not require an exception (bigger change, bwc considerations). @ywelsch seems to indicate this option here: Should we consider "Shard not available exception" as regular search failures #47700 (comment)
• Add a new "unavailable shards" section (seems like too big of a change to me, bwc considerations)
• Keep serializing the stacktrace between nodes, but when writing XContent, don't write the stacktrace or suppressed exceptions for unavailable shard failures (seems easiest, but still costs when serializing between nodes...)