fix: events: address sqlite index selection performance regressions

[rvagg]

Edit: If you want to take advantage of the performance gains from this change without upgrading to a version of Lotus that includes this fix, you can do it manually for yourself in a safe way that will still work when you eventually do upgrade Lotus to include this fix.

With sqlite3 installed, substituting the correct path for /path/to/lotus/repo, run:

sqlite3 /path/to/lotus/repo/sqlite/events.db '
  DROP INDEX IF EXISTS event_emitter_addr;
  DROP INDEX IF EXISTS event_reverted;
  DROP INDEX IF EXISTS event_entry_indexed_key;
  DROP INDEX IF EXISTS event_entry_codec_value;'

---

Fixes: #12255

SQLite was found to be avoiding the intended initial indexes for some types of complex queries and opting for minor indexes which don't narrow down the search space enough. Specifically we want queries to first either narrow by height or tipset_key_cid and then apply other criteria. Having alternative indexes when a query such as height>=X AND height<=Y are encountered cause SQLite to avoid the height index entirely. By removing additional indexes that could be used during the main query path (prefillFilter), we force SQLite to use the intended indexes and narrow the results without the use of indexes.

---

Some examples and numbers. These approximate the queries generated by prefillFilter() for various inputs and are run against my 40GiB events db containing 380,025 epochs. I've got 2 copies of it, one with the schema as in this PR and one with the schema as in master and I'm running and timing these queries directly on the commandline.

Basic select using height=

(note the current version doesn't even use this, it's always >= & <=)

SELECT event.height, hex(event.tipset_key_cid)
FROM event
JOIN event_entry ON event.id=event_entry.event_id
WHERE event.height=4098162 AND event.reverted=false
ORDER BY event.height DESC, event_entry._rowid_ ASC

Before: 3.05s, after: 0.0s - the reason for the slowness is that it chooses the reverted index instead of height, for reasons I can't fathom. This could be fixed by having a compound index on the two, but that fails for other cases.

Basic select using height>= and height <= for single height

Results in the same output as above.

SELECT event.height, hex(event.tipset_key_cid)
FROM event
JOIN event_entry ON event.id=event_entry.event_id
WHERE event.height>=4098162 AND event.height<=4098162 AND event.reverted=false
ORDER BY event.height DESC, event_entry._rowid_ ASC

Before: 3.10, after: 0.0s

Basic select using tipset_key_cid

Same results as above but using the tipset instead of height.

SELECT event.height, hex(event.tipset_key_cid)
FROM event
JOIN event_entry ON event.id=event_entry.event_id
WHERE event.tipset_key_cid=X'0171A0E40220DA1BA218ADC163101801DD8894A7378ABE5FDC4C62DC7056DCC96EC5F039EF64' AND event.reverted=false
ORDER BY event.height DESC, event_entry._rowid_ ASC

Before: 3.42, after: 0.0s - it's again using reverted for some reason, and then matching on tsk cid which is slower because it's a large bytes match.

Select at height and match emitter_addr

Again, this isn't quite what master does because we don't have height=X there, but close:

SELECT event.height, hex(event.tipset_key_cid)
FROM event
JOIN event_entry ON event.id=event_entry.event_id
WHERE event.height=4098162 AND event.reverted=false AND event.emitter_addr=X'040A60E1773636CF5E4A227D9AC24F20FECA034EE25A'
ORDER BY event.height DESC, event_entry._rowid_ ASC

Before: 3.15s, after: 0.0s - reverted again, but if I remove the reverted clause here it goes back to the height index rather than the emitter_addr index.

Select at height >= and <= and match emitter_addr

Same as above but with height>=X AND height <=X

Before: 3.19s, after: 0.0s

Select at tipset and match emitter_addr

Returns the same results as the above 2 but uses tsk

SELECT event.height, hex(event.tipset_key_cid)
FROM event
JOIN event_entry ON event.id=event_entry.event_id
WHERE event.tipset_key_cid=X'0171A0E40220DA1BA218ADC163101801DD8894A7378ABE5FDC4C62DC7056DCC96EC5F039EF64' AND event.reverted=false AND event.emitter_addr=X'040A60E1773636CF5E4A227D9AC24F20FECA034EE25A'
ORDER BY event.height DESC, event_entry._rowid_ ASC

Before: 3.35s, after: 0.0s

Select event_entry properties

Now we get interesting, this is the kind of thing you might get through GetActorEventsRaw or eth_getLogs searching for a particular log output. prefillFilter() builds these queries, they're a bit nested and complex but trigger a bunch of different indices:

SELECT event.height, hex(event.tipset_key_cid),hex(event.emitter_addr),event_entry.indexed,event_entry.codec,event_entry.key,hex(event_entry.value)
FROM event
JOIN event_entry ON event.id=event_entry.event_id, event_entry ee2 ON event.id=ee2.event_id
WHERE event.height>=4095282 AND event.height<=4098162 AND event.reverted=false AND event.emitter_addr=X'040A60E1773636CF5E4A227D9AC24F20FECA034EE25A' AND ee2.indexed=1 AND ee2.key='t1' AND ((ee2.value=X'DDF252AD1BE2C89B69C2B068FC378DAA952BA7F163C4A11628F55A4DF523B3EF' AND ee2.codec=85))
ORDER BY event.height DESC, event_entry._rowid_ ASC

Before: 0.73s, after: 0.1s.

EXPLAIN QUERY PLAN on this is interesting, it's still avoiding the index we want to hit (height) but it's finding one that'll narrow it down enough so the resultset isn't too big to filter further down:

Before:

QUERY PLAN
|--SEARCH ee2 USING INDEX event_entry_codec_value (codec=? AND value=?)
|--SEARCH event USING INTEGER PRIMARY KEY (rowid=?)
|--SEARCH event_entry USING INDEX event_entry_event_id (event_id=?)
`--USE TEMP B-TREE FOR ORDER BY

After:

QUERY PLAN
|--SEARCH event USING INDEX event_height (height>? AND height<?)
|--SEARCH ee2 USING INDEX event_entry_event_id (event_id=?)
|--SEARCH event_entry USING INDEX event_entry_event_id (event_id=?)
`--USE TEMP B-TREE FOR RIGHT PART OF ORDER BY

---

I'm using reverted in here because eth_* will use that, but it's not always the case if you use GetActorEventsRaw so the speed may vary if it hits alternative indexes.

After all of this I still don't know what criteria SQLite is using to select its index ordering. Perhaps it's to do with the efficiency of the indexes? Bool index is better than int index is better than bytes index? I don't really understand the use of event_entry_codec_value in the last one though, but maybe that's a special case because of nesting?

[aarshkshah1992]

lgtm

Let's test this on a mainnet node and ideally ask Glif/Mikers to deploy it and get the RPC timings out to see the impact.

[BigLep]

Wow - awesome work investigating this. I have been living in a world of assuming the DB engine would "do the right thing". Good to know that verification is needed. Well done!