feat: add estimate_pending_compaction_bytes metric to analyze fragmen…

…tation rate Add estimate_pending_compaction_bytes metric to analyze fragmentation rate. Fixes: #1665 Signed-off-by: yaoyinnan <[email protected]>
OpenAtomFoundation · Jul 15, 2023 · 73c2438 · 73c2438
1 parent 355bb10
commit 73c2438
Show file tree

Hide file tree

Showing 3 changed files with 48 additions and 27 deletions.
diff --git a/src/storage/src/redis.cc b/src/storage/src/redis.cc
@@ -136,6 +136,9 @@ void Redis::GetRocksDBInfo(std::string &info, const char *prefix) {
  write_stream_key_value(rocksdb::DB::Properties::kTotalSstFilesSize, "total_sst_files_size");
  write_stream_key_value(rocksdb::DB::Properties::kLiveSstFilesSize, "live_sst_files_size");
 
+ // pending compaction bytes
+ write_stream_key_value(rocksdb::DB::Properties::kEstimatePendingCompactionBytes, "estimate_pending_compaction_bytes");
+
  // block cache
  write_stream_key_value(rocksdb::DB::Properties::kBlockCacheCapacity, "block_cache_capacity");
  write_stream_key_value(rocksdb::DB::Properties::kBlockCacheUsage, "block_cache_usage");

diff --git a/tools/pika_exporter/README.md b/tools/pika_exporter/README.md
@@ -142,33 +142,34 @@ prometheus --config.file=./grafana/prometheus.yml
 
 ### Rocksdb Metrics
 
-| Serial Number | Metric | Meaning |
-| ------------- | ------ | ------- |
-| 11 | rocksdb.num-immutable-mem-table | Number of immutable memtables not yet flushed. |
-| 12 | rocksdb.num-immutable-mem-table-flushed | Number of immutable memtables that have been flushed. |
-| 13 | rocksdb.mem-table-flush-pending | Returns 1 if there is a pending memtable flush; otherwise returns 0. |
-| 14 | rocksdb.num-running-flushes | Number of currently running flush operations. |
-| 15 | rocksdb.compaction-pending | Returns 1 if at least one compaction operation is pending; otherwise returns 0. |
-| 16 | rocksdb.num-running-compactions | Number of running compactions. |
-| 17 | rocksdb.background-errors | Total number of background errors. |
-| 18 | rocksdb.cur-size-active-mem-table | Approximate size, in bytes, of the active memtable. |
-| 19 | rocksdb.cur-size-all-mem-tables | Total size in bytes of memtables not yet flushed, including the current active memtable and the unflushed immutable memtables. |
-| 20 | rocksdb.size-all-mem-tables | Total size in bytes of all memtables, including the active memtable, unflushed immutable memtables, and pinned immutable memtables. |
-| 25 | rocksdb.estimate-num-keys | Estimated number of keys in active memtable, unflushed immutable memtables, and flushed SST files. |
-| 26 | rocksdb.estimate-table-readers-mem | Estimated memory size used for reading SST files, excluding block cache (such as filter and index blocks). |
-| 28 | rocksdb.num-snapshots | Number of unreleased snapshots in the database. |
-| 31 | rocksdb.num-live-versions | Number of current versions. More current versions usually indicate more SST files being used by iterators or incomplete compactions. |
-| 32 | rocksdb.current-super-version-number | Current number of the LSM version. It is a uint64_t integer that increments after any changes in the LSM tree. This number is not preserved after restarting the database and starts from 0 after a database restart. |
-| 33 | rocksdb.estimate-live-data-size | Estimated size of the activity data in bytes.For BlobDB, it also includes the actual live bytes in the version's blob file. |
-| 36 | rocksdb.total-sst-files-size | Total size (in bytes) of all SST files.Note: If there are too many files, it may slow down the online query. |
-| 37 | rocksdb.live-sst-files-size | Total size in bytes of all SST files belonging to the latest LSM tree. |
-| 47 | rocksdb.block-cache-capacity | The capacity of the block cache. |
-| 48 | rocksdb.block-cache-usage | Memory size occupied by entries in the block cache. |
-| 49 | rocksdb.block-cache-pinned-usage | Memory size occupied by pinned entries in the block cache. |
-| 51 | rocksdb.num-blob-files | The number of blob files in the current version. |
-| 52 | rocksdb.blob-stats | The total and size of all blob files, and the total amount of garbage (in bytes) in blob files in the current version. |
-| 53 | rocksdb.total-blob-file-size | The total size of all blob files across all versions. |
-| 54 | rocksdb.live-blob-file-size | The total size of all blob files in the current version. |
+| Serial Number | Metric | Meaning |
+|---------------| ------ |-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| 11 | rocksdb.num-immutable-mem-table | Number of immutable memtables not yet flushed. |
+| 12 | rocksdb.num-immutable-mem-table-flushed | Number of immutable memtables that have been flushed. |
+| 13 | rocksdb.mem-table-flush-pending | Returns 1 if there is a pending memtable flush; otherwise returns 0. |
+| 14 | rocksdb.num-running-flushes | Number of currently running flush operations. |
+| 15 | rocksdb.compaction-pending | Returns 1 if at least one compaction operation is pending; otherwise returns 0. |
+| 16 | rocksdb.num-running-compactions | Number of running compactions. |
+| 17 | rocksdb.background-errors | Total number of background errors. |
+| 18 | rocksdb.cur-size-active-mem-table | Approximate size, in bytes, of the active memtable. |
+| 19 | rocksdb.cur-size-all-mem-tables | Total size in bytes of memtables not yet flushed, including the current active memtable and the unflushed immutable memtables. |
+| 20 | rocksdb.size-all-mem-tables | Total size in bytes of all memtables, including the active memtable, unflushed immutable memtables, and pinned immutable memtables. |
+| 25 | rocksdb.estimate-num-keys | Estimated number of keys in active memtable, unflushed immutable memtables, and flushed SST files. |
+| 26 | rocksdb.estimate-table-readers-mem | Estimated memory size used for reading SST files, excluding block cache (such as filter and index blocks). |
+| 28 | rocksdb.num-snapshots | Number of unreleased snapshots in the database. |
+| 31 | rocksdb.num-live-versions | Number of current versions. More current versions usually indicate more SST files being used by iterators or incomplete compactions. |
+| 32 | rocksdb.current-super-version-number | Current number of the LSM version. It is a uint64_t integer that increments after any changes in the LSM tree. This number is not preserved after restarting the database and starts from 0 after a database restart. |
+| 33 | rocksdb.estimate-live-data-size | Estimated size of the activity data in bytes.For BlobDB, it also includes the actual live bytes in the version's blob file. |
+| 36 | rocksdb.total-sst-files-size | Total size (in bytes) of all SST files.Note: If there are too many files, it may slow down the online query. |
+| 37 | rocksdb.live-sst-files-size | Total size in bytes of all SST files belonging to the latest LSM tree. |
+| 40 | rocksdb.estimate-pending-compaction-bytes | Estimated total number of bytes that compression needs to rewrite to bring all levels down below the target size. Has no effect on compression other than level-based compression. |
+| 47 | rocksdb.block-cache-capacity | The capacity of the block cache. |
+| 48 | rocksdb.block-cache-usage | Memory size occupied by entries in the block cache. |
+| 49 | rocksdb.block-cache-pinned-usage | Memory size occupied by pinned entries in the block cache. |
+| 51 | rocksdb.num-blob-files | The number of blob files in the current version. |
+| 52 | rocksdb.blob-stats | The total and size of all blob files, and the total amount of garbage (in bytes) in blob files in the current version. |
+| 53 | rocksdb.total-blob-file-size | The total size of all blob files across all versions. |
+| 54 | rocksdb.live-blob-file-size | The total size of all blob files in the current version. |
 
 ## Grafana Dashboard ##
 See [here](./grafana/grafana_prometheus_pika_dashboard.json)

diff --git a/tools/pika_exporter/exporter/metrics/rocksdb.go b/tools/pika_exporter/exporter/metrics/rocksdb.go
@@ -189,6 +189,23 @@ var collectRocksDBMetrics = map[string]MetricConfig{
  },
  },
 
+ // pending compaction bytes
+ "estimate_pending_compaction_bytes": {
+ Parser: &regexParser{
+ name: "estimate_pending_compaction_bytes",
+ reg: regexp.MustCompile(`(?P<data_type>\w+)_.*?estimate_pending_compaction_bytes:(?P<estimate_pending_compaction_bytes>\d+)`),
+ Parser: &normalParser{},
+ },
+ MetricMeta: &MetaData{
+ Name: "estimate_pending_compaction_bytes",
+ Help: "Estimated total number of bytes that compression needs to rewrite to bring all levels down
+ below the target size. Has no effect on compression other than level-based compression.",
+ Type: metricTypeGauge,
+ Labels: []string{LabelNameAddr, LabelNameAlias, "data_type"},
+ ValueName: "estimate_pending_compaction_bytes",
+ },
+ },
+
  // snapshots
  "num_snapshots": {
  Parser: &regexParser{