feat(metrics): monitor s3 sdk retry #9790

wcy-fdu · 2023-05-15T03:42:10Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Checklist For Contributors

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
I have demonstrated that backward compatibility is not broken by breaking changes and created issues to track deprecated features to be removed in the future. (Please refer to the issue)
All checks passed in ./risedev check (or alias, ./risedev c)

Checklist For Reviewers

I have requested macro/micro-benchmarks as this PR can affect performance substantially, and the results are shown.

Documentation

My PR DOES NOT contain user-facing changes.

Click here for Documentation

Types of user-facing changes

Please keep the types that apply to your changes, and remove the others.

Installation and deployment
Connector (sources & sinks)
SQL commands, functions, and operators
RisingWave cluster configuration changes
Other (please specify in the release note below)

Release note

src/object_store/src/object/s3.rs

grafana/risingwave-dev-dashboard.dashboard.py

src/cmd/src/lib.rs

src/common/src/metrics.rs

wcy-fdu · 2023-06-12T09:51:02Z

@BugenZhao PTAL❤️

wcy-fdu · 2023-06-12T11:43:30Z

Seems that some CI timeout, might affect performance?

src/utils/runtime/src/lib.rs

src/object_store/src/object/s3.rs

src/common/src/metrics.rs

BugenZhao · 2023-06-13T04:14:30Z

Seems that some CI timeout, might affect performance?

I think so...

src/object_store/src/object/s3.rs

src/utils/runtime/src/lib.rs

src/common/src/metrics.rs

src/utils/runtime/src/lib.rs

codecov · 2023-06-13T08:57:20Z

Codecov Report

Merging #9790 (5273c45) into main (2b2ea49) will decrease coverage by 0.02%.
The diff coverage is 10.11%.

@@            Coverage Diff             @@
##             main    #9790      +/-   ##
==========================================
- Coverage   70.54%   70.52%   -0.02%     
==========================================
  Files        1243     1243              
  Lines      212978   213037      +59     
==========================================
+ Hits       150246   150250       +4     
- Misses      62732    62787      +55

Flag	Coverage Δ
rust	`70.52% <10.11%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/cmd/src/lib.rs	`0.00% <0.00%> (ø)`
src/cmd_all/src/bin/risingwave.rs	`0.00% <0.00%> (ø)`
src/cmd_all/src/playground.rs	`0.00% <0.00%> (ø)`
src/common/src/metrics.rs	`14.70% <0.00%> (-16.55%)`	⬇️
src/compute/src/lib.rs	`0.00% <0.00%> (ø)`
src/compute/src/server.rs	`0.00% <0.00%> (ø)`
src/object_store/src/object/s3.rs	`3.55% <0.00%> (-0.18%)`	⬇️
src/utils/runtime/src/lib.rs	`0.00% <0.00%> (ø)`
src/object_store/src/object/object_metrics.rs	`100.00% <100.00%> (ø)`

... and 3 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

commit ec637af4f5458b1a951d591a3dd7fc6994192e8f Author: Little-Wallace <[email protected]> Date: Tue Jun 20 12:52:47 2023 +0800 fix config Signed-off-by: Little-Wallace <[email protected]> commit 14641c2 Author: Little-Wallace <[email protected]> Date: Mon Jun 19 20:47:43 2023 +0800 fix config Signed-off-by: Little-Wallace <[email protected]> commit bc252ee Author: Little-Wallace <[email protected]> Date: Mon Jun 19 20:10:51 2023 +0800 fix busy loop Signed-off-by: Little-Wallace <[email protected]> commit 5b816a6 Merge: 1059c15 02dfee5 Author: Wallace <[email protected]> Date: Mon Jun 19 13:59:04 2023 +0800 Merge branch 'main' into scheduler-split commit 02dfee5 Author: William Wen <[email protected]> Date: Mon Jun 19 13:52:03 2023 +0800 feat(log-store): implement a merge stream of kv-log-store (#10090) commit a6c9c39 Author: lmatz <[email protected]> Date: Mon Jun 19 13:28:28 2023 +0800 chore: use github action to auto cherry pick pr to release branch (#10383) commit 608e183 Author: Bohan Zhang <[email protected]> Date: Mon Jun 19 12:18:28 2023 +0800 fix: support variable scale decimal in avro (#10368) Co-authored-by: idx0-dev <[email protected]> commit 75f6025 Author: zwang28 <[email protected]> Date: Sun Jun 18 17:41:15 2023 +0800 feat(trace): enable await tree trace for compactor (#10381) commit 321d376 Author: wu <[email protected]> Date: Sun Jun 18 15:59:38 2023 +0800 feat(connector): sink support for elasticsearch (#10357) commit d13d862 Author: Eric Fu <[email protected]> Date: Sun Jun 18 00:26:47 2023 +0800 feat: add debug profile tools in docker image (#10380) commit 1059c15 Merge: 9ac9ed4 d26f4bb Author: Wallace <[email protected]> Date: Fri Jun 16 20:49:21 2023 +0800 Merge branch 'main' into scheduler-split commit d26f4bb Author: Yuhao Su <[email protected]> Date: Fri Jun 16 18:36:27 2023 +0800 feat(metrics): add metrics for the evicted watermark for each executors (#10379) commit 3dd1393 Author: William Wen <[email protected]> Date: Fri Jun 16 17:34:34 2023 +0800 feat(sink): enable delta lake sink (#10374) commit 9ac9ed4 Merge: 58d8562 5c6b25c Author: Wallace <[email protected]> Date: Fri Jun 16 17:08:38 2023 +0800 Merge branch 'main' into scheduler-split commit 7b66d55 Author: William Wen <[email protected]> Date: Fri Jun 16 16:49:57 2023 +0800 fix(docker): install sasl library in docker (#10365) Co-authored-by: Eric Fu <[email protected]> commit 5c6b25c Author: zwang28 <[email protected]> Date: Fri Jun 16 16:10:23 2023 +0800 feat(ctl): list serving fragment mappings (#10331) commit 2c2a2b7 Author: Renjie Liu <[email protected]> Date: Fri Jun 16 15:49:00 2023 +0800 fix: Memory counter leak (#10358) commit 1c1354c Author: lmatz <[email protected]> Date: Fri Jun 16 15:36:00 2023 +0800 chore: return a warning message when creating sink with order by (#10239) commit 558cef5 Author: zwang28 <[email protected]> Date: Fri Jun 16 13:55:08 2023 +0800 feat(frontend): support mask failed serving worker temporarily (#10328) commit 7dccfa3 Author: Bohan Zhang <[email protected]> Date: Fri Jun 16 13:03:21 2023 +0800 chore: fix kafka download path in risedev (#10363) commit 58d8562 Author: Little-Wallace <[email protected]> Date: Fri Jun 16 12:53:47 2023 +0800 fix config test Signed-off-by: Little-Wallace <[email protected]> commit e77b76b Author: Little-Wallace <[email protected]> Date: Fri Jun 16 12:21:35 2023 +0800 fix space reclaim miss Signed-off-by: Little-Wallace <[email protected]> commit 2e5a907 Author: Little-Wallace <[email protected]> Date: Fri Jun 16 11:10:19 2023 +0800 merge conflict Signed-off-by: Little-Wallace <[email protected]> commit 1af4ea1 Author: Little-Wallace <[email protected]> Date: Wed Jun 14 16:41:54 2023 +0800 do not check table size for large throughput Signed-off-by: Little-Wallace <[email protected]> commit ccc47a2 Merge: 35199c4 9d83f88 Author: Wallace <[email protected]> Date: Fri Jun 16 10:59:19 2023 +0800 Merge branch 'main' into scheduler-split commit 9d83f88 Author: idx0-dev <[email protected]> Date: Thu Jun 15 18:39:25 2023 +0800 refactor(source): unified message parser (#10096) Co-authored-by: Eric Fu <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> commit 171e212 Author: William Wen <[email protected]> Date: Thu Jun 15 16:35:33 2023 +0800 feat(pinot-demo): add demo for sink to pinot via kafka (#10294) commit 11d3092 Author: William Wen <[email protected]> Date: Thu Jun 15 16:32:54 2023 +0800 feat(java-binding): bundle jni library to jar (#10229) commit 56f4011 Author: Yuhao Su <[email protected]> Date: Thu Jun 15 16:27:28 2023 +0800 feat(metrics): add memory usage metrics for more executor (#10351) commit ea7f95b Author: Yuanxin Cao <[email protected]> Date: Thu Jun 15 15:30:31 2023 +0800 refactor(sink): prune out hidden columns within sink executor (#10276) commit d818a00 Author: Tesla Zhang‮ <[email protected]> Date: Thu Jun 15 02:58:47 2023 -0400 refactor(plan_node_fmt): 6 more impls for Distill, refactor all `columns_name` functions (#10344) commit 26750c9 Author: xxchan <[email protected]> Date: Thu Jun 15 08:34:38 2023 +0200 build: use debug=1 back for release (#10345) commit ca41717 Author: Renjie Liu <[email protected]> Date: Thu Jun 15 14:28:25 2023 +0800 fix: Batch memory maybe negative (#10338) commit d95d3a2 Author: zwang28 <[email protected]> Date: Thu Jun 15 14:09:33 2023 +0800 chore(metric): add metric for hummock full GC (#10264) commit 65f05dd Author: StrikeW <[email protected]> Date: Thu Jun 15 13:10:06 2023 +0800 test(integration-test): jdbc sink data type tests (#10202) commit a164ab7 Author: xxchan <[email protected]> Date: Thu Jun 15 06:06:18 2023 +0200 chore: bump typos version and fix typos (#10342) commit 5cf94c9 Author: xxchan <[email protected]> Date: Wed Jun 14 16:18:53 2023 +0200 feat: support scalar function in FROM clause (#10317) commit 9593d1b Author: Tesla Zhang‮ <[email protected]> Date: Wed Jun 14 08:40:29 2023 -0400 refactor(plan_node_fmt): 4 more impls for Distill (#10296) commit 5b38239 Author: xxchan <[email protected]> Date: Wed Jun 14 13:20:21 2023 +0200 fix: replace ouroboros with self_cell (#10316) commit 90ee868 Author: Xinjing Hu <[email protected]> Date: Wed Jun 14 19:00:27 2023 +0800 feat(expr, agg): support `PERCENTILE_CONT`, `PERCENTILE_DISC` and `MODE` aggregation (#10252) Signed-off-by: Richard Chien <[email protected]> Co-authored-by: Richard Chien <[email protected]> Co-authored-by: Noel Kwan <[email protected]> commit e3fe51b Author: congyi wang <[email protected]> Date: Wed Jun 14 17:41:39 2023 +0800 refactor(log): change `aws_credential_types::cache::lazy_caching` log level to WARN (#10333) commit 33694b1 Author: stonepage <[email protected]> Date: Wed Jun 14 17:11:22 2023 +0800 refactor(binder): bind create table (#10307) commit 02a110c Author: Noel Kwan <[email protected]> Date: Wed Jun 14 16:18:10 2023 +0800 feat(storage): support replicated `LocalHummockStorage` (#10226) commit ede3278 Author: Richard Chien <[email protected]> Date: Wed Jun 14 16:02:34 2023 +0800 refactor(common): add `MemcmpEncoded` struct to represent memcmp encoded data (#10319) Signed-off-by: Richard Chien <[email protected]> commit ff91a4a Author: Li0k <[email protected]> Date: Wed Jun 14 15:56:21 2023 +0800 refactor(storage): refactor hummock timer loop (#10164) commit 353da76 Author: Richard Chien <[email protected]> Date: Wed Jun 14 15:06:07 2023 +0800 fix(macro): support `derive(EstimateSize)` on tuple struct (#10318) Signed-off-by: Richard Chien <[email protected]> Co-authored-by: Yuhao Su <[email protected]> commit 7dd388b Author: Runji Wang <[email protected]> Date: Wed Jun 14 14:52:48 2023 +0800 doc(udf): document Java UDF (#10320) commit e4aec8b Author: xiangjinwu <[email protected]> Date: Wed Jun 14 14:15:36 2023 +0800 feat(binder): support `group by` output alias or index (#10305) commit 8eb0e43 Author: Huangjw <[email protected]> Date: Wed Jun 14 11:01:28 2023 +0800 fix(ci): fix release script (#10325) commit 86f734c Author: Shanicky Chen <[email protected]> Date: Wed Jun 14 03:45:42 2023 +0800 fix: Increase timeout for end-to-end test (parallel) (dev mode) (#10308) Co-authored-by: xxchan <[email protected]> commit e02ef6c Author: Eric Fu <[email protected]> Date: Wed Jun 14 03:42:56 2023 +0800 fix: jemalloc profiling (#10314) Co-authored-by: xxchan <[email protected]> commit 07f6b52 Author: xxchan <[email protected]> Date: Tue Jun 13 21:31:21 2023 +0200 fix: use alias as table function's column name (#10311) commit 3017aa2 Author: xxchan <[email protected]> Date: Tue Jun 13 20:58:37 2023 +0200 ci: download dependencies from s3 (#9782) commit f971965 Author: zwang28 <[email protected]> Date: Tue Jun 13 19:24:45 2023 +0800 refactor(batch): maintain serving vnode mapping in meta node (#10004) commit 2b2950d Author: Zhanxiang (Patrick) Huang <[email protected]> Date: Tue Jun 13 19:07:40 2023 +0800 refactor: replace minstant/minitrace with tokio instant/tracing (#10302) commit 9177034 Author: congyi wang <[email protected]> Date: Tue Jun 13 18:08:51 2023 +0800 feat(metrics): monitor s3 sdk retry (#9790) commit 16a0efc Author: Runji Wang <[email protected]> Date: Tue Jun 13 17:58:34 2023 +0800 feat(udf): Java UDF SDK (#10095) commit 2b2ea49 Author: Eric Fu <[email protected]> Date: Tue Jun 13 17:19:35 2023 +0800 fix(metrics): incorrect FP rate (#10300) commit 54c660b Author: lmatz <[email protected]> Date: Tue Jun 13 16:57:07 2023 +0800 chore: remove enable_stream_row_count config (#10261) commit a6f38d9 Author: Shanicky Chen <[email protected]> Date: Tue Jun 13 16:30:40 2023 +0800 feat: Add revision for rescheduling process (#10199) Signed-off-by: Shanicky Chen <[email protected]> Signed-off-by: Little-Wallace <[email protected]>

Co-authored-by: congyi wang <[email protected]>

wcy-fdu added 6 commits May 11, 2023 13:27

add log for s3 timeout retry

e720082

minor

eec3ce8

minor

1dc5074

empty commit for retry

5724ab3

monitor s3 retry

90bf13a

rebase main

8a04cf7

wcy-fdu marked this pull request as draft May 15, 2023 03:42

github-actions bot added the type/feature label May 15, 2023

wcy-fdu commented May 15, 2023

View reviewed changes

src/object_store/src/object/s3.rs Show resolved Hide resolved

empty commit for retry

0ff4f42

hzxa21 reviewed May 15, 2023

View reviewed changes

grafana/risingwave-dev-dashboard.dashboard.py Show resolved Hide resolved

wcy-fdu added 9 commits May 15, 2023 21:45

save work

8aa39f5

save work

19063dd

add tracing filter for retry

421c1ac

temp commit

a1e3972

run longevity

162b084

save work

a54c156

temp commit

c583a53

mointor log

f89b05e

minor

35abafb

wcy-fdu changed the title ~~feat(metrics): monitor s3 retry rate~~ feat(metrics): monitor s3 sdk retry Jun 9, 2023

monitor aws http timeout retry

988a175

wcy-fdu marked this pull request as ready for review June 9, 2023 08:19

wcy-fdu requested review from hzxa21 and zwang28 June 9, 2023 08:19

wcy-fdu added 2 commits June 9, 2023 16:20

typo

ff27467

sort .toml

03101e0

hzxa21 reviewed Jun 9, 2023

View reviewed changes

src/cmd/src/lib.rs Show resolved Hide resolved

src/common/src/metrics.rs Outdated Show resolved Hide resolved

src/common/src/metrics.rs Outdated Show resolved Hide resolved

wcy-fdu added 2 commits June 12, 2023 11:42

save workspace

d8fe04f

resolve comments

553b4e9

wcy-fdu added 2 commits June 12, 2023 18:29

fix

a2b7fbf

fix

28fbe89

github-actions bot added the user-facing-changes Contains changes that are visible to users label Jun 12, 2023

wcy-fdu removed the user-facing-changes Contains changes that are visible to users label Jun 12, 2023

wcy-fdu added 2 commits June 12, 2023 19:52

empty commit for retry

b64429d

raise timeout interval

5d269e1

hzxa21 reviewed Jun 13, 2023

View reviewed changes

fix

1291bf9

BugenZhao reviewed Jun 13, 2023

View reviewed changes

wcy-fdu added 2 commits June 13, 2023 13:54

hope to pass CI

93fa1e4

remove inner filter

71910d6

resolve comments

c426bb4

wcy-fdu requested a review from BugenZhao June 13, 2023 09:11

wcy-fdu enabled auto-merge June 13, 2023 09:11

wcy-fdu added this pull request to the merge queue Jun 13, 2023

github-merge-queue bot removed this pull request from the merge queue due to a conflict with the base branch Jun 13, 2023

resolve conflict

5273c45

wcy-fdu enabled auto-merge June 13, 2023 09:44

wcy-fdu added this pull request to the merge queue Jun 13, 2023

Merged via the queue into main with commit 9177034 Jun 13, 2023

wcy-fdu deleted the wcy/monitor_s3_retry branch June 13, 2023 10:35

lmatz pushed a commit that referenced this pull request Jun 22, 2023

feat(metrics): monitor s3 sdk retry (#9790)

5ac0f1d

lmatz pushed a commit that referenced this pull request Jun 22, 2023

feat(metrics): monitor s3 sdk retry (#9790)

50c298b

lmatz added a commit that referenced this pull request Jun 22, 2023

feat(metrics): monitor s3 sdk retry (#9790) (#10489)

acbc3bf

Co-authored-by: congyi wang <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(metrics): monitor s3 sdk retry #9790

feat(metrics): monitor s3 sdk retry #9790

wcy-fdu commented May 15, 2023 •

edited

Loading

wcy-fdu commented Jun 12, 2023

wcy-fdu commented Jun 12, 2023

BugenZhao commented Jun 13, 2023

codecov bot commented Jun 13, 2023 •

edited

Loading

feat(metrics): monitor s3 sdk retry #9790

feat(metrics): monitor s3 sdk retry #9790

Conversation

wcy-fdu commented May 15, 2023 • edited Loading

What's changed and what's your intention?

Checklist For Contributors

Checklist For Reviewers

Documentation

Types of user-facing changes

Release note

wcy-fdu commented Jun 12, 2023

wcy-fdu commented Jun 12, 2023

BugenZhao commented Jun 13, 2023

codecov bot commented Jun 13, 2023 • edited Loading

Codecov Report

wcy-fdu commented May 15, 2023 •

edited

Loading

codecov bot commented Jun 13, 2023 •

edited

Loading