Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added max async export support using separate AsyncBatchSpan/LogProcessor #1306

Merged
merged 11 commits into from
May 4, 2022
Merged

Added max async export support using separate AsyncBatchSpan/LogProcessor #1306

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
f218940
Added max async export support
DebajitDas Apr 1, 2022
0c8dfcf
Added max async export support
DebajitDas Apr 4, 2022
39df6c8
Incorporated Review Comments. Added async flags for most jobs barring…
DebajitDas Apr 6, 2022
e4bc6e8
Added CI flags
DebajitDas Apr 6, 2022
b15931f
Added Separate Async Batch Span/Log Processors to isolate async funct…
DebajitDas Apr 14, 2022
1421910
Added changelog
DebajitDas Apr 14, 2022
4f2fc3b
Incorporated Review Comments and added async processor as derived of …
DebajitDas Apr 19, 2022
ca556b1
Incorporated some review comments
DebajitDas Apr 25, 2022
c2c5860
Fixed Format
DebajitDas Apr 25, 2022
f5324c9
Incorporated Review Comments
DebajitDas Apr 28, 2022
3c3bb3e
Merge branch 'async-changes' into export-changes
DebajitDas Apr 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,28 @@ jobs:
- name: run tests
run: ./ci/do_ci.sh bazel.test

bazel_test_async:
name: Bazel with async export
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
submodules: 'recursive'
- name: Mount Bazel Cache
uses: actions/cache@v3
env:
cache-name: bazel_cache
with:
path: /home/runner/.cache/bazel
key: bazel_test
- name: setup
run: |
sudo ./ci/setup_thrift.sh dependencies_only
sudo ./ci/setup_ci_environment.sh
sudo ./ci/install_bazelisk.sh
- name: run tests
run: ./ci/do_ci.sh bazel.with_async_export
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you enable -DENABLE_ASYNC_EXPORT_PREVIEW for all jobs but left one without it?I think we also need benchmark,address sanitizer, thread sanitizer and so on for this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please point me to some examples to refer to for benchmark, address sanitizer and thread sanitizer for this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I missed something, bazel.asan, bazel.tsan and bazel.valgrind already enable async APIs now, but benchmark still use $BAZEL_OPTIONS, I think we should also use $BAZEL_OPTIONS_ASYNC.
And we should also add --copt=-DENABLE_ASYNC_EXPORT_PREVIEW or -DWITH_ASYNC_EXPORT_PREVIEW=ON into all actions in ci/do_ci.ps1, it's used on Windows.


bazel_with_abseil:
name: Bazel with external abseil
runs-on: ubuntu-latest
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Increment the:

## [Unreleased]

* [SDK] Async Batch Span/Log processor with max async support ([#1306](https:/open-telemetry/opentelemetry-cpp/pull/1306))
* [EXPORTER] OTLP http exporter allow concurrency session ([#1209](https:/open-telemetry/opentelemetry-cpp/pull/1209))

## [1.3.0] 2022-04-11
Expand Down
6 changes: 5 additions & 1 deletion ci/do_ci.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ $action = $args[0]

$SRC_DIR=(Get-Item -Path ".\").FullName

$BAZEL_OPTIONS="--copt=-DENABLE_METRICS_PREVIEW --copt=-DENABLE_LOGS_PREVIEW"
$BAZEL_OPTIONS="--copt=-DENABLE_METRICS_PREVIEW --copt=-DENABLE_LOGS_PREVIEW --copt=-DENABLE_ASYNC_EXPORT"
$BAZEL_TEST_OPTIONS="$BAZEL_OPTIONS --test_output=errors"

if (!(test-path build)) {
Expand All @@ -32,6 +32,7 @@ switch ($action) {
cd "$BUILD_DIR"
cmake $SRC_DIR `
-DVCPKG_TARGET_TRIPLET=x64-windows `
-DWITH_ASYNC_EXPORT_PREVIEW=ON `
"-DCMAKE_TOOLCHAIN_FILE=$VCPKG_DIR\scripts\buildsystems\vcpkg.cmake"
$exit = $LASTEXITCODE
if ($exit -ne 0) {
Expand All @@ -52,6 +53,7 @@ switch ($action) {
cd "$BUILD_DIR"
cmake $SRC_DIR `
-DVCPKG_TARGET_TRIPLET=x64-windows `
-DWITH_ASYNC_EXPORT_PREVIEW=ON `
-DWITH_OTPROTCOL=ON `
"-DCMAKE_TOOLCHAIN_FILE=$VCPKG_DIR\scripts\buildsystems\vcpkg.cmake"
$exit = $LASTEXITCODE
Expand All @@ -73,6 +75,7 @@ switch ($action) {
cd "$BUILD_DIR"
cmake $SRC_DIR `
-DVCPKG_TARGET_TRIPLET=x64-windows `
-DWITH_ASYNC_EXPORT_PREVIEW=ON `
"-DCMAKE_TOOLCHAIN_FILE=$VCPKG_DIR\scripts\buildsystems\vcpkg.cmake"
$exit = $LASTEXITCODE
if ($exit -ne 0) {
Expand All @@ -89,6 +92,7 @@ switch ($action) {
cd "$BUILD_DIR"
cmake $SRC_DIR `
-DVCPKG_TARGET_TRIPLET=x64-windows `
-DWITH_ASYNC_EXPORT_PREVIEW=ON `
"-DCMAKE_TOOLCHAIN_FILE=$VCPKG_DIR\scripts\buildsystems\vcpkg.cmake"
$exit = $LASTEXITCODE
if ($exit -ne 0) {
Expand Down
39 changes: 26 additions & 13 deletions ci/do_ci.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ function run_benchmarks

[ -z "${BENCHMARK_DIR}" ] && export BENCHMARK_DIR=$HOME/benchmark
mkdir -p $BENCHMARK_DIR
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS -c opt -- \
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS_ASYNC -c opt -- \
$(bazel query 'attr("tags", "benchmark_result", ...)')
echo ""
echo "Benchmark results in $BENCHMARK_DIR:"
Expand Down Expand Up @@ -66,8 +66,11 @@ if [[ "$1" != "bazel.nortti" ]]; then
fi
BAZEL_TEST_OPTIONS="$BAZEL_OPTIONS --test_output=errors"

BAZEL_OPTIONS_ASYNC="--copt=-DENABLE_METRICS_PREVIEW --copt=-DENABLE_LOGS_PREVIEW --copt=-DENABLE_TEST --copt=-DENABLE_ASYNC_EXPORT"
BAZEL_TEST_OPTIONS_ASYNC="$BAZEL_OPTIONS_ASYNC --test_output=errors"

# https:/bazelbuild/bazel/issues/4341
BAZEL_MACOS_OPTIONS="$BAZEL_OPTIONS --features=-supports_dynamic_linker --build_tag_filters=-jaeger"
BAZEL_MACOS_OPTIONS="$BAZEL_OPTIONS_ASYNC --features=-supports_dynamic_linker --build_tag_filters=-jaeger"
BAZEL_MACOS_TEST_OPTIONS="$BAZEL_MACOS_OPTIONS --test_output=errors"

BAZEL_STARTUP_OPTIONS="--output_user_root=$HOME/.cache/bazel"
Expand All @@ -85,6 +88,7 @@ if [[ "$1" == "cmake.test" ]]; then
-DWITH_METRICS_PREVIEW=ON \
-DWITH_LOGS_PREVIEW=ON \
-DCMAKE_CXX_FLAGS="-Werror" \
-DWITH_ASYNC_EXPORT_PREVIEW=ON \
"${SRC_DIR}"
make
make test
Expand All @@ -96,6 +100,7 @@ elif [[ "$1" == "cmake.abseil.test" ]]; then
-DWITH_METRICS_PREVIEW=ON \
-DWITH_LOGS_PREVIEW=ON \
-DCMAKE_CXX_FLAGS="-Werror" \
-DWITH_ASYNC_EXPORT_PREVIEW=ON \
-DWITH_ABSEIL=ON \
"${SRC_DIR}"
make
Expand All @@ -106,6 +111,7 @@ elif [[ "$1" == "cmake.c++20.test" ]]; then
rm -rf *
cmake -DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_CXX_FLAGS="-Werror" \
-DWITH_ASYNC_EXPORT_PREVIEW=ON \
-DCMAKE_CXX_STANDARD=20 \
"${SRC_DIR}"
make
Expand All @@ -118,6 +124,7 @@ elif [[ "$1" == "cmake.c++20.stl.test" ]]; then
-DWITH_METRICS_PREVIEW=ON \
-DWITH_LOGS_PREVIEW=ON \
-DCMAKE_CXX_FLAGS="-Werror" \
-DWITH_ASYNC_EXPORT_PREVIEW=ON \
-DWITH_STL=ON \
"${SRC_DIR}"
make
Expand Down Expand Up @@ -145,6 +152,7 @@ elif [[ "$1" == "cmake.legacy.exporter.otprotocol.test" ]]; then
cmake -DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_CXX_STANDARD=11 \
-DWITH_OTLP=ON \
-DWITH_ASYNC_EXPORT_PREVIEW=ON \
"${SRC_DIR}"
grpc_cpp_plugin=`which grpc_cpp_plugin`
proto_make_file="CMakeFiles/opentelemetry_proto.dir/build.make"
Expand All @@ -157,6 +165,7 @@ elif [[ "$1" == "cmake.exporter.otprotocol.test" ]]; then
rm -rf *
cmake -DCMAKE_BUILD_TYPE=Debug \
-DWITH_OTLP=ON \
-DWITH_ASYNC_EXPORT_PREVIEW=ON \
"${SRC_DIR}"
grpc_cpp_plugin=`which grpc_cpp_plugin`
proto_make_file="CMakeFiles/opentelemetry_proto.dir/build.make"
Expand All @@ -165,8 +174,8 @@ elif [[ "$1" == "cmake.exporter.otprotocol.test" ]]; then
cd exporters/otlp && make test
exit 0
elif [[ "$1" == "bazel.with_abseil" ]]; then
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS --//api:with_abseil=true //...
bazel $BAZEL_STARTUP_OPTIONS test $BAZEL_TEST_OPTIONS --//api:with_abseil=true //...
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS_ASYNC --//api:with_abseil=true //...
bazel $BAZEL_STARTUP_OPTIONS test $BAZEL_TEST_OPTIONS_ASYNC --//api:with_abseil=true //...
exit 0
elif [[ "$1" == "cmake.test_example_plugin" ]]; then
# Build the plugin
Expand Down Expand Up @@ -206,6 +215,10 @@ elif [[ "$1" == "bazel.test" ]]; then
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS //...
bazel $BAZEL_STARTUP_OPTIONS test $BAZEL_TEST_OPTIONS //...
exit 0
elif [[ "$1" == "bazel.with_async_export" ]]; then
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS_ASYNC //...
bazel $BAZEL_STARTUP_OPTIONS test $BAZEL_TEST_OPTIONS_ASYNC //...
exit 0
elif [[ "$1" == "bazel.benchmark" ]]; then
run_benchmarks
exit 0
Expand All @@ -216,14 +229,14 @@ elif [[ "$1" == "bazel.macos.test" ]]; then
elif [[ "$1" == "bazel.legacy.test" ]]; then
# we uses C++ future and async() function to test the Prometheus Exporter functionality,
# that make this test always fail. ignore Prometheus exporter here.
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS -- //... -//exporters/otlp/... -//exporters/prometheus/...
bazel $BAZEL_STARTUP_OPTIONS test $BAZEL_TEST_OPTIONS -- //... -//exporters/otlp/... -//exporters/prometheus/...
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS_ASYNC -- //... -//exporters/otlp/... -//exporters/prometheus/...
bazel $BAZEL_STARTUP_OPTIONS test $BAZEL_TEST_OPTIONS_ASYNC -- //... -//exporters/otlp/... -//exporters/prometheus/...
exit 0
elif [[ "$1" == "bazel.noexcept" ]]; then
# there are some exceptions and error handling code from the Prometheus and Jaeger Clients
# that make this test always fail. ignore Prometheus and Jaeger exporters in the noexcept here.
bazel $BAZEL_STARTUP_OPTIONS build --copt=-fno-exceptions --build_tag_filters=-jaeger $BAZEL_OPTIONS -- //... -//exporters/prometheus/... -//exporters/jaeger/...
bazel $BAZEL_STARTUP_OPTIONS test --copt=-fno-exceptions --build_tag_filters=-jaeger $BAZEL_TEST_OPTIONS -- //... -//exporters/prometheus/... -//exporters/jaeger/...
bazel $BAZEL_STARTUP_OPTIONS build --copt=-fno-exceptions --build_tag_filters=-jaeger $BAZEL_OPTIONS_ASYNC -- //... -//exporters/prometheus/... -//exporters/jaeger/...
bazel $BAZEL_STARTUP_OPTIONS test --copt=-fno-exceptions --build_tag_filters=-jaeger $BAZEL_TEST_OPTIONS_ASYNC -- //... -//exporters/prometheus/... -//exporters/jaeger/...
exit 0
elif [[ "$1" == "bazel.nortti" ]]; then
# there are some exceptions and error handling code from the Prometheus and Jaeger Clients
Expand All @@ -232,18 +245,18 @@ elif [[ "$1" == "bazel.nortti" ]]; then
bazel $BAZEL_STARTUP_OPTIONS test --cxxopt=-fno-rtti --build_tag_filters=-jaeger $BAZEL_TEST_OPTIONS -- //... -//exporters/prometheus/... -//exporters/jaeger/...
exit 0
elif [[ "$1" == "bazel.asan" ]]; then
bazel $BAZEL_STARTUP_OPTIONS test --config=asan $BAZEL_TEST_OPTIONS //...
bazel $BAZEL_STARTUP_OPTIONS test --config=asan $BAZEL_TEST_OPTIONS_ASYNC //...
exit 0
elif [[ "$1" == "bazel.tsan" ]]; then
bazel $BAZEL_STARTUP_OPTIONS test --config=tsan $BAZEL_TEST_OPTIONS //...
bazel $BAZEL_STARTUP_OPTIONS test --config=tsan $BAZEL_TEST_OPTIONS_ASYNC //...
exit 0
elif [[ "$1" == "bazel.valgrind" ]]; then
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS //...
bazel $BAZEL_STARTUP_OPTIONS test --run_under="/usr/bin/valgrind --leak-check=full --error-exitcode=1 --suppressions=\"${SRC_DIR}/ci/valgrind-suppressions\"" $BAZEL_TEST_OPTIONS //...
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS_ASYNC //...
bazel $BAZEL_STARTUP_OPTIONS test --run_under="/usr/bin/valgrind --leak-check=full --error-exitcode=1 --suppressions=\"${SRC_DIR}/ci/valgrind-suppressions\"" $BAZEL_TEST_OPTIONS_ASYNC //...
exit 0
elif [[ "$1" == "benchmark" ]]; then
[ -z "${BENCHMARK_DIR}" ] && export BENCHMARK_DIR=$HOME/benchmark
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS -c opt -- \
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS_ASYNC -c opt -- \
$(bazel query 'attr("tags", "benchmark_result", ...)')
echo ""
echo "Benchmark results in $BENCHMARK_DIR:"
Expand Down
Loading