Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mock for http exporters #1185

Merged
merged 46 commits into from
Feb 19, 2022
Merged

Mock for http exporters #1185

merged 46 commits into from
Feb 19, 2022

Conversation

esigo
Copy link
Member

@esigo esigo commented Jan 30, 2022

Fixes #1049 (issue)

Changes

removes http server from exporter tests, and uses mock for the clients.

For significant contributions please make sure you have completed the following items:

  • CHANGELOG.md updated for non-trivial changes
  • Unit tests have been added
  • Changes in public API reviewed

@esigo esigo changed the title Mock for http exporters [WIP] Mock for http exporters Jan 30, 2022
@codecov
Copy link

codecov bot commented Jan 30, 2022

Codecov Report

Merging #1185 (a643d1a) into main (9502339) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #1185   +/-   ##
=======================================
  Coverage   92.96%   92.96%           
=======================================
  Files         196      196           
  Lines        7040     7040           
=======================================
  Hits         6544     6544           
  Misses        496      496           
Impacted Files Coverage Δ
...ntelemetry/ext/http/client/curl/http_client_curl.h 91.67% <ø> (ø)
...t/src/http/client/curl/http_client_factory_curl.cc 50.00% <ø> (ø)
ext/test/http/curl_http_test.cc 95.17% <ø> (ø)

@esigo esigo changed the title [WIP] Mock for http exporters Mock for http exporters Jan 31, 2022
@esigo esigo marked this pull request as ready for review January 31, 2022 20:05
@esigo esigo requested a review from a team January 31, 2022 20:05
@@ -22,7 +22,7 @@ $VCPKG_DIR="$SRC_DIR\vcpkg"

switch ($action) {
"bazel.build" {
bazel build $BAZEL_OPTIONS -- //...
bazel build --copt=-DENABLE_TEST $BAZEL_OPTIONS -- //...
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit - --copt=-DENABLE_TEST is also part of BAZEL_OPTIONS, so can be removed here.


/**
* Export
* @param message message to export, it should be ExportTraceServiceRequest,
* ExportMetricsServiceRequest or ExportLogsServiceRequest
*/
sdk::common::ExportResult Export(const google::protobuf::Message &message) noexcept;

VIRTUAL_TEST sdk::common::ExportResult Export(const google::protobuf::Message &message) noexcept;
Copy link
Member

@lalitb lalitb Feb 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit - googletest suggests to mock the non-virtual function using template - https://chromium.googlesource.com/external/github.com/google/googletest/+/refs/tags/release-1.8.0/googlemock/docs/CookBook.md#mocking-nonvirtual-methods. Sharing if this is feasible, though I don't see much issue with the existing approach.

@@ -98,14 +104,16 @@ class OtlpHttpClient
* Create an OtlpHttpClient using the given options.
*/
explicit OtlpHttpClient(OtlpHttpClientOptions &&options);
#ifdef ENABLE_TEST
VIRTUAL_TEST ~OtlpHttpClient() {}
Copy link
Member

@lalitb lalitb Feb 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am thinking if mocking ext::http::client::HttpClient would be a better approach (the way we do it for Zipkin) as this will allow us to test most of the existing code in OtlpHttpClient::Export() method. We may have to add ext::http::client::HttpClient object as dependency injection to OtlpHttpClient to make this feasible.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't use template because I thought it will be too much of noise in code.
Thanks for the idea, I will use dependency injection.

return http_request_;
}

MOCK_METHOD(void,
Copy link
Member

@lalitb lalitb Feb 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to mock the SendRequest(), or can we create a empty (200 OK) response, and call the callback from same thread. Something like this

void SendRequest(EventHandler &callback) noexcept override
{
    auto response   = std::unique_ptr<Response>(new Response());
    .. populate response object..
    callback.OnResponse(response);
}

This will allow us to remove dependency of gtest/gmock from this target.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have used Mock for testing the request that is going to be sent. The other option would be to make the client test itself via:

std::function<void(http_client::nosend::Session *)> CreateJsonTester(
    const std::string report_trace_id)
{
  return [report_trace_id](http_client::nosend::Session *mock_session) {
    auto check_json    = nlohmann::json::parse(mock_session->GetRequest()->body_, nullptr, false);
    // extract received_trace_id from request
    EXPECT_EQ(received_trace_id, report_trace_id);
  };
}

Then store this function into session, so whenever there is a request going to be sent the session tests itself:

  void SendRequest(opentelemetry::ext::http::client::EventHandler &callback) noexcept override
  {
    if (test_input_)
    {
      test_input_(this);
    }
    // let the otlp_http_client to continue
    Response response;
    callback.OnResponse(response);
  }

Shall I keep gMock or let the session test itself?

Copy link
Member

@lalitb lalitb Feb 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank. Let's keep the gMock for now, we can fix it later.

I am just thinking another solution to change signature for EventHandler::OnEvent to something like:

virtual void OnEvent(SessionState, void* data = nullptr, size_t size = 0) noexcept = 0;

The method in its current form is not useful. Sending back client implementation-specific data would be useful in real-world scenarios like server certificate validation during SSL Handshake. And nosend client can then use this to return the Request object for validation, something like

void SendRequest(EventHandler &callback) noexcept override
{
    callback.OnEvent(SessionState::Connected, (void*)http_request_.get(), size);    
    auto response   = std::unique_ptr<Response>(new Response());
      .. populate response object..
    callback.OnResponse(response);
}

But don't want to delay this PR more, let's have this fix separately. We can have a github ticket to add this support / or remove mock from nosend client.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea, thanks.
yes I agree let's merge this PR and add the support in another issue.
Please assign the new issue to me.
client factory should be fixed.

Copy link
Member

@lalitb lalitb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. Changes look good, apart from minor issues to address

  • Remove mock dependency for nosend HTTP interface.
  • Small change in HTTP factory implementation (discussing offline)

OtlpHttpClient http_client_;
std::unique_ptr<OtlpHttpClient> http_client_;
// For testing
friend class OtlpHttpExporterTestPeer;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicated with line 95?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, cleaned.

OtlpHttpClient http_client_;
std::unique_ptr<OtlpHttpClient> http_client_;
// For testing
friend class OtlpHttpLogExporterTestPeer;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicated with line 94.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, cleaned

@lalitb lalitb merged commit 9157fd4 into open-telemetry:main Feb 19, 2022
@esigo esigo deleted the mock-http-exporters branch February 19, 2022 10:14
owent added a commit to owent/opentelemetry-cpp that referenced this pull request Feb 19, 2022
owent added a commit to owent/opentelemetry-cpp that referenced this pull request Feb 24, 2022
Export all datas before notify `ForceFlush` to  continue

Signed-off-by: owent <[email protected]>
Fix timeout for `wait_for`

Signed-off-by: owent <[email protected]>
Fix wakeup notify

Signed-off-by: owent <[email protected]>
1. Reduce busy waiting of `BatchSpanProcessor::ForceFlush` and `BatchSpanProcessor::ForceFlush` when another thread is exporting data.
2. Fix the deadlock when `Shutdown` is called and background thread is finished after first checking of `is_shutdown_` and before setting `is_shutdown_` in `ForceFlush` of BatchSpanProcessor and `BatchSpanProcessor`.

Signed-off-by: owent <[email protected]>
Fix initialization of `is_shutdown_` in internal constructor of `OtlpHttpClient`

Signed-off-by: owent <[email protected]>
Add `ForceFlush` for `LoggerProvider`

Signed-off-by: owent <[email protected]>
Merge open-telemetry#1185 and add `OnEvent` for `nosend::HttpClient`

Signed-off-by: owent <[email protected]>
Add changelog

Signed-off-by: owentou <[email protected]>
Fix asan of overflow of chrono

Signed-off-by: owentou <[email protected]>
Fix shutdown timeout

Signed-off-by: owentou <[email protected]>
Fix thread-safety problem when destroying `OtlpHttpClient`

Signed-off-by: owentou <[email protected]>
Fix some compiler do not generate the correct move construct/assignment operator implementation

Signed-off-by: owent <[email protected]>
Fix compatibility

Signed-off-by: owent <[email protected]>
Allow concurrency otlp http session for otlp http client

Signed-off-by: owent <[email protected]>
owent added a commit to owent/opentelemetry-cpp that referenced this pull request Mar 4, 2022
Export all datas before notify `ForceFlush` to  continue

Signed-off-by: owent <[email protected]>
Fix timeout for `wait_for`

Signed-off-by: owent <[email protected]>
Fix wakeup notify

Signed-off-by: owent <[email protected]>
1. Reduce busy waiting of `BatchSpanProcessor::ForceFlush` and `BatchSpanProcessor::ForceFlush` when another thread is exporting data.
2. Fix the deadlock when `Shutdown` is called and background thread is finished after first checking of `is_shutdown_` and before setting `is_shutdown_` in `ForceFlush` of BatchSpanProcessor and `BatchSpanProcessor`.

Signed-off-by: owent <[email protected]>
Fix initialization of `is_shutdown_` in internal constructor of `OtlpHttpClient`

Signed-off-by: owent <[email protected]>
Add `ForceFlush` for `LoggerProvider`

Signed-off-by: owent <[email protected]>
Merge open-telemetry#1185 and add `OnEvent` for `nosend::HttpClient`

Signed-off-by: owent <[email protected]>
Add changelog

Signed-off-by: owentou <[email protected]>
Fix asan of overflow of chrono

Signed-off-by: owentou <[email protected]>
Fix shutdown timeout

Signed-off-by: owentou <[email protected]>
Fix thread-safety problem when destroying `OtlpHttpClient`

Signed-off-by: owentou <[email protected]>
Fix some compiler do not generate the correct move construct/assignment operator implementation

Signed-off-by: owent <[email protected]>
Fix compatibility

Signed-off-by: owent <[email protected]>
Allow concurrency otlp http session for otlp http client

Signed-off-by: owent <[email protected]>
owent added a commit to owent/opentelemetry-cpp that referenced this pull request Mar 9, 2022
Export all datas before notify `ForceFlush` to  continue

Signed-off-by: owent <[email protected]>
Fix timeout for `wait_for`

Signed-off-by: owent <[email protected]>
Fix wakeup notify

Signed-off-by: owent <[email protected]>
1. Reduce busy waiting of `BatchSpanProcessor::ForceFlush` and `BatchSpanProcessor::ForceFlush` when another thread is exporting data.
2. Fix the deadlock when `Shutdown` is called and background thread is finished after first checking of `is_shutdown_` and before setting `is_shutdown_` in `ForceFlush` of BatchSpanProcessor and `BatchSpanProcessor`.

Signed-off-by: owent <[email protected]>
Fix initialization of `is_shutdown_` in internal constructor of `OtlpHttpClient`

Signed-off-by: owent <[email protected]>
Add `ForceFlush` for `LoggerProvider`

Signed-off-by: owent <[email protected]>
Merge open-telemetry#1185 and add `OnEvent` for `nosend::HttpClient`

Signed-off-by: owent <[email protected]>
Add changelog

Signed-off-by: owentou <[email protected]>
Fix asan of overflow of chrono

Signed-off-by: owentou <[email protected]>
Fix shutdown timeout

Signed-off-by: owentou <[email protected]>
Fix thread-safety problem when destroying `OtlpHttpClient`

Signed-off-by: owentou <[email protected]>
Fix some compiler do not generate the correct move construct/assignment operator implementation

Signed-off-by: owent <[email protected]>
Fix compatibility

Signed-off-by: owent <[email protected]>
Allow concurrency otlp http session for otlp http client

Signed-off-by: owent <[email protected]>
owent added a commit to owent/opentelemetry-cpp that referenced this pull request Mar 13, 2022
Export all datas before notify `ForceFlush` to  continue

Signed-off-by: owent <[email protected]>
Fix timeout for `wait_for`

Signed-off-by: owent <[email protected]>
Fix wakeup notify

Signed-off-by: owent <[email protected]>
1. Reduce busy waiting of `BatchSpanProcessor::ForceFlush` and `BatchSpanProcessor::ForceFlush` when another thread is exporting data.
2. Fix the deadlock when `Shutdown` is called and background thread is finished after first checking of `is_shutdown_` and before setting `is_shutdown_` in `ForceFlush` of BatchSpanProcessor and `BatchSpanProcessor`.

Signed-off-by: owent <[email protected]>
Fix initialization of `is_shutdown_` in internal constructor of `OtlpHttpClient`

Signed-off-by: owent <[email protected]>
Add `ForceFlush` for `LoggerProvider`

Signed-off-by: owent <[email protected]>
Merge open-telemetry#1185 and add `OnEvent` for `nosend::HttpClient`

Signed-off-by: owent <[email protected]>
Add changelog

Signed-off-by: owentou <[email protected]>
Fix asan of overflow of chrono

Signed-off-by: owentou <[email protected]>
Fix shutdown timeout

Signed-off-by: owentou <[email protected]>
Fix thread-safety problem when destroying `OtlpHttpClient`

Signed-off-by: owentou <[email protected]>
Fix some compiler do not generate the correct move construct/assignment operator implementation

Signed-off-by: owent <[email protected]>
Fix compatibility

Signed-off-by: owent <[email protected]>
Allow concurrency otlp http session for otlp http client

Signed-off-by: owent <[email protected]>
owent added a commit to owent/opentelemetry-cpp that referenced this pull request Mar 21, 2022
Export all datas before notify `ForceFlush` to  continue

Signed-off-by: owent <[email protected]>
Fix timeout for `wait_for`

Signed-off-by: owent <[email protected]>
Fix wakeup notify

Signed-off-by: owent <[email protected]>
1. Reduce busy waiting of `BatchSpanProcessor::ForceFlush` and `BatchSpanProcessor::ForceFlush` when another thread is exporting data.
2. Fix the deadlock when `Shutdown` is called and background thread is finished after first checking of `is_shutdown_` and before setting `is_shutdown_` in `ForceFlush` of BatchSpanProcessor and `BatchSpanProcessor`.

Signed-off-by: owent <[email protected]>
Fix initialization of `is_shutdown_` in internal constructor of `OtlpHttpClient`

Signed-off-by: owent <[email protected]>
Add `ForceFlush` for `LoggerProvider`

Signed-off-by: owent <[email protected]>
Merge open-telemetry#1185 and add `OnEvent` for `nosend::HttpClient`

Signed-off-by: owent <[email protected]>
Add changelog

Signed-off-by: owentou <[email protected]>
Fix asan of overflow of chrono

Signed-off-by: owentou <[email protected]>
Fix shutdown timeout

Signed-off-by: owentou <[email protected]>
Fix thread-safety problem when destroying `OtlpHttpClient`

Signed-off-by: owentou <[email protected]>
Fix some compiler do not generate the correct move construct/assignment operator implementation

Signed-off-by: owent <[email protected]>
Fix compatibility

Signed-off-by: owent <[email protected]>
Allow concurrency otlp http session for otlp http client

Signed-off-by: owent <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unit test failing in CI: OtlpHttpExporterTestPeer.ExportJsonIntegrationTest
3 participants