OTLP Serialization for Dimensional Metrics (#826)

* Add protos under packages for otlp * Add common otlp proto payload methods * Add new oltp protocol class * Remove ML event from log message * Remove params, add api-key header & expose path The params are not relevant to OTLP so remove these. The api-key header is how we provide the license key to OTLP so add this. The path to upload dimensional metrics and events are different in OTLP so expose the path so it can be overriden inside the coresponding data_collector methods. * Add metric protos * Use protos to create payload * Squashed commit of the following: commit 6f15520cea6a1098915c9ca340dbe42de6a5de1d Author: Tim Pansino <[email protected]> Date: Mon May 15 14:28:50 2023 -0700 TEMP commit 1a28d36 Author: Tim Pansino <[email protected]> Date: Thu May 11 17:28:27 2023 -0700 Cover tags as list not dict commit 71261e3 Merge: 459e085 c2d4629 Author: Timothy Pansino <[email protected]> Date: Thu May 11 16:59:11 2023 -0700 Merge branch 'main' into feature-dimensional-metrics commit 459e085 Author: Tim Pansino <[email protected]> Date: Thu May 11 16:57:16 2023 -0700 Add testing for dimensional metrics commit ed33957 Author: Tim Pansino <[email protected]> Date: Thu May 11 16:56:31 2023 -0700 Add attribute processing to metric identity commit 6caf71e Author: Tim Pansino <[email protected]> Date: Thu May 11 16:56:16 2023 -0700 Add dimensional stats table to stats engine commit 5e1cc9d Author: Tim Pansino <[email protected]> Date: Wed May 10 16:00:42 2023 -0700 Squashed commit of the following: commit c2d4629 Author: Timothy Pansino <[email protected]> Date: Wed May 10 15:59:13 2023 -0700 Add required option for tox v4 (#795) * Add required option for tox v4 * Update tox in GHA * Remove py27 no-cache-dir commit a963649 Author: Hannah Stepanek <[email protected]> Date: Tue May 9 10:46:39 2023 -0700 Run coverage around pytest (#813) * Run coverage around pytest * Trigger tests * Fixup * Add redis client_no_touch to ignore list * Temporarily remove kafka from coverage * Remove coverage for old libs commit 3d82845 Author: Lalleh Rafeei <[email protected]> Date: Wed May 3 14:50:30 2023 -0700 Omit some frameworks from coverage analysis (#810) * Omit some frameworks from coverage analysis * Remove commas * Change format of omit * Add relative_files option to coverage * Add absolute directory * Add envsitepackagedir * Add coveragerc file * Add codecov.yml * [Mega-Linter] Apply linters fixes * Revert coveragerc file settings * Add files in packages and more frameworks * Remove commented line --------- Co-authored-by: lrafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> commit fd0fa35 Author: Uma Annamalai <[email protected]> Date: Tue May 2 10:55:36 2023 -0700 Add testing for genshi and mako. (#799) * Add testing for genshi and mako. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: umaannamalai <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit be4fb3d Author: Lalleh Rafeei <[email protected]> Date: Mon May 1 16:01:09 2023 -0700 Add tests for Waitress (#797) * Change import format * Initial commit * Add more tests to adapter_waitress * Remove commented out code * [Mega-Linter] Apply linters fixes * Add assertions to all tests * Add more NR testing to waitress --------- Co-authored-by: lrafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 7103506 Author: Hannah Stepanek <[email protected]> Date: Mon May 1 14:12:31 2023 -0700 Add tests for pyodbc (#796) * Add tests for pyodbc * Move imports into tests to get import coverage * Fixup: remove time import * Trigger tests --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 363122a Author: Hannah Stepanek <[email protected]> Date: Mon May 1 13:34:35 2023 -0700 Pin virtualenv, fix pip arg deprecation & disable kafka tests (#803) * Pin virtualenv * Fixup: use 20.21.1 instead * Replace install-options with config-settings See pypa/pip#11358. * Temporarily disable kafka tests commit c2d4629 Author: Timothy Pansino <[email protected]> Date: Wed May 10 15:59:13 2023 -0700 Add required option for tox v4 (#795) * Add required option for tox v4 * Update tox in GHA * Remove py27 no-cache-dir commit a963649 Author: Hannah Stepanek <[email protected]> Date: Tue May 9 10:46:39 2023 -0700 Run coverage around pytest (#813) * Run coverage around pytest * Trigger tests * Fixup * Add redis client_no_touch to ignore list * Temporarily remove kafka from coverage * Remove coverage for old libs commit dc81a50 Author: Tim Pansino <[email protected]> Date: Sat May 6 14:16:14 2023 -0700 Wiring dimensional metrics commit 3d82845 Author: Lalleh Rafeei <[email protected]> Date: Wed May 3 14:50:30 2023 -0700 Omit some frameworks from coverage analysis (#810) * Omit some frameworks from coverage analysis * Remove commas * Change format of omit * Add relative_files option to coverage * Add absolute directory * Add envsitepackagedir * Add coveragerc file * Add codecov.yml * [Mega-Linter] Apply linters fixes * Revert coveragerc file settings * Add files in packages and more frameworks * Remove commented line --------- Co-authored-by: lrafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> commit fd0fa35 Author: Uma Annamalai <[email protected]> Date: Tue May 2 10:55:36 2023 -0700 Add testing for genshi and mako. (#799) * Add testing for genshi and mako. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: umaannamalai <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit be4fb3d Author: Lalleh Rafeei <[email protected]> Date: Mon May 1 16:01:09 2023 -0700 Add tests for Waitress (#797) * Change import format * Initial commit * Add more tests to adapter_waitress * Remove commented out code * [Mega-Linter] Apply linters fixes * Add assertions to all tests * Add more NR testing to waitress --------- Co-authored-by: lrafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 7103506 Author: Hannah Stepanek <[email protected]> Date: Mon May 1 14:12:31 2023 -0700 Add tests for pyodbc (#796) * Add tests for pyodbc * Move imports into tests to get import coverage * Fixup: remove time import * Trigger tests --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Squashed commit of the following: commit 7a384c5 Author: Tim Pansino <[email protected]> Date: Thu Jun 1 12:10:59 2023 -0700 Cleaning out agent protocol commit c87d31d Author: Tim Pansino <[email protected]> Date: Thu Jun 1 12:10:46 2023 -0700 Change content-type header commit 5750e54 Author: Tim Pansino <[email protected]> Date: Thu Jun 1 12:05:52 2023 -0700 Add common utilities for OTLP * Remove testing logic * Adding metric serialization helpers * Squashed commit of the following: commit a47e209 Author: Tim Pansino <[email protected]> Date: Tue Jun 6 11:11:30 2023 -0700 Commit suggestions from code review commit 1a28d36 Author: Tim Pansino <[email protected]> Date: Thu May 11 17:28:27 2023 -0700 Cover tags as list not dict commit 71261e3 Merge: 459e085 c2d4629 Author: Timothy Pansino <[email protected]> Date: Thu May 11 16:59:11 2023 -0700 Merge branch 'main' into feature-dimensional-metrics commit 459e085 Author: Tim Pansino <[email protected]> Date: Thu May 11 16:57:16 2023 -0700 Add testing for dimensional metrics commit ed33957 Author: Tim Pansino <[email protected]> Date: Thu May 11 16:56:31 2023 -0700 Add attribute processing to metric identity commit 6caf71e Author: Tim Pansino <[email protected]> Date: Thu May 11 16:56:16 2023 -0700 Add dimensional stats table to stats engine commit 5e1cc9d Author: Tim Pansino <[email protected]> Date: Wed May 10 16:00:42 2023 -0700 Squashed commit of the following: commit c2d4629 Author: Timothy Pansino <[email protected]> Date: Wed May 10 15:59:13 2023 -0700 Add required option for tox v4 (#795) * Add required option for tox v4 * Update tox in GHA * Remove py27 no-cache-dir commit a963649 Author: Hannah Stepanek <[email protected]> Date: Tue May 9 10:46:39 2023 -0700 Run coverage around pytest (#813) * Run coverage around pytest * Trigger tests * Fixup * Add redis client_no_touch to ignore list * Temporarily remove kafka from coverage * Remove coverage for old libs commit 3d82845 Author: Lalleh Rafeei <[email protected]> Date: Wed May 3 14:50:30 2023 -0700 Omit some frameworks from coverage analysis (#810) * Omit some frameworks from coverage analysis * Remove commas * Change format of omit * Add relative_files option to coverage * Add absolute directory * Add envsitepackagedir * Add coveragerc file * Add codecov.yml * [Mega-Linter] Apply linters fixes * Revert coveragerc file settings * Add files in packages and more frameworks * Remove commented line --------- Co-authored-by: lrafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> commit fd0fa35 Author: Uma Annamalai <[email protected]> Date: Tue May 2 10:55:36 2023 -0700 Add testing for genshi and mako. (#799) * Add testing for genshi and mako. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: umaannamalai <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit be4fb3d Author: Lalleh Rafeei <[email protected]> Date: Mon May 1 16:01:09 2023 -0700 Add tests for Waitress (#797) * Change import format * Initial commit * Add more tests to adapter_waitress * Remove commented out code * [Mega-Linter] Apply linters fixes * Add assertions to all tests * Add more NR testing to waitress --------- Co-authored-by: lrafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 7103506 Author: Hannah Stepanek <[email protected]> Date: Mon May 1 14:12:31 2023 -0700 Add tests for pyodbc (#796) * Add tests for pyodbc * Move imports into tests to get import coverage * Fixup: remove time import * Trigger tests --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 363122a Author: Hannah Stepanek <[email protected]> Date: Mon May 1 13:34:35 2023 -0700 Pin virtualenv, fix pip arg deprecation & disable kafka tests (#803) * Pin virtualenv * Fixup: use 20.21.1 instead * Replace install-options with config-settings See pypa/pip#11358. * Temporarily disable kafka tests commit c2d4629 Author: Timothy Pansino <[email protected]> Date: Wed May 10 15:59:13 2023 -0700 Add required option for tox v4 (#795) * Add required option for tox v4 * Update tox in GHA * Remove py27 no-cache-dir commit a963649 Author: Hannah Stepanek <[email protected]> Date: Tue May 9 10:46:39 2023 -0700 Run coverage around pytest (#813) * Run coverage around pytest * Trigger tests * Fixup * Add redis client_no_touch to ignore list * Temporarily remove kafka from coverage * Remove coverage for old libs commit dc81a50 Author: Tim Pansino <[email protected]> Date: Sat May 6 14:16:14 2023 -0700 Wiring dimensional metrics commit 3d82845 Author: Lalleh Rafeei <[email protected]> Date: Wed May 3 14:50:30 2023 -0700 Omit some frameworks from coverage analysis (#810) * Omit some frameworks from coverage analysis * Remove commas * Change format of omit * Add relative_files option to coverage * Add absolute directory * Add envsitepackagedir * Add coveragerc file * Add codecov.yml * [Mega-Linter] Apply linters fixes * Revert coveragerc file settings * Add files in packages and more frameworks * Remove commented line --------- Co-authored-by: lrafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> commit fd0fa35 Author: Uma Annamalai <[email protected]> Date: Tue May 2 10:55:36 2023 -0700 Add testing for genshi and mako. (#799) * Add testing for genshi and mako. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: umaannamalai <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit be4fb3d Author: Lalleh Rafeei <[email protected]> Date: Mon May 1 16:01:09 2023 -0700 Add tests for Waitress (#797) * Change import format * Initial commit * Add more tests to adapter_waitress * Remove commented out code * [Mega-Linter] Apply linters fixes * Add assertions to all tests * Add more NR testing to waitress --------- Co-authored-by: lrafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 7103506 Author: Hannah Stepanek <[email protected]> Date: Mon May 1 14:12:31 2023 -0700 Add tests for pyodbc (#796) * Add tests for pyodbc * Move imports into tests to get import coverage * Fixup: remove time import * Trigger tests --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add protobuf to agent features tests * Proper bucket dimensional metric serialization * Wiring up OTLP protocol for metrics * Correct metrics payloads * Make default content-encoding header configurable * Clean up otlp encoding * Expand OTLP metrics testing * Squashed commit of the following: commit 30f0bf5 Author: Hannah Stepanek <[email protected]> Date: Fri Jun 9 16:12:09 2023 -0700 Add OTLP protocol class & protos (#821) * Add protos under packages for otlp * Add common otlp proto payload methods * Add new oltp protocol class * Remove ML event from log message * Remove params, add api-key header & expose path The params are not relevant to OTLP so remove these. The api-key header is how we provide the license key to OTLP so add this. The path to upload dimensional metrics and events are different in OTLP so expose the path so it can be overriden inside the coresponding data_collector methods. * Add otlp_port and otlp_host settings * Default to JSON if protobuf not available & warn * Move otlp_utils to core * Call encode in protocol class * Patch issues with data collector * Move resource to utils & add log proto imports --------- Co-authored-by: Tim Pansino <[email protected]> commit e970884 Author: Timothy Pansino <[email protected]> Date: Thu Jun 8 13:17:28 2023 -0700 Dimensional Metrics (#815) * Wiring dimensional metrics * Squashed commit of the following: commit c2d4629 Author: Timothy Pansino <[email protected]> Date: Wed May 10 15:59:13 2023 -0700 Add required option for tox v4 (#795) * Add required option for tox v4 * Update tox in GHA * Remove py27 no-cache-dir commit a963649 Author: Hannah Stepanek <[email protected]> Date: Tue May 9 10:46:39 2023 -0700 Run coverage around pytest (#813) * Run coverage around pytest * Trigger tests * Fixup * Add redis client_no_touch to ignore list * Temporarily remove kafka from coverage * Remove coverage for old libs commit 3d82845 Author: Lalleh Rafeei <[email protected]> Date: Wed May 3 14:50:30 2023 -0700 Omit some frameworks from coverage analysis (#810) * Omit some frameworks from coverage analysis * Remove commas * Change format of omit * Add relative_files option to coverage * Add absolute directory * Add envsitepackagedir * Add coveragerc file * Add codecov.yml * [Mega-Linter] Apply linters fixes * Revert coveragerc file settings * Add files in packages and more frameworks * Remove commented line --------- Co-authored-by: lrafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> commit fd0fa35 Author: Uma Annamalai <[email protected]> Date: Tue May 2 10:55:36 2023 -0700 Add testing for genshi and mako. (#799) * Add testing for genshi and mako. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: umaannamalai <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit be4fb3d Author: Lalleh Rafeei <[email protected]> Date: Mon May 1 16:01:09 2023 -0700 Add tests for Waitress (#797) * Change import format * Initial commit * Add more tests to adapter_waitress * Remove commented out code * [Mega-Linter] Apply linters fixes * Add assertions to all tests * Add more NR testing to waitress --------- Co-authored-by: lrafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 7103506 Author: Hannah Stepanek <[email protected]> Date: Mon May 1 14:12:31 2023 -0700 Add tests for pyodbc (#796) * Add tests for pyodbc * Move imports into tests to get import coverage * Fixup: remove time import * Trigger tests --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 363122a Author: Hannah Stepanek <[email protected]> Date: Mon May 1 13:34:35 2023 -0700 Pin virtualenv, fix pip arg deprecation & disable kafka tests (#803) * Pin virtualenv * Fixup: use 20.21.1 instead * Replace install-options with config-settings See pypa/pip#11358. * Temporarily disable kafka tests * Add dimensional stats table to stats engine * Add attribute processing to metric identity * Add testing for dimensional metrics * Cover tags as list not dict * Commit suggestions from code review * Fix missing resource error * Add global settings override for otlp_host test * Fix unbound local variable * Remove redundant and miscategorized tests * Migrate and merge otlp utils to core. * Fix virtualenv for newer tox versions and Py27 * Fix validator for Py27 * Fix dimensional metric normalization * Fix lint errors * Fix pypy 27 naming * Add debug override for metric serialization * Fix exit code passthrough in tox script * Make otlp_encode more robust * Add json vs protobuf testing fixture * Remove sklearn py27 testing * Validate resource in OTLP * Revert unrelated changes from code review * Fixup: service.provider assertion --------- Co-authored-by: Hannah Stepanek <[email protected]>
newrelic · Jun 23, 2023 · ad05014 · ad05014
1 parent a69a704
commit ad05014
Show file tree

Hide file tree

Showing 12 changed files with 665 additions and 132 deletions.
diff --git a/.github/scripts/retry.sh b/.github/scripts/retry.sh
@@ -25,4 +25,4 @@ for i in $(seq 1 $retries); do
 done
 
 # Exit with status code of wrapped command
-exit $?
+exit $result
diff --git a/newrelic/config.py b/newrelic/config.py
@@ -495,6 +495,7 @@ def _process_configuration(section):
  _process_setting(section, "debug.disable_certificate_validation", "getboolean", None)
  _process_setting(section, "debug.disable_harvest_until_shutdown", "getboolean", None)
  _process_setting(section, "debug.connect_span_stream_in_developer_mode", "getboolean", None)
+ _process_setting(section, "debug.otlp_content_encoding", "get", None)
  _process_setting(section, "cross_application_tracer.enabled", "getboolean", None)
  _process_setting(section, "message_tracer.segment_parameters_enabled", "getboolean", None)
  _process_setting(section, "process_host.display_name", "get", None)

diff --git a/newrelic/core/config.py b/newrelic/core/config.py
@@ -851,6 +851,7 @@ def default_otlp_host(host):
 _settings.debug.log_untrusted_distributed_trace_keys = False
 _settings.debug.disable_harvest_until_shutdown = False
 _settings.debug.connect_span_stream_in_developer_mode = False
+_settings.debug.otlp_content_encoding = None
 
 _settings.message_tracer.segment_parameters_enabled = True
 

diff --git a/newrelic/core/data_collector.py b/newrelic/core/data_collector.py
@@ -32,6 +32,7 @@
 )
 from newrelic.core.agent_streaming import StreamingRpc
 from newrelic.core.config import global_settings
+from newrelic.core.otlp_utils import encode_metric_data
 
 _logger = logging.getLogger(__name__)
 
@@ -152,17 +153,12 @@ def send_dimensional_metric_data(self, start_time, end_time, metric_data):
  specific metrics.
 
  NOTE: This data is sent not sent to the normal agent endpoints but is sent
- to the MELT API endpoints to keep the entity separate. This is for use
+ to the OTLP API endpoints to keep the entity separate. This is for use
  with the machine learning integration only.
  """
 
- payload = (self.agent_run_id, start_time, end_time, metric_data)
- # return self._protocol.send("metric_data", payload)
-
- # TODO: REMOVE THIS. Replace with actual protocol.
- DIMENSIONAL_METRIC_DATA_TEMP.append(payload)
- _logger.debug("Dimensional Metrics: %r" % metric_data)
- return 200 
+ payload = encode_metric_data(metric_data, start_time, end_time)
+ return self._otlp_protocol.send("dimensional_metric_data", payload, path="/v1/metrics")
 
  def send_log_events(self, sampling_info, log_event_data):
  """Called to submit sample set for log events."""

diff --git a/newrelic/core/otlp_utils.py b/newrelic/core/otlp_utils.py
@@ -12,66 +12,87 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""This module provides common utilities for interacting with OTLP protocol buffers."""
+"""
+This module provides common utilities for interacting with OTLP protocol buffers.
+
+The serialization implemented here attempts to use protobuf as an encoding, but falls
+back to JSON when encoutering exceptions unless the content type is explicitly set in debug settings.
+"""
 
 import logging
 
-_logger = logging.getLogger(__name__)
+from newrelic.common.encoding_utils import json_encode
+from newrelic.core.stats_engine import CountStats, TimeStats
+from newrelic.core.config import global_settings
 
-try:
- from newrelic.packages.opentelemetry_proto.common_pb2 import AnyValue, KeyValue
- from newrelic.packages.opentelemetry_proto.logs_pb2 import (
- LogRecord,
- ResourceLogs,
- ScopeLogs,
- )
- from newrelic.packages.opentelemetry_proto.metrics_pb2 import (
- AggregationTemporality,
- Metric,
- MetricsData,
- NumberDataPoint,
- ResourceMetrics,
- ScopeMetrics,
- Sum,
- Summary,
- SummaryDataPoint,
- )
- from newrelic.packages.opentelemetry_proto.resource_pb2 import Resource
+_logger = logging.getLogger(__name__)
 
- AGGREGATION_TEMPORALITY_DELTA = AggregationTemporality.AGGREGATION_TEMPORALITY_DELTA
- ValueAtQuantile = SummaryDataPoint.ValueAtQuantile
+_settings = global_settings()
+otlp_content_setting = _settings.debug.otlp_content_encoding
+if not otlp_content_setting or otlp_content_setting == "protobuf":
+ try:
+ from newrelic.packages.opentelemetry_proto.common_pb2 import AnyValue, KeyValue
+ from newrelic.packages.opentelemetry_proto.logs_pb2 import (
+ LogRecord,
+ ResourceLogs,
+ ScopeLogs,
+ )
+ from newrelic.packages.opentelemetry_proto.metrics_pb2 import (
+ AggregationTemporality,
+ Metric,
+ MetricsData,
+ NumberDataPoint,
+ ResourceMetrics,
+ ScopeMetrics,
+ Sum,
+ Summary,
+ SummaryDataPoint,
+ )
+ from newrelic.packages.opentelemetry_proto.resource_pb2 import Resource
 
- otlp_encode = lambda payload: payload.SerializeToString()
- OTLP_CONTENT_TYPE = "application/x-protobuf"
+ ValueAtQuantile = SummaryDataPoint.ValueAtQuantile
+ AGGREGATION_TEMPORALITY_DELTA = AggregationTemporality.AGGREGATION_TEMPORALITY_DELTA
+ OTLP_CONTENT_TYPE = "application/x-protobuf"
 
-except ImportError:
- from newrelic.common.encoding_utils import json_encode
+ otlp_content_setting = "protobuf" # Explicitly set to overwrite None values
+ except Exception:
+ if otlp_content_setting == "protobuf":
+ raise # Reraise exception if content type explicitly set
+ else: # Fallback to JSON
+ otlp_content_setting = "json"
 
- def otlp_encode(*args, **kwargs):
- _logger.warn(
- "Using OTLP integration while protobuf is not installed. This may result in larger payload sizes and data loss."
- )
- return json_encode(*args, **kwargs)
 
- Resource = dict
- ValueAtQuantile = dict
+if otlp_content_setting == "json":
  AnyValue = dict
  KeyValue = dict
- NumberDataPoint = dict
- SummaryDataPoint = dict
- Sum = dict
- Summary = dict
  Metric = dict
  MetricsData = dict
- ScopeMetrics = dict
+ NumberDataPoint = dict
+ Resource = dict
  ResourceMetrics = dict
- AGGREGATION_TEMPORALITY_DELTA = 1
+ ScopeMetrics = dict
+ Sum = dict
+ Summary = dict
+ SummaryDataPoint = dict
+ ValueAtQuantile = dict
  ResourceLogs = dict
  ScopeLogs = dict
  LogRecord = dict
+
+ AGGREGATION_TEMPORALITY_DELTA = 1
  OTLP_CONTENT_TYPE = "application/json"
 
 
+def otlp_encode(payload):
+ if type(payload) is dict:
+ _logger.warning(
+ "Using OTLP integration while protobuf is not installed. This may result in larger payload sizes and data loss."
+ )
+ return json_encode(payload)
+ else:
+ return payload.SerializeToString()
+
+
 def create_key_value(key, value):
  if isinstance(value, bool):
  return KeyValue(key=key, value=AnyValue(bool_value=value))
@@ -85,11 +106,13 @@ def create_key_value(key, value):
  # those are not valid custom attribute types according to our api spec,
  # we will not bother to support them here either.
  else:
- _logger.warn("Unsupported attribute value type %s: %s." % (key, value))
+ _logger.warning("Unsupported attribute value type %s: %s." % (key, value))
 
 
 def create_key_values_from_iterable(iterable):
- if isinstance(iterable, dict):
+ if not iterable:
+ return None
+ elif isinstance(iterable, dict):
  iterable = iterable.items()
 
  # The create_key_value list may return None if the value is an unsupported type
@@ -103,5 +126,93 @@ def create_key_values_from_iterable(iterable):
 
 
 def create_resource(attributes=None):
- attributes = attributes or {"instrumentation.provider": "nr_performance_monitoring"}
+ attributes = attributes or {"instrumentation.provider": "newrelic-opentelemetry-python-ml"}
  return Resource(attributes=create_key_values_from_iterable(attributes))
+
+
+def TimeStats_to_otlp_data_point(self, start_time, end_time, attributes=None):
+ data = SummaryDataPoint(
+ time_unix_nano=int(end_time * 1e9), # Time of current harvest
+ start_time_unix_nano=int(start_time * 1e9), # Time of last harvest
+ attributes=attributes,
+ count=int(self[0]),
+ sum=float(self[1]),
+ quantile_values=[
+ ValueAtQuantile(quantile=0.0, value=float(self[3])), # Min Value
+ ValueAtQuantile(quantile=1.0, value=float(self[4])), # Max Value
+ ],
+ )
+ return data
+
+
+def CountStats_to_otlp_data_point(self, start_time, end_time, attributes=None):
+ data = NumberDataPoint(
+ time_unix_nano=int(end_time * 1e9), # Time of current harvest
+ start_time_unix_nano=int(start_time * 1e9), # Time of last harvest
+ attributes=attributes,
+ as_int=int(self[0]),
+ )
+ return data
+
+
+def stats_to_otlp_metrics(metric_data, start_time, end_time):
+ """
+ Generator producing protos for Summary and Sum metrics, for CountStats and TimeStats respectively.
+
+ Individual Metric protos must be entirely one type of metric data point. For mixed metric types we have to
+ separate the types and report multiple metrics, one for each type.
+ """
+ for name, metric_container in metric_data:
+ if any(isinstance(metric, CountStats) for metric in metric_container.values()):
+ # Metric contains Sum metric data points.
+ yield Metric(
+ name=name,
+ sum=Sum(
+ aggregation_temporality=AGGREGATION_TEMPORALITY_DELTA,
+ is_monotonic=True,
+ data_points=[
+ CountStats_to_otlp_data_point(
+ value,
+ start_time=start_time,
+ end_time=end_time,
+ attributes=create_key_values_from_iterable(tags),
+ )
+ for tags, value in metric_container.items()
+ if isinstance(value, CountStats)
+ ],
+ ),
+ )
+ if any(isinstance(metric, TimeStats) for metric in metric_container.values()):
+ # Metric contains Summary metric data points.
+ yield Metric(
+ name=name,
+ summary=Summary(
+ data_points=[
+ TimeStats_to_otlp_data_point(
+ value,
+ start_time=start_time,
+ end_time=end_time,
+ attributes=create_key_values_from_iterable(tags),
+ )
+ for tags, value in metric_container.items()
+ if isinstance(value, TimeStats)
+ ]
+ ),
+ )
+
+
+def encode_metric_data(metric_data, start_time, end_time, resource=None, scope=None):
+ resource = resource or create_resource()
+ return MetricsData(
+ resource_metrics=[
+ ResourceMetrics(
+ resource=resource,
+ scope_metrics=[
+ ScopeMetrics(
+ scope=scope,
+ metrics=list(stats_to_otlp_metrics(metric_data, start_time, end_time)),
+ )
+ ],
+ )
+ ]
+ )