Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dogstatsd metrics exporter #326

Merged
merged 111 commits into from
Nov 22, 2019
Merged
Show file tree
Hide file tree
Changes from 105 commits
Commits
Show all changes
111 commits
Select commit Hold shift + click to select a range
19346a4
Add MetricAggregator.Merge() implementations
jmacd Oct 30, 2019
cb9557a
Upstream
jmacd Oct 30, 2019
03f7854
Update from feedback
jmacd Oct 30, 2019
09d88a1
Type
jmacd Oct 30, 2019
865b7ab
Upstream
jmacd Oct 30, 2019
5719abe
Upstream
jmacd Oct 30, 2019
c6ca4fd
Ckpt
jmacd Oct 31, 2019
acc2450
Ckpt
jmacd Oct 31, 2019
5bebed5
Upstream
jmacd Oct 31, 2019
5b23af4
Add push controller
jmacd Oct 31, 2019
80e0df8
Ckpt
jmacd Oct 31, 2019
f533814
Upstream
jmacd Oct 31, 2019
3423242
Add aggregator interfaces, stdout encoder
jmacd Oct 31, 2019
a788631
Modify basic main.go
jmacd Oct 31, 2019
fbc50a1
Main is working
jmacd Oct 31, 2019
49aa969
Batch stdout output
jmacd Oct 31, 2019
c2c73be
Sum udpate
jmacd Oct 31, 2019
5ea9128
Rename stdout
jmacd Oct 31, 2019
f0d986c
Add stateless/stateful Batcher options
jmacd Oct 31, 2019
63d79a8
Undo a for-loop in the example, remove a done TODO
jmacd Oct 31, 2019
8716da3
Merge
jmacd Nov 3, 2019
837035d
Update imports
jmacd Nov 3, 2019
9586471
Add note
jmacd Nov 3, 2019
0c09f8c
Rename defaultkeys
jmacd Nov 4, 2019
03ff7d2
Support variable label encoder to speed OpenMetrics/Statsd export
jmacd Nov 4, 2019
88a236e
Lint
jmacd Nov 4, 2019
bf313da
Upstream
jmacd Nov 5, 2019
37a7c7a
Checkpoint
jmacd Nov 5, 2019
c8a321c
Checkpoint
jmacd Nov 5, 2019
db41c9d
Doc
jmacd Nov 5, 2019
dd6229c
Merge
jmacd Nov 5, 2019
0bc5ffe
Precommit/lint
jmacd Nov 6, 2019
c575b08
Simplify Aggregator API
jmacd Nov 6, 2019
3be8e6e
Record->Identifier
jmacd Nov 6, 2019
214b882
Remove export.Record a.k.a. Identifier
jmacd Nov 6, 2019
048c8d9
Checkpoint
jmacd Nov 6, 2019
657c064
Propagate errors to the SDK, remove a bunch of 'TODO warn'
jmacd Nov 6, 2019
0d78cfa
Checkpoint
jmacd Nov 6, 2019
f81aa34
Introduce export.Labels
jmacd Nov 6, 2019
94580ae
Comments in export/metric.go
jmacd Nov 7, 2019
2dee926
Comment
jmacd Nov 7, 2019
e4c9dde
Merge
jmacd Nov 7, 2019
a7623d8
More merge
jmacd Nov 7, 2019
8fcfe95
More doc
jmacd Nov 7, 2019
df3b3af
Complete example
jmacd Nov 7, 2019
4121362
Lint fixes
jmacd Nov 7, 2019
41d3f7b
Add a testable example
jmacd Nov 7, 2019
72872fa
Lint
jmacd Nov 7, 2019
f0f0906
Upstream
jmacd Nov 8, 2019
2425008
Dogstats
jmacd Nov 8, 2019
0160c3d
Let Export return an error
jmacd Nov 8, 2019
79eadad
Upstream
jmacd Nov 8, 2019
760fe57
Checkpoint
jmacd Nov 8, 2019
77e4c3f
add a basic stdout exporter test
jmacd Nov 8, 2019
dc23ae1
Add measure test; fix aggregator APIs
jmacd Nov 9, 2019
18b8340
Upstream
jmacd Nov 9, 2019
dfbc881
Use JSON numbers, not strings
jmacd Nov 9, 2019
9ecdf51
Test stdout exporter error
jmacd Nov 9, 2019
60ab98b
Add a test for the call to RangeTest
jmacd Nov 9, 2019
bf2f1e6
Add error handler API to improve correctness test; return errors from…
jmacd Nov 9, 2019
419ed4f
Undo the previous -- do not expose errors
jmacd Nov 9, 2019
0953112
Add simple selector variations, test
jmacd Nov 9, 2019
8034ebe
Repair examples
jmacd Nov 9, 2019
a472048
Test push controller error handling
jmacd Nov 9, 2019
08a26de
Add SDK label encoder tests
jmacd Nov 9, 2019
c09bd0e
Add a defaultkeys batcher test
jmacd Nov 9, 2019
9ef0a37
Add an ungrouped batcher test
jmacd Nov 9, 2019
557f912
Lint new tests
jmacd Nov 9, 2019
0133786
Respond to krnowak's feedback
jmacd Nov 10, 2019
2d3e3cf
Upstream
jmacd Nov 10, 2019
d12ab8b
Checkpoint
jmacd Nov 10, 2019
7749675
Funciontal example using unixgram
jmacd Nov 11, 2019
6bf8cad
Tidy the example
jmacd Nov 11, 2019
cc88c9d
Add a packet-split test
jmacd Nov 12, 2019
3ff9bf9
More tests
jmacd Nov 12, 2019
e518398
Undo comment
jmacd Nov 12, 2019
72e3d30
Use concrete receivers for export records and labels, since the const…
jmacd Nov 12, 2019
9acdc5a
Bug fix for stateful batchers; clone an aggregator for long term storage
jmacd Nov 12, 2019
4d331e6
Upstream
jmacd Nov 12, 2019
d7a5cda
Upstream
jmacd Nov 14, 2019
399c34c
Remove TODO addressed in #318
jmacd Nov 14, 2019
2a75566
Add errors to all aggregator interfaces
jmacd Nov 14, 2019
efb75ed
Handle ErrNoLastValue case in stdout exporter
jmacd Nov 14, 2019
1153503
Move aggregator API into sdk/export/metric/aggregator
jmacd Nov 15, 2019
623a63a
Update all aggregator exported-method comments
jmacd Nov 15, 2019
e298c94
Document the aggregator APIs
jmacd Nov 15, 2019
d75bc6e
More aggregator comments
jmacd Nov 15, 2019
723d084
Add multiple updates to the ungrouped test
jmacd Nov 15, 2019
8b5a4d6
Fixes for feedback from Gustavo and Liz
jmacd Nov 15, 2019
13e0580
Producer->CheckpointSet; add FinishedCollection
jmacd Nov 15, 2019
b75059e
Process takes an export.Record
jmacd Nov 15, 2019
d03709a
ReadCheckpoint->CheckpointSet
jmacd Nov 15, 2019
93fe58f
EncodeLabels->Encode
jmacd Nov 15, 2019
782cabf
Format a better inconsistent type error; add more aggregator API tests
jmacd Nov 15, 2019
eedaaab
More RangeTest test coverage
jmacd Nov 15, 2019
f67b47c
Make benbjohnson/clock a test-only dependency
jmacd Nov 15, 2019
2e7007d
Merge
jmacd Nov 15, 2019
046db4a
Handle ErrNoLastValue in stress_test
jmacd Nov 15, 2019
a294720
Upstream
jmacd Nov 15, 2019
2ed8fe1
Upstream
jmacd Nov 15, 2019
423fb82
Merge upstream
jmacd Nov 16, 2019
d507863
Upstream
jmacd Nov 19, 2019
12cb491
Upstream
jmacd Nov 20, 2019
4b2e5c3
Update comments; use a pipe vs a unix socket in the example test
jmacd Nov 20, 2019
f93dbe2
Update test
jmacd Nov 20, 2019
c831b6e
Spelling
jmacd Nov 20, 2019
c48ff79
Typo fix
jmacd Nov 20, 2019
260f5fe
Rename DefaultLabelEncoder to NewDefaultLabelEncoder for clarity
jmacd Nov 22, 2019
e50c004
Rename DefaultLabelEncoder to NewDefaultLabelEncoder for clarity
jmacd Nov 22, 2019
4cf2f3c
Test different adapters; add ForceEncode to statsd label encoder
jmacd Nov 22, 2019
889414d
Upstream
jmacd Nov 22, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions exporter/metric/dogstatsd/dogstatsd.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
// Copyright 2019, OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package dogstatsd // import "go.opentelemetry.io/otel/exporter/metric/dogstatsd"

import (
"bytes"

"go.opentelemetry.io/otel/exporter/metric/internal/statsd"
export "go.opentelemetry.io/otel/sdk/export/metric"
)

type (
Config = statsd.Config

// Exporter implements a dogstatsd-format statsd exporter,
// which encodes label sets as indepenent fields in the
// output.
//
// TODO: find a link for this syntax. It's been copied out of
// code, not a specification:
//
// https:/stripe/veneur/blob/master/sinks/datadog/datadog.go
Exporter struct {
*statsd.Exporter
*statsd.LabelEncoder
}
)

var (
_ export.Exporter = &Exporter{}
_ export.LabelEncoder = &Exporter{}
)

// New returns a new Dogstatsd-syntax exporter. This type implements
// the metric.LabelEncoder interface, allowing the SDK's unique label
// encoding to be pre-computed for the exporter and stored in the
// LabelSet.
func New(config Config) (*Exporter, error) {
jmacd marked this conversation as resolved.
Show resolved Hide resolved
exp := &Exporter{
LabelEncoder: statsd.NewLabelEncoder(),
}

var err error
exp.Exporter, err = statsd.NewExporter(config, exp)
return exp, err
}

// AppendName is part of the stats-internal adapter interface.
func (*Exporter) AppendName(rec export.Record, buf *bytes.Buffer) {
_, _ = buf.WriteString(rec.Descriptor().Name())
}

// AppendTags is part of the stats-internal adapter interface.
func (e *Exporter) AppendTags(rec export.Record, buf *bytes.Buffer) {
labels := rec.Labels()

if labels.Encoder() != e {
// TODO: This case could be handled by directly
// encoding the labels at this point, but presently it
// should not occur.
panic("Should have self-encoded labels")
}

_, _ = buf.WriteString(labels.Encoded())
}
87 changes: 87 additions & 0 deletions exporter/metric/dogstatsd/example_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
package dogstatsd_test

import (
"context"
"fmt"
"io"
"log"
"sync"
"time"

"go.opentelemetry.io/otel/api/key"
"go.opentelemetry.io/otel/api/metric"
"go.opentelemetry.io/otel/exporter/metric/dogstatsd"
"go.opentelemetry.io/otel/sdk/metric/batcher/ungrouped"
"go.opentelemetry.io/otel/sdk/metric/controller/push"
"go.opentelemetry.io/otel/sdk/metric/selector/simple"
)

func ExampleNew() {
// Create a "server"
wg := &sync.WaitGroup{}
wg.Add(1)

reader, writer := io.Pipe()

go func() {
defer wg.Done()

for {
var buf [4096]byte
n, err := reader.Read(buf[:])
if err == io.EOF {
return
} else if err != nil {
log.Fatal("Read err: ", err)
} else if n >= len(buf) {
log.Fatal("Read small buffer: ", n)
} else {
fmt.Print(string(buf[0:n]))
}
}
}()

// Create a meter
selector := simple.NewWithExactMeasure()
exporter, err := dogstatsd.New(dogstatsd.Config{
// The Writer field provides test support.
Writer: writer,

// In real code, use the URL field:
//
// URL: fmt.Sprint("unix://", path),
})
if err != nil {
log.Fatal("Could not initialize dogstatsd exporter:", err)
}
// The ungrouped batcher ensures that the export sees the full
// set of labels as dogstatsd tags.
batcher := ungrouped.New(selector, false)

// The pusher automatically recognizes that the exporter
// implements the LabelEncoder interface, which ensures the
// export encoding for labels is encoded in the LabelSet.
pusher := push.New(batcher, exporter, time.Hour)
pusher.Start()

ctx := context.Background()

key := key.New("key")

// pusher implements the metric.MeterProvider interface:
meter := pusher.GetMeter("example")

// Create and update a single counter:
counter := meter.NewInt64Counter("a.counter", metric.WithKeys(key))
labels := meter.Labels(key.String("value"))

counter.Add(ctx, 100, labels)

// Flush the exporter, close the pipe, and wait for the reader.
pusher.Stop()
writer.Close()
wg.Wait()

// Output:
// a.counter:100|c|#key:value
}
Loading