Metric benchmark investigation

[dashpole]

Looking into #5542. Closing, as this is not meant to be merged.

My original local benchmark:

$ go test -benchmem -benchtime=2s -bench=Bench .
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/opentelemetry-go/sdk/benchmark
cpu: AMD EPYC 7B12
BenchmarkPrometheusCounter-24       	762422826	         3.154 ns/op	       0 B/op	       0 allocs/op
BenchmarkOTELCounter-24             	21401206	       113.2 ns/op	       0 B/op	       0 allocs/op
BenchmarkOTELCounterWithLabel-24    	 8772984	       271.5 ns/op	      16 B/op	       1 allocs/op

Exemplar collection accounts for ~50ns of the overhead (a1bead9), even though I believe we shouldn't be collecting exemplars by default, and we aren't doing tracing. This is probably a good area to optimize.

BenchmarkOTELCounter-24             	42491396	        56.06 ns/op	       0 B/op	       0 allocs/op
BenchmarkOTELCounterWithLabel-24    	10961318	       220.1 ns/op	      16 B/op	       1 allocs/op

Attribute cardinality limiting seems to account for a very small (~2ns) portion of the overhead.

Lookup based on the attribute set accounts for ~40ns of the overhead for the no-attributes case, and the vast majority of the overhead for the with-attributes case. OTel would need to introduce bound instruments to remove this chunk of overhead.

BenchmarkOTELCounter-24             	148504042	        16.23 ns/op	       0 B/op	       0 allocs/op
BenchmarkOTELCounterWithLabel-24    	38502283	        61.73 ns/op	      16 B/op	       1 allocs/op

Our counter increment function (with locking) accounts for ~8ns of the overhead. We use a simple lock and increment a counter value. Prometheus appears to have implemented some optimizations for this. Benchmarks without any measurement whatsoever:

BenchmarkOTELCounter-24             	299598176	         7.974 ns/op	       0 B/op	       0 allocs/op
BenchmarkOTELCounterWithLabel-24    	43538810	        55.06 ns/op	      16 B/op	       1 allocs/op

The remaining overhead is from the API, and from the Options pattern which requires calling NewAddConfig. This would presumably be eliminated if instruments were already bound to attributes.