loki-> jaeger [[email protected]+incompatible/reporter.go]

[eskuai]

Hello,

I am trying to run your demo, i think it's great!
I am using main git branch
As a user,

[vagrant@fedora boot-opentelemetry-tempo]$ docker-compose up

checking dockers ..

[vagrant@fedora boot-opentelemetry-tempo]$ docker ps
CONTAINER ID   IMAGE                                              COMMAND                  CREATED          STATUS          PORTS                                                                                                                                                                                    NAMES
387377145b68   grafana/grafana:7.4.0-ubuntu                       "/run.sh"                56 minutes ago   Up 43 seconds   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp                                                                                                                                                boot-opentelemetry-tempo_grafana_1
4af1c2b2156d   grafana/promtail:2.2.0                             "/usr/bin/promtail -…"   56 minutes ago   Up 43 seconds                                                                                                                                                                                            boot-opentelemetry-tempo_promtail_1
9df372e9b436   prom/prometheus:latest                             "/bin/prometheus --c…"   56 minutes ago   Up 43 seconds   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp                                                                                                                                                boot-opentelemetry-tempo_prometheus_1
699cf488dc37   mnadeem/boot-otel-tempo-api:0.0.1-SNAPSHOT         "sh /app/bin/start-a…"   56 minutes ago   Up 44 seconds   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp                                                                                                                                                boot-opentelemetry-tempo_boot-otel-tempo-api_1
c7a113f40b01   mnadeem/boot-otel-tempo-provider1:0.0.1-SNAPSHOT   "sh /app/bin/start-a…"   56 minutes ago   Up 44 seconds   0.0.0.0:8090->8090/tcp, :::8090->8090/tcp                                                                                                                                                boot-opentelemetry-tempo_boot-otel-tempo-provider1_1
7bb1454e015f   dpage/pgadmin4                                     "/entrypoint.sh"         56 minutes ago   Up 44 seconds   443/tcp, 0.0.0.0:7070->80/tcp, :::7070->80/tcp                                                                                                                                           boot-opentelemetry-tempo_pgadmin_1
40e01d4ef144   grafana/tempo-query:0.7.0                          "/go/bin/query-linux…"   56 minutes ago   Up 44 seconds   0.0.0.0:16686->16686/tcp, :::16686->16686/tcp                                                                                                                                            boot-opentelemetry-tempo_tempo-query_1
91464ff22136   postgres                                           "docker-entrypoint.s…"   56 minutes ago   Up 44 seconds   0.0.0.0:5432->5432/tcp, :::5432->5432/tcp                                                                                                                                                boot-opentelemetry-tempo_provider1-db_1
6c518fb52f80   grafana/tempo:0.7.0                                "/tempo -config.file…"   56 minutes ago   Up 44 seconds   0.0.0.0:14250->14250/tcp, :::14250->14250/tcp, 0.0.0.0:14268->14268/tcp, :::14268->14268/tcp, 0.0.0.0:55680->55680/tcp, :::55680->55680/tcp, 0.0.0.0:3102->3100/tcp, :::3102->3100/tcp   boot-opentelemetry-tempo_tempo_1
c416a8970576   mnadeem/volume_exporter                            "/bin/volume_exporte…"   56 minutes ago   Up 45 seconds   0.0.0.0:9889->9888/tcp, :::9889->9888/tcp                                                                                                                                                boot-opentelemetry-tempo_volume_exporter_1
2208cc1c597e   grafana/loki:2.2.0                                 "/usr/bin/loki -conf…"   56 minutes ago   Up 44 seconds   0.0.0.0:3101->3100/tcp, :::3101->3100/tcp                                                                                                                                                boot-opentelemetry-tempo_loki_1
[vagrant@fedora boot-opentelemetry-tempo]$

I can do sql "insert" into postgres, ok ..
I can get a web resquest-response

[
{
"origin": "SEA",
"destination": "LAS",
"airline": "Deltoid",
"departureTime": "2020-12-01T18:10:00.000+00:00"
},
{
"origin": "SEA",
"destination": "LAS",
"airline": "Unitely",
"departureTime": "2020-12-01T19:25:00.000+00:00"
},
{
"origin": "SEA",
"destination": "LAS",
"airline": "Pan-Um",
"departureTime": "2020-12-01T15:15:00.000+00:00"
}
]

Console messages show that there is somethings running ok

loki_1                       | level=info ts=2023-01-25T09:50:08.153612696Z caller=table_manager.go:476 msg="creating table" table=index_18981
loki_1                       | level=info ts=2023-01-25T09:50:08.153619912Z caller=table_manager.go:476 msg="creating table" table=index_19075
loki_1                       | level=info ts=2023-01-25T09:50:08.153629425Z caller=table_manager.go:476 msg="creating table" table=index_19167
loki_1                       | level=info ts=2023-01-25T09:50:08.153634125Z caller=table_manager.go:476 msg="creating table" table=index_19314
loki_1                       | level=info ts=2023-01-25T09:50:08.153638346Z caller=table_manager.go:476 msg="creating table" table=index_19361
loki_1                       | level=info ts=2023-01-25T09:50:08.153642468Z caller=table_manager.go:476 msg="creating table" table=index_18714
loki_1                       | level=info ts=2023-01-25T09:50:08.153646605Z caller=table_manager.go:476 msg="creating table" table=index_18894
loki_1                       | level=info ts=2023-01-25T09:50:08.153650688Z caller=table_manager.go:476 msg="creating table" table=index_18933
loki_1                       | level=info ts=2023-01-25T09:50:08.15365477Z caller=table_manager.go:476 msg="creating table" table=index_19216
loki_1                       | level=info ts=2023-01-25T09:50:08.153663394Z caller=table_manager.go:476 msg="creating table" table=index_18588
loki_1                       | level=info ts=2023-01-25T09:50:08.153670899Z caller=table_manager.go:476 msg="creating table" table=index_18939
loki_1                       | level=info ts=2023-01-25T09:50:08.153675518Z caller=table_manager.go:476 msg="creating table" table=index_19014
tempo_1                      | level=info ts=2023-01-25T09:50:08.183434929Z caller=compactor.go:91 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo_1                      | level=info ts=2023-01-25T09:50:08.183468568Z caller=compactor.go:95 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant
tempo_1                      | level=info ts=2023-01-25T09:50:38.183476029Z caller=compactor.go:91 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo_1                      | level=info ts=2023-01-25T09:50:38.183556963Z caller=compactor.go:95 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant
boot-otel-tempo-api_1        | 2023-01-25 03:50:45.028  INFO [boot-otel-tempo-api,f0ba2bd160dca1f79ed6ccf0d0b8135d,aacf0ccfaaccffaf] 7 --- [nio-8080-exec-1] i.o.example.flight.FlightController      : Before Service Method Call
boot-otel-tempo-api_1        | 2023-01-25 03:50:45.028  INFO [boot-otel-tempo-api,f0ba2bd160dca1f79ed6ccf0d0b8135d,aacf0ccfaaccffaf] 7 --- [nio-8080-exec-1] i.o.example.flight.FlightService         : Getting flights for "SEA"
boot-otel-tempo-api_1        | 2023-01-25 03:50:45.029  INFO [boot-otel-tempo-api,f0ba2bd160dca1f79ed6ccf0d0b8135d,ae7ab8e6bccd017b] 7 --- [nio-8080-exec-1] i.o.example.flight.FlightClient          : Getting Flights from http://boot-otel-tempo-provider1:8090/flights
boot-otel-tempo-provider1_1  | 2023-01-25 03:50:45.033  INFO [boot-otel-tempo-provider1,f0ba2bd160dca1f79ed6ccf0d0b8135d,a764d501bf0e2483] 7 --- [io-8090-exec-10] i.o.example.flight.FlighController       : processing Request
boot-otel-tempo-provider1_1  | 2023-01-25 03:50:45.033  INFO [boot-otel-tempo-provider1,f0ba2bd160dca1f79ed6ccf0d0b8135d,519e77d05f580ee9] 7 --- [io-8090-exec-10] i.o.example.flight.FlightService         : Doing some work In New span
boot-otel-tempo-api_1        | 2023-01-25 03:50:45.040  INFO [boot-otel-tempo-api,f0ba2bd160dca1f79ed6ccf0d0b8135d,4b36a7c39b9ca986] 7 --- [nio-8080-exec-1] i.o.example.flight.FlightService         : Doing some work In New span
loki_1                       | level=info ts=2023-01-25T09:51:08.142503097Z caller=table_manager.go:208 msg="syncing tables"
loki_1                       | level=info ts=2023-01-25T09:51:08.147230987Z caller=table_manager.go:171 msg="uploading tables"
loki_1                       | level=info ts=2023-01-25T09:51:08.147296101Z caller=table.go:336 msg="uploading table index_19382"
loki_1                       | level=info ts=2023-01-25T09:51:08.147306867Z caller=table.go:363 msg="finished uploading table index_19382"
loki_1                       | level=info ts=2023-01-25T09:51:08.147311227Z caller=table.go:421 msg="cleaning up unwanted dbs from table index_19382"
tempo_1                      | level=info ts=2023-01-25T09:51:08.183746382Z caller=compactor.go:91 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo_1                      | level=info ts=2023-01-25T09:51:08.183771985Z caller=compactor.go:95 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant
provider1-db_1               | 2023-01-25 09:51:08.192 UTC [27] LOG:  checkpoint starting: time
tempo_1                      | level=info ts=2023-01-25T09:51:08.209778748Z caller=flush.go:256 msg="flushing block" userid=single-tenant block=2c376cae-b17a-4cf4-a41c-1593cbe51812
provider1-db_1               | 2023-01-25 09:51:08.221 UTC [27] LOG:  checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.023 s, total=0.029 s; sync files=2, longest=0.021 s, average=0.012 s; distance=0 kB, estimate=0 kB

and again

boot-otel-tempo-api_1        | 2023-01-25 03:52:30.820  INFO [boot-otel-tempo-api,e00a373d52614d324a56f23cd91dd564,eb88cd7496af3413] 7 --- [nio-8080-exec-5] i.o.example.flight.FlightController      : Before Service Method Call
boot-otel-tempo-api_1        | 2023-01-25 03:52:30.820  INFO [boot-otel-tempo-api,e00a373d52614d324a56f23cd91dd564,eb88cd7496af3413] 7 --- [nio-8080-exec-5] i.o.example.flight.FlightService         : Getting flights for "SEA"
boot-otel-tempo-api_1        | 2023-01-25 03:52:30.821  INFO [boot-otel-tempo-api,e00a373d52614d324a56f23cd91dd564,c18b60005f6f3a67] 7 --- [nio-8080-exec-5] i.o.example.flight.FlightClient          : Getting Flights from http://boot-otel-tempo-provider1:8090/flights
boot-otel-tempo-provider1_1  | 2023-01-25 03:52:30.827  INFO [boot-otel-tempo-provider1,e00a373d52614d324a56f23cd91dd564,41b8c42ba1bc195b] 7 --- [io-8090-exec-10] i.o.example.flight.FlighController       : processing Request
boot-otel-tempo-provider1_1  | 2023-01-25 03:52:30.827  INFO [boot-otel-tempo-provider1,e00a373d52614d324a56f23cd91dd564,80270a556d6a59c4] 7 --- [io-8090-exec-10] i.o.example.flight.FlightService         : Doing some work In New span
boot-otel-tempo-api_1        | 2023-01-25 03:52:30.833  INFO [boot-otel-tempo-api,e00a373d52614d324a56f23cd91dd564,b7a2906447b9ad7b] 7 --- [nio-8080-exec-5] i.o.example.flight.FlightService         : Doing some work In New span
tempo_1                      | level=info ts=2023-01-25T09:52:38.183271501Z caller=compactor.go:91 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo_1                      | level=info ts=2023-01-25T09:52:38.183332092Z caller=compactor.go:95 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant

Running tempo, i cannot get anything, it only shows:

tempo_1                      | level=info ts=2023-01-25T09:49:08.184133771Z caller=compactor.go:91 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo_1                      | level=info ts=2023-01-25T09:49:08.184275657Z caller=compactor.go:95 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant
tempo_1                      | level=error ts=2023-01-25T09:49:08.311343846Z caller=frontend_processor.go:59 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = transport is closing"
tempo_1                      | level=error ts=2023-01-25T09:49:08.311410872Z caller=frontend_processor.go:59 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = transport is closing"
tempo_1                      | level=error ts=2023-01-25T09:49:08.311407186Z caller=frontend_processor.go:59 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = transport is closing"
tempo_1                      | level=error ts=2023-01-25T09:49:08.311434239Z caller=frontend_processor.go:59 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = transport is closing"
tempo_1                      | level=error ts=2023-01-25T09:49:08.311450076Z caller=frontend_processor.go:59 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = transport is closing"
tempo_1                      | level=info ts=2023-01-25T09:49:38.183278219Z caller=compactor.go:91 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo_1                      | level=info ts=2023-01-25T09:49:38.183351782Z caller=compactor.go:95 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant

It shows:

Then, if i use jaeger, put a traceid, response is empty and lokie shows:

tempo-query_1                | {"level":"error","ts":1674639544.7755914,"caller":"zap/logger.go:33","msg":"failed to flush Jaeger spans to server: write udp 127.0.0.1:46646->127.0.0.1:6831: write: connection refused","stacktrace":"github.com/uber/jaeger-client-go/log/zap.(*Logger).Error\n\tgithub.com/uber/[email protected]+incompatible/log/zap/logger.go:33\ngithub.com/uber/jaeger-client-go.(*remoteReporter).processQueue.func1\n\tgithub.com/uber/[email protected]+incompatible/reporter.go:288\ngithub.com/uber/jaeger-client-go.(*remoteReporter).processQueue\n\tgithub.com/uber/[email protected]+incompatible/reporter.go:298"}

And, jaeger info is empty ...

what am i doing wrong?

Tx

PD

loki is ok ( it seems that loki doesn't like touching 'job string':

Temp seems ok:

Prometheus ok:

Jaeger ... [no service]

Requesting using url browser, runs ok ... http://localhost:6650/trace/5ad80347b409a4b5d9aa7317c4e7ca3c