Loki log output: Maximum active stream limit exceeded, reduce the number of active streams when 1000s of connection resets received

[stevenebutler]

Brief summary

I'm using Loki to store my log files from test execution and this works well when they are log messages that I generate as the text of my messages is not converted to a label.

However, when K6 generates errors from a huge number of connection resets I start seeing errors from Loki (HTTP 429) with text Maximum active stream limit exceeded, reduce the number of active streams

I'm not sure what's happening as my Loki instance doesn't seem to have recorded any of the connection reset messages. If K6 is logging these kinds of messages as a Loki label value (e.g. for error) it may be exceeding the Loki stream limit.

loki, version 2.6.1 (branch: HEAD, revision: 6bd05c9a4)
build user: root@af90ed01061f
build date: 2022-07-18T08:41:09Z
go version: go1.17.6
platform: linux/amd64

Using Loki config

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /var/lib/loki
  storage:
    filesystem:
      chunks_directory: /var/lib/loki/chunks
      rules_directory: /var/lib/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093
analytics:
  reporting_enabled: false

k6 version

0.40

OS

Ubuntu Linux 20.04

Docker version and image (if applicable)

No response

Steps to reproduce the problem

1. Run K6 using Loki '--log-output=loki=http://localhost:3100/loki/api/v1/push,label.k6=y'
2. Receive a large number of connection resets during a test when other system malfunctions

Expected behaviour

Connection reset errors are logged in Loki along with my fail() and throw message/stacktraces that I use to detect logical problems in the test.

Actual behaviour

Lots of errors logged on the console that error messages couldn't be logged in Loki.
Maximum active stream limit exceeded, reduce the number of active streams

[mstoykov]

Hi @stevenebutler,

Yeah the thing is that k6 has had "labels" in the logs from before we had the loki log output and as such some labels are arguably high cardinality. error being a pretty good example.

Due to that we did add the option of unlabelling labels in the loki output through the usage of allowedLabels documented in the loki log output documetion. It's value will be an array of labels and those will stay labels, while everything else will be appended to the message as key=value which seemed like what loki recommedend at the time.

Maybe we should set a default value for this, but this is a bit of a ... gray area:

• it will be a breaking change
• loki is the only log output that is a service and has such configuration. After(or as part of) Log output as xk6 extension #1939 and Multiple log outputs #2287 we would likely have some "standard" configurations and this seems like one of those.

So I am not certain if we should set it to anything by default, also what will the default be? It seems like the kind of thing that might just need better documentation or a blog post written about it so it's easier to find.

I am somewhat hopeful that this issue will also serve this purpose ;) .

I will keep this open so we can have more discussion about it but will remove the bug label for now.

[stevenebutler]

Oh I wasn't aware of that setting. I have only recently started using Loki with k6 and tried to set it up based on the tutorial in the documents page but don't recall seeing that particular configuration. I think that's because when I searched for Loki and K6 on Google the only relevant result that came back was this blog: https://k6.io/blog/using-loki-to-store-and-query-k6-logs/ and I didn't find the documentation you linked above at all. I did a lot of searching for the error message I received and didn't see any relevant discussion of it online, except to understand what the underlying cause was. Updating the reference documentation with the specific error message may help people find the answer, and it's a shame the reference docs aren't found with a simple search either. I think this is a big gotcha so deserves a prominent warning / advice on the aforementioned blog (due to its front-page prominence in Google search results). I do note the blog links to the documentation you provided, so that's on me for not noticing. [edit] added link to blog article which is the tutorial I used to setup my Loki output.

[stevenebutler]

As a follow-up on my experience trying to configure this: just reading the documentation I am struggling to find
what labels K6 will send by default so I don't know which ones I should allow through. Is there somewhere the
default labels are documented? Can the labels change depending on the kind of error being logged?

At the moment I think all I can do is try to analyze the logs sent to my Loki instance and try to work out what labels are
being sent and which ones would be valuable to me but if they vary on message type I may miss something.

[stevenebutler]

By excluding the error label I'm now getting my errors logged in Loki when I get mass connection reset events across a range of URLs. Thanks very much for the help.

[mstoykov]

Thanks for the more inside. Unfortunately the log output system (as mentioned above) is underdeveloped, in no small part due to it not being the primary thing for k6

There is no list of labels - the code may emit logs with any labels it wants. And at least at one point so did js scripts could also through console.log add labels such as 0, 1 and so on ... which was removed as it seemed pretty pointless 😅 .

Arguably two labels that are always present is level - I would argue most others aren't that interesting. source is also nice but given how loki works, I would expect using just level will be enough 🤷 . And it will be only useful to distinguish between console.log messages and exceptions for example.

You can also use --log-format=json to see all the "labels" that will be emitted and as you can see some of the early debug messages have a bunch of labels that you likely are never going to be using such as phase. You will need to add -v to see the debug messages.

Hope this helps you

[stevenebutler]

Thanks, I settled on type,executor,scenario,level though I probably didn't need any of them.

I also added my own labels to the allowedList but wasn't sure if that was necessary or not. I need to have at least one label always present in the stream (I used K6=yes) so I could identify a stream in a Grafana dashboard to view the logs.

I'm happy to close this issue now.