cloudRunLocally command

[mstoykov]

This is issue is about making a k6 run -o cloud like command that does know and does more than just outputing metrics to the cloud. But still runs locally.

Even now the cloud output:

1. outputs metrics to the cloud - which is exactly what a metric output should do
2. since v0.46.0 now it can also output trace data ... which is a bit more "iffy" IMO but at least it uses the same input as a metric output, so this seems pretty okay.
3. It also can tell the cloud why the test stopped ... which is more of QoL thing than anything to do with metrics
4. It also can stop the test (if a particular env variable is set) if the output tells it.
5. Probably something else I forget about.

And with #3259 we now intend it to be able to configure log outputs.

Not only does this seem like too many thing a metric output should do ... it makes a bunch of things harder and harder to do.

As in this case for example log outputs are configured earlier than practically anything else. Moving this to a command might not remove the need for some log output refactoring, but at least means that we don't make metric outputs "all powerful".

Also issues such as adding log outputs as extensions and having multiple log outputs might cross this as well.

Solution 0/Current situation

We keep adding Output additions they keep getting more and more complicated. This also means that other parts of the system need to know about outputs and how they handle stuff and that they can actually do something to them.

Solution 1

Make a new command cloudRunLocally(variant 1) or make k6 cloud take an argument --run-locally (variant 2) 🤷‍♂ - naming IMO will be the hardest here.

But that command can do all of those things without keep adding stuff to one output and features that every other output will practically never use.

Solution 2

While writing this I did notice that the argument is called "output" instead of "metric-output" which is likely an oversight, because .. well that was the only thing k6 cared about years ago.

As such if we decide that we want to rename a bunch of stuff we can have a new type Output that is combination of many outputs. There a few problems here the major one of which is naming - we already have cli arg --output which is for metric outputs, so breaking this will be a problem.

The benefit of this is mostly that users will not change the command but add cli args - which IMO isn't that big of a benefit.

From implementation standpoint we will likely want to make this extendable through xk6 which will add complexity.

So Solution 1 is kind of easier in that regard.

Implementation note:

In all cases for the logs the bigger problem is that log outputs is configured way earlier, so there will need to be internal refactor to even enable this without breaking stuff.

But pushing this features to metric outputs is likely going to require more work and definitely a k6 command will have to have access to this.

[dgzlopes]

Part of that extensive list of things this could unlock is sending archives to the Cloud by default.

It would get us closer to Cloud, Local, and PLZ behaving similarly. Right now, for our users, it is challenging to navigate local runs in the Cloud. They have no script attached, and they all look the same.

This could be disabled with a flag (as with the logs) if the customer wants to disable it. Archives should omit env vars by default.

Also, if we had support for this, helping our customers troubleshoot their local runs would be much easier.

cc @codebien

[mpandurovic]

This would be very helpful to have. Many exceptions happen around backend because of the --o cloud test runs and it's difficult to maintain on our end. If we're able to customize the new command (or the extended k6 cloud) command, it would simplify test provisioning, queueing, and some other things.

Q: If we opt to start using the new command - what do we do with the legacy k6 run -o cloud command? Do we keep supporting it for a while and then after some time deprecate it? Or are we going to maintain both indefinitely?

[mstoykov]

Q: If we opt to start using the new command - what do we do with the legacy k6 run -o cloud command? Do we keep supporting it for a while and then after some time deprecate it? Or are we going to maintain both indefinitely?

If the plan for it ends up being to configure multiple "output" (not only metric ones) and coordinate with backend more. Then in that case it still needs to output metrics to the cloud - which is what the -o cloud says.

This would be very helpful to have. Many exceptions happen around backend because of the --o cloud test runs and it's difficult to maintain on our end. If we're able to customize the new command (or the extended k6 cloud) command, it would simplify test provisioning, queueing, and some other things.

I am not certain to what extent those will be fixable - but will make a lot more sense in a command like that then in a metric output definitely.

It might also be beneficial to look this from the perspective of #3218 and think if that will play nicely with it as well.