Create Metrics Dashboard

[Amzani]

Problem

We aim to provide a dashboard for monitoring our Developer Experience (DX) metrics, which should be collected in asyncapi/community#879

Solution

The dashboard must have the capability to showcase Developer Experience Working group metrics and enable other working groups to add their respective metrics.

• Time to first API Design
• System errors
• Validation errors
• Time to fix a validation error
• Number of created AsyncAPI files
• AsyncAPI 3.0x Adoption

Look & Feel

Link to Figma

Where do we get metrics:

This initiative focuses on the CLI metrics

Definition of metrics

Time to first API Design

It's the time between the creation of a new AsyncAPI file and generation (doc, code...) for a specific user.

 asyncapi generate - asyncapi new file

System errors

Refer to any issue or malfunction of our tools that are not generated by the user (eg. validation errors).
Example: asyncapi/cli#1137

Validation errors

Errors from asyncapi validate

Time to fix a validation error

The delta between a failing validation and successful validation.

 asyncapi validate (success) - asyncapi validate (error)

Number of created AsyncAPI File

From asyncapi new file

AsyncAPI 3.0x Adoption

We can parse the version when a user triggers asyncapi validate

Tech stack

• React/NextJS
• Library: https://www.tremor.so/ or NewRelic Widgets

Note: This is a suggestion! anything else relevant with a valid argument can work as well.

Rabbit holes

1. Assurance in the Metrics Collection:

   • Due to the pending merge of Measure AsyncAPI Adoption asyncapi/community#879, uncertainty surrounds whether we'll have the required metrics in time. To proactively address this risk, let's proceed without being blocked and develop using mocked data.

2. Rate Limit Consideration for New Relic API:

   • Should we opt to construct our own component rather than relying on NewRelic widgets, it's imperative that the backend incorporates a caching mechanism. Refer to [this discussion]((Measure AsyncAPI Adoption asyncapi/community#879 (comment)) for more details.

Scope

• Metrics readiness #2
• Tech stack decision - Tremor or NewRelic Widgets #3
• Metric dashboard Implementation #4

Out of bounds

• We show only Developer Experience (DX) metrics

Success criteria

• The dashboard should be accessible to the public either through the website or a designated URL such as metrics.asyncapi.com.
• Maintainers need to have a clear understanding of the metrics presented on the dashboard.

[KhudaDad414]

@Amzani

AsyncAPI 3.0x Adoption
We can parse the version when a user triggers asyncapi validate

Right now, the query only provides a raw count of AsyncAPI v3 validations. To get a better understanding of adoption trends, could we calculate the percentage of AsyncAPI v3 validations out of the total validations? This would give us a much clearer picture of v3 adoption over time.