Add Support for JSON Format in ULS Logs

[sethumadhav07]

Is your feature request related to a problem? Please describe.

Currently, the ULS logs are displayed in a plain text format, which limits their usability with third-party integrations like Kibana. Integrating these logs with Kibana requires them to be in JSON format, which can include mandatory fields populated with certain values for proper ingestion.

Example of current log format:

2024-08-06 05:37:43,988 ULS D Logging initialized
2024-08-06 05:37:43,988 ULS I No install ID file found - creating ID + File
2024-08-06 05:37:43,989 ULS I Created & saved a new installation id: b'NlJNMDk2LTIwMjQwODA2LTEuNy41'
2024-08-06 05:37:43,989 ULS D UlsMonitoring monitoring thread started...
2024-08-06 05:37:43,989 ULS D UlsMonitoring sleeping 300 sec...
2024-08-06 05:37:43,989 ULS I UlsOutput Selected Output Type: HTTP
2024-08-06 05:37:43,989 ULS D UlsTransformation - No Transformation loaded
2024-08-06 05:37:43,990 ULS I UlsInputCli - selected product: EAA
2024-08-06 05:37:43,990 ULS I UlsInputCli - selected feed: ACCESS

Describe the solution you'd like
We would like the ULS logs to support JSON format output. This change would enable the logs to be sent to and ingested by third-party integrations such as Kibana more effectively. Additionally, it would be beneficial to support adding a few custom fields to the JSON logs to meet Kibana's mandatory field requirements for our use case.

Describe alternatives you've considered
An alternative could be using an external log parser to convert the current plain text logs into JSON format before sending them to Kibana. However, this adds complexity and overhead, and it's not as efficient or straightforward as having native JSON support in the ULS logs.

Additional context
Supporting JSON format directly within ULS logs would streamline integration with tools like Kibana and enhance log analysis and monitoring capabilities.

Below is a sample of how the JSON log format might look:
{
"timestamp": "2024-08-06T05:37:43.988Z",
"log_level": "Debug",
"component": "ULS",
"message": "Logging initialized",
"host": "XXXX",
"app": "EAA"
}

[MikeSchiessl]

Hi @sethumadhav07, sorry for the delay in my reply - I have just returned from OOO.

You are correct, a "JSON" formatted log output would in deed make a lot more sense for logging systems. I will have a look at how much work this would involve.

As it is not a vital function of ULS, it might take a while until the fix makes it into an active sprint.
So please be a little patient upon this topic.

Best
Mike

[sethumadhav07]

Thanks. It would be nice if it can be done soon. SAP uses Akamai products (for e.g Akamai EAA), and it would be nice to have this feature to fulfill our compliance needs.

[MikeSchiessl]

ACK,

I just had a look at the codebase and it seemed to be just a minor change in the logging format.
I have implemented the change in the latest development version, you can find here or as a pre-built docker container here.

You can enable the "json" log output either by using the "--json-log" command line argument or by setting the "ULS_JSONLOG=True" ENV var.

Please give it a shot if this solves the above enhancement request.

Best
Mike

[sethumadhav07]

Hi Mike,

The JSON output generated by the current flag is quite useful, but in our use case, it's not entirely sufficient for ingestion into Kibana. Kibana expects certain default fields to be present and populated for the logs to be processed correctly.

It would be really helpful if we could have the ability to transform the ULS logs so that we can add our own default fields alongside the existing ones. Specifically, it would be great if the timestamp value could be mapped to a different field, such as @timestamp. This would make the logs compatible with a broader range of tools.

For our use case, the following fields need to be present in the JSON logs for successful ingestion into Kibana:

• @timestamp: This should be in ISO8601 format.
• type: Hardcoded as "log" or "metric".
• data_version: A hardcoded value.
• application: The name of our application, set by us.
• roletype

Please let me know if you need any additional details or examples to clarify the requirements.

[MikeSchiessl]

Hey @sethumadhav07 ,
got you !

I've added another "flag" into the development branch, that should actually give you full control of how the logs look like:

--ulslogformat '{"timestamp": "%(asctime)s", "log_level": "%(levelname)s", "component": "%(name)s", "message": "%(message)s", "pathname": "%(pathname)s"}'

Please read the following doc for a full wrap up on how to use the log formatting option.

As this line overwrites plain text AND the json format, you don't need necessarily set the --json-log flag any longer ;)
I have kept the --json-logif someone wants json logs but does not have the requirement to adopt his own logging scheme.

Please have a look and let me know if this suits your requirements ;)

Best regards & a happy weekend
Mike

[sethumadhav07]

Hi @MikeSchiessl,

This looks great and meets our requirements. One suggestion: it would be beneficial to add a flag for setting the date format for the timestamp. The logging.Formatter in Python allows you to specify the date format using the datefmt parameter, as detailed here: https://docs.python.org/3/library/logging.html#formatter-objects. We could add a --datefmt '%Y-%m-%dT%H:%M:%S%z' flag to define the format accordingly.

[MikeSchiessl]

Hi @sethumadhav07 ,

thanks for confirming. I am happy to hear that the provided approach suits your requirements.
I'll review the your proposal on the date format change - likely it will also make it in ULS 1.8.0.

Besides the format addition, are you ok to close this ticket ?

Best regards
Mike

[sethumadhav07]

I would like to know the release date of ULS 1.8.0? I appreciate your prompt response. Yes, this ticket can be closed.

[MikeSchiessl]

We have not yet planned an official date but I would say it's safe to expect it somewhere mid / end next week or the beginning /mid of the week after the next. This all depends a little on other things floating around my desk and I wanted to chuck in one or two additional things i had in mind.

Thanks for confirming the closure - let me know if you have any other issues on the ULS topic

[sethumadhav07]

Sure Thanks

[MikeSchiessl]

FYI: I have added the additional datefmt feature to the development branch.
Next week ill be OOO for a couple of days, but i will try to update my testing scripts in order to run tests for all the new stuff and then I think we're good to go on the release road ;)

[MikeSchiessl]

ULS 1.8.0 is out since yesterday !

Thanks again for contributing with your feature requests.
This is really appreciated

[sethumadhav07]

Thanks @MikeSchiessl for the update. Appreciate the quick response for resolving this issue.