Logs intake endpoint

[felixbarny]

To enable agents to ship logs, the APM Server should implement a dedicated logs intake endpoint.

See this PR for a specification of the endpoint: elastic/apm#252

Notes on metadata:
The intake metadata event is not ECS compliant. Therefore, it has to be converted, just as we already do it in the event intake API.

Also, it contains a lot of information that we probably don't want to add to every log event, for example process.argv

Should the agents make sure to omit those fields or should the server do that?

Index
Which index should the APM Server write to? The Logs App should include that index by default.

There should also be an ILM setup for this index.

De-dotting
When I said the logs lines are all ECS compatible, that's actually not entirely true. The log events use dotted property names as an equivalent for nesting. For example: log.level. The main reasons are readability and easier and more performant JSON serialization in the loggers.

I've created an issue for a de-dotting filter in beats a while ago: elastic/beats#17021. This filter would be useful to apply within the APM Server logs endpoint processing.

I think that de-dotting is not required to get started. But it does mean that someone who wants to process log events in an ingest node pipeline, for example, will face issues.

[graphaelli]

@ruflin Can we consider writing to to logs-apm-* for this?

[ruflin]

Yes, that is possible. But I wonder if this might become confusing as these are not really APM logs itself (like from apm-server) but logs from a specific service? Should these also go to logs-generic-default and only if users decide to give it a dataset / namespace to have a more dedicated index?

[felixbarny]

Is the logs-generic-default index already actively used for Filebeat? Or is it just a concept atm and will only actually be used in the future by the Elastic Agent?

Should we go with apm-logs-* now and migrate the indices when we migrate the rest of the APM indices?

[ruflin]

Filebeat is and will not be using the new indexing strategy. The new indices will exist in 7.9 and will be used by the Agent and all the processes it runs.

What is the timeline of the above? It kind of makes sense to migrate everything at the same time but also data migrations are painful.

[graphaelli]

What is the timeline of the above?

The timeline is not set but certainly not before 7.9. We're still in the information gathering step for this issue and it's good to know logs-* is a possibility.

Filebeat is and will not be using the new indexing strategy. The new indices will exist in 7.9 and will be used by the Agent and all the processes it runs.

The processes it runs are other beats right?

• The logs for filebeat, metricbeat, etc will end up in logs-generic-default?
• The logs for the services filebeat is monitoring, even when running as an agent process, will continue to go to filebeat-* for now?
• The logs for the services filebeat is monitoring will eventually go to logs-{dataset}-{namespace}?

It kind of makes sense to migrate everything at the same time but also data migrations are painful.

Agreed, that's a big part of why I want to explore this option. The other part is that this proposes to introduce an entirely new intake and I would like to optimize for what users would expect and provides the most value for them, even if it means managing a non apm- prefixed index. In fact, I think this even a good opportunity to build the infrastructure for that.

Should these also go to logs-generic-default and only if users decide to give it a dataset / namespace to have a more dedicated index?

Two options I had in mind here:

1. Dataset is always apm and the namespace is the {service.name}[-{service.environment}] resulting in a datastream index logs-apm-{service.name}[-{service_environment}]

There's nothing apm about these logs other than the collection method, though that does define the mapping. Should APM Server avoid writing to a generically defined logs-* index? I expect that due to ECS the mappings would at least be compatible, if not identical.

2. Dataset is the service.name and the namespace is service.environment resulting in a datastream index logs-{service.name}[-{service_environment}].

Also, I'm not clear on whether it's compliant with datastreams to have an optional namespace. If required, an absent service.environment, since it's not required by APM, could be none, undefined, etc.

[felixbarny]

What's the correlation between the dataset, which is part of the index name, and the ECS field event.dataset, which is also set in log events? Is it overlap in terminology or does the event.dataset determine the dataset in the index name?

I don't want to go too much off-track here but the thin line between logs and events is going to blur more and more: elastic/apm#47 (comment)

[ruflin]

• Logs for FB and MB: Currently they end up in logs-agent-default. But we are discussing what do do with it for 7.9
• If agent is running FB, all the logs will end up in logs-*-* and not filebeat-*. logs-{dataset}-{namespace} is already the case today. My note was only referring to the case, where Filebeat is not run by the Agent.
• Mappings ECS: I expect all mappings under logs-*-* to be on top of ECS
• Mapping v1 vs v2: I think a few questions can help answer which on is better:
  • Is the mapping really the same for all these different logs?
  • How will the user query these logs?
  • Can some of these logs have different retention periods?
• Namespace: If not namespace is set, default should be used. But APM could also define none or undefined.

@felixbarny The fields this is based on are stream.type, stream.dataset, stream.namespace. event.dataset and stream.dataset are equal but event.dataset should be deprecated. One of the main difference is that stream.dataset is a constant_keyword.

To not go off track, will not comment on logs vs events :-)

[graphaelli]

Thanks for all of the details @ruflin .

we are discussing what do do with it for 7.9

Can you link to an issue or doc once there is one? We'd be happy to participate in a discussion as well.

Is the mapping really the same for all these different logs?

The should be at least compatible (fully ECS) if not identical- we enforce the format via intake. @felixbarny please correct me if I missed something in elastic/apm#252

How will the user query these logs?

The initial proposal doesn't have special UI needs for consumption, they will be used like any other application logs.

Can some of these logs have different retention periods?

That's possible but it would help to understand why this is relevant to decide if its significant.

[felixbarny]

The should be at least compatible (fully ECS) if not identical- we enforce the format via intake.

That's correct. But note that users can add their own custom fields to log events (see what I did there?).

How will the user query these logs?

The initial proposal doesn't have special UI needs for consumption, they will be used like any other application logs.

Yep, mainly through the Logs App.

Can some of these logs have different retention periods?

That's possible but it would help to understand why this is relevant to decide if its significant.

It'd be great if the user could customize the {dataset}-{namespace} part of the index. Either via an ES ingest pipeline or APM Server configuration. The former has the advantage that it can be changed at runtime and without coordinating between different APM Server instances.
The user should be able to use fields from the logs, like service.name and event.dataset to define the index name.

[ruflin]

• Docs/ Issue on monitoring: Will provide it as soon as we get to it.
• How to query: The part I was more referring to, will the user always use service.name = foo AND blabla-query, meaning the user will filter per service anyway?
• Different retention periods: The reason it matters is if you send all to one index, you can't have different retention periods as the ILM policy is per index.
• ECS fields: All indices under logs-*-* must be ECS but with their own fields on top. The question is probably how many users will add ...
• Customise on ingest: It is something we also discussed in some other context. Definitively possible. +1 on not doing it in apm-server.

[axw]

We're changing tack on logging a bit, and this will no longer be required. If that changes, let's reopen.