Discuss: deprecation path for packages

[mtojek]

We'd like to deprecate cyberark in favor of the cyberarkpas package. Unfortunately it's precedence now and we should decide what is the recommended flow here.

A set of loose thoughts:

1. Introduce additional property "deprecated" to the package manifest, that hides this package from listing in Kibana.
2. Add deprecation notice to package's README. Suggest an alternative?

cc @ruflin @jsoriano @marc-gr @endorama
cc (kibana) @jen-huang

[ruflin]

We need to define what happens with such a package in Kibana / Fleet in case a users installed it. How long will it still be usable? Will it be removed automatically at one stage? How to we inform the user?

[mtojek]

This is of course opinionated:

How long will it still be usable?

The safe option would be to leave as is, just not push any further updates to it and hide in the UI. It should be still possible to force-install it using API.

Will it be removed automatically at one stage?

Never to keep the consistency with any Kibana deployment.

[endorama]

that hides this package from listing in Kibana.

I would not mandate a behaviour, and leave the team responsible to choose how to handle it.
What I think is relevant is to have it in the package spec, so every consumer can automatically act depending on the use case (from hiding in the UI to displaying an alert to being able in Elastic Cloud to contact users of a deprecated package).

How long will it still be usable? Will it be removed automatically at one stage?

Is viable to stop supporting deprecated packages (maybe after a grace period)? If yes I don't see downsides of keeping it deprecated but available. Is there any disadvantage to this approach?

How to we inform the user?

I think there are multiple "kinds" of users and I would expect different ways:

• developers/contributors: who either contributes directly to the package or build features on top of a package
  How to inform: package README, package releases or deprecation notice in logs for installed instances of the package (this require to release a new version with the deprecation, which I think is acceptable and good practice)
• ES operators: is there any "available update" logic for packages? That where I would expect this message to be
• Cloud ES users: like ES operators plus we should be able to contact them directly

[mtojek]

Thanks for your input, Edoardo!

so every consumer can automatically act depending on the use case (from hiding in the UI to displaying an alert to being able in Elastic Cloud to contact users of a deprecated package)

Currently the most important consumer is Fleet UI, so I would leave this decision to Jen and the team on how to perform in case of detecting a deprecated package (which might be already installed).

ES operators: is there any "available update" logic for packages? That where I would expect this message to be

Same here, a question for Kibana team.

Cloud ES users: like ES operators plus we should be able to contact them directly

Actually this is a good question, I'm sure if we have any channel to reach out to paid customers. Should we do something extra than displaying a deprecation notice?
WDYT @mostlyjason ?

[mostlyjason]

Thanks it makes sense that we need a path for deprecating packages.

@mtojek what is the timeline for deprecating it and how long until its no longer supported? Are there any others that will be deprecated soon?

I just looked at our telemetry and I only see 5 clusters using cyberark, so my initial inclination is that this is not one of our highest priorities and it should go on the backlog unless we have more packages being deprecated soon.

[marc-gr]

@mtojek what is the timeline for deprecating it and how long until its no longer supported? Are there any others that will be deprecated soon?

I think @jamiehynds have more context on this

[jamiehynds]

@mostlyjason we shipped a Cyberark Beats module also, which probably explains why we're not seeing strong adoption of the package yet. There will probably be at least one security integration to deprecate per release for the foreseeable, as we're building our experimental integrations from scratch and deprecating the old ones.

If there isn't a clear path for deprecation yet, maybe we could update the package name + description to indicate they are deprecated? Want to avoid confusion as to which integration to use as users onboard sources, eg:

[mostlyjason]

@jamiehynds how many integrations are you planning to deprecate? I'm worried about the user experience if we make deprecation a standard practice because its a breaking change for users. Our new corporate level theme is "make it minor" which means to minimize breaking changes and put extra effort in to make upgrades seamless for users. Is there anything we can do to upgrade integrations in place to make it easier for users?

[jamiehynds]

@mostlyjason we could be looking at roughly 20 integrations eventually. These integrations originated from open source RSA Netwitness parsers that we converted to integrations in an attempt to quickly expand our coverage of security data sources. We shipped them as experimental with a view to iterating and moving towards Beta and GA. However, we've since realised they are exceptionally difficult to maintain/update and easier to build these integrations from scratch. The new integrations will provide a much better user experience (lots of known issues with the RSA parsing).

Outside of these RSA-based integrations, we have no intention of making deprecation a standard practice. We can look at improving the upgrade experience for these particular integrations, but don't believe there's a straight forward path. Worth noting that based on Fleet telemetry, adoption for most of these integrations is pretty low.

[mostlyjason]

Ok thanks for the context! 20 integrations sounds like a lot and will impact the UX if we don't handle this use case. Updating the package name seems straightforward and doesn't require extra investment on the Kibana side. It seems like the number of users is still low so I'm not sure its worth doing more than showing they are deprecated in the package name.

We need a way to filter deprecated packages out from the integrations UI browse tab. ++ on having something in the package spec. I imagine we'll want to update the EPR API to filter them out too?