[master] Security CCS automation POC #99042
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cavokz
force-pushed
the
security_ccs_automation_poc
branch
2 times, most recently
from
d54a93e
to
a07179c
Compare
cavokz
force-pushed
the
security_ccs_automation_poc
branch
from
a07179c
to
5d40cbe
Compare
cavokz
added
release_note:skip
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
MadameSheema
reviewed
May 5, 2021
| @@ -203,5 +224,51 @@ export default ({ getService, getPageObjects }) => { | |||
| expect(hitCount).to.be.lessThan(originalHitCount); | |||
| }); | |||
| }); | |||
|
|
|||
| it('should add detection rule', async () => { | |||
There was a problem hiding this comment.
We don't need to add a test to create a rule, the execution of the rule should be part of it since we need it to perform the assertion.
There was a problem hiding this comment.
Should this reasoning be applied also to the index pattern creation then?
There was a problem hiding this comment.
That would be great! Take a look to my comment above.
MadameSheema
reviewed
May 5, 2021
| await retry.tryForTime(40000, async () => { | ||
| const hitCount = await PageObjects.discover.getHitCount(); | ||
| log.debug('### hit count = ' + hitCount); | ||
| expect(hitCount).to.be.greaterThan('0'); |
There was a problem hiding this comment.
It is better if we assert the number of alerts we are expecting.
There was a problem hiding this comment.
This requires tuning, depends on the observation period and on the time elapsed when the reading is done. I also suspect that the observed index (.monitoring-es-*) get updated. Essentially, I'm not sure we can have a deterministic value here. Maybe I should investigate some other index for applying the rule onto.
There was a problem hiding this comment.
It is complex to use our own index of data in the CCS configuration?
There was a problem hiding this comment.
As anticipated, none of the indices on the remote (except makelogs*, which is non ECS-compatible) has a deterministic number of entries that can be checked here.
Should we consider ingesting some other data then?
There was a problem hiding this comment.
This is probably a good place to use esArchiver, but it's going to take a little bit of work.
The integration-test repo no longer uses node.js which would be required to call esArchiver.
After the CCS config is provisioned we basically do pushd ../kibana, and then start the FTR tests.
FTR tests do load data with esArchiver, but currently only to the local cluster. While it's possible to add other configuration for FTR so that esArchiver could also load data in a remote cluster, I think it would be easier to use the CLI to call it from the jenkins_test.sh script in the integration-test just before starting the FTR.
If you want to go this route, let me know and I'll help with it.
MadameSheema
reviewed
May 5, 2021
| const patternName = await PageObjects.settings.getIndexPageHeading(); | ||
| expect(patternName).to.be('*:makelogs工程-*'); | ||
| }); | ||
|
|
||
| it('create local siem signals index pattern', async () => { |
There was a problem hiding this comment.
As in this file it is already tested that an index pattern can be added, in order to improve execution time, as part of the data preparation we need for our test, we can probably add the index we need through the API.
cavokz
force-pushed
the
security_ccs_automation_poc
branch
from
5d40cbe
to
b250aee
Compare
cavokz
changed the title
MadameSheema
approved these changes
May 11, 2021
There was a problem hiding this comment.
Approving the PR in order to have something already merged and working, but I think we should refactor it to differentiate between the preparation of the test and the test itself. We need also to work to make the test deterministic to make sure that the CCS configuration is fully working.
| await retry.tryForTime(40000, async () => { | ||
| const hitCount = await PageObjects.discover.getHitCount(); | ||
| log.debug('### hit count = ' + hitCount); | ||
| expect(hitCount).to.be.greaterThan('0'); |
There was a problem hiding this comment.
It is complex to use our own index of data in the CCS configuration?
| .expect(200); | ||
| }); | ||
|
|
||
| it('create remote monitoring ES index pattern', async () => { |
There was a problem hiding this comment.
The code inside this test and the above is a preparation of the test we want to perform. So I would refactor it in order to reflect that.
There was a problem hiding this comment.
We actually have 2 other ways to automate creating the index pattern without using the API.
-
The old way we're trying to move away from is using
es_archiver, which let's us save and load a whole .kibana index, which includes the index-pattern saved object. There's still lots of this in use in functional tests. And we do still need it for loading data other than the .kibana index. -
kbnarchiver is pretty new. It basically just wraps the saved object API. There's a CLI for it. So the basic steps would be;
- manually create your index pattern, and any other saved objects you need for your test
- run the CLI to export the saved objects to a .json file
- add the .json file to your PR and code to load that file in your test (often in the before method)
Here's an example PR switching from using esArchiver to kbnArchiver #98063
|
Friendly reminder: Looks like this PR hasn’t been backported yet. |
kibanamachine
added
the
backport missing
|
Friendly reminder: Looks like this PR hasn’t been backported yet. |
1 similar comment
|
Friendly reminder: Looks like this PR hasn’t been backported yet. |
|
Friendly reminder: Looks like this PR hasn’t been backported yet. |
5 similar comments
|
Friendly reminder: Looks like this PR hasn’t been backported yet. |
|
Friendly reminder: Looks like this PR hasn’t been backported yet. |
|
Friendly reminder: Looks like this PR hasn’t been backported yet. |
|
Friendly reminder: Looks like this PR hasn’t been backported yet. |
|
Friendly reminder: Looks like this PR hasn’t been backported yet. |
kibanamachine
removed
the
backport missing
|
This is just a first step and it's not worth backporting. |
💔 Build Failed
Failed CI Steps
Test FailuresKibana Pipeline / general / task-queue-process-23 / X-Pack Endpoint API Integration Tests.x-pack/test/security_solution_endpoint_api_int/apis/resolver/entity·ts.Endpoint plugin Resolver tests Resolver tests for the entity route winlogbeat tests "before all" hook for "returns a winlogbeat sysmon event when the event matches the schema correctly"Standard OutStack TraceKibana Pipeline / general / task-queue-process-23 / X-Pack Endpoint API Integration Tests.x-pack/test/security_solution_endpoint_api_int/apis/resolver/entity·ts.Endpoint plugin Resolver tests Resolver tests for the entity route winlogbeat tests "before all" hook for "returns a winlogbeat sysmon event when the event matches the schema correctly"Standard OutStack TraceKibana Pipeline / general / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/infra/metrics_anomalies·ts.InfraOps App Metrics UI Metrics UI Anomaly Flyout with anomalies present renders more anomalies on threshold changeStandard OutStack Traceand 4 more failures, only showing the first 3. Metrics [docs]
History
To update your PR or re-run it, just comment with: cc @cavokz |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Automate testing of the security signal generation when detection rules are applied to remote clusters. See https:/elastic/security-team/issues/1030.