Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Metricbeat] Enterprise Search Module #27549

Merged
merged 56 commits into from
Sep 30, 2021
Merged

[Metricbeat] Enterprise Search Module #27549

merged 56 commits into from
Sep 30, 2021

Conversation

carlosdelest
Copy link
Member

@carlosdelest carlosdelest commented Aug 23, 2021 •
edited by kovyrin
Loading

Originally, a Space Time ON project from @carlosdelest, later adopted into the roadmap and developed by @kovyrin.

What does this PR do?

Created a new Enterprise Search module, based on current App Search module. Created the first MetricSet that uses the Health API to get metrics. We're planning to add a couple more metricsets going forward, but want to merge the module first to allow for better parallelization of efforts around this project.

Why is it important?

Retrieve observability metrics for Enterprise Search. It will allow Monitoring plugin creation in Kibana, and Dashboards for retrieving metrics information from Enterprise Search.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

  • Unit tests included: mage unitTest
  • Integration tests included: `MODULE=enterprisesearch mage integTest``
  • Run Enterprise Search locally and run this module manually to generate metrics

Additional notes

Please note: There are some changes in this PR we had to make to improve the development experience (or enable it) on M1 macs and a few cosmetic changes to the code outside of our module. We have extracted those changes into separate PRs, but those PRs are still in review, so the changes are visible here as well:

These have been merged, so it is not an issue anymore.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 23, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Aug 23, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-09-30T12:47:33.639+0000

  • Duration: 98 min 7 sec

  • Commit: 9c94e03

Test stats 🧪

Test Results
Failed 0
Passed 9747
Skipped 2560
Total 12307

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

@kovyrin
Make it work on M1 macs
cda34cd
@kovyrin
Make integration tests work on M1 Macs by using the correct architect…
4ebcab0 
…ure for mage binaries
@kovyrin
Switch Enterprise Search Health metrics set to using the TestData magic
d5c98fb
@kovyrin
Remove unused boilerplate
67d70a5
@kovyrin
Make it possible to filter down python integration tests to a specifi…
1f8bf0c 
…c module when running with mage against x-pack metricbeat directory
@kovyrin
Add enterprise search credentials when performing python integration …
f1998c1 
…tests
@kovyrin
Provide a better feedback in test logs when testing is scoped to a si…
c58482f 
…ngle module
@kovyrin
Reverted go module updates because it is really hard to merge from th…
35bbc97 
…e master now.

This reverts commit 5bf6fb2.
@kovyrin
Merge remote-tracking branch 'elastic/master' into carlosdelest/enter…
3a8dc2c 
…prise-search-metricbeat-module
@kovyrin kovyrin changed the title Create Enterprise Search module - SpaceTime ON [Metricbeat] Enterprise Search Module Sep 16, 2021
@kovyrin kovyrin self-assigned this Sep 16, 2021
@kovyrin kovyrin added Metricbeat Metricbeat and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 16, 2021
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 16, 2021
@kovyrin
We need python development headers to be able to install pynacl (noti…
c1d22f6 
…ced this on M1, but not sure if it is M1-specific)
@kovyrin kovyrin mentioned this pull request Sep 20, 2021
Merged
7 tasks
@kovyrin
Make cluster_uuid required
6670dbe
@kovyrin
Final schema for the stats metricset
63e0f90
@kovyrin
Merge remote-tracking branch 'elastic/master' into carlosdelest/enter…
804a979 
…prise-search-metricbeat-module
@kovyrin
Sort all keys before checking them for documentation to make sure mis…
c195502 
…sing documentation error messages are produced in a predictable and repeatable order.

This is really helpful when someone is trying to fix multiple issues and wants to make sure a specific issue is fixed before moving one to the next one. Otherwise, the errors are thrown in a random order and every time a new message is presented to the developer making it very confusing if the previous one had been resolved or not.
@kovyrin
Latest version of the stats schema for Enterprise Search
e537a4d
@kovyrin
go fmt
9d5914e
@kovyrin kovyrin requested a review from sayden September 29, 2021 21:26
@kovyrin
Copy link
Contributor

kovyrin commented Sep 30, 2021

Ok, all of those changes unrelated to this PR have been merged, so now the diff should only contain the new module

sayden
sayden approved these changes Sep 30, 2021
View reviewed changes
Copy link
Contributor

@sayden sayden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the field rework. I leave you the "honor" of merging 🙂

@kovyrin kovyrin merged commit 3c77c9a into elastic:master Sep 30, 2021
@kovyrin kovyrin added backport-v7.16.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Oct 1, 2021
@kovyrin
Copy link
Contributor

kovyrin commented Oct 1, 2021

@mergify backport

mergify bot pushed a commit that referenced this pull request Oct 1, 2021
@carlosdelest
[Metricbeat] Enterprise Search Module (#27549)
e6de9aa 
Added Enterprise Search module with two metricsets: health and stats, each using a corresponding API endpoint

Co-authored-by: Oleksiy Kovyrin <[email protected]>
(cherry picked from commit 3c77c9a)

# Conflicts:
#	metricbeat/docs/fields.asciidoc
@mergify mergify bot mentioned this pull request Oct 1, 2021
Merged
@mergify
Copy link
Contributor

mergify bot commented Oct 1, 2021

Command backport: failure

No backport have been created
No destination branches found

Hey, I reacted but my real name is @Mergifyio

kovyrin added a commit that referenced this pull request Oct 1, 2021
@carlosdelest @kovyrin
[Metricbeat] Enterprise Search Module (#27549)
96be82e 
Added Enterprise Search module with two metricsets: health and stats, each using a corresponding API endpoint

Co-authored-by: Oleksiy Kovyrin <[email protected]>
kovyrin added a commit that referenced this pull request Oct 1, 2021
@mergify @carlosdelest @kovyrin
[7.x](backport #27549) [Metricbeat] Enterprise Search Module (#28214)
d27aa4b 
* [Metricbeat] Enterprise Search Module (#27549)

Added Enterprise Search module with two metricsets: health and stats, each using a corresponding API endpoint

Co-authored-by: Oleksiy Kovyrin <[email protected]>
(cherry picked from commit 3c77c9a)

# Conflicts:
#	metricbeat/docs/fields.asciidoc

* [Metricbeat] Enterprise Search Module (#27549)

Added Enterprise Search module with two metricsets: health and stats, each using a corresponding API endpoint

Co-authored-by: Oleksiy Kovyrin <[email protected]>

* Fix the issue with the linux module not being loaded on a mac

Co-authored-by: Carlos Delgado <[email protected]>
Co-authored-by: Oleksiy Kovyrin <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
@kovyrin kovyrin mentioned this pull request Oct 6, 2021
Merged
7 tasks
Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021
@carlosdelest @kovyrin
[Metricbeat] Enterprise Search Module (elastic#27549)
63878f5 
Added Enterprise Search module with two metricsets: health and stats, each using a corresponding API endpoint

Co-authored-by: Oleksiy Kovyrin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kovyrin kovyrin kovyrin left review comments

@sayden sayden sayden approved these changes

@ruflin ruflin Awaiting requested review from ruflin

Assignees

@kovyrin kovyrin

Labels
7.16-candidate 7.16 candidate backport-v7.16.0 Automated backport with mergify Metricbeat Metricbeat Team:Integrations Label for the Integrations team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@carlosdelest @elasticmachine @kovyrin @sayden @ruflin