Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add copy ingest processor #56985

Closed
wants to merge 1 commit into from

Conversation

gaobinlong
Copy link
Contributor

@gaobinlong gaobinlong commented May 20, 2020

Relates to #55682 and #51046.
The main point of this PR is adding a copy ingest processor which can copy the value of one field to another and even the top level of the document. The supported data types which can be copied are boolean, number, array, object, string, date.

@astefan astefan added the :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP label May 20, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Ingest)

@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label May 20, 2020
@danhermann danhermann self-requested a review May 20, 2020 11:08
@danhermann
Copy link
Contributor

@gaobinlong, thank you for this contribution. It's useful functionality that we would like to incorporate into ingest pipelines. We are reconsidering how we've organized the processors and are trying to determine whether this fits best as part of the existing set processor or as its own separate processor. If you'll bear with us while we consider that, we will let you know and then work to get this PR refactored if necessary and then merged.

@gaobinlong
Copy link
Contributor Author

@danhermann, I have tried to use set processor to copy one field to another, but found that we cannot copy array,object and boolean, because the template snippet always produces string value. I'm eager to know your idea about that and glad to do more work.

@danhermann
Copy link
Contributor

@gaobinlong, apologies for the delay on this one. We decided that this is how the set processor should work instead of producing string values as you note that it does now. If you would like to continue this PR by modifying the set processor to correctly handle, I would be happy to review it.

@gaobinlong
Copy link
Contributor Author

@danhermann, I'm glad to continue doing that and will refractor the code soon.

@gaobinlong
Copy link
Contributor Author

@danhermann , I have refactored the code and opened a new PR #63540, this PR can be closed lately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP Team:Data Management Meta label for data/management team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants