Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for runtime fields #61776

Merged
merged 109 commits into from
Sep 3, 2020
Merged

Add support for runtime fields #61776

merged 109 commits into from
Sep 3, 2020

Conversation

javanna
Copy link
Member

@javanna javanna commented Sep 1, 2020

This PR includes the work that has been done on the runtime fields feature branch until now. The high level tasks are listed in #59332. The tasks that have not yet been completed can be worked on after merging the feature branch to master.

We are adding a new x-pack plugin called runtime-fields that plugs in a custom mapper which allows to define runtime fields based on a script.
The changes included in this PR that were made outside of the x-pack/plugin/runtime-fields directory are minimal and revolve around 1) making the ScriptService available while parsing index mappings so that the scripts associated to runtime fields can be compiled 2) sharing code to manipulate ranges etc. as it can be reused in runtime fields .

javanna and others added 30 commits June 17, 2020 16:10
Put together some example script contexts for scripted fields. This is
almost certainly wrong, but it gets us something to think about.
This reworks the script contexts to support multiple values for each
hit. Just like before, these are almost certainly not the final
implementation, but they give us something to iterate on.
Painless extensions work differently now.
Uses a temporary hack to allow us to use painless in unit tests.
This is the first draft implementation of scripted field of type keyword. Still lots of TODOs to address, this PR targets a feature branch.
This adds what I think is just about the simplest possible `term` query
implementation for `keyword` script fields and wires it into the field
mapper that we build for them.
Adds tests for the `keyword` typed `script` field's fielddata
implementation. One unit test which runs quickly and an integretion test
for fetching and an integration test for the `terms` agg.
This adds the `exits` and `terms` query for the `keyword` style of
scripted field.
Adds the remaining queries for scripted keyword fields and handles a few
leftover TODOs, mostly around `hashCode` and `equals`.
This updates a test so that runtime fields depend on one another and
fixes support for it.
Instead of relying no createComponents to get the script service, we rely on the query shard context supplier that's available in the parser context, yet we only carry around as callback that allows us to compile the script
This will cause them to fail if you don't have permission to execute
expensive queries.
There seems to be no longer a way to require a field in mapper paramaters. This needs to be fixed upstream.
The error was calling toString on the parameter object. we need to instead print out the returned value by calling getValue
This addresses a TODO around using the script params, which are now parsed from the mappings. It also expand existing tests to verify that params are carried around and accessible in script for both fielddata and queries.

Relates to #59332
This adds the `long` runtime type to `script` fields and implements doc
values, field data, the `term` and `exists` query.
Copy link
Member

@nik9000 nik9000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recognize this code!

Like you told me yesterday, most of the stuff outside of the plugin is about plumbing the script service and handling ranges. Thanks to all of your hard work getting the rest of what we needed into master already.

I found a couple of small things to add to my personal TODO list that I'd forgot about but none of those should block merging.

LGTM

@nik9000
Copy link
Member

nik9000 commented Sep 1, 2020

Looks like there are some core tests failing with runtime fields:

08:51:41   2> REPRODUCE WITH: ./gradlew ':x-pack:plugin:runtime-fields:qa:rest:yamlRestTest' --tests "org.elasticsearch.xpack.runtimefields.rest.CoreTestsWithRuntimeFieldsIT.test {yaml=async_search/20-with-poin-in-time/Async search with point in time}" -Dtests.seed=5BF1520839E0DD82 -Dtests.security.manager=true -Dtests.locale=vi -Dtests.timezone=Asia/Anadyr -Druntime.java=11 

I'm cool skipping those for now to unblock this.

This adds support for the `format` parameter on `long` and `double`
valued long fields. It only adds support for it in the request - in the
mapping it is not yet supported.
Runtime fields needs to see the point in time api for its tests.
We had it for testing early on and forgot to remove it.
@nik9000
Copy link
Member

nik9000 commented Sep 1, 2020

Looks like there are some core tests failing with runtime fields:

I pushed 1ea67d2 to fix this.

nik9000 and others added 7 commits September 1, 2020 12:10
Adds explicit tests for nested fields to the runtime fields project.
And skip all of the core tests for nested fields. They don't really have
a chance of resolving the right fields from the source.
…pts (#61775)

This commit exposes stored fields to runtime fields (through params._fields) and replaces the current way to access _source (source['field']) in favour of the documented way for all existing scripts: params._source .
This commit introduces integration tests for Document level security with queries against runtime fields, as well as field level security used in combination with runtime fields.

Relates to #59332
)

This commit adds tests around field_caps and search against multiple indices, where one index has some fields defined as runtime fields, while another index has the same fields defined as concrete fields. We want to make sure that as long as the data type matches, the two can integrate seamlessly.

Relates to #59332
This commit extends the coverage around max chain depth in runtime fields. We test that direct references are not counted, that 5 references are allowed while 6 are not, and we test that the same error is returned no matter where the runtime field is used.

Relates to #59332
@javanna
Copy link
Member Author

javanna commented Sep 2, 2020

run elasticsearch-ci/docs

Adds a test for `runtime_script` who's script contains
`currentTimeMillis`.
@javanna
Copy link
Member Author

javanna commented Sep 3, 2020

run elasticsearch-ci/packaging-sample-windows

@javanna javanna merged commit 60cda2a into master Sep 3, 2020
@javanna javanna deleted the feature/runtime_fields branch September 3, 2020 07:54
javanna added a commit that referenced this pull request Sep 7, 2020
This commit includes the work that has been done on the runtime fields feature branch until now. The high level tasks are listed in #59332. The tasks that have not yet been completed can be worked on after merging the feature branch.

We are adding a new x-pack plugin called runtime-fields that plugs in a custom mapper which allows to define runtime fields based on a script.
The changes included in this commit that were made outside of the x-pack/plugin/runtime-fields directory are minimal and revolve around 1) making the ScriptService available while parsing index mappings so that the scripts associated to runtime fields can be compiled 2) sharing code to manipulate ranges etc. as it can be reused in runtime fields.

Co-authored-by: Nik Everett <[email protected]>
@javanna javanna added v7.11.0 and removed v7.10.0 labels Oct 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement release highlight :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team v7.11.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants