Add FlatObject FieldMapper (opensearch-project#6507)

* Add FlatObject FieldMapper

Signed-off-by: Mingshi Liu <[email protected]>

* resolve import package for HttpHost

Signed-off-by: Mingshi Liu <[email protected]>

* Dynamic Create FlatObjectFieldType for dotpath field

Signed-off-by: Mingshi Liu <[email protected]>

* Rename flat-object to flat_object and fix CI tests

Signed-off-by: Mingshi Liu <[email protected]>

* Organized package

Signed-off-by: Mingshi Liu <[email protected]>

* resolved compile error

Signed-off-by: Mingshi Liu <[email protected]>

* organize package

Signed-off-by: Mingshi Liu <[email protected]>

* Add integration tests and remove benchmark

Signed-off-by: Mingshi Liu <[email protected]>

* fix IT tests

Signed-off-by: Mingshi Liu <[email protected]>

* Skip IT tests before 2.7.0

Signed-off-by: Mingshi Liu <[email protected]>

* Revert "Skip IT tests before 2.7.0"

Signed-off-by: Mingshi Liu <[email protected]>

* Add more IT tests for supported queries

Signed-off-by: Mingshi Liu <[email protected]>

* Removed license head and add tests for wildcard query

Signed-off-by: Mingshi Liu <[email protected]>

* Add tests for array, nested-arrary, number and float

Signed-off-by: Mingshi Liu <[email protected]>

* Upgrade FlatObjectFieldMapperTests to  MapperTestCase

- Upgrading `FlatObjectFieldMapperTests` from `MapperServiceTestCase` to `MapperTestCase`. The `MapperTestCase` explicitly forces us to:
	- Test parameter updates (empty now)
	- Explicitly specify if the field supports Meta and Boost (if not, relevant tests are automatically skipped)
- Test also the substring fields
- Add new test `testMapperServiceHasParser` to verify the new `flat_object` field mapper is present in mapper service registry
- Remove duplicated test and assertions methods
- Removed `testExistsQueryDocValuesDisabledWithNorms` as this test was not adding much. We shall reintroduce similar test later if we decide that we want to support ExistsQuery and what to do if DocValue are disabled and Norms enabled.

Signed-off-by: Lukáš Vlček <[email protected]>

* Add exist query in FlatObjectFieldMapperTests and FlatObjectFieldDataTests

Signed-off-by: Mingshi Liu <[email protected]>

* Add IT tests for painless query in testDocValues

Signed-off-by: Mingshi Liu <[email protected]>

* Add filter script (Painless) test for flat_object

While it is not possible to use flat_object field in scripting filter context to access doc values (like `doc[<flat_object>.<field_x>]`) it is possible to call `doc[<flat_object>].size()` to get number of fields inside the flat_object field.

- Reorganize flat_object yaml tests into sections:
  - Mappings
  - Supported
  - Unsupported
- Scripting (Painless) yamlRest tests need to go into lang-painless module

Signed-off-by: Lukáš Vlček <[email protected]>

* Removed Normalizer

Signed-off-by: Mingshi Liu <[email protected]>

* removed unused codes

Signed-off-by: Mingshi Liu <[email protected]>

* Remove non-relevant Javadoc from DynamicKeyFieldMapper

Signed-off-by: Lukáš Vlček <[email protected]>

* Improve flat_object scripting test

Make it more obvious what the `doc[<flat_field>].size()` number represents.

Signed-off-by: Lukáš Vlček <[email protected]>

* Add test for mapping parameters

Mapping parameters are not allowed in the initial version. This commit adds a test to demonstrate that trying to specify index/search analyzer for the flat_object field will fail.

Signed-off-by: Lukáš Vlček <[email protected]>

* remove IndexAnalyzer from Builder

Signed-off-by: Mingshi Liu <[email protected]>

* remove IndexAnalyzer from Builder

Signed-off-by: Mingshi Liu <[email protected]>

---------

Signed-off-by: Mingshi Liu <[email protected]>
Signed-off-by: Lukáš Vlček <[email protected]>
Co-authored-by: Lukáš Vlček <[email protected]>

CHANGELOG.md

@@ -118,6 +118,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 ### Changed
 - Require MediaType in Strings.toString API ([#6009](https:/opensearch-project/OpenSearch/pull/6009))
 - [Refactor] XContent base classes from xcontent to core library ([#5902](https:/opensearch-project/OpenSearch/pull/5902))
+- Added a new field type: flat_object ([#6507](https:/opensearch-project/OpenSearch/pull/6507))
 
 ### Deprecated
 - Map, List, and Set in org.opensearch.common.collect ([#6609](https:/opensearch-project/OpenSearch/pull/6609))

modules/lang-painless/src/yamlRestTest/resources/rest-api-spec/test/painless/30_search.yml

@@ -482,3 +482,100 @@
  }]
  - match: { error.root_cause.0.type: "illegal_argument_exception" }
  - match: { error.root_cause.0.reason: "script score function must not produce negative scores, but got: [-9.0]"}
+
+---
+
+"Flat-object fields from within the scripting":
+ - skip:
+ version: " - 2.99.99"
+ reason: "flat_object is introduced in 3.0.0 in main branch"
+
+ - do:
+ indices.create:
+ index: test
+ body:
+ mappings:
+ properties:
+ flat:
+ type : "flat_object"
+
+ # This document has 6 distinct parts in its flat_object field paths:
+ # - flat.field_1
+ # - flat.field_2
+ # - flat.field_3
+ # - flat.inner
+ # - flat.field_A
+ # - flat.field_B
+ - do:
+ index:
+ index: test
+ id: 1
+ body: {
+ "flat": {
+ "field_1": "John Doe",
+ "field_2": 33,
+ "field_3": false,
+ "inner": {
+ "field_A": ["foo", "bar"],
+ "field_B": false
+ }
+ }
+ }
+
+ - do:
+ index:
+ index: test
+ id: 2
+ body: {
+ "flat": {
+ "field_1": "Joe Public",
+ "field_2": 45
+ }
+ }
+
+ - do:
+ indices.refresh:
+ index: test
+
+ # It is possible to filter based on the number of distinct parts of flat_object field paths
+ - do:
+ search:
+ body: {
+ _source: true,
+ query: {
+ bool: {
+ filter: {
+ script: {
+ script: {
+ source: "doc['flat'].size() == 6",
+ lang: "painless"
+ }
+ }
+ }
+ }
+ }
+ }
+
+ - length: { hits.hits: 1 }
+ - match: { hits.hits.0._source.flat.field_1: "John Doe" }
+
+ - do:
+ search:
+ body: {
+ _source: true,
+ query: {
+ bool: {
+ filter: {
+ script: {
+ script: {
+ source: "doc['flat'].size() < 6",
+ lang: "painless"
+ }
+ }
+ }
+ }
+ }
+ }
+
+ - length: { hits.hits: 1 }
+ - match: { hits.hits.0._source.flat.field_1: "Joe Public" }