Enable Partial Flat Object

[mingshl]

Description

Before this PR, flat_object can only be defined at the root field of JSON object in 2.7 and 2.8, because the json object is flatten so it lost the structure of a tree. Now, this PR enables json object to be partially mapped as flat_object field type while other fields in the JSON can be other field type.

The main idea is to remember the flat object field name as mappedFieldTypeName in the dynamic created field (only in query time and it will vanish after returning result), which helps the logic to identify the starting point of a flat object, to decide if a query contains a dot path after flat object field name.

In the example given in the issue,

issue.label is defined as a flat_object, so issue.labels is the flat object field name, andissue.label.comment contains a dot path after the flat_object field name .

Related Issues

#7136

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ❌
• URL:
• CommitID: b78a505
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testPitCreatedOnReplica
      1 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure

• URL: https://build.ci.opensearch.org/job/gradle-check/17265/
• CommitID: 220dc24
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #7997 (e9a25c7) into main (c1c23b4) will increase coverage by 0.07%.
The diff coverage is 87.65%.

@@             Coverage Diff              @@
##               main    #7997      +/-   ##
============================================
+ Coverage     70.90%   70.97%   +0.07%     
- Complexity    56903    56939      +36     
============================================
  Files          4758     4758              
  Lines        269225   269234       +9     
  Branches      39407    39403       -4     
============================================
+ Hits         190881   191092     +211     
+ Misses        62256    62041     -215     
- Partials      16088    16101      +13     

Impacted Files	Coverage Δ
...opensearch/index/translog/NoOpTranslogManager.java	63.63% <0.00%> (-1.99%)	⬇️
...search/index/shard/RemoteStoreRefreshListener.java	82.84% <66.66%> (-1.16%)	⬇️
...opensearch/index/mapper/FlatObjectFieldMapper.java	54.25% <80.00%> (+10.06%)	⬆️
...va/org/opensearch/index/store/RemoteDirectory.java	88.37% <87.50%> (-0.52%)	⬇️
...h/cluster/metadata/MetadataCreateIndexService.java	79.64% <100.00%> (-0.11%)	⬇️
...earch/index/store/RemoteSegmentStoreDirectory.java	92.09% <100.00%> (+0.17%)	⬆️
...search/index/translog/InternalTranslogManager.java	72.02% <100.00%> (+0.19%)	⬆️

... and 470 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testCancelPrimaryAllocation
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/17331/
• CommitID: b78a505
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[mingshl]

PR Ready for reviews and comments @reta @dblock

[owaiskazi19]

@mingshl can you add tests for the new changes?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication

• URL: https://build.ci.opensearch.org/job/gradle-check/19099/
• CommitID: 3ec3362
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[macohen]

• closes [Enhancement] Enable Partial Flat_object Field Type #7136

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/19310/
• CommitID: e9a25c7

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessOnThirdAttemptAttempt

• URL: https://build.ci.opensearch.org/job/gradle-check/19348/
• CommitID: 4db5e12
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.