Add Transform reset logic #2

gwbrown · 2021-02-25T21:44:23Z

This is a draft of reset logic for Transforms. It's not on the main Elasticsearch repo as it depends on infrastructure which has not yet been merged (elastic#69469), but I would like to get eyes on it and this is the easiest way to do so.

hendrikmuhs

LGTM, after that indexes must be deleted, but we can do this in a follow-up

gwbrown · 2021-03-02T22:33:05Z

Thanks for the review @hendrikmuhs! This will be merged as part of elastic#69469.

Regarding deleting the indices, after stopping the transforms, we call into the default implementation of cleanUpFeature, which deletes all the system indices and associated indices registered to that plugin - unless there's specific index cleanup other than that, plugins should rely on that default logic to delete the indices in question.

…astic#69765) Previously we did not resolve the attributes recursively which meant that if a field or expression was re-aliased multiple times (through multiple levels of subqueries), the aliases were only resolved one level down. This led to failed query translation because `ReferenceAttribute`s were pointing to non-existing attributes during query translation. For example the query ```sql SELECT i AS j FROM ( SELECT int AS i FROM test) ORDER BY j ``` failed during translation because the `OrderBy` resolved the `j` ReferenceAttribute to another `i` ReferenceAttribute that was later removed by an Optimization: ``` OrderBy[[Order[j{r}#4,ASC,LAST]]] ! OrderBy[[Order[i{r}#2,ASC,LAST]]] \_Project[[j]] = \_Project[[j]] \_Project[[i]] ! \_EsRelation[test][date{f}elastic#6, some{f}elastic#7, some.string{f}elastic#8, some.string..] \_EsRelation[test][date{f}elastic#6, some{f}elastic#7, some.string{f}elastic#8, some.string..] ! ``` By resolving the `Attributes` recursively both `j{r}` and `i{r}` will resolve to `test.int{f}` above: ``` OrderBy[[Order[test.int{f}elastic#22,ASC,LAST]]] = OrderBy[[Order[test.int{f}elastic#22,ASC,LAST]]] \_Project[[j]] = \_Project[[j]] \_Project[[i]] ! \_EsRelation[test][date{f}elastic#6, some{f}elastic#7, some.string{f}elastic#8, some.string..] \_EsRelation[test][date{f}elastic#6, some{f}elastic#7, some.string{f}elastic#8, some.string..] ! ``` The scope of recursive resolution depends on how the `AttributeMap` is constructed and populated. Fixes elastic#67237

…astic#69765) (elastic#70322) Previously we did not resolve the attributes recursively which meant that if a field or expression was re-aliased multiple times (through multiple levels of subqueries), the aliases were only resolved one level down. This led to failed query translation because `ReferenceAttribute`s were pointing to non-existing attributes during query translation. For example the query ```sql SELECT i AS j FROM ( SELECT int AS i FROM test) ORDER BY j ``` failed during translation because the `OrderBy` resolved the `j` ReferenceAttribute to another `i` ReferenceAttribute that was later removed by an Optimization: ``` OrderBy[[Order[j{r}#4,ASC,LAST]]] ! OrderBy[[Order[i{r}#2,ASC,LAST]]] \_Project[[j]] = \_Project[[j]] \_Project[[i]] ! \_EsRelation[test][date{f}elastic#6, some{f}elastic#7, some.string{f}elastic#8, some.string..] \_EsRelation[test][date{f}elastic#6, some{f}elastic#7, some.string{f}elastic#8, some.string..] ! ``` By resolving the `Attributes` recursively both `j{r}` and `i{r}` will resolve to `test.int{f}` above: ``` OrderBy[[Order[test.int{f}elastic#22,ASC,LAST]]] = OrderBy[[Order[test.int{f}elastic#22,ASC,LAST]]] \_Project[[j]] = \_Project[[j]] \_Project[[i]] ! \_EsRelation[test][date{f}elastic#6, some{f}elastic#7, some.string{f}elastic#8, some.string..] \_EsRelation[test][date{f}elastic#6, some{f}elastic#7, some.string{f}elastic#8, some.string..] ! ``` The scope of recursive resolution depends on how the `AttributeMap` is constructed and populated. Fixes elastic#67237

…c#100592) * Don't print synthetic source in mapping for bwc tests * Move comment. * Don't print synthetic source in mapping for bwc tests #2 * Don't print synthetic source in mapping for bwc tests #2 * Revert "Don't print synthetic source in mapping for bwc tests #2" This reverts commit 034262c. * Revert "Don't print synthetic source in mapping for bwc tests #2" This reverts commit 44e8156. * Revert "Don't print synthetic source in mapping for bwc tests (elastic#100572)" This reverts commit 9322ab9. * Exclude synthetic source test from mixedClusterTests * Update comment.

….10 (elastic#100805) * Don't print synthetic source in mapping for bwc tests * Move comment. * Don't print synthetic source in mapping for bwc tests #2 * Don't print synthetic source in mapping for bwc tests #2 * Revert "Don't print synthetic source in mapping for bwc tests #2" This reverts commit 034262c. * Revert "Don't print synthetic source in mapping for bwc tests #2" This reverts commit 44e8156. * Revert "Don't print synthetic source in mapping for bwc tests (elastic#100572)" This reverts commit 9322ab9. * Exclude synthetic source test from mixedClusterTests * Update comment. * Mute all tsdb tests in mixedClusterTests This is an interim step to stop sporadic test failures, while we try to fix version skip for mixed cluster tests. * Remove old exclusion * Add aggregations too * Mute tests for versions between 8.7-8.10 * Remove mute

* Don't print synthetic source in mapping for bwc tests * Move comment. * Don't print synthetic source in mapping for bwc tests #2 * Don't print synthetic source in mapping for bwc tests #2 * Revert "Don't print synthetic source in mapping for bwc tests #2" This reverts commit 034262c. * Revert "Don't print synthetic source in mapping for bwc tests #2" This reverts commit 44e8156. * Revert "Don't print synthetic source in mapping for bwc tests (elastic#100572)" This reverts commit 9322ab9. * Exclude synthetic source test from mixedClusterTests * Update comment. * Mute all tsdb tests in mixedClusterTests This is an interim step to stop sporadic test failures, while we try to fix version skip for mixed cluster tests. * Remove old exclusion * Add aggregations too * Mute tests for versions between 8.7-8.10 * Remove mute * Restore version skipping for position fields * Restore version skip for synthetic source

….10 (elastic#100805) (elastic#100814) * Don't print synthetic source in mapping for bwc tests * Move comment. * Don't print synthetic source in mapping for bwc tests #2 * Don't print synthetic source in mapping for bwc tests #2 * Revert "Don't print synthetic source in mapping for bwc tests #2" This reverts commit 034262c. * Revert "Don't print synthetic source in mapping for bwc tests #2" This reverts commit 44e8156. * Revert "Don't print synthetic source in mapping for bwc tests (elastic#100572)" This reverts commit 9322ab9. * Exclude synthetic source test from mixedClusterTests * Update comment. * Mute all tsdb tests in mixedClusterTests This is an interim step to stop sporadic test failures, while we try to fix version skip for mixed cluster tests. * Remove old exclusion * Add aggregations too * Mute tests for versions between 8.7-8.10 * Remove mute

…c#100823) * Don't print synthetic source in mapping for bwc tests * Move comment. * Don't print synthetic source in mapping for bwc tests #2 * Don't print synthetic source in mapping for bwc tests #2 * Revert "Don't print synthetic source in mapping for bwc tests #2" This reverts commit 034262c. * Revert "Don't print synthetic source in mapping for bwc tests #2" This reverts commit 44e8156. * Revert "Don't print synthetic source in mapping for bwc tests (elastic#100572)" This reverts commit 9322ab9. * Exclude synthetic source test from mixedClusterTests * Update comment. * Mute all tsdb tests in mixedClusterTests This is an interim step to stop sporadic test failures, while we try to fix version skip for mixed cluster tests. * Remove old exclusion * Add aggregations too * Mute tests for versions between 8.7-8.10 * Remove mute * Restore version skipping for position fields * Restore version skip for synthetic source

…8.7 - 8.10 (elastic#100805) (elastic#100815) * [TEST] Mute all tsdb tests in mixedClusterTests, for versions 8.7 - 8.10 (elastic#100805) * Don't print synthetic source in mapping for bwc tests * Move comment. * Don't print synthetic source in mapping for bwc tests #2 * Don't print synthetic source in mapping for bwc tests #2 * Revert "Don't print synthetic source in mapping for bwc tests #2" This reverts commit 034262c. * Revert "Don't print synthetic source in mapping for bwc tests #2" This reverts commit 44e8156. * Revert "Don't print synthetic source in mapping for bwc tests (elastic#100572)" This reverts commit 9322ab9. * Exclude synthetic source test from mixedClusterTests * Update comment. * Mute all tsdb tests in mixedClusterTests This is an interim step to stop sporadic test failures, while we try to fix version skip for mixed cluster tests. * Remove old exclusion * Add aggregations too * Mute tests for versions between 8.7-8.10 * Remove mute * Fix skip for position fields

…rce (elastic#100827) * [TEST] Mute all tsdb tests in mixedClusterTests, for versions 8.7 - 8.10 (elastic#100805) * Don't print synthetic source in mapping for bwc tests * Move comment. * Don't print synthetic source in mapping for bwc tests #2 * Don't print synthetic source in mapping for bwc tests #2 * Revert "Don't print synthetic source in mapping for bwc tests #2" This reverts commit 034262c. * Revert "Don't print synthetic source in mapping for bwc tests #2" This reverts commit 44e8156. * Revert "Don't print synthetic source in mapping for bwc tests (elastic#100572)" This reverts commit 9322ab9. * Exclude synthetic source test from mixedClusterTests * Update comment. * Mute all tsdb tests in mixedClusterTests This is an interim step to stop sporadic test failures, while we try to fix version skip for mixed cluster tests. * Remove old exclusion * Add aggregations too * Mute tests for versions between 8.7-8.10 * Remove mute * Fix skip for position fields * Restore version skip for synthetic source

When we run the csv-spec tests for ESQL against a real http endpoint we actually run them twice - once async and once sync. But the names of the tests didn't reflect that - they just looked like they were accidentally duplicated. This updates the format. So this: ``` test {string.Trim} test {string.Trim #2} ``` becomes: ``` test {string.Trim ASYNC} test {string.Trim SYNC} ```

Implement Transform feature state reset

5ea27d6

hendrikmuhs approved these changes Mar 2, 2021

View reviewed changes

gwbrown merged commit a61cf38 into williamrandolph:si/reset-features-api Mar 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Transform reset logic #2

Add Transform reset logic #2

gwbrown commented Feb 25, 2021

hendrikmuhs left a comment

gwbrown commented Mar 2, 2021

Add Transform reset logic #2

Add Transform reset logic #2

Conversation

gwbrown commented Feb 25, 2021

hendrikmuhs left a comment

Choose a reason for hiding this comment

gwbrown commented Mar 2, 2021