Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split test suite with separate docker-compose. #472

Merged
merged 3 commits into from
Aug 7, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions .github/workflows/test-spec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,18 @@ jobs:
- version: 2.0.0
admin_password: admin
- version: 2.15.0
- version: 2.15.0
tests: plugins/index_state_management
- version: 2.15.0
tests: snapshot
- version: 2.16.0
hub: opensearchstaging
ref: '@sha256:bcd7f5d5d30231f24f266064248cc8d3306574948190f7bf93016dff29acf17e'
- version: 3.0.0
hub: opensearchstaging
ref: '@sha256:db1918b2b8f7ef6c22dd6ff54a0640877c3d395a392a53864745024933981e3b'

name: test-opensearch-spec (version=${{ matrix.entry.version }}, hub=${{ matrix.entry.hub || 'opensearchproject' }})
name: test-opensearch-spec (version=${{ matrix.entry.version }}, hub=${{ matrix.entry.hub || 'opensearchproject' }}, tests=${{ matrix.entry.tests || 'default' }})
runs-on: ubuntu-latest

env:
Expand All @@ -58,7 +62,7 @@ jobs:
run: npm ci

- name: Run OpenSearch Cluster
working-directory: .github/opensearch-cluster
working-directory: tests/${{ matrix.entry.tests || 'default' }}
run: docker compose up -d

- name: Get Container ID
Expand All @@ -71,7 +75,8 @@ jobs:
npm run test:spec -- \
--opensearch-insecure \
--opensearch-version=${{ matrix.entry.version }} \
--coverage coverage/test-spec-coverage-${{ matrix.entry.version }}.json
--coverage coverage/test-spec-coverage-${{ matrix.entry.version }}-${{ matrix.entry.tests || 'default' }}.json \
--tests=tests/${{ matrix.entry.tests || 'default' }}

- name: Get Container Logs
if: failure() && steps.container.outputs.CONTAINER_ID
Expand All @@ -82,8 +87,8 @@ jobs:
- name: Upload Test Coverage Results
uses: actions/upload-artifact@v4
with:
name: coverage-${{ matrix.entry.version }}
path: coverage/test-spec-coverage-${{ matrix.entry.version }}.json
name: coverage-${{ matrix.entry.version }}-${{ hashFiles(format('tests/{0}', matrix.entry.tests || 'default')) }}
path: coverage/test-spec-coverage-${{ matrix.entry.version }}-${{ matrix.entry.tests || 'default' }}.json

merge-coverage:
runs-on: ubuntu-latest
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test-tools-integ.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ jobs:
uses: actions/checkout@v4

- name: Run OpenSearch Cluster
working-directory: .github/opensearch-cluster
working-directory: tests/default
run: |
docker compose up -d
sleep 15
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Tester tool now provides better context for non-2XX responses when --verbose is used ([#359](https:/opensearch-project/opensearch-api-specification/pull/359))
- Lock testing for next release of OpenSearch to a specific SHA ([#431](https:/opensearch-project/opensearch-api-specification/pull/431))
- Replace nullable with null type ([#436](https:/opensearch-project/opensearch-api-specification/pull/436))
- Split test suite ([#472])(https:/opensearch-project/opensearch-api-specification/pull/472)

### Deprecated

Expand Down
16 changes: 9 additions & 7 deletions TESTING_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Set up an OpenSearch cluster with Docker:
(Replace `<<your_password>>` with your desired password. If not provided, the default password inside the `docker-compose.yml` file will be used.)
```bash
export OPENSEARCH_PASSWORD=<<your_password>>
cd .github/opensearch-cluster
cd tests/default
docker compose up -d
```

Expand All @@ -36,7 +36,7 @@ npm run test:spec -- --opensearch-insecure

Run a specific test story:
```bash
npm run test:spec -- --opensearch-insecure --tests tests/_core/info.yaml
npm run test:spec -- --opensearch-insecure --tests tests/default/_core/info.yaml
```

Verbose output:
Expand Down Expand Up @@ -69,7 +69,9 @@ curl -k -X PUT --user "admin:${OPENSEARCH_PASSWORD}" https://localhost:9200/_clu

## Writing Spec Tests

The spec tests reside in the [tests/](tests) directory. Tests are organized in folders that match [namespaces](spec/namespaces). For example, tests for APIs defined in [spec/namespaces/indices.yaml](spec/namespaces/indices.yaml) can be found in [tests/indices/index.yaml](tests/indices/index.yaml) (for `/{index}`), and [tests/indices/doc.yaml](tests/indices/doc.yaml) (for `/{index}/_doc`).
The spec tests reside in the [tests/](tests) directory. Tests are organized in suites ([default](tests/default/), etc.), and subsequently in folders that match [namespaces](spec/namespaces). For example, tests for APIs defined in [spec/namespaces/indices.yaml](spec/namespaces/indices.yaml) can be found in [tests/default/indices/index.yaml](tests/default/indices/index.yaml) (for `/{index}`), and [tests/default/indices/doc.yaml](tests/default/indices/doc.yaml) (for `/{index}/_doc`).

Additional suites require custom configuration that is defined in a separate `docker-compose.yml`. For example [tests/plugins/index_state_management/docker-compose.yml](tests/plugins/index_state_management/docker-compose.yml) uses a custom setting of `plugins.index_state_management.job_interval=1` to cause the `/_nodes` API to return plugin information tested in [tests/plugins/index_state_management/nodes/plugins/index_state_management.yaml](tests/plugins/index_state_management/nodes/plugins/index_state_management.yaml).

Each yaml file in the tests directory represents a test story that tests a collection of related operations.

Expand All @@ -82,9 +84,9 @@ Check the [test_story JSON Schema](json_schemas/test_story.schema.yaml) for the

### Simple Test Story

Below is the simplified version of the test story that tests the [index operations](tests/indices/index.yaml):
Below is the simplified version of the test story that tests the [index operations](tests/default/indices/index.yaml):
```yaml
$schema: ../json_schemas/test_story.schema.yaml # The schema of the test story. Include this line so that your editor can validate the test story on the fly.
$schema: ../../json_schemas/test_story.schema.yaml # The schema of the test story. Include this line so that your editor can validate the test story on the fly.

description: This story tests all endpoints relevant the lifecycle of an index, from creation to deletion.

Expand Down Expand Up @@ -134,7 +136,7 @@ chapters:

### Using Output from Previous Chapters

Consider the following chapters in [ml/model_groups](tests/ml/model_groups.yaml) test story:
Consider the following chapters in [ml/model_groups](tests/default/ml/model_groups.yaml) test story:
```yaml
- synopsis: Create model group.
id: create_model_group # Only needed if you want to refer to this chapter in another chapter.
Expand Down Expand Up @@ -167,7 +169,7 @@ Consider the following chapters in [ml/model_groups](tests/ml/model_groups.yaml)
```
As you can see, the `output` field in the first chapter saves the `model_group_id` from the response body. This value is then used in the subsequent chapters to query and delete the model group.

You can also reuse output in payload expectations. See [tests/nodes/plugins/index_state_management.yaml](tests/nodes/plugins/index_state_management.yaml) for an example.
You can also reuse output in payload expectations. See [tests/plugins/index_state_management/nodes/plugins/index_state_management.yaml](tests/plugins/index_state_management/nodes/plugins/index_state_management.yaml) for an example.

### Managing Versions

Expand Down
2 changes: 1 addition & 1 deletion tests/_core/bulk.yaml → tests/default/_core/bulk.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test bulk endpoint.
epilogues:
Expand Down
2 changes: 1 addition & 1 deletion tests/_core/info.yaml → tests/default/_core/info.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@

$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test root endpoint.
chapters:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test mappings endpoints.
prologues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test reindex.
epilogues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../../json_schemas/test_story.schema.yaml
$schema: ../../../../json_schemas/test_story.schema.yaml

description: Test reindex with a Search pipeline.
epilogues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../../json_schemas/test_story.schema.yaml
$schema: ../../../../json_schemas/test_story.schema.yaml

description: Test search endpoint (_source).
prologues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../../json_schemas/test_story.schema.yaml
$schema: ../../../../json_schemas/test_story.schema.yaml

description: Test search endpoint with cancel_after_time_interval.
prologues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../../json_schemas/test_story.schema.yaml
$schema: ../../../../json_schemas/test_story.schema.yaml

description: Test search endpoint with geo_distance query.
prologues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../../json_schemas/test_story.schema.yaml
$schema: ../../../../json_schemas/test_story.schema.yaml

description: Test search endpoint with match query.
prologues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../../json_schemas/test_story.schema.yaml
$schema: ../../../../json_schemas/test_story.schema.yaml

description: Test search endpoint with multi_match query.
prologues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../../json_schemas/test_story.schema.yaml
$schema: ../../../../json_schemas/test_story.schema.yaml

description: Test search endpoint with phase_took.
prologues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../../json_schemas/test_story.schema.yaml
$schema: ../../../../json_schemas/test_story.schema.yaml

description: Test search endpoint with rest_total_hits_as_int.
prologues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../../json_schemas/test_story.schema.yaml
$schema: ../../../../json_schemas/test_story.schema.yaml

description: Test search endpoint with seq_no_primary_term.
prologues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../../json_schemas/test_story.schema.yaml
$schema: ../../../../json_schemas/test_story.schema.yaml

description: Test search endpoint with size.
prologues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/aliases endpoints.
epilogues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/allocation endpoints.
chapters:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/cluster_manager endpoints.
chapters:
Expand Down
2 changes: 1 addition & 1 deletion tests/cat/count.yaml → tests/default/cat/count.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/count endpoints.
chapters:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/fielddata endpoints.
chapters:
Expand Down
2 changes: 1 addition & 1 deletion tests/cat/health.yaml → tests/default/cat/health.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/health endpoints.
chapters:
Expand Down
2 changes: 1 addition & 1 deletion tests/cat/index.yaml → tests/default/cat/index.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat endpoints.
chapters:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/indices endpoints.
prologues:
Expand Down
2 changes: 1 addition & 1 deletion tests/cat/master.yaml → tests/default/cat/master.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/master endpoints.
chapters:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/nodeattrs endpoints.
chapters:
Expand Down
2 changes: 1 addition & 1 deletion tests/cat/nodes.yaml → tests/default/cat/nodes.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/nodes endpoints.
chapters:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/pending_tasks endpoints.
chapters:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../../json_schemas/test_story.schema.yaml
$schema: ../../../../json_schemas/test_story.schema.yaml

description: Test cat/pit_segments/_all endpoints.
version: '>= 2.4'
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../../json_schemas/test_story.schema.yaml
$schema: ../../../../json_schemas/test_story.schema.yaml

description: Test cat/pit_segments endpoints.
version: '>= 2.4'
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/plugins endpoints.
chapters:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/recovery endpoints.
chapters:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/segment_replication endpoints.
version: '>= 2.7'
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/segments endpoints.
chapters:
Expand Down
2 changes: 1 addition & 1 deletion tests/cat/shards.yaml → tests/default/cat/shards.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/shards endpoints.
epilogues:
Expand Down
2 changes: 1 addition & 1 deletion tests/cat/tasks.yaml → tests/default/cat/tasks.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/tasks endpoints.
chapters:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/templates endpoints.
epilogues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cat/thread_pool endpoints.
chapters:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test component templates.
epilogues:
Expand Down
12 changes: 12 additions & 0 deletions tests/default/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
version: '3'

services:
opensearch-cluster:
image: ${OPENSEARCH_DOCKER_HUB_PROJECT:-opensearchproject}/opensearch:${OPENSEARCH_VERSION:-latest}${OPENSEARCH_DOCKER_REF}
ports:
- 9200:9200
- 9600:9600
environment:
- OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_PASSWORD:-myStrongPassword123!}
- OPENSEARCH_JAVA_OPTS=${OPENSEARCH_JAVA_OPTS}
- discovery.type=single-node
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../../json_schemas/test_story.schema.yaml
$schema: ../../../../json_schemas/test_story.schema.yaml

description: Test alias endpoints.
epilogues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../../json_schemas/test_story.schema.yaml
$schema: ../../../../json_schemas/test_story.schema.yaml

description: Test aliases endpoints.
epilogues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../../json_schemas/test_story.schema.yaml
$schema: ../../../../json_schemas/test_story.schema.yaml

description: Test put_alias variants.
epilogues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test index clear cache.
prologues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test cloning an index.
prologues:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$schema: ../../json_schemas/test_story.schema.yaml
$schema: ../../../json_schemas/test_story.schema.yaml

description: Test closing an index.
prologues:
Expand Down
Loading
Loading