Update DEVELOPER_GUIDE.md to document bwc and feature flag mechanims (#…

…6028)

OpenSearch uses @opensearch.internal, @opensearch.api, @opensearch.experimental,
and feature flags to signal maturity and usage intent of the various
implementation components. Each one of these annotations carry a level of
backwards compatibility guarantees and support requirements that is not
officially documented. This change officially documents those bwc requirements,
level of guarantee, and recommended approaches in how to best use the mechanisms
to communicate this to the development and user community.

Signed-off-by: Nicholas Walter Knize <[email protected]>

DEVELOPER_GUIDE.md

@@ -50,6 +50,11 @@
  - [Installation](#installation)
  - [Configuration](#configuration)
  - [Submitting Changes](#submitting-changes)
+ - [Backwards Compatibility](#backwards-compatibility)
+ - [Data](#data)
+ - [Developer API](#developer-api)
+ - [User API](#user-api)
+ - [Experimental Development](#experimental-development)
  - [Backports](#backports)
  - [LineLint](#linelint)
  - [Lucene Snapshots](#lucene-snapshots)
@@ -382,7 +387,7 @@ The Distribution Download plugin downloads the latest version of OpenSearch by d
 
 A fat-JAR (or an uber-JAR) is the JAR, which contains classes from all the libraries, on which your project depends and, of course, the classes of current project.
 
-There might be cases where a developer would like to add some custom logic to the code of a module (or multiple modules) and generate a fat-JAR that can be directly used by the dependency management tool. For example, in [#3665](https:/opensearch-project/OpenSearch/pull/3665) a developer wanted to provide a tentative patch as a fat-JAR to a consumer for changes made in the high level REST client. 
+There might be cases where a developer would like to add some custom logic to the code of a module (or multiple modules) and generate a fat-JAR that can be directly used by the dependency management tool. For example, in [#3665](https:/opensearch-project/OpenSearch/pull/3665) a developer wanted to provide a tentative patch as a fat-JAR to a consumer for changes made in the high level REST client.
 
 Use [Gradle Shadow plugin](https://imperceptiblethoughts.com/shadow/).
 Add the following to the `build.gradle` file of the module for which you want to create the fat-JAR, e.g. `client/rest-high-level/build.gradle`:
@@ -400,7 +405,7 @@ This will generate a fat-JAR in the `build/distributions` folder of the module,
 
 You can further customize your fat-JAR by customising the plugin, More information about shadow plugin can be found [here](https://imperceptiblethoughts.com/shadow/).
 
-To use the generated JAR, install the JAR locally, e.g. 
+To use the generated JAR, install the JAR locally, e.g.
 ```
 mvn install:install-file -Dfile=src/main/resources/opensearch-rest-high-level-client-1.4.0-SNAPSHOT.jar -DgroupId=org.opensearch.client -DartifactId=opensearch-rest-high-level-client -Dversion=1.4.0-SNAPSHOT -Dpackaging=jar -DgeneratePom=true
 ```
@@ -518,6 +523,45 @@ git secrets --register-aws
 
 See [CONTRIBUTING](CONTRIBUTING.md).
 
+### Backwards Compatibility
+
+OpenSearch strives for a smooth and easy upgrade experience that is resilient to data loss and corruption while minimizing downtime and ensuring integration with
+external systems does not unexpectedly break.
+
+To provide these guarantees each version must be designed and developed with [forward compatibility](https://en.wikipedia.org/wiki/Forward_compatibility) in mind.
+OpenSearch addresses backward and forward compatibility at three different levels: 1. Data, 2. Developer API, 3. User API. These levels and the developer mechanisms
+to ensure backwards compatibility are provided below.
+
+#### Data
+The data level consists of index and application data file formats. OpenSearch guarantees file formats and indexes are compatible only back to the first release of
+the previous major version. If on disk formats or encodings need to be changed (including index data, cluster state, or any other persisted data) developers must
+use Version checks accordingly (e.g., `Version.onOrAfter`, `Version.before`) to guarantee backwards compatibility.
+
+#### Developer API
+The Developer API consists of interfaces and foundation software implementations that enable external users to develop new OpenSearch features. This includes
+obvious components such as the Plugin framework and less obvious components such as REST Action Handlers. When developing a new feature of OpenSearch it is important
+to explicitly mark which implementation components may, or may not, be extended by external implementations. For example, all new API classes with `@opensearch.api`
+signal that the new component may be extended by an external implementation and therefore provide backwards compatibility guarantees. Similarly, any class explicitly
+marked with the `@opensearch.internal` annotation, or not explicitly marked by an annotation should not be extended by external implementation components as it does not
+guarantee backwards compatibility and may change at any time. The `@deprecated` annotation should also be added to any `@opensearch.api` classes or methods that are
+either changed or planned to be removed across minor versions.
+
+#### User API
+The User API consists of integration specifications (e.g., [Query Domain Specific Language](https://opensearch.org/docs/latest/opensearch/query-dsl/index/),
+[field mappings](https://opensearch.org/docs/latest/opensearch/mappings/)) and endpoints (e.g., [`_search`](https://opensearch.org/docs/latest/api-reference/search/),
+[`_cat`](https://opensearch.org/docs/latest/api-reference/cat/index/)) users rely on to integrate and use OpenSearch. Backwards compatibility is critical to the
+User API, therefore OpenSearch commits to using [semantic versioning](https://opensearch.org/blog/what-is-semver/) for all User facing APIs. To support this
+developers must leverage `Version` checks for any user facing endpoints or API specifications that change across minor versions. Developers must also inform
+users of any changes by adding the `>breaking` label on Pull Requests, adding an entry to the [CHANGELOG](https:/opensearch-project/OpenSearch/blob/main/CHANGELOG.md)
+and a log message to the OpenSearch deprecation log files using the `DeprecationLogger`.
+
+#### Experimental Development
+Rapidly developing new features often benefit from several release cycles before committing to an official and long term supported (LTS) API. To enable this cycle OpenSearch
+uses an Experimental Development process leveraging [Feature Flags](https://featureflags.io/feature-flags/). This allows a feature to be developed using the same process as
+a LTS feature but with additional guard rails and communication mechanisms to signal to the users and development community the feature is not yet stable, may change in a future
+release, or be removed altogether. Any Developer or User APIs implemented along with the experimental feature should be marked with the `@opensearch.experimental` annotation to
+signal the implementation is not subject to LTS and does not follow backwards compatibility guidelines.
+
 ### Backports
 
 The Github workflow in [`backport.yml`](.github/workflows/backport.yml) creates backport PRs automatically when the original PR with an appropriate label `backport <backport-branch-name>` is merged to main with the backport workflow run successfully on the PR. For example, if a PR on main needs to be backported to `1.x` branch, add a label `backport 1.x` to the PR and make sure the backport workflow runs on the PR along with other checks. Once this PR is merged to main, the workflow will create a backport PR to the `1.x` branch.
@@ -551,7 +595,7 @@ If you encounter a build/test failure in CI that is unrelated to the change in y
 
 1. Follow failed CI links, and locate the failing test(s).
 2. Copy-paste the failure into a comment of your PR.
-3. Search through [issues](https:/opensearch-project/OpenSearch/issues?q=is%3Aopen+is%3Aissue+label%3A%22flaky-test%22) using the name of the failed test for whether this is a known flaky test. 
+3. Search through [issues](https:/opensearch-project/OpenSearch/issues?q=is%3Aopen+is%3Aissue+label%3A%22flaky-test%22) using the name of the failed test for whether this is a known flaky test.
 5. If an existing issue is found, paste a link to the known issue in a comment to your PR.
 6. If no existing issue is found, open one.
 7. Retry CI via the GitHub UX or by pushing an update to your PR.