Add repositories metering API 7.x

docs/reference/repositories-metering-api/apis/clear-repositories-metering-archive.asciidoc

@@ -0,0 +1,35 @@
+[role="xpack"]
+[testenv="basic"]
+[[clear-repositories-metering-archive-api]]
+=== Clear repositories metering archive
+++++
+<titleabbrev>Clear repositories metering archive</titleabbrev>
+++++
+
+Removes the archived repositories metering information present in the cluster.
+
+[[clear-repositories-metering-archive-api-request]]
+==== {api-request-title}
+
+`DELETE /_nodes/<node_id>/_repositories_metering/<max_version_to_clear>`
+
+[[clear-repositories-metering-archive-api-desc]]
+==== {api-description-title}
+
+You can use this API to clear the archived repositories metering information in the cluster.
+
+[[clear-repositories-metering-archive-api-path-params]]
+==== {api-path-parms-title}
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=node-id]
+
+`<max_version_to_clear>`::
+ (long) Specifies the maximum <<get-repositories-metering-api-response-body, archive_version>> to be cleared from the archive.
+
+All the nodes selective options are explained <<cluster-nodes,here>>.
+[role="child_attributes"]
+[[clear-repositories-metering-archive-api-response-body]]
+==== {api-response-body-title}
+Returns the deleted archived repositories metering information.
+
+include::{es-repo-dir}/repositories-metering-api/apis/repositories-meterings-body.asciidoc[tag=repositories-metering-body]

docs/reference/repositories-metering-api/apis/get-repositories-metering.asciidoc

@@ -0,0 +1,35 @@
+[role="xpack"]
+[testenv="basic"]
+[[get-repositories-metering-api]]
+=== Get repositories metering information
+++++
+<titleabbrev>Get repositories metering information</titleabbrev>
+++++
+
+Returns cluster repositories metering information.
+
+[[get-repositories-metering-api-request]]
+==== {api-request-title}
+
+`GET /_nodes/<node_id>/_repositories_metering`
+
+[[get-repositories-metering-api-desc]]
+==== {api-description-title}
+
+You can use the cluster repositories metering API to retrieve repositories metering information in a cluster.
+
+This API exposes monotonically non-decreasing counters and it's expected that clients would durably store
+the information needed to compute aggregations over a period of time. Additionally, the information
+exposed by this API is volatile, meaning that it won't be present after node restarts.
+
+[[get-repositories-metering-api-path-params]]
+==== {api-path-parms-title}
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=node-id]
+
+All the nodes selective options are explained <<cluster-nodes,here>>.
+
+[role="child_attributes"]
+[[get-repositories-metering-api-response-body]]
+==== {api-response-body-title}
+include::{es-repo-dir}/repositories-metering-api/apis/repositories-meterings-body.asciidoc[tag=repositories-metering-body]

docs/reference/repositories-metering-api/apis/repositories-meterings-body.asciidoc

@@ -0,0 +1,178 @@
+tag::repositories-metering-body[]
+`_nodes`::
+(object)
+Contains statistics about the number of nodes selected by the request.
++
+.Properties of `_nodes`
+[%collapsible%open]
+====
+`total`::
+(integer)
+Total number of nodes selected by the request.
+
+`successful`::
+(integer)
+Number of nodes that responded successfully to the request.
+
+`failed`::
+(integer)
+Number of nodes that rejected the request or failed to respond. If this value
+is not `0`, a reason for the rejection or failure is included in the response.
+====
+
+`cluster_name`::
+(string)
+Name of the cluster. Based on the <<cluster.name>> setting.
+
+`nodes`::
+(object)
+Contains repositories metering information for the nodes selected by the request.
++
+.Properties of `nodes`
+[%collapsible%open]
+====
+`<node_id>`::
+(array)
+An array of repository metering information for the node.
++
+.Properties of objects in `node_id`
+[%collapsible%open]
+=====
+`repository_name`::
+(string)
+Repository name.
+
+`repository_type`::
+(string)
+Repository type.
+
+`repository_location`::
+(object)
+Represents an unique location within the repository.
++
+.Properties of `repository_location` for repository type `Azure`
+[%collapsible%open]
+======
+`base_path`::
+(string)
+The path within the container where the repository stores data.
+
+`container`::
+(string)
+Container name.
+======
++
+.Properties of `repository_location` for repository type `GCP`
+[%collapsible%open]
+======
+`base_path`::
+(string)
+The path within the bucket where the repository stores data.
+
+`bucket`::
+(string)
+Bucket name.
+======
++
+.Properties of `repository_location` for repository type `S3`
+[%collapsible%open]
+======
+`base_path`::
+(string)
+The path within the bucket where the repository stores data.
+
+`bucket`::
+(string)
+Bucket name.
+======
+`repository_ephemeral_id`::
+(string)
+An identifier that changes every time the repository is updated.
+
+`repository_started_at`::
+(long)
+Time the repository was created or updated. Recorded in milliseconds
+since the https://en.wikipedia.org/wiki/Unix_time[Unix Epoch].
+
+`repository_stopped_at`::
+(Optional, long)
+Time the repository was deleted or updated. Recorded in milliseconds
+since the https://en.wikipedia.org/wiki/Unix_time[Unix Epoch].
+
+`archived`::
+(boolean)
+A flag that tells whether or not this object has been archived.
+When a repository is closed or updated the repository metering information
+is archived and kept for a certain period of time. This allows retrieving
+the repository metering information of previous repository instantiations.
+
+`archive_version`::
+(Optional, long)
+The cluster state version when this object was archived, this field
+can be used as a logical timestamp to delete all the archived metrics up
+to an observed version. This field is only present for archived
+repository metering information objects. The main purpose of this
+field is to avoid possible race conditions during repository metering
+information deletions, i.e. deleting archived repositories metering
+information that we haven't observed yet.
+
+`request_counts`::
+(object)
+An object with the number of request performed against the repository
+grouped by request type.
++
+.Properties of `request_counts` for repository type `Azure`
+[%collapsible%open]
+======
+`GetBlobProperties`::
+(long) Number of https://docs.microsoft.com/en-us/rest/api/storageservices/get-blob-properties[Get Blob Properties] requests.
+`GetBlob`::
+(long) Number of https://docs.microsoft.com/en-us/rest/api/storageservices/get-blob[Get Blob] requests.
+`ListBlobs`::
+(long) Number of https://docs.microsoft.com/en-us/rest/api/storageservices/list-blobs[List Blobs] requests.
+`PutBlob`::
+(long) Number of https://docs.microsoft.com/en-us/rest/api/storageservices/put-blob[Put Blob] requests.
+`PutBlock`::
+(long) Number of https://docs.microsoft.com/en-us/rest/api/storageservices/put-block[Put Block].
+`PutBlockList`::
+(long) Number of https://docs.microsoft.com/en-us/rest/api/storageservices/put-block-list[Put Block List] requests.
+
+Azure storage https://azure.microsoft.com/en-us/pricing/details/storage/blobs/[pricing].
+======
++
+.Properties of `request_counts` for repository type `GCP`
+[%collapsible%open]
+======
+`GetObject`::
+(long) Number of https://cloud.google.com/storage/docs/json_api/v1/objects/get[get object] requests.
+`ListObjects`::
+(long) Number of https://cloud.google.com/storage/docs/json_api/v1/objects/list[list objects] requests.
+`InsertObject`::
+(long) Number of https://cloud.google.com/storage/docs/json_api/v1/objects/insert[insert object] requests,
+including https://cloud.google.com/storage/docs/uploading-objects[simple], https://cloud.google.com/storage/docs/json_api/v1/how-tos/multipart-upload[multipart] and
+https://cloud.google.com/storage/docs/resumable-uploads[resumable] uploads. Resumable uploads can perform multiple http requests to
+insert a single object but they are considered as a single request since they are https://cloud.google.com/storage/docs/resumable-uploads#introduction[billed] as an individual operation.
+
+Google Cloud storage https://cloud.google.com/storage/pricing[pricing].
+======
++
+.Properties of `request_counts` for repository type `S3`
+[%collapsible%open]
+======
+`GetObject`::
+(long) Number of https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html[GetObject] requests.
+`ListObjects`::
+(long) Number of https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html[ListObjects] requests.
+`PutObject`::
+(long) Number of https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html[PutObject] requests.
+`PutMultipartObject`::
+(long) Number of https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html[Multipart] requests,
+including https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html[CreateMultipartUpload],
+https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html[UploadPart] and https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html[CompleteMultipartUpload]
+requests.
+
+Amazon Web Services Simple Storage Service https://aws.amazon.com/s3/pricing/[pricing].
+======
+=====
+====
+end::repositories-metering-body[]

docs/reference/repositories-metering-api/repositories-metering-apis.asciidoc

@@ -0,0 +1,16 @@
+[role="xpack"]
+[testenv="basic"]
+[[repositories-metering-apis]]
+== Repositories metering APIs
+
+experimental[]
+
+You can use the following APIs to retrieve repositories metering information.
+
+This is an API used by Elastic's commercial offerings.
+
+* <<get-repositories-metering-api,Get repositories metering information>>
+* <<clear-repositories-metering-archive-api,Clear repositories metering archive>>
+
+include::apis/get-repositories-metering.asciidoc[]
+include::apis/clear-repositories-metering-archive.asciidoc[]

docs/reference/rest-api/index.asciidoc

@@ -30,6 +30,7 @@ endif::[]
 * <<ml-df-analytics-apis,{ml-cap} {dfanalytics} APIs>>
 * <<migration-api,Migration APIs>>
 * <<indices-reload-analyzers,Reload Search Analyzers API>>
+* <<repositories-metering-apis,Repositories Metering APIs>>
 * <<rollup-apis,Rollup APIs>>
 * <<search, Search APIs>>
 ifdef::permanently-unreleased-branch[]
@@ -63,6 +64,7 @@ include::{es-repo-dir}/ml/anomaly-detection/apis/index.asciidoc[]
 include::{es-repo-dir}/ml/df-analytics/apis/index.asciidoc[]
 include::{es-repo-dir}/migration/migration.asciidoc[]
 include::{es-repo-dir}/indices/apis/reload-analyzers.asciidoc[]
+include::{es-repo-dir}/repositories-metering-api/repositories-metering-apis.asciidoc[]
 include::{es-repo-dir}/rollup/rollup-api.asciidoc[]
 include::{es-repo-dir}/search.asciidoc[]
 ifdef::permanently-unreleased-branch[]

plugins/repository-azure/src/internalClusterTest/java/org/elasticsearch/repositories/azure/AzureBlobStoreRepositoryTests.java

@@ -40,7 +40,6 @@
 import java.util.Base64;
 import java.util.Collection;
 import java.util.Collections;
-import java.util.List;
 import java.util.Map;
 import java.util.regex.Pattern;
 
@@ -76,11 +75,6 @@ protected HttpHandler createErroneousHttpHandler(final HttpHandler delegate) {
  return new AzureErroneousHttpHandler(delegate, randomIntBetween(2, 3));
  }
 
- @Override
- protected List<String> requestTypesTracked() {
- return org.elasticsearch.common.collect.List.of("GET", "LIST", "HEAD", "PUT", "PUT_BLOCK");
- }
-
  @Override
  protected Settings nodeSettings(int nodeOrdinal) {
  final String key = Base64.getEncoder().encodeToString(randomAlphaOfLength(10).getBytes(StandardCharsets.UTF_8));
@@ -180,23 +174,28 @@ private AzureHTTPStatsCollectorHandler(HttpHandler delegate) {
  @Override
  protected void maybeTrack(String request, Headers headers) {
  if (Regex.simpleMatch("GET /*/*", request)) {
- trackRequest("GET");
+ trackRequest("GetBlob");
  } else if (Regex.simpleMatch("HEAD /*/*", request)) {
- trackRequest("HEAD");
+ trackRequest("GetBlobProperties");
  } else if (listPattern.matcher(request).matches()) {
- trackRequest("LIST");
- } else if (isBlockUpload(request)) {
- trackRequest("PUT_BLOCK");
+ trackRequest("ListBlobs");
+ } else if (isPutBlock(request)) {
+ trackRequest("PutBlock");
+ } else if (isPutBlockList(request)) {
+ trackRequest("PutBlockList");
  } else if (Regex.simpleMatch("PUT /*/*", request)) {
- trackRequest("PUT");
+ trackRequest("PutBlob");
  }
  }
 
- // https://docs.microsoft.com/en-us/rest/api/storageservices/put-block-list
  // https://docs.microsoft.com/en-us/rest/api/storageservices/put-block
- private boolean isBlockUpload(String request) {
- return Regex.simpleMatch("PUT /*/*?*comp=blocklist*", request)
- || (Regex.simpleMatch("PUT /*/*?*comp=block*", request) && request.contains("blockid="));
+ private boolean isPutBlock(String request) {
+ return Regex.simpleMatch("PUT /*/*?*comp=block*", request) && request.contains("blockid=");
+ }
+
+ // https://docs.microsoft.com/en-us/rest/api/storageservices/put-block-list
+ private boolean isPutBlockList(String request) {
+ return Regex.simpleMatch("PUT /*/*?*comp=blocklist*", request);
  }
  }
 }

plugins/repository-azure/src/main/java/org/elasticsearch/repositories/azure/AzureBlobStore.java

@@ -120,19 +120,19 @@ public AzureBlobStore(RepositoryMetadata metadata, AzureStorageService service,
  this.uploadMetricsCollector = (httpURLConnection -> {
  assert httpURLConnection.getRequestMethod().equals("PUT");
  String queryParams = httpURLConnection.getURL().getQuery();
- if (queryParams != null && isBlockUpload(queryParams)) {
- stats.putBlockOperations.incrementAndGet();
- } else {
+ if (queryParams == null) {
  stats.putOperations.incrementAndGet();
+ return;
  }
- });
- }
 
- private boolean isBlockUpload(String queryParams) {
- // https://docs.microsoft.com/en-us/rest/api/storageservices/put-block
- // https://docs.microsoft.com/en-us/rest/api/storageservices/put-block-list
- return (queryParams.contains("comp=block") && queryParams.contains("blockid="))
- || queryParams.contains("comp=blocklist");
+ // https://docs.microsoft.com/en-us/rest/api/storageservices/put-block
+ // https://docs.microsoft.com/en-us/rest/api/storageservices/put-block-list
+ if (queryParams.contains("comp=block") && queryParams.contains("blockid=")) {
+ stats.putBlockOperations.incrementAndGet();
+ } else if (queryParams.contains("comp=blocklist")) {
+ stats.putBlockListOperations.incrementAndGet();
+ }
+ });
  }
 
  @Override
@@ -385,14 +385,15 @@ private static class Stats {
 
  private final AtomicLong putBlockOperations = new AtomicLong();
 
+ private final AtomicLong putBlockListOperations = new AtomicLong();
+
  private Map<String, Long> toMap() {
- return org.elasticsearch.common.collect.Map.of(
- "GET", getOperations.get(),
- "LIST", listOperations.get(),
- "HEAD", headOperations.get(),
- "PUT", putOperations.get(),
- "PUT_BLOCK", putBlockOperations.get()
- );
+ return org.elasticsearch.common.collect.Map.of("GetBlob", getOperations.get(),
+ "ListBlobs", listOperations.get(),
+ "GetBlobProperties", headOperations.get(),
+ "PutBlob", putOperations.get(),
+ "PutBlock", putBlockOperations.get(),
+ "PutBlockList", putBlockListOperations.get());
  }
  }
 }