Sink healthcheck failed for aws_s3

[Rory-Z]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I can't push logs to aws_s3, got this error:

2024-01-16T10:20:10.700740Z ERROR vector::topology::builder: msg="Healthcheck failed." error=dispatch failure component_kind="sink" component_type="aws_s3" component_id=aws_s3

Configuration

I'm deploy vector on K8s by Helm chart, this is my values.yaml

role: Agent
env:
  - name: VECTOR_LOG
    value: debug
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: aws-secret
        key: key_id
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: aws-secret
        key: access_key
  - name: AWS_DEFAULT_REGION
    value: us-esat-1
customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: true
    address: 0.0.0.0:8686
    playground: false
  sources:
    kubernetes_logs:
      type: kubernetes_logs
  sinks:
    aws_s3:
      type: aws_s3
      inputs:
        - kubernetes_logs
      bucket: "public-broker-logs"
      encoding:
        codec: json
    stdout:
      type: console
      inputs: [kubernetes_logs]
      encoding:
        codec: json

Version

0.35.0-distroless-libc

Debug Output

2024-01-17T06:11:49.957137Z  WARN sink{component_kind="sink" component_id=aws_s3 component_type=aws_s3}:request{request_id=1}: vector::sinks::util::retries: Retrying after error. error=dispatch failure internal_log_rate_limit=true
2024-01-17T06:11:54.680147Z  WARN sink{component_kind="sink" component_id=aws_s3 component_type=aws_s3}:request{request_id=1}: vector::sinks::util::retries: Internal log [Retrying after error.] is being suppressed to avoid flooding.
2024-01-17T06:12:19.923882Z  WARN sink{component_kind="sink" component_id=aws_s3 component_type=aws_s3}:request{request_id=1}: vector::sinks::util::retries: Internal log [Retrying after error.] has been suppressed 1 times.
2024-01-17T06:12:19.923898Z  WARN sink{component_kind="sink" component_id=aws_s3 component_type=aws_s3}:request{request_id=1}: vector::sinks::util::retries: Retrying after error. error=dispatch failure internal_log_rate_limit=true

Example Data

No response

Additional Context

I'm using terraform to create aws s3 bucket, this is my script

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.16"
    }
  }

  required_version = ">= 1.2.0"
}

resource "aws_s3_bucket" "broker" {
  bucket = "public-broker-logs"

  tags = {
    Name = "public-broker-logs"
  }
}

resource "aws_s3_bucket_policy" "broker" {
  bucket = aws_s3_bucket.broker.id

  // Text of the policy.
  // Although this is a bucket policy rather than an IAM policy, the aws_iam_policy_document data source may be used, so long as it specifies a principal. 
  policy = data.aws_iam_policy_document.allow_vector.json
}

data "aws_iam_policy_document" "allow_vector" {
  statement {
    principals {
      type        = "AWS"
      identifiers = ["1234567890"]
    }

    actions = [
      "s3:GetObject",
      "s3:PutObject",
      "s3:ListBucket",
    ]

    resources = [
      "arn:aws:s3:::public-broker-logs",
      "arn:aws:s3:::public-broker-logs/*"
    ]
  }
}

References

No response

[DavidGitter]

Have you found any solution? @Rory-Z