AWS S3 sink provides unclear errors when an incorrect endpoint is provided

[calebschoepp]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Here's a simple use case where I'm generating demo logs and then attempting to store them in an AWS S3 sink (technically an S3 compatible store Linode Object Storage).

This config is incorrect. The endpoint should include a https:// prefix in order for it to work. However, this config fails with a very non-obvious error:

It was only by fluke that I discovered I had an invalid endpoint. Ideally we can more clearly indicate to the user that their endpoint is incorrect.

Configuration

api:
  enabled: true

sources:
  generate_syslog:
    type:   "demo_logs"
    format: "syslog"
    count:  100

transforms:
  remap_syslog:
    inputs:
      - "generate_syslog"
    type:   "remap"
    source: |
            structured = parse_syslog!(.message)
            . = merge(., structured)

sinks:
  store_syslog_in_s3:
    inputs:
      - "remap_syslog"
    type: "aws_s3"
    bucket: "caleb-test"
    region: "us-sea-1"
    auth:
      access_key_id: "<REDACTED>"
      secret_access_key: "<REDACTED>"
    endpoint: "us-sea-1.linodeobjects.com"
    encoding:
      codec: "json"

Version

vector 0.41.1 (x86_64-apple-darwin 745babd 2024-09-11 14:55:36.802851761)

Debug Output

VECTOR_LOG=debug vector -c vector.yaml
2024-10-04T15:24:56.853085Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2024-10-04T15:24:56.853144Z  INFO vector::app: Log level is enabled. level="debug"
2024-10-04T15:24:56.853162Z DEBUG vector::app: messaged="Building runtime." worker_threads=8
2024-10-04T15:24:56.853963Z  INFO vector::app: Loading configs. paths=["vector.yaml"]
2024-10-04T15:24:56.855494Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2024-10-04T15:24:56.858083Z DEBUG vector::topology::builder: Building new source. component=generate_syslog
2024-10-04T15:24:56.858381Z DEBUG vector::topology::builder: Building new transform. component=remap_syslog
2024-10-04T15:24:56.858712Z DEBUG vector::topology::builder: Building new sink. component=store_syslog_in_s3
2024-10-04T15:24:56.868482Z DEBUG sink{component_kind="sink" component_id=store_syslog_in_s3 component_type=aws_s3}: vector_core::tls::settings: Fetching system root certs.
2024-10-04T15:24:57.052740Z DEBUG sink{component_kind="sink" component_id=store_syslog_in_s3 component_type=aws_s3}: vector_core::tls::settings: Fetching system root certs.
2024-10-04T15:24:57.223842Z DEBUG sink{component_kind="sink" component_id=store_syslog_in_s3 component_type=aws_s3}: aws_config::fs_util: loaded home directory src="HOME"
2024-10-04T15:24:57.223894Z DEBUG sink{component_kind="sink" component_id=store_syslog_in_s3 component_type=aws_s3}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/Users/<your_username>" path="~/.aws/config"
2024-10-04T15:24:57.223913Z DEBUG sink{component_kind="sink" component_id=store_syslog_in_s3 component_type=aws_s3}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/Users/<your_username>/.aws/config"
2024-10-04T15:24:57.225333Z DEBUG sink{component_kind="sink" component_id=store_syslog_in_s3 component_type=aws_s3}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/Users/<your_username>/.aws/config") size=10
2024-10-04T15:24:57.225363Z DEBUG sink{component_kind="sink" component_id=store_syslog_in_s3 component_type=aws_s3}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/Users/<your_username>" path="~/.aws/credentials"
2024-10-04T15:24:57.225377Z DEBUG sink{component_kind="sink" component_id=store_syslog_in_s3 component_type=aws_s3}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/Users/<your_username>/.aws/credentials"
2024-10-04T15:24:57.225594Z DEBUG sink{component_kind="sink" component_id=store_syslog_in_s3 component_type=aws_s3}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/Users/<your_username>/.aws/credentials") size=116
2024-10-04T15:24:57.228038Z  INFO vector::topology::running: Running healthchecks.
2024-10-04T15:24:57.228058Z DEBUG vector::topology::running: Connecting changed/added component(s).
2024-10-04T15:24:57.228071Z DEBUG vector::topology::running: Configuring outputs for source. component=generate_syslog
2024-10-04T15:24:57.228285Z DEBUG invoke{service=s3 operation=HeadBucket sdk_invocation_id=<redacted>}:apply_configuration: aws_smithy_runtime::client::orchestrator: timeout settings for this operation: TimeoutConfig { connect_timeout: Disabled, read_timeout: Disabled, operation_timeout: Disabled, operation_attempt_timeout: Disabled }
2024-10-04T15:24:57.228333Z DEBUG invoke{service=s3 operation=HeadBucket sdk_invocation_id=<redacted>}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'serialization' phase
2024-10-04T15:24:57.228377Z DEBUG invoke{service=s3 operation=HeadBucket sdk_invocation_id=<redacted>}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'before transmit' phase
2024-10-04T15:24:57.228399Z DEBUG invoke{service=s3 operation=HeadBucket sdk_invocation_id=<redacted>}:try_op: aws_smithy_runtime::client::retries::strategy::standard: no client rate limiter configured, so no token is required for the initial request.
2024-10-04T15:24:57.228507Z DEBUG vector::topology::running: Configuring output for component. component=generate_syslog output_id=None
2024-10-04T15:24:57.228520Z DEBUG vector::topology::running: Configuring outputs for transform. component=remap_syslog
2024-10-04T15:24:57.228525Z DEBUG vector::topology::running: Configuring output for component. component=remap_syslog output_id=None
2024-10-04T15:24:57.228532Z DEBUG vector::topology::running: Connecting inputs for transform. component=remap_syslog
2024-10-04T15:24:57.228545Z DEBUG vector::topology::running: Adding component input to fanout. component=remap_syslog fanout_id=generate_syslog
2024-10-04T15:24:57.228558Z DEBUG vector::topology::running: Connecting inputs for sink. component=store_syslog_in_s3
2024-10-04T15:24:57.228565Z DEBUG vector::topology::running: Adding component input to fanout. component=store_syslog_in_s3 fanout_id=remap_syslog
2024-10-04T15:24:57.228611Z DEBUG invoke{service=s3 operation=HeadBucket sdk_invocation_id=<redacted>}:try_op: aws_smithy_runtime::client::orchestrator: retry strategy has OKed initial request
2024-10-04T15:24:57.228633Z DEBUG vector::topology::running: Spawning new source. key=generate_syslog
2024-10-04T15:24:57.228815Z DEBUG invoke{service=s3 operation=HeadBucket sdk_invocation_id=<redacted>}:try_op: aws_smithy_runtime::client::orchestrator: beginning attempt #1
2024-10-04T15:24:57.228882Z DEBUG vector::topology::running: Spawning new transform. key=remap_syslog
2024-10-04T15:24:57.228996Z DEBUG source{component_kind="source" component_id=generate_syslog component_type=demo_logs}: vector::topology::builder: Source pump supervisor starting.
2024-10-04T15:24:57.229006Z  INFO vector: Vector has started. debug="false" version="0.41.1" arch="x86_64" revision="745babd 2024-09-11 14:55:36.802851761"
2024-10-04T15:24:57.229019Z DEBUG transform{component_kind="transform" component_id=remap_syslog component_type=remap}: vector::topology::builder: Synchronous transform starting.
2024-10-04T15:24:57.229009Z DEBUG source{component_kind="source" component_id=generate_syslog component_type=demo_logs}: vector::topology::builder: Source starting.
2024-10-04T15:24:57.229046Z DEBUG sink{component_kind="sink" component_id=store_syslog_in_s3 component_type=aws_s3}: vector::topology::builder: Sink starting.
2024-10-04T15:24:57.229090Z DEBUG source{component_kind="source" component_id=generate_syslog component_type=demo_logs}: vector::topology::builder: Source pump starting.
2024-10-04T15:24:57.229052Z DEBUG invoke{service=s3 operation=HeadBucket sdk_invocation_id=<redacted>}:try_op:try_attempt: aws_smithy_runtime::client::orchestrator::endpoints: resolving endpoint endpoint_params=EndpointResolverParams(TypeErasedBox[!Clone]:Params { bucket: Some("<bucket-name>"), region: Some("us-sea-1"), use_fips: false, use_dual_stack: false, endpoint: Some("<endpoint-url>"), force_path_style: true, accelerate: false, use_global_endpoint: false, use_object_lambda_endpoint: None, key: None, prefix: None, disable_access_points: None, disable_multi_region_access_points: false, use_arn_region: None, use_s3_express_control_endpoint: None, disable_s3_express_session_auth: None }) endpoint_prefix=None
2024-10-04T15:24:57.229373Z DEBUG invoke{service=s3 operation=HeadBucket sdk_invocation_id=<redacted>}:try_op:try_attempt: aws_smithy_runtime::client::orchestrator: encountered orchestrator error; halting
2024-10-04T15:24:57.229403Z DEBUG invoke{service=s3 operation=HeadBucket sdk_invocation_id=<redacted>}:try_op: aws_smithy_runtime::client::retries::strategy::standard: not retrying because we are out of attempts attempts=1 max_attempts=1
2024-10-04T15:24:57.229412Z DEBUG invoke{service=s3 operation=HeadBucket sdk_invocation_id=<redacted>}:try_op: aws_smithy_runtime::client::orchestrator: a retry is either unnecessary or not possible, exiting attempt loop
2024-10-04T15:24:57.229471Z ERROR vector::topology::builder: msg="Healthcheck failed." error=dispatch failure component_kind="sink" component_type="aws_s3" component_id=store_syslog_in_s3

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Thanks @calebschoepp . I agree the error could be made much more obvious in this situation.