AWS Kinesis Firehose Terraform module

Terraform module which creates a Kinesis Firehose delivery stream towards Observe.

Additionally, this repository provides submodules to interact with the Firehose delivery stream set up by this module:

• Subscribe CloudWatch Logs to Kinesis Firehose
• Collect CloudWatch Metrics Stream
• Collect EventBridge
• Collect EKS Fargate logs

Usage

module "observe_kinesis_firehose" {
  source = "observeinc/kinesis-firehose/aws"

  name                        = "observe-kinesis-firehose"
  observe_collection_endpoint = "https://<id>.collect.observeinc.com"
  observe_token               = var.observe_token
}

This module will create a Kinesis Firehose delivery stream, as well as a role and any required policies. An S3 bucket will be created to store messages that failed to be delivered to Observe.

Providing an S3 bucket

If you prefer providing an existing S3 bucket, you can pass it as a module parameter:

resource "aws_s3_bucket" "bucket" {
  bucket        = "observe-kinesis-firehose-bucket"
  acl           = "private"
  force_destroy = true
}

module "observe_kinesis_firehose" {
  source = "observeinc/kinesis-firehose/aws"

  name                        = "observe-kinesis-firehose"
  observe_collection_endpoint = "https://<id>.collect.observeinc.com"
  observe_token               = var.observe_token
  s3_delivery_bucket          = aws_s3_bucket.bucket
}

Configuring Kinesis Data Stream as a source

You can specify a Kinesis Data Stream to act as a source to the Kinesis Firehose delivery stream. Only one data stream can be specified, and configuring this option disables all other inputs to your Kinesis Firehose.

resource "aws_kinesis_stream" "example" {
  name             = "observe-kinesis-stream-example"
  shard_count      = 1
  retention_period = 24
}

module "observe_kinesis_firehose" {
  source = "observeinc/kinesis-firehose/aws"

  name                        = "observe-kinesis-firehose"
  observe_collection_endpoint = "https://<id>.collect.observeinc.com"
  observe_token               = "<token>"
  kinesis_stream              = aws_kinesis_stream.example
}

For more details, see the Kinesis Data Stream example.

Configuring other sources

If you have not specified a Kinesis Data Stream as a source, you are free to configure other sources to send directly to your Kinesis Firehose delivery stream. You can use the module output policy when adding sources:

resource "aws_iam_role_policy_attachment" "invoke_firehose" {
  role       = aws_iam_role.role.name
  policy_arn = module.observe_kinesis_firehose.firehose_iam_policy.arn
}

See the provided EventBridge example for a more complete example.

Cloudwatch Logs

A Cloudwatch Log Group can optionally be provided in order to surface logs for both S3 and HTTP endpoint delivery.

resource "aws_cloudwatch_log_group" "group" {
  name              = "my-log-group"
  retention_in_days = 14
}

module "observe_kinesis_firehose" {
  source = "observeinc/kinesis-firehose/aws"

  name                        = "observe-kinesis-firehose"
  observe_collection_endpoint = "https://<id>.collect.observeinc.com"
  observe_token               = "<token>"
  cloudwatch_log_group        = aws_cloudwatch_log_group.group
}

Currently the module configures two output streams: one for S3 delivery, and another for HTTP endpoint delivery. You can disable either stream by setting s3_delivery_cloudwatch_log_stream_name and http_endpoint_cloudwatch_log_stream_name respectively to an empty string.

Examples

This repository contains examples of how to solve for concrete usecases:

• EventBridge to Kinesis Firehose
• Kinesis Data Stream to Kinesis Firehose
• Cross Account Role for Firehose
• EKS to Observe

Requirements

Name	Version
terraform	>= 1.1.9
aws	>= 5.0
random	>= 3.0.0

Providers

Name	Version
aws	>= 5.0
random	>= 3.0.0

Modules

No modules.

Resources

Name	Type
aws_cloudwatch_log_stream.http_endpoint_delivery	resource
aws_cloudwatch_log_stream.s3_delivery	resource
aws_iam_policy.firehose_cloudwatch	resource
aws_iam_policy.firehose_s3	resource
aws_iam_policy.kinesis_firehose	resource
aws_iam_policy.put_record	resource
aws_iam_role.firehose	resource
aws_iam_role_policy_attachment.firehose_cloudwatch	resource
aws_iam_role_policy_attachment.firehose_s3	resource
aws_iam_role_policy_attachment.kinesis_firehose	resource
aws_kinesis_firehose_delivery_stream.this	resource
aws_s3_bucket.bucket	resource
aws_s3_bucket_acl.bucket	resource
aws_s3_bucket_ownership_controls.bucket	resource
random_string.bucket_suffix	resource

Inputs

Name	Description	Type	Default	Required
cloudwatch_log_group	The CloudWatch group for logging. Providing this value enables logging.	object({ name = string arn = string })	null	no
common_attributes	Key value pairs sent as payload metadata	map(string)	{}	no
http_endpoint_buffering_interval	Buffer incoming data for the specified period of time, in seconds, before delivering it to the destination.	number	60	no
http_endpoint_buffering_size	Buffer incoming data to the specified size, in MiBs, before delivering it to the destination.	number	1	no
http_endpoint_cloudwatch_log_stream_name	Log stream name for HTTP endpoint logs. If empty, log stream will be disabled	string	"HttpEndpointDelivery"	no
http_endpoint_content_encoding	Kinesis Data Firehose uses the content encoding to compress the body of a request before sending the request to the destination.	string	"GZIP"	no
http_endpoint_name	Name of Kinesis Firehose target HTTP endpoint	string	"Observe"	no
http_endpoint_retry_duration	The total amount of time that Kinesis Data Firehose spends on retries. This duration starts after the initial attempt to send data to the custom destination via HTTPS endpoint fails. It doesn't include the periods during which Kinesis Data Firehose waits for acknowledgment from the specified destination after each attempt.	number	300	no
http_endpoint_s3_backup_mode	S3 backup mode for Kinesis Firehose HTTP endpoint. By default, only data that cannot be delivered to Observe via HTTP is written to S3. To backup all data to S3, set this to AllData.	string	"FailedDataOnly"	no
iam_name_prefix	Prefix used for all created IAM roles and policies	string	"observe-kinesis-firehose-"	no
kinesis_stream	Kinesis Data Stream ARN to configure as source	object({ arn = string })	null	no
name	Name of Kinesis Firehose resource	string	n/a	yes
observe_collection_endpoint	Observe Collection Endpoint, e.g https://123456789012.collect.observeinc.com	string	null	no
observe_customer	Observe Customer ID. Deprecated, please use observe_collection_endpoint instead	string	null	no
observe_domain	Observe domain. Deprecated, please use observe_collection_endpoint instead	string	"observeinc.com"	no
observe_token	Observe Token	string	n/a	yes
observe_url	Observe URL. Deprecated.	string	""	no
s3_delivery_bucket	S3 bucket to be used as backup for message delivery	object({ arn = string })	null	no
s3_delivery_cloudwatch_log_stream_name	Log stream name for S3 delivery logs. If empty, log stream will be disabled	string	"S3Delivery"	no
s3_delivery_compression_format	The compression format. If no value is specified, the default is UNCOMPRESSED.	string	"UNCOMPRESSED"	no
s3_delivery_prefix	The "YYYY/MM/DD/HH" time format prefix is automatically used for delivered Amazon S3 files	string	null	no
tags	A map of tags to add to all resources	map(string)	{}	no

Outputs

Name	Description
firehose_delivery_stream	Kinesis Firehose delivery stream towards Observe
firehose_iam_policy	IAM policy to publish records to Kinesis Firehose. If a Kinesis Data Stream is set as a source, no policy is provided since Firehose will not allow any other event source.

License

Apache 2 Licensed. See LICENSE for full details.