Unable to remove the filter_policy from aws_sns_topic_subscription

[oanasabau]

Hi there,

After updating to new version of the aws terraform provider we are unable to remove the filter_policy from a sns subscription when running terraform apply.

The terraform configuration for the sns subscription was created without the filter policy and the policy was added using aws cli. Since the filter policy is not in the configuration file terraform will try to remove it and fail.

The same issue can be reproduced if you remove the filter_policy argument from aws_sns_topic_subscription resource after applying the configuration.

Terraform Version

• Downloading plugin for provider "template" (1.0.0)...
• Downloading plugin for provider "aws" (1.9.0)...
• Downloading plugin for provider "consul" (1.0.0)...

Affected Resource(s)

• aws_sns_topic_subscription

Terraform Configuration Files

  endpoint = "${aws_sqs_queue.test.arn}"
  protocol = "sqs"
  topic_arn = "${var.sns_test_topic_arn}"
  raw_message_delivery = true
}

Terraform plan

Terraform will perform the following actions:
~ module.test.aws_sns_topic_subscription.test-subscription
filter_policy: "{"TYPE":["TEST"]}" => ""

Expected Behavior

The filter policy should have been removed.

Actual Behavior

Error: Error applying plan:

1 error(s) occurred:

• module.test.aws_sns_topic_subscription.test-subscription: 1 error(s) occurred:

• aws_sns_topic_subscription.test-subscription: Unable to set filter policy attribute on subscription: InvalidParameter: Invalid parameter: FilterPolicy: Filter is not an object
  at [Source: ; line: 1, column: 1]

[bflad]

@oanasabau sorry you're running into trouble with this. I'm able to reproduce by adding an additional step to the existing acceptance test that tries to remove the filter policy and receive the same error from the SNS API. What's even more curious is that I'm not sure how to coerce the SNS SDK to "unset" the filter policy easily. 😢

• When you do SetSubscriptionAttribute with AttributeValue not set at all: InvalidParameter: Invalid parameter: FilterPolicy: missing required
• When you do SetSubscriptionAttribute with AttributeValue as {}: InvalidParameter: Invalid parameter: FilterPolicy: Empty objects are not allowed
• When you do SetSubscriptionAttribute with AttributeValue as null: InvalidParameter: Invalid parameter: FilterPolicy: Filter is not an object

Does it happen to work from the AWS CLI without the --attribute-value specified?

[oanasabau]

I get same errors when running the command using aws cli.

• without --atribute-value
  An error occurred (InvalidParameter) when calling the SetSubscriptionAttributes operation: Invalid parameter: FilterPolicy: missing required

• --attribute-value null

An error occurred (InvalidParameter) when calling the SetSubscriptionAttributes operation: Invalid parameter: FilterPolicy: Filter is not an object
at [Source: null; line: 1, column: 9]

• --attribute-value ""

An error occurred (InvalidParameter) when calling the SetSubscriptionAttributes operation: Invalid parameter: FilterPolicy: Filter is not an object
at [Source: ; line: 1, column: 1]

• --attribute-value "{}"

An error occurred (InvalidParameter) when calling the SetSubscriptionAttributes operation: Invalid parameter: FilterPolicy: Empty objects are not allowed
at [Source: {}; line: 1, column: 3]

[oanasabau]

This is the info displayed when trying to edit subscription filter policy in the amazon console:

A subscription filter policy is a simple JSON object that enables your subscriber to receive only the subset of messages it’s actually interested in, as opposed to receiving every single message published to the topic.

When a message is published to the topic, Amazon SNS will attempt to match the incoming message attributes to the subscription attribute that defines a filter policy. If they match, Amazon SNS will then deliver the message to the corresponding subscriber.

Although a filter policy can be updated at any point, it cannot be deleted.

Maybe this can be solved by deleting and recreating the resource without the policy filter?

[bflad]

That sounds like it might be our only option given the lack of backend API support. It certainly deserves a documentation note on our end and we can add a CustomizeDiff function to check for removal and mark the resource as ForceNew.

[oanasabau]

One thing to consider when doing this is what happens with the messages if the new resource cannot be created before the old one is destroyed (some messages might be lost).

[grimm26]

FYI, have you tried a workaround of removing the filter_policy from the config and then tainting the subscription resource so it is recreated? That would delete the whole subscription (and the filter_policy) and recreate it (without the filter_policy).

[Ninir]

Since we can update the value but can't remove it, we might solve this issue by implementing the CustomizeDiff and marking the resource as ForceNew (example).

Thoughts? :)

[otaviofff]

You can unset a subscription filter policy by providing an empty JSON object.

Have a look at the CLI example below.

sns set-subscription-attributes --subscription-arn <INSERT-ARN-HERE> --attribute-name FilterPolicy --attribute-value "{ }"

[otaviofff]

More info about removing filter policies can be found here:
https://docs.aws.amazon.com/sns/latest/dg/message-filtering.html#message-filtering-policy-remove

[bflad]

Pull request submitted: #6023

[bflad]

The fix for this issue has been merged and will release with version 1.39.0 of the AWS provider, likely later today.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!