New service: AWS Lake Formation

[ewbankkit]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

AWS Lake Formation is now GA.

New or Affected Resource(s)

• aws_XXXXX

Potential Terraform Configuration

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.

References

Announcement.
Blog post.

Requires:

• Update module aws/aws-sdk-go to v1.23.0 #9670

[slapula]

I'm also looking into writing up these resources as well.

[rafiparvez]

We are using AWS lakeformation for creating data lake at my organization. I am interested in contributing to this.

[ewbankkit]

@rafiparvez Could you please propose some resources that would match your uses cases?

[dthauvin]

@ewbankkit one off major features that brings AWS LakeFormation , is the capabilities to control fine-grained privilege on GlueTable ( like GRANT at column level) and s3 location with course-grained Permissions. So far to enable use case around fine grained i think we need 3 ressources:

AWS::LakeFormation::DataLakeSettings ( config between IAM policy and LakeFormation policy)
AWS::LakeFormation::Permissions (represents the permissions that a principal has on catalog)
AWS::LakeFormation::Resource (represents the data (Amazon S3 buckets and folders)

BR

[gmazelier]

Here is a proposal to support Lake Formation settings (admins configuration), resources (de/register data locations) and permissions. Feedbacks are welcome!

[tomaszdudek7]

Any ETA?

[packetlag]

bump for an update, please

[ewbankkit]

@packetlag This has been added to the August - October 2020 Roadmap. Thanks.

[peramnaveen29]

#14411
I guess it is related with the current AWS Lake Formation enhancement

[theophilechevalier]

Hi @ewbankkit,
I'm willing to finish the implementation but there are already numerous open pull requests on this matter. Can you plan a review so we can fix what's missing? We're a lot to need this feature

[roberth-k]

Resource Links (#15296) and CreateTableDefaultPermissions (#14411) should be implemented as part of this epic, as they are essential for accessing cross-account Lake Formation resources using Athena and Redshift Spectrum.

[YakDriver]

The first release with Lake Formation support is likely to include these data sources and resources:

• resource/aws_lakeformation_resource (Lake Formation Resource #13267)
• data source/aws_lakeformation_resource (Lake Formation Permissions #13396)
• resource/aws_lakeformation_data_lake_settings (Lake Formation Data Lake Settings #13250)
• data source/aws_lakeformation_data_lake_settings (Lake Formation Permissions #13396)
• resource/aws_lakeformation_permissions (Lake Formation Permissions #13396)
• data source/aws_lakeformation_permissions (Lake Formation Permissions #13396)

Subsequent releases (including additional contributions from community members) may include these data sources and resources:

• data source/aws_lakeformation_resources (see ds/lakeformation_resources: New data source #16845)
• data source/aws_lakeformation_effective_permissions (see ds/lakeformation_effective_permissions: New data source #16846)

[priyen]

@YakDriver Do you have any idea of any sort of ETA for lake formation support being available? I see lots of in flight PRs above.

[YakDriver]

Should be available with version 3.22!

[YakDriver]

Closing this since significant Lake Formation funcitonality has been merged. Remaining functionality is identified in #16845 and #16846. If there are other issues and features needed, please open a new issue! Thank you for your interest in Lake Formation!

[ghost]

This has been released in version 3.22.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!