Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(glue): Iceberg Table Support on S3Table construct #29660

Open
1 of 2 tasks
lawofcycles opened this issue Mar 31, 2024 · 4 comments
Open
1 of 2 tasks

(glue): Iceberg Table Support on S3Table construct #29660

lawofcycles opened this issue Mar 31, 2024 · 4 comments
Labels
@aws-cdk/aws-glue Related to AWS Glue effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p3

Comments

@lawofcycles
Copy link
Contributor

lawofcycles commented Mar 31, 2024

Describe the feature

Add cfn-glue-table-opentableformatinput to the Glue S3 Table construct

In addition, create new construct TableOptimizer supporting AWS::Glue::TableOptimizer

Then add option param that create TableOptimizer corresponding to the Open Table Format based table to the Glue S3 Table construct.

Use Case

  • Users will be able to define Glue Tables in CDK that support Iceberg and Open Table Format
  • Users can easily create a corresponding TableOptimizer when defining Iceberg-based Glue Tables

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change
@lawofcycles lawofcycles added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Mar 31, 2024
@github-actions github-actions bot added the @aws-cdk/aws-glue Related to AWS Glue label Mar 31, 2024
@lawofcycles
Copy link
Contributor Author

I'm willing to work on implementing this feature myself if my idea looks good.

@tim-finnigan tim-finnigan self-assigned this Apr 1, 2024
@tim-finnigan tim-finnigan added investigating This issue is being investigated and/or work is in progress to resolve the issue. and removed needs-triage This issue or PR still needs to be triaged. labels Apr 1, 2024
@tim-finnigan
Copy link

Thanks for creating this feature request and offering to work on a PR - I think this sounds like a good idea.

@tim-finnigan tim-finnigan added p2 effort/medium Medium work item – several days of effort and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Apr 1, 2024
@tim-finnigan tim-finnigan removed their assignment Apr 1, 2024
@pahud pahud added p3 and removed p2 labels Jun 11, 2024
@Smotrov
Copy link

Smotrov commented Aug 6, 2024

So far you can do like this to create an Iceberg table

  const myTable = new glue.S3Table(props.scope, 'IcebergTest2', {
    database: props.database,
    tableName: 'iceberg_test2',
    bucket: props.bucket,
    s3Prefix: 'iceberg_test2',
    dataFormat: glue.DataFormat.PARQUET,
    columns: [{
      name: 'col1',
      type: glue.Schema.STRING,
    }],
  });


  // Hack starts here to make the table Iceberg
  const cfnTable = myTable.node.defaultChild as mainGlue.CfnTable;

  cfnTable.openTableFormatInput = {
    icebergInput: {
      metadataOperation: 'CREATE',
      version: '2',
    }
  };

@Rizxcviii
Copy link
Contributor

Just as a side note, there is currently an open issue in CloudFormation whereby you cannot update a iceberg table after creation, as the table format will be deleted when an update takes place in the CloudFormation template. Just a little thing to make note of
aws-cloudformation/cloudformation-coverage-roadmap#1919 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-glue Related to AWS Glue effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p3
Projects
None yet
Development

No branches or pull requests

5 participants