Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Key in dictionary does not always exist #998

Closed
jkhairal opened this issue Nov 10, 2021 · 1 comment · Fixed by #999
Closed

Key in dictionary does not always exist #998

jkhairal opened this issue Nov 10, 2021 · 1 comment · Fixed by #999
Assignees
Labels
bug Something isn't working minor release Will be addressed in the next minor release ready to release
Milestone

Comments

@jkhairal
Copy link

Describe the bug

In the following line of the code:
https:/awslabs/aws-data-wrangler/blob/main/awswrangler/catalog/_create.py#L53 (And/Or Line 96)
If the Glue table actually exists and does not have any tags, it will not return table_inputs["Parameters"] and there will be a Key Error at the above lines.

Environment
Python:

 File "test.py", line 345, in main
    upload_data_to_s3(entries, bucket, database, "ticket_detailed_entries", 'modified_date', log)
  File "test.py", line 201, in upload_data_to_s3
    wr.s3.to_parquet(
  File "/apollo/env/HWEngDSADataIngester/lib/python3.8/site-packages/awswrangler/_config.py", line 417, in wrapper
    return function(**args)
  File "/apollo/env/HWEngDSADataIngester/lib/python3.8/site-packages/awswrangler/s3/_write_parquet.py", line 582, in to_parquet
    catalog._create_parquet_table(  # pylint: disable=protected-access
  File "/apollo/env/HWEngDSADataIngester/lib/python3.8/site-packages/awswrangler/catalog/_create.py", line 262, in _create_parquet_table
    _create_table(
  File "/apollo/env/HWEngDSADataIngester/lib/python3.8/site-packages/awswrangler/catalog/_create.py", line 96, in _create_table
    table_input["Parameters"]["projection.enabled"] = "false"

To Reproduce

Create a simple glue table with no tags, call the following function:

wr.s3.to_parquet(
    df=dataframe,
    path=f"s3://{output_bucket}/{output_s3_prefix}",
    index=False,
    dataset=True,
    mode='overwrite_partitions',
    partition_cols=[partition],
    concurrent_partitioning=True,
    database=database,  # Athena/Glue database
    table=table,  # Athena/Glue table
    dtype=all_types
)
@jkhairal jkhairal added the bug Something isn't working label Nov 10, 2021
@kukushking kukushking self-assigned this Nov 11, 2021
@kukushking kukushking linked a pull request Nov 11, 2021 that will close this issue
@kukushking kukushking added minor release Will be addressed in the next minor release ready to release labels Nov 11, 2021
@kukushking kukushking added this to the 2.13.0 milestone Nov 11, 2021
@jaidisido
Copy link
Contributor

Should be fixed by #999 and will be available in next release. Thank you for raising

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working minor release Will be addressed in the next minor release ready to release
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants