Timeout waiting for long running Amazon EBS AMI copies on v1.2.5 #6536

nathandines · 2018-07-25T04:37:14Z

This is probably more a lack of documentation around the attempts and polling delay than a functional issue. Also, the error should be more verbose by default. The output below was with PACKER_LOG=1 set

[14:22:45]	==> amazon-ebs: Waiting for AMI to become ready...
[14:22:45]	2018/07/25 04:22:45 packer.exe: 2018/07/25 04:22:45 No AWS timeout and polling overrides have been set. Packer will defalt to waiter-specific delays and timeouts. If you would like to customize the length of time between retries and max number of retries you may do so by setting the environment variables AWS_POLL_DELAY_SECONDS and AWS_MAX_ATTEMPTS to your desired values.
[14:33:19]	2018/07/25 04:33:19 packer.exe: 2018/07/25 04:33:19 Error waiting for AMI: ResourceNotReady: exceeded wait attempts
[14:33:19]	==> amazon-ebs: Error waiting for AMI. Reason: <nil>
[14:33:19]	2018/07/25 04:33:19 ui error: ==> amazon-ebs: Error waiting for AMI. Reason: <nil>
[14:33:19]	2018/07/25 04:33:19 ui: ==> amazon-ebs: Terminating the source AWS instance...
[14:33:19]	==> amazon-ebs: Terminating the source AWS instance...
[14:33:19]	2018/07/25 04:33:19 packer.exe: 2018/07/25 04:33:19 No AWS timeout and polling overrides have been set. Packer will defalt to waiter-specific delays and timeouts. If you would like to customize the length of time between retries and max number of retries you may do so by setting the environment variables AWS_POLL_DELAY_SECONDS and AWS_MAX_ATTEMPTS to your desired values.
[14:33:36]	2018/07/25 04:33:36 ui: ==> amazon-ebs: Cleaning up any extra volumes...
[14:33:36]	==> amazon-ebs: Cleaning up any extra volumes...
[14:33:36]	2018/07/25 04:33:36 ui: ==> amazon-ebs: Deleting temporary security group...
[14:33:36]	==> amazon-ebs: Deleting temporary security group...
[14:33:37]	==> amazon-ebs: Deleting temporary keypair...
[14:33:37]	2018/07/25 04:33:37 ui: ==> amazon-ebs: Deleting temporary keypair...
[14:33:37]	2018/07/25 04:33:37 [INFO] (telemetry) ending amazon-ebs
[14:33:37]	Build 'amazon-ebs' errored: Error waiting for AMI. Reason: <nil>
[14:33:37]	2018/07/25 04:33:37 ui error: Build 'amazon-ebs' errored: Error waiting for AMI. Reason: <nil>
[14:33:37]	2018/07/25 04:33:37 Builds completed. Waiting on interrupt barrier...
[14:33:37]	
[14:33:37]	2018/07/25 04:33:37 machine readable: error-count []string{"1"}
[14:33:37]	==> Some builds didn't complete successfully and had errors:
[14:33:37]	2018/07/25 04:33:37 ui error:
[14:33:37]	--> amazon-ebs: Error waiting for AMI. Reason: <nil>
[14:33:37]	==> Some builds didn't complete successfully and had errors:
[14:33:37]	
[14:33:37]	2018/07/25 04:33:37 machine readable: amazon-ebs,error []string{"Error waiting for AMI. Reason: <nil>"}
[14:33:37]	==> Builds finished but no artifacts were created.
[14:33:37]	2018/07/25 04:33:37 ui error: --> amazon-ebs: Error waiting for AMI. Reason: <nil>

The text was updated successfully, but these errors were encountered:

xtimon · 2018-07-25T10:36:08Z

I have the same issue

jjuarez · 2018-07-25T10:42:41Z

I'm facing some issues copying the generated AMIs in different AWS regions, using the same Packer version (1.2.5)

2018/07/25 12:43:12 ui: ==> amazon-ebs: Deregistering the AMI because cancellation or error...
==> amazon-ebs: * Error waiting for AMI (ami-c6cf96be) in region (us-west-2): ResourceNotReady: exceeded wait attempts
==> amazon-ebs: Deregistering the AMI because cancellation or error...
2018/07/25 12:43:12 ui: ==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Terminating the source AWS instance...
2018/07/25 12:43:13 packer: 2018/07/25 12:43:13 No AWS timeout and polling overrides have been set. Packer will defalt to waiter-specific delays and timeouts. If you would like to customize the length of time between retries and max number of retries you may do so by setting the environment variables AWS_POLL_DELAY_SECONDS and AWS_MAX_ATTEMPTS to your desired values.
==> amazon-ebs: Cleaning up any extra volumes...
2018/07/25 12:43:28 ui: ==> amazon-ebs: Cleaning up any extra volumes...
2018/07/25 12:43:28 ui: ==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: No volumes to clean up, skipping
2018/07/25 12:43:28 ui: ==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary security group...
2018/07/25 12:43:28 [INFO] (telemetry) ending amazon-ebs
2018/07/25 12:43:28 ui error: Build 'amazon-ebs' errored: 1 error(s) occurred:

* Error waiting for AMI (ami-c6cf96be) in region (us-west-2): ResourceNotReady: exceeded wait attempts
2018/07/25 12:43:28 Builds completed. Waiting on interrupt barrier...
2018/07/25 12:43:28 machine readable: error-count []string{"1"}
2018/07/25 12:43:28 ui error:
==> Some builds didn't complete successfully and had errors:
2018/07/25 12:43:28 machine readable: amazon-ebs,error []string{"1 error(s) occurred:\n\n* Error waiting for AMI (ami-c6cf96be) in region (us-west-2): ResourceNotReady: exceeded wait attempts"}
2018/07/25 12:43:28 ui error: --> amazon-ebs: 1 error(s) occurred:

* Error waiting for AMI (ami-c6cf96be) in region (us-west-2): ResourceNotReady: exceeded wait attempts
2018/07/25 12:43:28 ui:
==> Builds finished but no artifacts were created.
2018/07/25 12:43:28 [INFO] (telemetry) Finalizing.
Build 'amazon-ebs' errored: 1 error(s) occurred:

* Error waiting for AMI (ami-c6cf96be) in region (us-west-2): ResourceNotReady: exceeded wait attempts

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: 1 error(s) occurred:

* Error waiting for AMI (ami-c6cf96be) in region (us-west-2): ResourceNotReady: exceeded wait attempts

==> Builds finished but no artifacts were created.
2018/07/25 12:43:29 waiting for all plugin processes to complete...
2018/07/25 12:43:29 /usr/local/bin/packer: plugin process exited

mwhooker · 2018-07-25T16:05:22Z

I think we'll try to bump the defaults but please try increasing AWS_MAX_ATTEMPTS as a workaround

SwampDragons · 2018-07-25T16:18:36Z

This is probably due to the same underlying issues as #6526, so I'm going to mark as a duplicate. Not closing because this seems to affect enough people that I want to have an issue open for them to find.

Please let us know if setting the env variable AWS_MAX_ATTEMPTS helps, and what you set it to. It'll help us figure out the best default value.

nathandines · 2018-07-25T23:13:24Z

Yeah, setting AWS_MAX_ATTEMPTS and AWS_POLL_DELAY_SECONDS was the resolution in the environment I'm working in.

The copy would take ~20 minutes total, but I set the values of each env var to 60 and 60 respectively (so 1 hour of polling at 1 minute intervals), which worked well for me.

nodesocket · 2018-07-26T00:33:34Z

👍 Have this issue copying AMI from us-west-2 to eu-central-1. This issue cropped up when I updated to 1.2.5. Did not occur in 1.2.4.

SwampDragons · 2018-07-27T17:56:06Z

So it sounds like for most people the waiter whose defaults are two low is the AMI Copy one. I can manually modify the defaults on that waiter, or we can try to give feedback to AWS to bump the defaults in their SDK.

mwchristiansen · 2018-07-31T05:20:13Z

brew switch packer 1.2.4 fixed the problem for me

embusalacchi · 2018-08-03T18:44:34Z

I am seeing this issue as well - I will try the AWS_MAX_ATTEMPTS and AWS_POLL_DELAY_SECONDS settings and report back.

chenghuang-mdsol · 2018-08-15T21:34:09Z

any updates?

SwampDragons · 2018-08-15T23:23:48Z

@chenghuang-mdsol Not really; I'm still waiting for more user input about what values of AWS_MAX_ATTEMPTS and AWS_POLL_DELAY_SECONDS are preventing this issue. I still intend to address this by the next release, but I'm letting this collect user input in the meantime.

SwampDragons · 2018-08-17T20:34:17Z

Okay, I figured out why this wait was affecting so many of you check out the linked pr for more details; I'm attaching a linux build of #6601 which should have the timeout set higher -- 30 minutes instead of 10. Let me know if this is too low still.
packer.zip

SwampDragons · 2018-08-23T21:37:39Z

Can any of you comment as to whether #6601 solved your problem?

nathandines · 2018-08-23T21:47:05Z

I'm not currently working in the environment where I was experiencing this issue.

I'll try and spin up a test AMI using the existing build (to replicate the issue), and then using this build as soon as I get the chance.

EDIT: Testing now. I'll let you know how it goes.

embusalacchi · 2018-08-23T21:48:54Z

I have to use Docker in the AWS build environment that I'm having this issue in so I'm pulling down the latest official version each time. Is there a ready made docker image of preview builds by any chance?

nathandines · 2018-08-23T23:09:43Z

@SwampDragons I can confirm that I replicated the issue again in Packer v1.2.5, and it appears to be resolved in the build which you posted above. Seems like #6601 does fix the problem for me.

Thanks.

SwampDragons · 2018-08-24T16:29:58Z

@embusalacchi nope, sorry.

The slaves can have quite a lot of containers on them now and by default Packer isn't providing enough time for the AMI to be successfully copied and saved. Suggestion taken from here: hashicorp/packer#6536

ghost · 2020-03-31T02:01:31Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

mwhooker added bug builder/amazon labels Jul 25, 2018

mwhooker added this to the upcoming-patch milestone Jul 25, 2018

SwampDragons added stage/duplicate regression labels Jul 25, 2018

SwampDragons mentioned this issue Jul 27, 2018

[1.2.5] amazon-import post-processor #6542

Closed

SwampDragons mentioned this issue Aug 17, 2018

Increase waiter timeout for AWS WaitUntilImageAvailable command #6601

Merged

SwampDragons mentioned this issue Aug 21, 2018

Windows-Restart provisioner: Lots of usage messages from shutdown.exe after switching to SSH communicator #6301

Closed

SwampDragons closed this as completed in #6601 Aug 24, 2018

dnr1503 mentioned this issue Apr 5, 2019

Packer ami creation time increase #7472

Closed

bhaskardegala mentioned this issue May 15, 2019

Issue with Packer , Not able to launch aws machine QT-DevOps/DevOpsIssues#452

Closed

jacderida mentioned this issue Oct 16, 2019

fix/job: larger timeout for ami build maidsafe-archive/jenkins-jobs#35

Closed

jacderida mentioned this issue Oct 16, 2019

fix/job: larger timeout for ami build maidsafe-archive/jenkins-jobs#36

Merged

ghost locked and limited conversation to collaborators Mar 31, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Timeout waiting for long running Amazon EBS AMI copies on v1.2.5 #6536

Timeout waiting for long running Amazon EBS AMI copies on v1.2.5 #6536

nathandines commented Jul 25, 2018 •

edited

Loading

xtimon commented Jul 25, 2018

jjuarez commented Jul 25, 2018 •

edited

Loading

mwhooker commented Jul 25, 2018

SwampDragons commented Jul 25, 2018

nathandines commented Jul 25, 2018

nodesocket commented Jul 26, 2018 •

edited

Loading

SwampDragons commented Jul 27, 2018

mwchristiansen commented Jul 31, 2018

embusalacchi commented Aug 3, 2018

chenghuang-mdsol commented Aug 15, 2018

SwampDragons commented Aug 15, 2018

SwampDragons commented Aug 17, 2018

SwampDragons commented Aug 23, 2018

nathandines commented Aug 23, 2018 •

edited

Loading

embusalacchi commented Aug 23, 2018

nathandines commented Aug 23, 2018 •

edited

Loading

SwampDragons commented Aug 24, 2018

ghost commented Mar 31, 2020

Timeout waiting for long running Amazon EBS AMI copies on v1.2.5 #6536

Timeout waiting for long running Amazon EBS AMI copies on v1.2.5 #6536

Comments

nathandines commented Jul 25, 2018 • edited Loading

xtimon commented Jul 25, 2018

jjuarez commented Jul 25, 2018 • edited Loading

mwhooker commented Jul 25, 2018

SwampDragons commented Jul 25, 2018

nathandines commented Jul 25, 2018

nodesocket commented Jul 26, 2018 • edited Loading

SwampDragons commented Jul 27, 2018

mwchristiansen commented Jul 31, 2018

embusalacchi commented Aug 3, 2018

chenghuang-mdsol commented Aug 15, 2018

SwampDragons commented Aug 15, 2018

SwampDragons commented Aug 17, 2018

SwampDragons commented Aug 23, 2018

nathandines commented Aug 23, 2018 • edited Loading

embusalacchi commented Aug 23, 2018

nathandines commented Aug 23, 2018 • edited Loading

SwampDragons commented Aug 24, 2018

ghost commented Mar 31, 2020

nathandines commented Jul 25, 2018 •

edited

Loading

jjuarez commented Jul 25, 2018 •

edited

Loading

nodesocket commented Jul 26, 2018 •

edited

Loading

nathandines commented Aug 23, 2018 •

edited

Loading

nathandines commented Aug 23, 2018 •

edited

Loading