Add SageMaker support #2493

jckuester · 2017-11-30T12:12:34Z

Amazon has announced the new service SageMaker to build, train, and deploy machine learning models at scale.

Prerequisite

aws-sdk-go v1.12.36 (#2474)

Affected Resource(s)

The following resources are needed.

For building model and training:

aws_sagemaker_notebook_instance (New Resource: aws_sagemaker_notebook_instance #2999)
aws_presigned_notebook_instance_url
aws_sagemaker_training_job (New Resource: aws_sagemaker_training_job #2955)
aws_sagemaker_lifecycle_config resource (New resource: aws_sagemaker_lifecycle_config #7585)

For the deployment/hosting part:

aws_sagemaker_model (open PR is New Resource: aws_sagemaker_model #2478)
aws_sagemaker_endpoint (open PR is New Resource: aws_sagemaker_endpoint #2479)
aws_sagemaker_endpoint_configuration (open PR is New Resource: aws_sagemaker_endpoint_configuration #2477)

Expected Behavior

Create, update, delete, and import for above listed SageMaker resources of deployment part:

resource "aws_sagemaker_endpoint" "foo" {
	name = "endpoint-foo"
	endpoint_config_name = "${aws_sagemaker_endpoint_configuration.ec.name}"
}

resource "aws_sagemaker_endpoint_configuration" "ec" {
	name = "endpoint-config-foo"

	production_variants {
                variant_name            = "variant-1"
		model_name = "${aws_sagemaker_model.m.name}"
                initial_instance_count  = 1
                instance_type           = "m3.xlarge"
                initial_variant_weight  = 1
	}
}

resource "aws_sagemaker_model" "m" {
    name = "my-model"

    primary_container {
       image = "111111111111.ecr.us-west-2.amazonaws.com/my-docker-image:latest"
       model_data_url  = "s3://111111111111-foo/model.tar.gz"
    }
}

Example

A terraform example that shows how to use the above resources to deploy your own model is here: #2585

References

https:/aws/aws-sdk-go/tree/master/service/sagemaker

darrenhaken · 2017-12-13T08:27:01Z

Has anyone picked this one up yet?

jckuester · 2017-12-13T09:21:13Z

@darrenhaken yes, I am working on it. See the linked PRs in the issue :-)

darrenhaken · 2017-12-15T14:55:44Z

@jckuester I can't see a PR for the notebook instances, are you working on that too?

jckuester · 2017-12-15T21:29:33Z

No, please start working on them if you like :)

ddcprg · 2017-12-20T14:43:34Z

Hi @jckuester @darrenhaken , anyone working on the training job resource? I'd like to contribute to this if possible

jckuester · 2018-01-03T17:27:28Z

@ddcprg I am only working on the linked PRs above (i.e. for deployment). So please contribute the training job resource if you like.

ddcprg · 2018-01-04T15:55:47Z

cool @jckuester ! I'll be taking a look soon and comment back on this ticket with an initial prototype

ddcprg · 2018-01-10T11:39:18Z

@jckuester I now you've done this in your PR but I've raised #2924 just to add the vendor as per the README

I have the training job nearly done, I'm working on testing and I'll add the PR to this ticket as soon as I consider it ready

ddcprg · 2018-01-12T08:54:38Z

Training Job PR -> #2955

If no one has picked up the notebook resource yet I'll take it

ddcprg · 2018-01-15T18:03:29Z

Notebook Instance PR -> #2999

ddcprg · 2018-05-16T09:20:36Z

How can we help to put some traction on these PR's? I'm going to go over the code and add any new features included by AWS recently if needed

ddcprg · 2018-05-16T10:34:29Z

@jckuester would you mind adding a new item to the list above to include support for "lifecycle configurations"? I'll take a look and post the PR number later

ddcprg · 2018-06-13T09:16:02Z

@randomcamel @radeksimko how can we help to push these PR's forward?

smrtslckr · 2018-07-12T23:57:17Z

+1, very interested in this functionality

ashleyjkell · 2018-08-20T12:52:08Z

Not been any movement on this for a while (looks like failed integration tests), any chance of getting it pushed on?

ddcprg · 2018-08-20T12:56:46Z

This has been open for long time, we are still waiting feedback from Hashicorp. The failures in the build are due to other issues in master at the time of merging, they may well have gone away by now

jckuester · 2018-08-21T08:01:56Z

@ashleyjkell from time to time, I rebased my PRs (#2477 - #2479) to make integration test green again (I guess they still are). But as long as there is no feedback coming from HashiCorp I will not wasting any more energy here to keeping PRs uptodate/resolving conflicts with main branch.

rainmanh · 2018-10-05T10:11:37Z

Any updates on this one?
Anyone from Hashicorp?

chasmosis · 2018-10-29T14:18:17Z

I am interested in this functionality as well. Updates?

ddcprg · 2018-11-01T19:02:35Z

I've just updated my PR's so they are green again. It'd be nice to have some feedback from @bflad or other TF committers. The PR's have been opened for about 10 months now

bcatubig · 2018-11-05T17:29:03Z

@bflad Is there any way to get these PR's approved and merged?

t-lanigan · 2018-11-05T17:32:30Z

Also adding my voice to this one. Interested in getting this merged

mbfrahry · 2018-11-08T15:41:01Z

Hello! I'm sorry for the delay and I'm actively looking at getting SageMaker support merged into the aws provider.

mbfrahry · 2018-11-08T15:42:04Z

I've just finished my review of the instance and training job resources and will be moving down the list noted here shortly.

ddcprg · 2018-11-09T10:56:42Z

awesome @mbfrahry ! I'll try to go through you review today and make the necessary changes

mbfrahry · 2018-11-27T15:36:39Z

Hey @reynico, you won't have to do anything with modifying the version of Terraform. If you're pinned to a specific version of the aws provider, then you'll have to modify that to get these changes when they're released.

rainmanh · 2018-11-28T17:06:22Z

Any estimate at all when this will be released?

ddcprg · 2019-01-04T09:33:49Z

Hello everyone. Unfortunately I've not been able to do the modifications as per @mbfrahry 's review and right now my access to AWS is limited and won't be able to test the code with the changes.

If anyone is willing to pick the first two bullets in the description up that'd be great, the PR is #2999 Perhaps @jckuester ? Whoever takes this on can squash my commits or start from scratch if you prefer.

mbfrahry · 2019-01-07T17:36:49Z

Hey @ddcprg, sorry to hear about your troubles. I'll take this on with @tracypholmes if it hasn't been picked up yet

ddcprg · 2019-01-08T09:55:38Z

Sure @mbfrahry pick it up, apologies for the inconveniences

bflad · 2019-01-16T04:17:51Z

The new aws_sagemaker_notebook_instance resource has been released in version 1.56.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

smrtslckr · 2019-01-16T23:38:04Z

That is a great milestone! I am sorry to be "that person", but I have been more interested in the 3 outstanding deployment and hosting PRs: 2477, 2478, and 2479. Is there a current ETA on those now too?

jckuester · 2019-01-21T17:15:31Z

@Erstwild we are very close :) I addressed all comments on the sagemaker PRs (which are in the 2nd feedback round now) and am now waiting for approval.

rainmanh · 2019-01-22T16:14:22Z

@julesjcraske
Is there any plan for implementing the ability of disabling the Direct internet access
And what about of attaching Lifecycle configurations ?

Many thanks

bcatubig · 2019-01-22T17:04:17Z

I would say the following are required in a followup pr.

LifecycleConfigName
DirectInternetAccess
DefaultCodeRepository
AdditionalCodeRepositories
VolumeSizeInGB (?)
AcceleratorTypes (?)

saritajoshi9389 · 2019-02-07T23:58:18Z

Hi there,
Do we have any specific roadmap or timelines by which we are certain about PRs: 2477, 2478, and 2479? Along with attaching Lifecycle configurations ?

bflad · 2019-02-08T08:38:20Z

The new aws_sagemaker_model resource has been released in version 1.58.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

jckuester · 2019-02-15T20:30:38Z

Added a new PR for the aws_sagemaker_lifecycle_config resource (#7585). I also have updates for the aws_sagemaker_notebook_instance in the pipeline (@saritajoshi9389, @bcatubig), as we need those resources in our company @yoyolabsio.

bcatubig · 2019-03-10T19:13:23Z

Added a PR for DirectInternetAccess support #7884 . Would love some feedback to see if things could be improved.

jckuester · 2019-03-19T15:59:31Z

Here is another PR (#8011) that adds all missing attributes to aws_sagemaker_instance, ie. adapts the resource to the latest AWS API.

bflad · 2019-03-29T06:36:02Z

The aws_sagemaker_endpoint_configuration resource has been released in version 2.4.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

gattil · 2019-04-04T11:45:53Z

It would be really great to have the notebook lifecycle implemented. This is crucial for one of my clients.

bflad · 2019-04-05T10:00:24Z

Hi folks 👋 The following will be releasing in version 2.5.0 of the Terraform AWS Provider shortly:

New Resource: aws_sagemaker_endpoint
New Resource: aws_sagemaker_notebook_instance_lifecycle_configuration
resource/aws_sagemaker_notebook_instance: Add lifecycle_config_name argument

rverma-nikiai · 2019-05-13T06:18:45Z

@bflad Is this released?

tdmalone · 2019-05-13T07:16:15Z

@rverma-nikiai The resources available for the AWS provider can be found at https://www.terraform.io/docs/providers/aws/ - if you type 'sagemaker' in the search box at the top left the resources mentioned above (and the new argument) have indeed been released.

bcatubig · 2019-05-13T18:09:04Z

Created a new PR for DirectInternetAccess support as I nuked my previous PR by mistake. The new PR is #8618

LarsNeR · 2019-08-01T13:23:38Z

Is it possible to change the timeout for creating the sagemaker endpoint? For me it stops after 1m saying

ResourceNotReady: failed waiting for successful resource state

Usually creating the endpoint I have takes around 15 minutes. Adding
timeouts { create = "30m" delete = "30m" }
like it is possible on other resources gives me:

Error decoding timeout: Timeout Key (create) is not supported

bflad · 2019-08-01T13:30:33Z

Hi @LarsNeR 👋 My recommendation would be to open submit new GitHub issues following the bug report and feature request templates for existing functionality. These large "support service X" GitHub issues tend to not have a clear definition of done and usually get closed out in preference of more targeted issues for tracking individual asks.

LarsNeR · 2019-08-01T14:04:25Z

I've just seen it was an error on my site. Works as expected now.

bflad · 2020-01-29T16:36:48Z

Hi folks 👋

We just merged direct_internet_access argument support into the aws_sagemaker_notebook_instance resource. It will release with version 2.47.0 of the Terraform AWS Provider, tomorrow. Thanks to @bcatubig for the implementation. 👍

Please note that I'm going to close this GitHub issue since the majority of the original work for it has been completed and since there is not a clear definition of done over time. If you have specific bug reports with existing functionality or feature requests for additional functionality please use those issue templates. Thanks.

ghost · 2020-03-27T17:33:01Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

randomcamel added the enhancement Requests to existing resources that expand the functionality or scope. label Nov 30, 2017

ddcprg mentioned this issue Jan 10, 2018

Add SageMaker vendor #2924

Merged

This was referenced Jan 12, 2018

New Resource: aws_sagemaker_training_job #2955

Closed

New Resource: aws_sagemaker_notebook_instance #2999

Merged

radeksimko added the service/sagemaker Issues and PRs that pertain to the sagemaker service. label Jan 28, 2018

saritajoshi9389 mentioned this issue Feb 7, 2019

Terraform + SageMaker full support- Roadmap? hashicorp/terraform#20262

Closed

LarsNeR mentioned this issue Aug 1, 2019

Add timeout support for aws_sagemaker_endpoint #9584

Closed

bflad closed this as completed Jan 29, 2020

ghost locked and limited conversation to collaborators Mar 27, 2020

Add SageMaker support #2493

Add SageMaker support #2493

Comments

jckuester commented Nov 30, 2017 • edited Loading

Prerequisite

Affected Resource(s)

Expected Behavior

Example

References

darrenhaken commented Dec 13, 2017

jckuester commented Dec 13, 2017

darrenhaken commented Dec 15, 2017

jckuester commented Dec 15, 2017

ddcprg commented Dec 20, 2017 • edited Loading

jckuester commented Jan 3, 2018

ddcprg commented Jan 4, 2018

ddcprg commented Jan 10, 2018

ddcprg commented Jan 12, 2018 • edited Loading

ddcprg commented Jan 15, 2018

ddcprg commented May 16, 2018

ddcprg commented May 16, 2018

ddcprg commented Jun 13, 2018

smrtslckr commented Jul 12, 2018

ashleyjkell commented Aug 20, 2018

ddcprg commented Aug 20, 2018

jckuester commented Aug 21, 2018 • edited Loading

rainmanh commented Oct 5, 2018 • edited Loading

chasmosis commented Oct 29, 2018

ddcprg commented Nov 1, 2018

bcatubig commented Nov 5, 2018

t-lanigan commented Nov 5, 2018

mbfrahry commented Nov 8, 2018

mbfrahry commented Nov 8, 2018

ddcprg commented Nov 9, 2018 • edited Loading

mbfrahry commented Nov 27, 2018

rainmanh commented Nov 28, 2018

ddcprg commented Jan 4, 2019 • edited Loading

mbfrahry commented Jan 7, 2019

ddcprg commented Jan 8, 2019

bflad commented Jan 16, 2019

smrtslckr commented Jan 16, 2019

jckuester commented Jan 21, 2019

rainmanh commented Jan 22, 2019

bcatubig commented Jan 22, 2019 • edited Loading

saritajoshi9389 commented Feb 7, 2019

bflad commented Feb 8, 2019

jckuester commented Feb 15, 2019 • edited Loading

bcatubig commented Mar 10, 2019

jckuester commented Mar 19, 2019 • edited Loading

bflad commented Mar 29, 2019

gattil commented Apr 4, 2019

bflad commented Apr 5, 2019

rverma-nikiai commented May 13, 2019

tdmalone commented May 13, 2019

bcatubig commented May 13, 2019

LarsNeR commented Aug 1, 2019

bflad commented Aug 1, 2019

LarsNeR commented Aug 1, 2019

bflad commented Jan 29, 2020

ghost commented Mar 27, 2020

jckuester commented Nov 30, 2017 •

edited

Loading

ddcprg commented Dec 20, 2017 •

edited

Loading

ddcprg commented Jan 12, 2018 •

edited

Loading

jckuester commented Aug 21, 2018 •

edited

Loading

rainmanh commented Oct 5, 2018 •

edited

Loading

ddcprg commented Nov 9, 2018 •

edited

Loading

ddcprg commented Jan 4, 2019 •

edited

Loading

bcatubig commented Jan 22, 2019 •

edited

Loading

jckuester commented Feb 15, 2019 •

edited

Loading

jckuester commented Mar 19, 2019 •

edited

Loading