Instruction to setup on AWS is not clear

[Jeffwan]

I can not successfully finish the aws setup. I use CloudFormation Template here https:/Netflix/metaflow-tools/tree/master/aws/cloudformation and it gives me all the resources I need to use here.

When I do metaflow configure aws, As I understand, I need to put output resources arns there. But I notice

1. It's hard for me to map resources to configurations. For example,

• Please enter the job queue to use for batch: -> Queue name or ARN?

• Please enter the IAM role to use for the container to get AWS S3 access -> Is it ECSJobRole from CF outputs?

• Please enter the URL for your metadata service: -> Is it the ServiceUrl? There's another one InternalServiceUrl.

2. Some configuration we can not find in the CloudFormation outputs

• Please enter the default container image to use -> Can not find any instruction here. I assume we at least need to have python env for the base container image?

• Please enter the container registry -> should be <account_id>.dkr.ecr.us-west-2.amazonaws.com

I would suggest to improve the doc here.
https://docs.metaflow.org/metaflow-on-aws/deploy-to-aws

[savingoyal]

@Jeffwan - Thanks for trying out Metaflow!

1. Mapping resources to configurations -

• You can use either the queue name or the ARN
• Yes, indeed it is the ECSJobRole
• Please use ServiceUrl

2. Image configuration - these were intended to be optional. Our next release will fix them

• Yes, indeed. For now (till we release 2.0.1), please use python:latest image
• You can use docker.io

These issues will be fixed promptly in the next release.
cc @seeravikiran

[crk-codaio]

@Jeffwan Thanks indeed for the thoughtful feedback.

[anishpdalal]

I really like the design and feel of Metaflow so far! I'd like to get it working with AWS, and I'm also having trouble with configuration as well.

I exported the environment variable:

export METAFLOW_DATASTORE_SYSROOT_S3=metaflow-tutorial-metaflows3bucket-85sqqazcntjh

but when I run python 05-helloaws/helloaws.py run it appears that the bucket name doesn't register and I get the following message:

I made sure that export METAFLOW_DEFAULT_METADATA=service and export METAFLOW_DEFAULT_DATASTORE=s3 were ran.

Also, I ran metaflow configure aws and it generated the config.json file. But when I run python 05-helloaws/helloaws.py run I got a message saying that METAFLOW_SERVICE_URL wasn't defined which leads me to believe the config file isn't getting loaded. Do I need to specify a path to the config file when I call run?

Thanks for your help!

[savingoyal]

@anishpdalal Yes, there is a bug report for this. It should be fixed in the next release, which is scheduled in the next couple of days. Can you prepend s3:// to METAFLOW_DATASTORE_SYSROOT_S3?

[anishpdalal]

Yes that worked thanks!

[Jeffwan]

@anishpdalal Yeah, you got it. The instruction needs to be improved. Same on the code side, for this case, probably check if there's no protocols, use s3:// by default.

[savingoyal]

There is a new README accompanying the cfn template now, as well as the metaflow configure aws command has been revamped in v2.0.1.

[Jeffwan]

Great work! Thanks