Stack Mapping Fails (V2 Transformer Migration)

[sacrampton]

Before opening, please confirm:

• I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
• I have searched for duplicate or closed issues.
• I have read the guide for submitting bug reports.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

How did you install the Amplify CLI?

npm install -g @aws-amplify/cli

If applicable, what version of Node.js are you using?

v17.1.0

Amplify CLI Version

8.1.0

What operating system are you using?

AWS LINUX

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

Modified transform.conf.json

{
  "StackMapping": {
     "ItemclassificationResolver": "ConnectionStack2",
     "ItemlocationResolver": "ConnectionStack2",
     "ItemmaterialResolver": "ConnectionStack2",
     "ItemoperatingUnitResolver": "ConnectionStack2",
     "ItempackageResolver": "ConnectionStack2"
  }
}

Amplify Categories

api

Amplify Commands

push

Describe the bug

Trying to migrate existing database to the V2 transformer and the ConnectionStack.json exceeds 500 resources.

So modifying the transform.conf.json file to split into multiple ConnectionStacks.

I compiled the schema without any StackMapping first and extracted a list of 427 resources to put in the transform.conf.json file. I put them all in ConnectionStack2 as shown (partially below) - but when I compiled it comes back with the error that there are more than 500 resources in ConnectionStack2.

  "StackMapping":{
    "UomscaleResolver": "ConnectionStack2",
    "ScaleuomResolver": "ConnectionStack2",
    "ItemplantResolver": "ConnectionStack2",
    "PartbrandResolver": "ConnectionStack2",
    "PhotoitemResolver": "ConnectionStack2"
   }

So I spread the resources across 6 additional Connection Stacks and that compiled fine - but when I push I get the following error.

Circular dependency between resources: [CustomResourcesjson, ConnectionStack6, ConnectionStack2, ConnectionStack3, ConnectionStack4, ConnectionStack5]

The Custom Resources is attached - there are no custom resources.
CustomResources.zip

Alternately, if I then change up the StackMapping to get beyond the error above, namely by reducing the numbers of records in StackMapping or changing how they are mapped I can get beyond the above circular error - but then I'll get the following error saying it failed...

CREATE_FAILED AssetVisitseriesReferenceResolver AWS::AppSync::Resolver Tue May 10 2022 23:18:22 GMT+0000 (Coordinated Universal Time) Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: 8cf3287a-6301-4222-874c-6f42cfbcf62c; Proxy: null)

Expected behavior

Expect it to work as per V1 where I put any resource from the original ConnectionStack.json into the StackMapping table and that resource moves to the separate json file. This is not the case as evidenced by my addition of 423 resources exceeds the 500 limit in the ConnectionStack2.json

Reproduction steps

See above

GraphQL schema(s)

# Put schemas below this line

Log output

# Put your logs below this line

Additional information

No response

[sacrampton]

This issue aws-amplify/amplify-cli#9762 is very similar and lots of people in this thread were reporting the same issue.

aws-amplify/amplify-cli#9762 (comment)
aws-amplify/amplify-cli#9762 (comment)

[sacrampton]

Hi @edwardfoyle - you mentioned this in your comment aws-amplify/amplify-cli#9762 (comment) - but I've tried for quite some time and there seems to be no way to get this working.

In V1 if I put 200 resources in StackMapping to go to say ConnectionStack2 then there are exactly 200 resources placed in ConnectionStack2 - but V2 is doing something substantially different with StackMapping. For example above, I put 427 into StackMapping - but it generates more than 500 resources into ConnectionStack2.

Something seems to be really messed up with the way StackMapping is generating the stacks. For example, if you split the same resource up between different stacks you can get different failure results.

If you follow the instructions it just doesn't work.

Once again, this is another show stopper from moving to V2.

[sacrampton]

Here is an example - there are 100 entries in the StackMapping for ConnectionStack2 and the compile generates 227 resources. Its adding all the functions in the stack in addition to the records I entered.

[sacrampton]

Hi @josefaidt - just wanted to follow up on this one. The documentation (https://docs.amplify.aws/cli/graphql/override/#place-appsync-resolvers-in-custom-named-stacks) for this does not appear to be accurate.

[sacrampton]

Hi @alharris-at - any thoughts on this issue?

[sacrampton]

Hi @alharris-at @josefaidt - any thoughts on this issue over the last couple of weeks. This is a roadblock to be able to migrate a production (ie. large) database to the V2 Transformer.

[josefaidt]

Hey @sacrampton 👋 apologies for the delay here!! I've promoted this with the team to look into a bit deeper but this is something we are actively investigating! Sorry for not posting an update sooner

[josefaidt]

@sacrampton in your transform.conf.json file I noticed the StackMapping keys are not prefixed with the operation (<Get | List | Create | Update | Delete><model name>Resolver).

  "StackMapping":{
    "UomscaleResolver": "ConnectionStack2",
    "ScaleuomResolver": "ConnectionStack2",
    "ItemplantResolver": "ConnectionStack2",
    "PartbrandResolver": "ConnectionStack2",
    "PhotoitemResolver": "ConnectionStack2"
   }

Are these the relational resolvers you're attempting to extract to their own stack?

Looking at a sample schema where we have a Todo model, if I specify the following in my transform.conf.json file:

{
  "Version": 5,
  "ElasticsearchWarning": true,
  "transformers": [],
  "StackMapping": {
    "CreateTodoResolver": "MyCustomStack"
  }
}

And run amplify api gql-compile, we see the Create* resolver is split out of build/stacks/Todo.json and into build/stacks/MyCustomStack.json

[sacrampton]

Hi @josefaidt

I've included the full transform.conf.json with this link below.

transform.conf.zip

Using "amplify api gql-compile --minify" the GraphQL Schema successfully compiled.

But when I do the push "amplify push --no-gql-override --yes" it fails with this error.

UPDATE_FAILED               apidatastore2                AWS::CloudFormation::Stack Wed Jun 01 2022 04:19:19 GMT+0000 (Coordinated Universal Time) Circular dependency between resources: [Company, EquipmentReferenceAttribute, SearchableStack, Photo, Document, GlobalPickList, Item, Classification, ProcessingQueue, Asset, M2MConnection, Bom, BrandSynonym, Messages, CustomResourcesjson, CrowdQueue, Brand, QuarantineArea, Part, ExternalCharacteristicValue, Package, OrgRole, CrowdResults, ProcessingQueueErrors, Group, User, ConnectionStack2, ConnectionStack3, OperatingUnit, ConnectionStack4, ConnectionStack5, Risk, EquipmentReference, FailureMode, FieldList, PotentialResult, FieldSet, Plant, ClientPickList, Defect, Division, PartsList, Component, AssetVisit, Submission, DefectMedia, Material, Scale, AssetVisitAttribute, OperatingGroup, CrowdSourcingJob, Series, Uom, CrowdWorker, ExternalAsset, CharacteristicValue, PhotoButton, Location]

[josefaidt]

Hey @sacrampton apologies for the delay here, however I was able to successfully reproduce this issue after attempting to split off my CreateTodoResolver as shown in my previous reply. Marking this as a bug 🙂

[sacrampton]

Hi @josefaidt - glad you were able to reproduce.

[alharris-at]

I looked into this today, focusing primarily on the Only one resolver is allowed issue first. I'm not sure if this is a miss on documentation, but I'm not sure how reassigning a resolver across stacks should work generally. Reviewing the Cloudformation docs there's no mention or guarantee of atomicity across child/peer stacks, so the behavior you're seeing makes sense to me, given the constraints of the system.

When we migrate an already-deployed resource to another stack, at the point of deployment one of two things is likely to happen (based mostly on timing between which stack deploys when). Either 1) the stack which we've removed the resolver from will attempt to remove the resolver first, or the 2) the stack which we've added the resolver to will attempt to deploy the resolver first. In the case of the former, I think this may succeed, though for a short time there will be no resolver connected. This is not desirable. In the latter case, we are attempting to attach a duplicate resolver to the field/pipeline, which also fails with a service error from Appsync like Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: ***; Proxy: ***). This is also not desirable.

Having read through the Cfn docs a bit more, there are certain resource types which can be refactored between stacks, though unfortunately AppSync resolvers on not on that list.

Since that appears to not be an option, I can think of two paths forward, depending on the lifecycle of the project. I've confirmed that using stack overrides works with a newly-deployed backend. In this case fully deleting the backend and re-creating may be an option, though obviously not for existing systems.

For systems that are already running in production, and where we need to migrate resources between stacks, we may be able to temporarily move the Resolvers out of management from cloudformation (by adding a Resource deletion policy, so they remain after being removed), and then manually importing those resources back into the new stack. This would likely need to be done outside the context of the Amplify CLI, and we'll need to draft up a process to do so safely.

I'll also kick off a thread with the AppSync and cloudformation teams to see if they have any insight into less invasive ways to effect the desired outcome here.

[sacrampton]

Thanks @alharris-at - I confirm we have a large production system that we need to migrate.

[sacrampton]

Hi @alharris-at - just checking back in on this issue. At the moment it is impossible to migrate a large production system to the V2 resolvers.

[alharris-at]

Hi Sacrampton, I apologize for the delayed response. Unfortunately we don't yet have a path forward regarding these stack overrides, due to the CFN limitations detailed above.

[sacrampton]

Hi @alharris-at - appreciate the update on status.

[oe-bayram]

We are facing the same problem.

Have there been any updates or solutions since the last conversation?
We are currently at a standstill.

[alharris-at]

Just to update here, I've spent a bit of time with the AppSync team to investigate alternative ways to resolve the duplicate resolver on field XX type error with no clear path forward on the service-side. I'm now thinking about ways we can mitigate this in user-space (less ideal) but one such approach would be ensuring we have support in Amplify for StackMapping functions (doesn't seem to be supported today), then vending a few more utils that can be used to manage StackMappings more safely.

In my mind for customers migrate to GQLv2, we can snap the state of your current app pre-migration, and place all functions into new stacks (e.g. { api-root-stack } -> { model1-stack, model2-stack, connections-stack1, connections-stack2, searchable-stack, etc. } -> { fn-stack1, fn-stack2, etc. } to maintain a one-way dependency flow here.

On Friday evening I teased at this a little bit in the linked PR, but this will need a bit more thinking to ensure we're building reusable tools that can help mitigate the different cases on this, and also to ensure there are no blockers to supporting stack mappings for functions (I don't see any technical reason for this, but it doesn't seem to have been built when the initial GQLv2 implementation was written).

Q for @sacrampton Would having a utility like 'assign-new-resources-of-type-to-stack --resource-type --stack-name ' mapping help get you unblocked in the V1->V2 migration flow?

[DougalW]

@sacrampton - did you ever get this working? I'm having very similar problems with an Amplify app with 55 dynamoDB tables, and even backing this down to 52 tables it still won't deploy successfully.

Currently a showstopper for us.

Hi @josefaidt

I've included the full transform.conf.json with this link below.

transform.conf.zip

Using "amplify api gql-compile --minify" the GraphQL Schema successfully compiled.

But when I do the push "amplify push --no-gql-override --yes" it fails with this error.

UPDATE_FAILED               apidatastore2                AWS::CloudFormation::Stack Wed Jun 01 2022 04:19:19 GMT+0000 (Coordinated Universal Time) Circular dependency between resources: [Company, EquipmentReferenceAttribute, SearchableStack, Photo, Document, GlobalPickList, Item, Classification, ProcessingQueue, Asset, M2MConnection, Bom, BrandSynonym, Messages, CustomResourcesjson, CrowdQueue, Brand, QuarantineArea, Part, ExternalCharacteristicValue, Package, OrgRole, CrowdResults, ProcessingQueueErrors, Group, User, ConnectionStack2, ConnectionStack3, OperatingUnit, ConnectionStack4, ConnectionStack5, Risk, EquipmentReference, FailureMode, FieldList, PotentialResult, FieldSet, Plant, ClientPickList, Defect, Division, PartsList, Component, AssetVisit, Submission, DefectMedia, Material, Scale, AssetVisitAttribute, OperatingGroup, CrowdSourcingJob, Series, Uom, CrowdWorker, ExternalAsset, CharacteristicValue, PhotoButton, Location]

[oe-bayram]

I was able to circumvent this issue by first completely removing the SearchableStack and then migrating to the V2 Transformer.

With over 90 models (DynamoDB tables), I could first deploy the ConnectionStack with 348 resources and then re-added the SearchableStack with 225 resources.

I believe that if the ConnectionStack and the SearchableStack are migrated together, the resource count during deployment is probably aggregated, thus reaching the 500 mark. However, this error does not occur when deploying them separately.

If one of the stacks has more than 500 resources, I would then recommend incrementally adding resources and using the Logical IDs from CloudFormation to add them to custom stacks.