feat(conversation): per message items and lambda history retrieval pattern

[atierian]

Problem Statement

Within conversation routes, the AppSync resolver function invokes the Lambda asynchronously with a payload including the entire conversation history.

The maximum event payload for an asynchronously invoked lambda is 256KB (reference).

This limit can be hit rather quickly in either or both of these conditions:

1. Conversation is long enough.
2. Conversation contains media (base64 encoded images).

Description of changes

1. Lambda History Retrieval

AppSync resolver function invokes Lambda function asynchronously with a payload containing only the information necessary (no conversation history) for the Lambda function to retrieve the conversation history from AppSync in a subsequent request.

Current Sequence Diagram

New Sequence Diagram

Changes to Lambda Function Payload

{
  // ...
- messages: Array<ConversationMessage>;
+ messageHistoryQuery: { 
+  getQueryName: string;
+  getQueryInputTypeName: string;
+  listQueryName: string;
+  listQueryInputTypeName: string;
+  listQueryLimit?: number;
+ };
  // ...
};

A Note on Eventual Consistency / Read After Write

Reference: DynamoDB read consistency

The listConversationMessage<RouteName> query resolver uses a global secondary index (PK: conversationId, SK: createdAt) to retrieve messages in descending sorted order. GSIs are eventually consistent. This can, in theory, be problematic with this new read-after-write pattern introduced here. The fact that there are a few network hops (1. AppSync -> Lambda 2. Lambda <-> AppSync <-> DDB) between write and read reduce the likelihood of this being an issue. However, it's a real possibility and the end-user experience of missing the current user message is unacceptable.

The Lambda function has logic to handle this case — if the currentMessageId is not included in the list query response, the Lambda function subsequently invokes a getConversationMessage<RouteName> query with the id (ConversationMessages<RouteName> table partition key), which we make a strongly consistent read using the ConsistentRead parameter.

2. Item Level Messages

Current Situation

Conversation routes currently treat messages as user:assistant pairs internally (single DDB item), while exposing them through APIs (e.g. list query) as individual messages. This is bad for several reasons -- the following are the key motivators:

• when we write an assistant response to DDB, we mutate an existing item.
• it's already leading to gnarly resolver code to map the paired object into the DTO used by the GraphQL API. This is only going to get worse over time.
• DynamoDB has an item size limit of 400 KB (reference). We're much less likely to hit that with individual messages than with user:assistant tuples.

The motivation for this approach was to create a GSI that accommodates querying based on the needs of the system ([user, assistant, user, assistant, ...] descending createdAt ordering, discarding user messages without an assistant response).

Changes

1 message (user or assistant) --> 1 item in ConversationMessage<RouteName> DynamoDB table.

This is accomplished by:

1. ConversationMessage model now includes optional associatedUserMessageId field. This is only populated for assistant messages and follows the paradigm set by hasOne / belongsTo references based relationships.
2. The assistant response mutation no longer mutates an existing item. Instead it writes a new item to the ConversationMessage<RouteName> table. Because we can no longer use a conditional UpdateItem here, we have to use the same technique for owner auth used by the send message mutation. See the sequence diagrams below for more details:

EXISTING

NEW

CDK / CloudFormation Parameters Changed

None

Issue #, if available

Related amplify-backend PR

• Use message history instead of event payload for conversation handler amplify-backend#2047

Description of how you validated changes

• E2E CI run
• Manual testing

Checklist

• PR description included
• yarn test passes
• E2E test run linked
• Tests are changed or added
• Relevant documentation is changed or added (and PR referenced)
• New AWS SDK calls or CloudFormation actions have been added to relevant test and service IAM policies
• Any CDK or CloudFormation parameter changes are called out explicitly

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[atierian]

Including this in a follow-up PR that consolidates the pipeline resolver generation logic.

[palpatim]

I assume this means that you want to derive the values closer to the ConversationDirective implementation, not that you're going to allow customers to configure these values?

[atierian]

correct

[atierian]

Preemptively answering the question "isn't this breaking?"

It's not because this field was never exposed in the MIS. It was strictly an implementation detail used between resolver code and the conversation messages DynamoDB table.

The associatedUserMessageId will be exposed in the MIS (codegen PR pending) as a way to represent user - assistant relationships to consumers.

[palpatim]

OK, I'm calling it: we need a central method for template substitution. Please refactor.

[atierian]

It's in progress in the ai.conversation-consolidate-resolver-generation branch.

I needed prioritize these changes (this and the other PRs from this/last week) and am allergic to mixing functional and (non-trivial) non-functional changes in PRs. I also wanted to make sure whatever abstraction I use here will cover these changes. Thanks for your patience, I know I've tested it with these last few PRs 😅

[palpatim]

I assume this means that you want to derive the values closer to the ConversationDirective implementation, not that you're going to allow customers to configure these values?

[palpatim]

Do we care that the GSI name doesn't include the base type?

[atierian]

The generated type ConversationMessage<RouteName>? Nope, or at least I don't. It's fine that the GSIs across multiple routes (DDB tables) have the same name.