Load Testing of Passport Application and API

[erichfi]

User Story:

As a developer,
I want to conduct thorough load testing on both the Passport application and its APIs,
so that I can verify the system's ability to handle increased traffic and ensure a seamless user experience under various load conditions.

Acceptance Criteria

GIVEN the Passport application or API has been updated or is expected to face increased traffic,
WHEN I perform load testing,
THEN the system should perform without significant slowdowns or failures,
AND any performance bottlenecks or scalability issues should be identified and remedied.

Product & Design Links:

Tech Details:

• Focus Areas:
  • Conduct tests on both the Passport application's user interface and backend APIs to assess the entire system's performance.
  • Test under scenarios mimicking real user behaviors and high-traffic conditions to identify performance limits.
• Performance Metrics:
  • Monitor critical metrics such as response time, throughput, error rates, and system resource utilization during the tests.
  • Evaluate the system's scalability by gradually increasing the load until the performance threshold is reached.
• Infrastructure and Setup:
  • Ensure the test environment closely replicates the production setting to gather accurate insights.
  • Use automated scripts to simulate load and stress on the system.
• Collaboration and Reporting:
  • Work closely with the infrastructure and operations teams to prepare and review test setups.
  • Document test results and share findings with all relevant stakeholders to guide further development and optimization efforts.

Open Questions:

• What are the key performance indicators that will define success for the load tests?
• How will we address identified bottlenecks, and what are the escalation processes for critical performance issues?

Notes/Assumptions:

• Assume that the existing infrastructure can support initial load testing scenarios without additional investment.
• Results from the load testing will inform decisions on necessary infrastructure enhancements or application optimizations.

[tim-schultz]

• Note for self, try to test for database issue from Q3 last year

[tim-schultz]

During testing the support banner endpoint experienced issues under high load. @erichfi/ @jesuloluwao do you all still use the banner feature or do you prefer using the intercom banner? We could deprecate the banner if intercom is the preferred solution.

I added a rundown of the test results and suggestions here: https://docs.google.com/document/d/1ps7jYYBa-xFAeTFt32-D2-s526oon0mxkJlnzrj2k74/edit

Just waiting on reviewing them with the team

[tim-schultz]

Here are my suggestions based on load testing.

• Move /nonce endpoint over to a lambda
• Move /banner endpoint to a lambda
• Move /check endpoint to a lambda. The request continues to return 504 errors due to server load