Custom access logging of API gateways is disabled

[dsotirho-ucsc]

"Custom Access Logging" of API gateways has been disabled for anvildev and dev, but not prod. Looking at the Cloudwatch logs it appears this happened around 12/02/2022 06:14 GMT. This time roughly correlates with the merge & deploy of PR #4770 for issues #4752, #3652, and #2992 .

Cloudwatch logs for /aws/apigateway/azul-indexer-anvildev:

anvildev API gateway settings:

Cloudwatch logs for /aws/apigateway/azul-service-dev:

Note: There are logs on the graph for the other days before 12/02 however they are being overshadowed by the large spike of 73,192 matches.

---

• Security design review completed; the Resolution of this issue does not …
  • … affect authentication; for example:
    • OAuth 2.0 with the application (API or Swagger UI)
    • Authentication of developers with Google Cloud APIs
    • Authentication of developers with AWS APIs
    • Authentication with a GitLab instance in the system
    • Password and 2FA authentication with GitHub
    • API access token authentication with GitHub
    • Authentication with
  • … affect the permissions of internal users like access to
    • Cloud resources on AWS and GCP
    • GitLab repositories, projects and groups, administration
    • an EC2 instance via SSH
    • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • … affect the permissions of external users like access to
    • TDR snapshots
  • … affect permissions of service or bot accounts
    • Cloud resources on AWS and GCP
  • … affect audit logging in the system, like
    • adding, removing or changing a log message that represents an auditable event
    • changing the routing of log messages through the system
  • … affect monitoring of the system
  • … introduce a new software dependency like
    • Python packages on PYPI
    • Command-line utilities
    • Docker images
    • Terraform providers
  • … add an interface that exposes sensitive or confidential data at the security boundary
  • … affect the encryption of data at rest
  • … require persistence of sensitive or confidential data that might require encryption at rest
  • … require unencrypted transmission of data within the security boundary
  • … affect the network security layer; for example by
    • modifying, adding or removing firewall rules
    • modifying, adding or removing security groups
    • changing or adding a port a service, proxy or load balancer listens on
• Documentation on any unchecked boxes is provided in comments below

[theathorn]

Spike to diagnose.

[hannes-ucsc]

When PR #4770 introduced the explicit stage, the dependency of the API Gateway deployment on the custom provisioner script became problematic. Either the script wasn't invoked or it was invoked too early and its effect was undone by TF during the creation of the explicit stage. Now that we manage the stage with Terraform, we can apply the log settings directly through Terraform, which is actually what #3412 is about.

[hannes-ucsc]

Freebie of #3412.

[hannes-ucsc]

For demo, attempt to reproduce.

[hannes-ucsc]

Because the regression and the fix were promoted to prod simultaneously, prod logging was never affected by this issue. I just checked /aws/apigateway/azul-indexer-prod and /aws/apigateway/azul-service-prod and they had a continuous series of log messages.