From ef729164033573d2ff487c803f6e1b3d63007cfb Mon Sep 17 00:00:00 2001
From: Lee Miller <lemiller@microsoft.com>
Date: Tue, 14 Mar 2023 15:58:05 -0700
Subject: [PATCH] Add HTTP retry logic and cancellation support for OpenAI
 services (#58)

### Motivation and Context

This set of code changes is required to improve the performance and
usability of the OpenAI completion backend by refactoring the kernel
retry mechanism and adding a new http retry delegating handler.
Specifically, it addresses the following problems:

1. The existing retry mechanism for kernel requests was using a custom
interface that made it difficult to customize and maintain the retry
logic. By refactoring it to use a delegating handler factory, we can
simplify the configuration of semantic skills and improve the
consistency and flexibility of the retry logic.

2. The existing retry handler did not take advantage of the Retry-After
value from the response headers to determine the backoff duration, which
can improve the efficiency and reliability of the requests. By adding a
new handler that does this, we can reduce the number of unnecessary
retries and improve the response times for the requests.

### Description

#### Kernel
- Added a new abstraction for creating HTTP retry handlers, the
`IDelegatingHandlerFactory` interface, which allows using the built-in
`HttpClient` retry logic and injecting custom retry policies.
- Added a new class `DefaultHttpRetryHandlerFactory` that implements a
configurable retry policy based on the `KernelConfig.HttpRetryConfig`
settings. This factory is used by default when creating a kernel, unless
a different factory is specified.
- Added a new class `NullHttpRetryHandlerFactory` that creates a no-op
retry handler, useful for testing or disabling retries.
- Added support for cancellation tokens and retry handlers in the OpenAI
services classes, such as `AzureTextCompletion`, `AzureTextEmbeddings`,
and `OpenAITextCompletion`. This allows the callers to cancel the
requests and handle transient failures more gracefully.
- Added unit tests for the `DefaultHttpRetryHandler` class, which is
responsible for handling HTTP retries. The tests cover various scenarios
such as retrying on different status codes, retrying on different
exception types, and respecting the retry configuration parameters.
- Removed the unused `IRetryMechanism` and `PassThroughWithoutRetry`
classes, and the `Example08_RetryMechanism.cs` file.
- Refactored the `KernelConfig` class to expose the `HttpRetryConfig`
and the `HttpHandlerFactory` properties.
- Refactored the `Kernel` class to use the `HttpHandlerFactory` instead
of the `RetryMechanism` for invoking the pipeline functions.
- Updated the `KernelBuilder` sample to show how to use a custom retry
handler factory.
- Updated the documentation and the code style accordingly.

#### Kernel-Syntax-Examples
This pull request adds two new classes to the Reliability namespace that
implement different retry policies for HTTP requests. The first class,
`RetryThreeTimesWithBackoff`, retries a request three times with an
exponential backoff if it encounters a 429 (Too Many Requests) or 401
(Unauthorized) status code. The second class,
`RetryThreeTimesWithRetryAfterBackoff`, also retries three times, but
uses the value of the Retry-After header to determine the backoff
duration. Both classes use Polly to implement the retry logic and log
the outcome of each attempt using the ILogger interface.

The pull request also modifies the `ConsoleLogger` class to filter out
log messages from the System namespace with a level lower than Warning.
This is to reduce the noise from the HttpClient and DelegatingHandler
classes.
This pull request introduces several improvements and features related
to the HTTP retry logic and the OpenAI services. The main changes are:
---
 .vscode/tasks.json | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/.vscode/tasks.json b/.vscode/tasks.json
index be95901abec2..c8b65d365445 100644
--- a/.vscode/tasks.json
+++ b/.vscode/tasks.json
@@ -74,10 +74,10 @@
       "detail": "Runs tasks to validate changes before checking in.",
       "group": "test",
       "dependsOn": [
+        "R# cleanup",
         "Build - Semantic-Kernel",
         "Test - Semantic-Kernel",
-        "Run - Kernel-Demo",
-        "R# cleanup"
+        "Run - Kernel-Demo"
       ],
       "dependsOrder": "sequence"
     },
@@ -103,7 +103,7 @@
       "label": "Test - Semantic-Kernel",
       "command": "dotnet",
       "type": "process",
-      "args": ["test", "SemanticKernel.Test.csproj"],
+      "args": ["test", "SemanticKernel.UnitTests.csproj"],
       "problemMatcher": "$msCompile",
       "group": "test",
       "presentation": {
@@ -112,7 +112,7 @@
         "group": "PR-Validate"
       },
       "options": {
-        "cwd": "${workspaceFolder}/dotnet/src/SemanticKernel.Test/"
+        "cwd": "${workspaceFolder}/dotnet/src/SemanticKernel.UnitTests/"
       }
     },
     {
@@ -123,7 +123,7 @@
         "test",
         "--collect",
         "XPlat Code Coverage;Format=lcov",
-        "SemanticKernel.Test.csproj"
+        "SemanticKernel.UnitTests.csproj"
       ],
       "problemMatcher": "$msCompile",
       "group": "test",
@@ -132,7 +132,7 @@
         "panel": "shared"
       },
       "options": {
-        "cwd": "${workspaceFolder}/dotnet/src/SemanticKernel.Test/"
+        "cwd": "${workspaceFolder}/dotnet/src/SemanticKernel.UnitTests/"
       }
     },
     {