Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HttpStress tests failing in difference scenarios on sending data to server #40388

Closed
alnikola opened this issue Aug 5, 2020 · 8 comments
Closed

HttpStress tests failing in difference scenarios on sending data to server #40388

alnikola opened this issue Aug 5, 2020 · 8 comments
Labels
area-System.Net.Http bug tenet-reliability Reliability/stability related issue (stress, load problems, etc.)
Milestone
6.0.0

Comments

@alnikola
Copy link
Contributor

alnikola commented Aug 5, 2020 •
edited by ManickaP
Loading

Updated (28.8.2020) based on current pipeline. Collected runs:

Error Windows Count Linux Count
System.Net.Http.HttpRequestException: An error occurred while sending the request.
System.IO.IOException: The request was aborted.
System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.		 0 2
System.Threading.Tasks.TaskCanceledException: A task was canceled. 0 1
System.Net.Http.HttpRequestException: An error occurred while sending the request.
System.IO.IOException: The request was aborted.
System.Net.NetworkException: The connection was forcibly closed by the remote host.
System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.		 1 0
System.Net.Http.HttpRequestException: An error occurred while sending the request.
System.IO.IOException: The request was aborted.
System.Net.NetworkException: A network error has occured, see InnerException for more details.
System.Net.Sockets.SocketException (32): Broken pipe		 0 2
System.Net.Http.HttpRequestException: Error while copying content to a stream.
System.IO.IOException: The request was aborted.
System.Net.NetworkException: A network error has occured, see InnerException for more details.
System.Net.Sockets.SocketException (32): Broken pipe		 0 2
System.Net.Http.HttpRequestException: An error occurred while sending the request.
System.IO.IOException: The request was aborted.
System.Net.NetworkException: The connection was forcibly closed by the remote host.
System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.		 6 0

Total | 7 | 7

System.Net.Http.HttpRequestException: An error occurred while sending the request.:

PUT Slow 
System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The request was aborted.
---> System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.
  at System.Net.Http.Http2Connection.<ReadFrameAsync>g__ThrowMissingFrame|49_1() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 286
  at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 240
  at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 340
  --- End of inner exception stack trace ---
  at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 2008
  at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 814
  at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 843
  at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 889
  at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1906
  --- End of inner exception stack trace ---
  at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1906
  at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 898
  at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
  at System.Net.Http.HttpClient.FinishSendAsync(ValueTask`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, Boolean buffered, Boolean async, CancellationToken callerToken, Int64 timeoutTime) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 605
  at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /app/ClientOperations.cs:line 93
  at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_13>d.MoveNext() in /app/ClientOperations.cs:line 464
--- End of stack trace from previous location ---
  at HttpStress.StressClient.<>c__DisplayClass15_0.<<StartCore>g__RunWorker|2>d.MoveNext() in /app/StressClient.cs:line 164

System.Threading.Tasks.TaskCanceledException: A task was canceled.:

PUT Slow 
System.Threading.Tasks.TaskCanceledException: A task was canceled.
  at System.Net.Http.Http2Connection.SendStreamDataAsync(Int32 streamId, ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1455
  at System.Net.Http.Http2Connection.Http2Stream.SendDataAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 1131
  at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in /app/ClientOperations.cs:line 620
  at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 286
  at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1906
  at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 898
  at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
  at System.Net.Http.HttpClient.FinishSendAsync(ValueTask`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, Boolean buffered, Boolean async, CancellationToken callerToken, Int64 timeoutTime) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 605
  at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /app/ClientOperations.cs:line 93
  at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_13>d.MoveNext() in /app/ClientOperations.cs:line 464
client_1  | --- End of stack trace from previous location ---
  at HttpStress.StressClient.<>c__DisplayClass15_0.<<StartCore>g__RunWorker|2>d.MoveNext() in /app/StressClient.cs:line 164

System.Net.Http.HttpRequestException: An error occurred while sending the request.:

PUT Slow 
System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The request was aborted.
---> System.Net.NetworkException: The connection was forcibly closed by the remote host.
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
  --- End of inner exception stack trace ---
  at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 917
  at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 883
  at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 806
  at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 327
  --- End of inner exception stack trace ---
  at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 2008
  at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 814
  at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 843
  --- End of stack trace from previous location ---
  at HttpStress.StressClient.<>c__DisplayClass16_0.<<StartCore>g__RunWorker|2>d.MoveNext() in C:\app\StressClient.cs:line 178

System.Net.Http.HttpRequestException: An error occurred while sending the request.:

PUT Slow + POST Multipart Data 
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The request was aborted.
 ---> System.Net.NetworkException: A network error has occured, see InnerException for more details.
 ---> System.Net.Sockets.SocketException (32): Broken pipe
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 926
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendAsyncForNetworkStream(Socket socket, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 727
   at System.Net.Sockets.Socket.SendAsyncForNetworkStream(ReadOnlyMemory`1 buffer, SocketFlags socketFlags, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 299
   at System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs:line 720
   at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 672
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 1012
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 42
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 807
   at System.Net.Http.Http2Connection.FlushOutgoingBytesAsync() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 213
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 42
   at System.Net.Http.Http2Connection.FlushOutgoingBytesAsync()
   at System.Net.Http.Http2Connection.ProcessOutgoingFramesAsync() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 967
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 186
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 327
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs:line 640
--- End of stack trace from previous location ---

   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|177_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 724
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 1030
   at System.Net.Http.Http2Connection.FlushOutgoingBytesAsync() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 223
   at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in /app/ClientOperations.cs:line 621
   at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 286
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 2017
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 814
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 843
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 889
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1915
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1915
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 898
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.FinishSendAsync(ValueTask`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, Boolean buffered, Boolean async, CancellationToken callerToken, Int64 timeoutTime) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 605
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /app/ClientOperations.cs:line 93
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_6>d.MoveNext() in /app/ClientOperations.cs:line 349
--- End of stack trace from previous location ---
   at HttpStress.StressClient.<>c__DisplayClass16_0.<<StartCore>g__RunWorker|2>d.MoveNext() in /app/StressClient.cs:line 178

System.Net.Http.HttpRequestException: Error while copying content to a stream.:

POST Duplex Slow
Probably the same problem as the previous one. 
System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.IO.IOException: The request was aborted.
 ---> System.Net.NetworkException: A network error has occured, see InnerException for more details.
 ---> System.Net.Sockets.SocketException (32): Broken pipe
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 926
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendAsyncForNetworkStream(Socket socket, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 727
   at System.Net.Sockets.Socket.SendAsyncForNetworkStream(ReadOnlyMemory`1 buffer, SocketFlags socketFlags, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 299
   at System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs:line 720
   at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 672
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 1012
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 42
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 807
   at System.Net.Http.Http2Connection.FlushOutgoingBytesAsync() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 213
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 42
   at System.Net.Http.Http2Connection.FlushOutgoingBytesAsync()
   at System.Net.Http.Http2Connection.ProcessOutgoingFramesAsync() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 967
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 186
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 327
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs:line 640
--- End of stack trace from previous location ---

   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|177_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 724
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 1030
   at System.Net.Http.Http2Connection.FlushOutgoingBytesAsync() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 223
   at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in /app/ClientOperations.cs:line 621
   at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 286
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 2017
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 814
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 945
   at System.Net.Http.Http2Connection.Http2Stream.CopyToAsync(HttpResponseMessage responseMessage, Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 66
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 556
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 556
   at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 847
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in /app/ClientOperations.cs:line 390
--- End of stack trace from previous location ---
   at HttpStress.StressClient.<>c__DisplayClass16_0.<<StartCore>g__RunWorker|2>d.MoveNext() in /app/StressClient.cs:line 178

System.Net.Http.HttpRequestException: An error occurred while sending the request.

https://dev.azure.com/dnceng/public/_build/results?buildId=794179&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=e0810b27-de1e-5f7f-0df7-1a1042e925c2
https://dev.azure.com/dnceng/public/_build/results?buildId=795432&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=e0810b27-de1e-5f7f-0df7-1a1042e925c2
https://dev.azure.com/dnceng/public/_build/results?buildId=794179&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=e0810b27-de1e-5f7f-0df7-1a1042e925c2
https://dev.azure.com/dnceng/public/_build/results?buildId=792358&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=e0810b27-de1e-5f7f-0df7-1a1042e925c2
https://dev.azure.com/dnceng/public/_build/results?buildId=792168&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=e0810b27-de1e-5f7f-0df7-1a1042e925c2

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.NetworkException: The connection was forcibly closed by the remote host.
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 926
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendAsyncForNetworkStream(Socket socket, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 727
   at System.Net.Sockets.Socket.SendAsyncForNetworkStream(ReadOnlyMemory`1 buffer, SocketFlags socketFlags, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 299
   at System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs:line 720
   at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 672
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 1012
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 42
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 807
   at System.Net.Http.Http2Connection.FlushOutgoingBytesAsync() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 212
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 42
   at System.Net.Http.Http2Connection.FlushOutgoingBytesAsync()
   at System.Net.Http.Http2Connection.ProcessOutgoingFramesAsync() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 966
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 186
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 327
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs:line 640
--- End of stack trace from previous location ---

   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|177_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 724
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 1030
   at System.Net.Http.Http2Connection.FlushOutgoingBytesAsync() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 222
   at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in C:\app\ClientOperations.cs:line 621
   at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 286
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1906
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1906
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 898
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.FinishSendAsync(ValueTask`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, Boolean buffered, Boolean async, CancellationToken callerToken, Int64 timeoutTime) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 605
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in C:\app\ClientOperations.cs:line 93
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_13>d.MoveNext() in C:\app\ClientOperations.cs:line 464
--- End of stack trace from previous location ---
   at HttpStress.StressClient.<>c__DisplayClass16_0.<<StartCore>g__RunWorker|2>d.MoveNext() in C:\app\StressClient.cs:line 178
@alnikola alnikola added this to the 6.0.0 milestone Aug 5, 2020
@ghost
Copy link

ghost commented Aug 5, 2020

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Aug 5, 2020
@geoffkizer
Copy link
Contributor

Do we have past data here to compare with? Would be good to have a sense of when these were introduced.

@alnikola
Copy link
Contributor Author

alnikola commented Aug 6, 2020

Unfortunately, we don't have historic data because the pipelines were broken for several months, and as soon as I fixed them those errors appeared.

@geoffkizer
Copy link
Contributor

Thanks.

We should investigate all of this.

The "The WriteAsync method cannot be called when another write operation is pending" errors concern me, but may be a test issue? Unclear.

The TaskCanceledException stuff is unclear to me. Generally we shouldn't throw this unless the task was actually canceled (maybe a timeout?). But it also could be that stuff is incorrectly timing out.

@marklio marklio added the tenet-reliability Reliability/stability related issue (stress, load problems, etc.) label Aug 6, 2020
@karelz karelz modified the milestones: 6.0.0, 5.0.0 Aug 11, 2020
@karelz karelz added bug and removed untriaged New issue has not been triaged by the area owner labels Aug 11, 2020
@alnikola alnikola mentioned this issue Aug 13, 2020
Closed
@ManickaP
Copy link
Member

The NotSupportedException: The WriteAsync method cannot be called when another write operation is pending. in Http2Connection.FlushOutgoingBytesAsync is not happening anymore.
I have reproduced it on 5.0.0-preview.8.20361.2 and it disappeared when ran on the master. So it must have been fixed sometimes in between. I suspect #39166 PR as a fix.

@alnikola
Copy link
Contributor Author

Good news!

This was referenced Aug 28, 2020
Merged
Merged
@karelz karelz mentioned this issue Sep 14, 2020
Merged
@ManickaP ManickaP mentioned this issue Sep 14, 2020
Open
@karelz
Copy link
Member

karelz commented Sep 21, 2020

Triage: Remaining issues don't seem to be critical for 5.0, moving to 6.0.

@karelz karelz modified the milestones: 5.0.0, 6.0.0 Sep 21, 2020
@karelz
Copy link
Member

karelz commented Sep 21, 2020

Duplicate of #42211

@karelz karelz marked this as a duplicate of #42211 Sep 21, 2020
@karelz karelz closed this as completed Sep 21, 2020
@ghost ghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Net.Http bug tenet-reliability Reliability/stability related issue (stress, load problems, etc.)
Projects
None yet
Milestone
6.0.0
Development

No branches or pull requests
6 participants
@marklio @karelz @ManickaP @geoffkizer @Dotnet-GitSync-Bot @alnikola