Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++][FlightSQL] Add session management primitives #34865

Closed
indigophox opened this issue Apr 3, 2023 · 1 comment · Fixed by #34817
Closed

[C++][FlightSQL] Add session management primitives #34865

indigophox opened this issue Apr 3, 2023 · 1 comment · Fixed by #34817

Comments

@indigophox
Copy link
Contributor

indigophox commented Apr 3, 2023

Describe the enhancement requested

In order to support Flight SQL server session persistence within a server application, we are introducing DoAction messages to support handling setting and checking session parameters, as well as closing sessions (housekeeping).

Additionally, a ServerSessionMiddleware has been provided as an example or utility implementation of handling session persistence via an opaque cookie.

(Note that updates to Location have been removed as it was impossible to design them to be consistent and elegant, so clients (and *DBC drivers) will simply set the desired session options independent of a Location URI.)

See also

The discussion on [email protected]: DISCUSS: [FlightSQL] Catalog support: https://lists.apache.org/thread/fd6r1n7vt91sg2c7fr35wcrsqz6x4645

Component(s)

C++, FlightRPC

@indigophox indigophox changed the title Add Session management primitives and Location path parsing Add Flight SQL Session management primitives Jun 28, 2023
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Aug 18, 2023
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Sep 19, 2023
@kou kou changed the title Add Flight SQL Session management primitives [C++][FlightSQL] Add session management primitives Oct 25, 2023
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 19, 2024
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 19, 2024
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 22, 2024
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 22, 2024
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 23, 2024
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 23, 2024
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 23, 2024
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 24, 2024
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 24, 2024
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 24, 2024
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 26, 2024
lidavidm pushed a commit that referenced this issue Feb 20, 2024
…4817)

### Rationale for this change

Flight presently contains no formal mechanism for managing connection/query configuration options; instead, request headers and/or non-query SQL statements are often used in lieu, with unnecessary overhead and poor failure handling.  A stateless (from Flight's perspective) Flight format extension is desirable to close this gap for server implementations that use/want connection state/context.

### What changes are included in this PR?

"Session" set/get/close Actions and server-side helper middleware.

### Are these changes tested?

Integration tests (C++ currently broken due to middleware-related framework issue) and some complex-case unit testing are included.

### Are there any user-facing changes?

Non-breaking extensions to wire format and corresponding client/server Flight RPC API extensions.

* Closes: #34865

Lead-authored-by: Paul Nienaber <[email protected]>
Co-authored-by: Paul Nienaber <[email protected]>
Co-authored-by: James Duong <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: David Li <[email protected]>
@lidavidm lidavidm added this to the 16.0.0 milestone Feb 20, 2024
@lidavidm
Copy link
Member

lidavidm commented Feb 20, 2024

#40155 to add this to Golang; apache/arrow-adbc#1557 to add this to the client.

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Feb 20, 2024
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Feb 21, 2024
stevelorddremio pushed a commit to stevelorddremio/arrow that referenced this issue Feb 26, 2024
…es (apache#34817)

### Rationale for this change

Flight presently contains no formal mechanism for managing connection/query configuration options; instead, request headers and/or non-query SQL statements are often used in lieu, with unnecessary overhead and poor failure handling.  A stateless (from Flight's perspective) Flight format extension is desirable to close this gap for server implementations that use/want connection state/context.

### What changes are included in this PR?

"Session" set/get/close Actions and server-side helper middleware.

### Are these changes tested?

Integration tests (C++ currently broken due to middleware-related framework issue) and some complex-case unit testing are included.

### Are there any user-facing changes?

Non-breaking extensions to wire format and corresponding client/server Flight RPC API extensions.

* Closes: apache#34865

Lead-authored-by: Paul Nienaber <[email protected]>
Co-authored-by: Paul Nienaber <[email protected]>
Co-authored-by: James Duong <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: David Li <[email protected]>
# Conflicts:
#	java/flight/flight-integration-tests/src/main/java/org/apache/arrow/flight/integration/tests/Scenarios.java
#	java/flight/flight-integration-tests/src/test/java/org/apache/arrow/flight/integration/tests/IntegrationTest.java
#	testing
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Feb 26, 2024
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Feb 26, 2024
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Feb 26, 2024
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Feb 26, 2024
zanmato1984 pushed a commit to zanmato1984/arrow that referenced this issue Feb 28, 2024
…es (apache#34817)

### Rationale for this change

Flight presently contains no formal mechanism for managing connection/query configuration options; instead, request headers and/or non-query SQL statements are often used in lieu, with unnecessary overhead and poor failure handling.  A stateless (from Flight's perspective) Flight format extension is desirable to close this gap for server implementations that use/want connection state/context.

### What changes are included in this PR?

"Session" set/get/close Actions and server-side helper middleware.

### Are these changes tested?

Integration tests (C++ currently broken due to middleware-related framework issue) and some complex-case unit testing are included.

### Are there any user-facing changes?

Non-breaking extensions to wire format and corresponding client/server Flight RPC API extensions.

* Closes: apache#34865

Lead-authored-by: Paul Nienaber <[email protected]>
Co-authored-by: Paul Nienaber <[email protected]>
Co-authored-by: James Duong <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: David Li <[email protected]>
zeroshade pushed a commit that referenced this issue Mar 1, 2024
…0284)

### Rationale for this change

Brings Go implementation in parity with recent session management additions to Java and C++: #34865

### What changes are included in this PR?

- Go Flight/FlightSQL implementations of session management RPC handlers
- Implementation of cookie-based session middleware
  - Implementation of stateful (id-lookup based) sessions/tokens
  - Implementation of stateless (fully encoded) sessions/tokens
- Fix minor C++ logic bug when closing sessions
- Update Java integration test server to return an empty session if `getSessionOptions` is called before `setSessionOptions`
- Refactor of `DoAction` handlers to consolidate the code that is essentially copied between them.
  - As part of this I found an issue with `CancelFlightInfo` where a copy of the message was being returned instead of a pointer as is typically the case with `proto.Message`'s. I updated the return type and any usage throughout the code base as part of the refactor.

### Are these changes tested?

Yes, both integration and unit tests are included.

A few tests were added in the Go integration suite beyond the existing coverage in the Java/C++ suites. These tests aim to demonstrate my understanding of session semantics in those scenarios, please let me know if you believe the details are not accurate.

Some of the new integration tests failed in the Java/C++ scenarios. I made very minor changes to those implementations to fix certain failures but there are still some remaining bugs (assuming these are testing the right semantics). Specifically:
- The integration test for reopening a previously closed session passes on Go/Java, but fails for C++ so it is commented out.
- This implementation prefers to set any cookies in the gRPC trailer which works fine for Go/C++, but not for Java. As a temporary workaround this implementation will _also_ set the cookie in the gRPC header if a new session was created. This is sufficient to maintain compatibility with Java stateful sessions where the session ID token can be known at the time of creation, but is not robust to other scenarios such as stateless sessions where in many cases the token cannot be known until after the RPC has completed.

### Are there any user-facing changes?
Yes, session management RPC as well as middleware implementations are included. Functionality is entirely additive

* GitHub Issue: #40155

Authored-by: joel <[email protected]>
Signed-off-by: Matt Topol <[email protected]>
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Mar 8, 2024
thisisnic pushed a commit to thisisnic/arrow that referenced this issue Mar 8, 2024
…es (apache#34817)

### Rationale for this change

Flight presently contains no formal mechanism for managing connection/query configuration options; instead, request headers and/or non-query SQL statements are often used in lieu, with unnecessary overhead and poor failure handling.  A stateless (from Flight's perspective) Flight format extension is desirable to close this gap for server implementations that use/want connection state/context.

### What changes are included in this PR?

"Session" set/get/close Actions and server-side helper middleware.

### Are these changes tested?

Integration tests (C++ currently broken due to middleware-related framework issue) and some complex-case unit testing are included.

### Are there any user-facing changes?

Non-breaking extensions to wire format and corresponding client/server Flight RPC API extensions.

* Closes: apache#34865

Lead-authored-by: Paul Nienaber <[email protected]>
Co-authored-by: Paul Nienaber <[email protected]>
Co-authored-by: James Duong <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: David Li <[email protected]>
thisisnic pushed a commit to thisisnic/arrow that referenced this issue Mar 8, 2024
…nt (apache#40284)

### Rationale for this change

Brings Go implementation in parity with recent session management additions to Java and C++: apache#34865

### What changes are included in this PR?

- Go Flight/FlightSQL implementations of session management RPC handlers
- Implementation of cookie-based session middleware
  - Implementation of stateful (id-lookup based) sessions/tokens
  - Implementation of stateless (fully encoded) sessions/tokens
- Fix minor C++ logic bug when closing sessions
- Update Java integration test server to return an empty session if `getSessionOptions` is called before `setSessionOptions`
- Refactor of `DoAction` handlers to consolidate the code that is essentially copied between them.
  - As part of this I found an issue with `CancelFlightInfo` where a copy of the message was being returned instead of a pointer as is typically the case with `proto.Message`'s. I updated the return type and any usage throughout the code base as part of the refactor.

### Are these changes tested?

Yes, both integration and unit tests are included.

A few tests were added in the Go integration suite beyond the existing coverage in the Java/C++ suites. These tests aim to demonstrate my understanding of session semantics in those scenarios, please let me know if you believe the details are not accurate.

Some of the new integration tests failed in the Java/C++ scenarios. I made very minor changes to those implementations to fix certain failures but there are still some remaining bugs (assuming these are testing the right semantics). Specifically:
- The integration test for reopening a previously closed session passes on Go/Java, but fails for C++ so it is commented out.
- This implementation prefers to set any cookies in the gRPC trailer which works fine for Go/C++, but not for Java. As a temporary workaround this implementation will _also_ set the cookie in the gRPC header if a new session was created. This is sufficient to maintain compatibility with Java stateful sessions where the session ID token can be known at the time of creation, but is not robust to other scenarios such as stateless sessions where in many cases the token cannot be known until after the RPC has completed.

### Are there any user-facing changes?
Yes, session management RPC as well as middleware implementations are included. Functionality is entirely additive

* GitHub Issue: apache#40155

Authored-by: joel <[email protected]>
Signed-off-by: Matt Topol <[email protected]>
stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Mar 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants