From 198f6ff5d3eff56c266be63f3450a868b8375e93 Mon Sep 17 00:00:00 2001 From: Josh Basch Date: Tue, 6 Aug 2024 16:50:07 -0700 Subject: [PATCH 01/10] SPICE-0009: External Readers --- spices/SPICE-0009-external-readers.adoc | 120 ++++++++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 spices/SPICE-0009-external-readers.adoc diff --git a/spices/SPICE-0009-external-readers.adoc b/spices/SPICE-0009-external-readers.adoc new file mode 100644 index 0000000..d5f5553 --- /dev/null +++ b/spices/SPICE-0009-external-readers.adoc @@ -0,0 +1,120 @@ += External Readers + +* Proposal: link:./SPICE-0009-external-readers.adoc[SPICE-0009] +* Status: TBD +* Implemented in: Pkl 0.27 () +* Category: Language + +== Introduction + +This SPICE proposes a way for custom module and resource readers to be used when evaluating modules with `pkl eval`. + +== Motivation + +When used via language binding libraries, custom module and resource readers schemes may be implemented by client code. +Custom readers help bridge Pkl into systems where it does't fit natively, enabling a wide variety of use cases: + +* Mediating access to resources that require authentication (eg. a secret store) +* Querying data accessible via non-HTTP protocols (eg. LDAP) +* Providing filesystem abstractions for remote or in-memory modules +* And more! + +One of the primary drawbacks to custom readers is that using one instantly turns a Pkl module into a _incompatible language_: the module may only be evaoluated by the client tool or application providing the reader and can no longer be evaluated directly using `pkl eval`. +This hinders development and debugging of such modules, as typical workflows involving `trace()` or `pkl eval -x ''` behave differently or are no longer possible. + +== Proposed Solution + +The existing message passing API and client libraries will be expanded to support external readers. +Instead of client libraries invoking `pkl server`, `pkl eval` invocations, when configured with an external reader, will run the reader executable as a subprocess. +External readers can provide any number of resource and/or module schemes and will configured as an executable and list of arguments via: + +* CLI flag +* PklProject property +* Java API +* Message passing API + + +[source,pkl] +---- +/// May be specified as an URI to a readable resource, eg. `file:`, `https:`, or `package:` +/// May also be specified as just an executable name, in which case it will be resolved according to the PATH environment variable +executable: String + +/// Command line arguments that will be passed to the reader process +arguments: Listing +---- + +=== Example + +Consider this module: + +[source,pkl] +---- +username = "john_appleseed" + +email = read("ldap://ds.example.com:389/dc=example,dc=com?mail?sub?(uid=\(username))").text +---- + +Pkl doesn't implement the `ldap:` resource URI scheme natively, but an external reader can provide it. +Assuming a hypothetical `pkl-ldap` binary implementing the external reader protocol and the `ldap:` scheme is in the `$PATH`, this module can be evaluated as: + +[source,text] +---- +$ pkl eval --external-reader=pkl-ldap --allowed-resources prop:,ldap: +username = "john_appleseed" +email = "appleseed@example.com" +---- + +For security reasons, schemes provided by external readers are not allowed by default and clients must explicitly allow the desired scheme(s). +In this example, the external reader may provide both `ldap:` and `ldaps:` schemes. +If a codebase or user wishes to enforce that only secure LDAP connections are made, the `ldaps:` scheme may be allowed while `ldap:` is not. + +== Detailed design + +To avoid terminology confusion with the existing language binding message passing model, the `pkl` process will remain known as the "server" and the reader process will be known as the "client", despite all requests in this proposal being initiated by the `pkl` process. + +* Message Passing API +** Add DiscoverReadersRequest +** Add DiscoverReadersResponse +** CreateEvaluatorRequest +*** `externalReaders` +** Pull into a new pkl-msgapi(?) package shared by pkl-code and pkl-server +*** Message*.kt files at least + +* Java API +** New `ExternalModuleKeyFactory` type +*** Entrypoints (CLI, Server) construct based on `externalReaders` +*** On (initialization? explicit startup?), launch subprocess +*** send each DiscoverReadersRequest, await DiscoverReadersResponse (timeout? 1s?) + +* stdlib +** `pkl:EvaluatorSettings` (used in `pkl:Project`) +*** `externalReaders: Listing` + +* CLI +** `--external-reader` +*** use `clikt.multiple` +*** may be passed as a space-separated string where the first element becomes `executable` and any remainder becomes `arguments` +*** should this be shlex'd? or just split on spaces? + +* Language Bindings +** Add `ExternalReaderRuntime` +** Add `externalReaders` to `EvaluatorOptions` + +== Compatibility + +This proposal is purely additive. + +In the case where newer language bindings configure external readers against an older `pkl` binary, the new `CreateEvaluatorRequest.externalReaders` field will be ignored silently. If module evaluation relies on configured external readers, it will fail accordingly. + +== Future directions + +* Java library for bindings to support being an external reader client +* To improve CLI ergonomics, could implement additive `--allow-resources`/`--allow-modules` args (current flags replace full list) +* Manage external reader processes separately from EvaluatorImpl lifetime +** Potential large savings in per-evaluator overhead for Java API and Language Binding usage +** Savings for CLI usage (primary use case) would be minimal +** Code is more complicated (need an ExternalReaderManager sort of mechanism tracking unique commands => processes) +** Change could be made as followup work with only changes to Java APIs and internals + +== Alternatives considered From d08001f3e4dab80fdb38e4f1b0c1e9d8ec14005f Mon Sep 17 00:00:00 2001 From: Josh B <421772+HT154@users.noreply.github.com> Date: Thu, 8 Aug 2024 10:44:27 -0700 Subject: [PATCH 02/10] Update spices/SPICE-0009-external-readers.adoc Co-authored-by: Islon Scherer --- spices/SPICE-0009-external-readers.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spices/SPICE-0009-external-readers.adoc b/spices/SPICE-0009-external-readers.adoc index d5f5553..1611110 100644 --- a/spices/SPICE-0009-external-readers.adoc +++ b/spices/SPICE-0009-external-readers.adoc @@ -19,7 +19,7 @@ Custom readers help bridge Pkl into systems where it does't fit natively, enabli * Providing filesystem abstractions for remote or in-memory modules * And more! -One of the primary drawbacks to custom readers is that using one instantly turns a Pkl module into a _incompatible language_: the module may only be evaoluated by the client tool or application providing the reader and can no longer be evaluated directly using `pkl eval`. +One of the primary drawbacks to custom readers is that using one instantly turns a Pkl module into a _incompatible language_: the module may only be evaluated by the client tool or application providing the reader and can no longer be evaluated directly using `pkl eval`. This hinders development and debugging of such modules, as typical workflows involving `trace()` or `pkl eval -x ''` behave differently or are no longer possible. == Proposed Solution From bfc092db03851cffa67fc346ecd67cb13d308550 Mon Sep 17 00:00:00 2001 From: Josh B <421772+HT154@users.noreply.github.com> Date: Thu, 8 Aug 2024 10:44:35 -0700 Subject: [PATCH 03/10] Update spices/SPICE-0009-external-readers.adoc Co-authored-by: Islon Scherer --- spices/SPICE-0009-external-readers.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spices/SPICE-0009-external-readers.adoc b/spices/SPICE-0009-external-readers.adoc index 1611110..9341234 100644 --- a/spices/SPICE-0009-external-readers.adoc +++ b/spices/SPICE-0009-external-readers.adoc @@ -26,7 +26,7 @@ This hinders development and debugging of such modules, as typical workflows inv The existing message passing API and client libraries will be expanded to support external readers. Instead of client libraries invoking `pkl server`, `pkl eval` invocations, when configured with an external reader, will run the reader executable as a subprocess. -External readers can provide any number of resource and/or module schemes and will configured as an executable and list of arguments via: +External readers can provide any number of resource and/or module schemes and will be configured as an executable and list of arguments via: * CLI flag * PklProject property From ed2dcf35b3fd8f7134ac98aa3fe87424e6d641a0 Mon Sep 17 00:00:00 2001 From: Josh Date: Mon, 12 Aug 2024 23:43:31 -0700 Subject: [PATCH 04/10] Document alternatives considered, move URI support for executables to future directions --- spices/SPICE-0009-external-readers.adoc | 80 ++++++++++++++++++++++++- 1 file changed, 78 insertions(+), 2 deletions(-) diff --git a/spices/SPICE-0009-external-readers.adoc b/spices/SPICE-0009-external-readers.adoc index 9341234..55e670e 100644 --- a/spices/SPICE-0009-external-readers.adoc +++ b/spices/SPICE-0009-external-readers.adoc @@ -33,10 +33,9 @@ External readers can provide any number of resource and/or module schemes and wi * Java API * Message passing API - [source,pkl] ---- -/// May be specified as an URI to a readable resource, eg. `file:`, `https:`, or `package:` +/// May be specified as an absolute path to an executable /// May also be specified as just an executable name, in which case it will be resolved according to the PATH environment variable executable: String @@ -109,6 +108,41 @@ In the case where newer language bindings configure external readers against an == Future directions +* Support for specifying URIs for external reader executables so they may be distributed in Pkl packages. +This is potentially very valuable for statically compiled reader binaries, but significantly complicates the implementation. +The design, as proposed, does not prohibit implementing this as a future enhancement. +This would also make it very convenient to provide friendly, type-safe Pkl APIs for complex reader URI schemes instead of having the "stringly-typed" URI as the primary API, e.g. building on the `ldap:` example: ++ +[source,pkl] +---- +typealias LDAPResult = Mapping> + +class LDAPQuery { + protocol: *"ldap"|"ldaps" + host: String + port: UInt16 = 389 + baseDN: String + attributes: Listing + scope: *"base"|"one"|"sub" + filter: String = "(&)" // matches anything + + fixed results: Listing = new json.Parser { useMapping = true }.parse( + read("\(protocol)://\(host):\(port)/\(baseDN)?\(attributes.join(","))?\(scope)?\(filter)").text + ) +} + +local queryResults = new LDAPQuery { + host = "ds.example.com" + baseDN = "dc=example,dc=com" + attributes { "mail" } + scope = "sub" + filter = "(uid=\(username))" +}.results + +username = "john_appleseed" + +email = queryResults[0]["mail"][0] +---- * Java library for bindings to support being an external reader client * To improve CLI ergonomics, could implement additive `--allow-resources`/`--allow-modules` args (current flags replace full list) * Manage external reader processes separately from EvaluatorImpl lifetime @@ -118,3 +152,45 @@ In the case where newer language bindings configure external readers against an ** Change could be made as followup work with only changes to Java APIs and internals == Alternatives considered + +=== One shot, per-read subprocesses + +Instead of "persistent" reader processes invoked during evaluator initialization. +Instead of using the msgpack message-passing API, reader binaries could be invoked with the read URI as a CLI argument and return their result on standard output. +This potentially greatly lowers the barrier to entry for implementing external readers, even allowing them to be implemented by shell scripts. + +This approach does not have a clean way to support globbed reads. +To resolve globs, Pkl can require many list modules/resources requests. +It's not clear one-shot reader processes would be invoked differently to distinguish read requests from list requests. +Multiple invocations would also have potentially significant overhead, especially for readers implemented in interpreted languages. + +There is definitely value in supporting significantly reduced barrier to reader implementation, especially when globbing is not required. +One way this gap might be closed is with a "shim" reader process that translates the message passing API calls to subprocess invocations: + +[source,text] +---- +$ pkl eval --external-reader 'pkl-cmd ldap=pkl-ldap.sh' --allowed-resources prop:,ldap: +username = "john_appleseed" +email = "appleseed@example.com" +---- + +It may even make sense for the `pkl` binary itself to provide this functionality. + +=== Up-front scheme -> reader registration + +Instead of Pkl starting reader subprocesses and discovering supported schemes during evaluator initialization, an alternative approach would be to explicit register this mapping. +This would allow reader processes to be launched on first read instead of during evaluator initialization. +This is more efficient in cases where the reader is not actually needed, but requires a greater amount of up-front configuration, especially when the same reader executable will be used for multiple schemes. + +[source,text] +---- +$ pkl eval --external-resource-reader ldap=pkl-ldap --allowed-resources prop:,ldap: +username = "john_appleseed" +email = "appleseed@example.com" +---- + +This approach raises a few questions: + +* Should declaring an external reader automatically allow reads for its scheme? +Declaring explicit allowed resources should probably disable this behavior. +* What happens when a reader doesn't support the scheme it is declared for? From 82ebff420c98cbd43818f578caba60919b796990 Mon Sep 17 00:00:00 2001 From: Josh Basch Date: Fri, 16 Aug 2024 18:38:49 -0700 Subject: [PATCH 05/10] Expand detailed design --- spices/SPICE-0009-external-readers.adoc | 173 ++++++++++++++++++++---- 1 file changed, 148 insertions(+), 25 deletions(-) diff --git a/spices/SPICE-0009-external-readers.adoc b/spices/SPICE-0009-external-readers.adoc index 55e670e..fa3552f 100644 --- a/spices/SPICE-0009-external-readers.adoc +++ b/spices/SPICE-0009-external-readers.adoc @@ -72,39 +72,156 @@ If a codebase or user wishes to enforce that only secure LDAP connections are ma To avoid terminology confusion with the existing language binding message passing model, the `pkl` process will remain known as the "server" and the reader process will be known as the "client", despite all requests in this proposal being initiated by the `pkl` process. -* Message Passing API -** Add DiscoverReadersRequest -** Add DiscoverReadersResponse -** CreateEvaluatorRequest -*** `externalReaders` -** Pull into a new pkl-msgapi(?) package shared by pkl-code and pkl-server -*** Message*.kt files at least +=== Overall Flow -* Java API -** New `ExternalModuleKeyFactory` type -*** Entrypoints (CLI, Server) construct based on `externalReaders` -*** On (initialization? explicit startup?), launch subprocess -*** send each DiscoverReadersRequest, await DiscoverReadersResponse (timeout? 1s?) +1. User requests an evaluator (via Java API, Message Passing API, or CLI) with an external reader. +2. Entrypoint (client Java code, `CliEvaluator`, or `Server`) instantiates an `ExternalReader`. +3. The `ExternalReader` is started, which starts the configured child process and sends a `DiscoverReadersRequest` message. +4. A `DiscoverReadersResponse` message is received (within some timeout). +5. The entrypoint requests resource readers (`ExternalResourceReader`) and module key factories (`ExternalModuleKeyFactory`) from the `ExternalReader` and uses them to build an evaluator. +`ExternalResourceReader` and `ExternalModuleKeyFactory` function similarly to the existing `ClientResourceReader` and `ClientModuleKeyFactory`, implementation may be reusable entirely. +6. Evaluation begins. +7. A resource or module read is encountered. +8. The appropriate `Read*Request` message is sent to the child process, the corresponding `Read*Response` message is awaited. +9. Evaluation continues. +10. Evaluation completes. +11. The `ExternalReader` is closed, terminating the child process. + +=== Java API + +New APIs: + +* `ExternalResourceReader` - a `ResourceReader` implementation similar (identical?) to `ClientResourceReader`. +* `ExternalModuleKeyFactory` - a `ModuleKeyFactory` implementation similar (identical?) to `ClientModuleKeyFactory`. +* `ExternalModuleKey` - a `ModuleKey` implementation similar (identical?) to `ClientModuleKey`. +* `ExternalReader` - manages the lifecycle of child processes. + ** Explicit `open`/`close` methods to manage child process lifecycle. + ** Opening an `ExternalReader` spawns the subprocess, sets up the `MessageTransport`, sends a `DiscoverReadersRequest` message, and awaits a `DiscoverReadersResponse` response. + ** Methods to procure `ExternalResourceReader` and `ExternalModuleKeyFactory` instances. + +This proposal requires that the message passing API functionality move out of pkl-server. +A new project pkl-msgapi will be created to contain the new APIs and the core messaging code current part of pkl-server (`pkl-server/src.main/kotlin/org.pkl.server/Message*.kt`). +The pkl-msgapi project will be depended on by pkl-cli, pkl-server, and Java API clients using external readers. + +=== Message Passing API + +Two new message types will be added: + +[source,pkl] +---- +/// Code: 0x2e +/// Type: Server Request +class DiscoverReadersRequest { + /// A number identifying this request + requestId: Int +} + +/// Code: 0x2f +/// Type: Client Response +class DiscoverReadersResponse { + /// A number identifying this request + requestId: Int + + /// Register client-side module readers. + /// + /// [ClientModuleReader] is defined at https://pkl-lang.org/main/current/bindings-specification/message-passing-api.html#create-evaluator-request + clientModuleReaders: Listing? + + /// Register client-side resource readers. + /// + /// [ClientResourceReader] is defined at https://pkl-lang.org/main/current/bindings-specification/message-passing-api.html#create-evaluator-request + clientResourceReaders: Listing? +} +---- + +`CreateEvaluatorRequest` will be expanded with an additional property: +[source,pkl] +---- +externalReaders: Listing? + +class ExternalReader { + /// May be specified as an absolute path to an executable + /// May also be specified as just an executable name, in which case it will be resolved according to the PATH environment variable + executable: String + + /// Command line arguments that will be passed to the reader process + arguments: Listing +} +---- + +=== CLI + +A new `--external-reader` CLI argument will be added to configure external readers. +This argument can be provided multiple times (using `clikt.multiple`) to configure multiple external readers. +The argument may be passed as a space-separated string where the first element becomes `executable` and any remainder becomes `arguments`. + +TBD: It might be best if the argument value is link:https://docs.python.org/3/library/shlex.html#shlex.split[shlex'd] instead of split to support passing arguments to the reader process that contain spaces. + +=== Standard Library -* stdlib -** `pkl:EvaluatorSettings` (used in `pkl:Project`) -*** `externalReaders: Listing` +The `EvaluatorSettings` module will be expanded to enable configuring external readers in `PklProject` files: -* CLI -** `--external-reader` -*** use `clikt.multiple` -*** may be passed as a space-separated string where the first element becomes `executable` and any remainder becomes `arguments` -*** should this be shlex'd? or just split on spaces? +[source,pkl] +---- +externalReaders: Listing? + +class ExternalReader { + /// May be specified as an absolute path to an executable + /// May also be specified as just an executable name, in which case it will be resolved according to the PATH environment variable + executable: String + + /// Command line arguments that will be passed to the reader process + arguments: Listing +} +---- + +=== Language Binding Libraries -* Language Bindings -** Add `ExternalReaderRuntime` -** Add `externalReaders` to `EvaluatorOptions` +The language binding libraries `pkl-go` and `pkl-swift` will be expanded to support using and implementing external readers. +For the purpose of illustration, examples will be provided using Golang. + +The `EvaluatorOptions` type will be expanded to include a new property for external readers: + +[source,go] +---- +type EvaluatorOptions struct { + // ... + ExternalReaders []ExternalReader + // ... +} + +type ExternalReader struct { + Executable string + Arguments []string +} +---- + +A new `ExternalReaderRuntime` type will be introduced to implement the child process message passing interface. +It makes sense to expand the existing libraries to add this functionality as much of the message passing infrastructure and types for implementing resource and module readers can be reused. +An `ExternalReaderRuntime` is configured with zero or more `ResourceReader` instances and zero or more `ModuleReader`. +When started, the runtime will consume messages from standard input, dispatch calls to the configured readers, and send responses over standard output. + +[source,go] +---- +type ExternalReaderRuntime interface { + Run() + Close() +} + +func NewExternalReaderRuntime(resourceReaders []ResourceReader, moduleReaders []ModuleReader) ExternalReaderRuntime { + // ... +} +---- == Compatibility -This proposal is purely additive. +From a language perspective, this proposal is purely additive. + +In the case where newer language bindings configure external readers against an older `pkl` binary, the new `CreateEvaluatorRequest.externalReaders` field will be ignored silently. +If module evaluation relies on configured external readers, it will fail accordingly. -In the case where newer language bindings configure external readers against an older `pkl` binary, the new `CreateEvaluatorRequest.externalReaders` field will be ignored silently. If module evaluation relies on configured external readers, it will fail accordingly. +Any usage of the pkl-server APIs that are moving to pkl-msgapi will break. +It's unlikely there are clients of these APIs outside the apple/pkl repo. == Future directions @@ -115,6 +232,8 @@ This would also make it very convenient to provide friendly, type-safe Pkl APIs + [source,pkl] ---- +import "pkl:json" + typealias LDAPResult = Mapping> class LDAPQuery { @@ -153,6 +272,10 @@ email = queryResults[0]["mail"][0] == Alternatives considered +=== Message Passing code in pkl-core + +Instead of creating a new project for pkl-msgapi, move the message passing code from pkl-server directly into pkl-core. + === One shot, per-read subprocesses Instead of "persistent" reader processes invoked during evaluator initialization. From 90261d96ee06184b88fee3d406f40e433fe40d58 Mon Sep 17 00:00:00 2001 From: Josh Basch Date: Tue, 20 Aug 2024 17:35:48 -0700 Subject: [PATCH 06/10] Address review feedback --- spices/SPICE-0009-external-readers.adoc | 38 ++++++++++++++++++------- 1 file changed, 27 insertions(+), 11 deletions(-) diff --git a/spices/SPICE-0009-external-readers.adoc b/spices/SPICE-0009-external-readers.adoc index fa3552f..0382985 100644 --- a/spices/SPICE-0009-external-readers.adoc +++ b/spices/SPICE-0009-external-readers.adoc @@ -85,7 +85,8 @@ To avoid terminology confusion with the existing language binding message passin 8. The appropriate `Read*Request` message is sent to the child process, the corresponding `Read*Response` message is awaited. 9. Evaluation continues. 10. Evaluation completes. -11. The `ExternalReader` is closed, terminating the child process. +11. The `ExternalReader` is closed, the child process is sent SIGTERM. +12. If the child process has not terminated after some timeout (3 seconds?), it is forcefully stopped via SIGKILL. === Java API @@ -99,9 +100,8 @@ New APIs: ** Opening an `ExternalReader` spawns the subprocess, sets up the `MessageTransport`, sends a `DiscoverReadersRequest` message, and awaits a `DiscoverReadersResponse` response. ** Methods to procure `ExternalResourceReader` and `ExternalModuleKeyFactory` instances. -This proposal requires that the message passing API functionality move out of pkl-server. -A new project pkl-msgapi will be created to contain the new APIs and the core messaging code current part of pkl-server (`pkl-server/src.main/kotlin/org.pkl.server/Message*.kt`). -The pkl-msgapi project will be depended on by pkl-cli, pkl-server, and Java API clients using external readers. +This proposal requires that the message passing API functionality move out of pkl-server and into pkl-core. +The code added to pkl-core will include the new APIs and the core messaging code currently part of pkl-server (`pkl-server/src.main/kotlin/org.pkl.server/Message*.kt`). === Message Passing API @@ -199,7 +199,7 @@ type ExternalReader struct { A new `ExternalReaderRuntime` type will be introduced to implement the child process message passing interface. It makes sense to expand the existing libraries to add this functionality as much of the message passing infrastructure and types for implementing resource and module readers can be reused. An `ExternalReaderRuntime` is configured with zero or more `ResourceReader` instances and zero or more `ModuleReader`. -When started, the runtime will consume messages from standard input, dispatch calls to the configured readers, and send responses over standard output. +When started, the runtime will consume messages from the configured `Reader`, dispatch calls to the configured readers, and send responses to the configured `Writer`. [source,go] ---- @@ -208,9 +208,29 @@ type ExternalReaderRuntime interface { Close() } -func NewExternalReaderRuntime(resourceReaders []ResourceReader, moduleReaders []ModuleReader) ExternalReaderRuntime { +type ExternalReaderRuntimeOptions struct { + // ResourceReaders are the resource readers to be used by the evaluator. + ResourceReaders []ResourceReader + + // ModuleReaders are the set of custom module readers to be used by the evaluator. + ModuleReaders []ModuleReader + + // Input reader to consume messages from Pkl from + // Defaults to os.Stdin if not set + Input io.Reader + + // Output writer to produce message to Pkl + // Defaults to os.Stdout if not set + Output io.Writer +} + +func NewExternalReaderRuntime(opts ...func(options *ExternalReaderRuntimeOptions)) ExternalReaderRuntime { // ... } + +var WithResourceReaders = // ... +var WithModuleReaders = // ... +var WithStreams = // ... ---- == Compatibility @@ -220,7 +240,7 @@ From a language perspective, this proposal is purely additive. In the case where newer language bindings configure external readers against an older `pkl` binary, the new `CreateEvaluatorRequest.externalReaders` field will be ignored silently. If module evaluation relies on configured external readers, it will fail accordingly. -Any usage of the pkl-server APIs that are moving to pkl-msgapi will break. +Any usage of the pkl-server APIs that are moving to pkl-core will break. It's unlikely there are clients of these APIs outside the apple/pkl repo. == Future directions @@ -272,10 +292,6 @@ email = queryResults[0]["mail"][0] == Alternatives considered -=== Message Passing code in pkl-core - -Instead of creating a new project for pkl-msgapi, move the message passing code from pkl-server directly into pkl-core. - === One shot, per-read subprocesses Instead of "persistent" reader processes invoked during evaluator initialization. From 8130a3e9667c0c1b5d6a12450e04cb1f8ce76892 Mon Sep 17 00:00:00 2001 From: Josh Date: Sat, 21 Sep 2024 16:26:32 -0700 Subject: [PATCH 07/10] Update to use explicit scheme registration --- spices/SPICE-0009-external-readers.adoc | 152 +++++++++++++----------- 1 file changed, 81 insertions(+), 71 deletions(-) diff --git a/spices/SPICE-0009-external-readers.adoc b/spices/SPICE-0009-external-readers.adoc index 0382985..bdc2be5 100644 --- a/spices/SPICE-0009-external-readers.adoc +++ b/spices/SPICE-0009-external-readers.adoc @@ -12,7 +12,7 @@ This SPICE proposes a way for custom module and resource readers to be used when == Motivation When used via language binding libraries, custom module and resource readers schemes may be implemented by client code. -Custom readers help bridge Pkl into systems where it does't fit natively, enabling a wide variety of use cases: +Custom readers help bridge Pkl into systems where it doesn't fit natively, enabling a wide variety of use cases: * Mediating access to resources that require authentication (eg. a secret store) * Querying data accessible via non-HTTP protocols (eg. LDAP) @@ -59,14 +59,20 @@ Assuming a hypothetical `pkl-ldap` binary implementing the external reader proto [source,text] ---- -$ pkl eval --external-reader=pkl-ldap --allowed-resources prop:,ldap: +$ pkl eval --external-resource ldap=pkl-ldap username = "john_appleseed" email = "appleseed@example.com" ---- -For security reasons, schemes provided by external readers are not allowed by default and clients must explicitly allow the desired scheme(s). In this example, the external reader may provide both `ldap:` and `ldaps:` schemes. -If a codebase or user wishes to enforce that only secure LDAP connections are made, the `ldaps:` scheme may be allowed while `ldap:` is not. +External readers are registered by their scheme, so to support both schemes both would need to be specified on the command line: +[source,text] +---- +$ pkl eval --external-resource ldap=pkl-ldap --external-resource ldaps=pkl-ldap +---- + +Registering an external reader for a scheme automatically adds that scheme to the default allowed modules/resources. +As with Pkl's built-in module and resource schemes, setting explicit allowed module or resources overrides this behavior and appropriate patterns must be specified to allow use of external readers. == Detailed design @@ -75,69 +81,94 @@ To avoid terminology confusion with the existing language binding message passin === Overall Flow 1. User requests an evaluator (via Java API, Message Passing API, or CLI) with an external reader. -2. Entrypoint (client Java code, `CliEvaluator`, or `Server`) instantiates an `ExternalReader`. -3. The `ExternalReader` is started, which starts the configured child process and sends a `DiscoverReadersRequest` message. -4. A `DiscoverReadersResponse` message is received (within some timeout). -5. The entrypoint requests resource readers (`ExternalResourceReader`) and module key factories (`ExternalModuleKeyFactory`) from the `ExternalReader` and uses them to build an evaluator. -`ExternalResourceReader` and `ExternalModuleKeyFactory` function similarly to the existing `ClientResourceReader` and `ClientModuleKeyFactory`, implementation may be reusable entirely. -6. Evaluation begins. -7. A resource or module read is encountered. -8. The appropriate `Read*Request` message is sent to the child process, the corresponding `Read*Response` message is awaited. -9. Evaluation continues. -10. Evaluation completes. -11. The `ExternalReader` is closed, the child process is sent SIGTERM. -12. If the child process has not terminated after some timeout (3 seconds?), it is forcefully stopped via SIGKILL. +2. The entrypoint (client Java code, `CliEvaluator`, or `Server`) instantiates an `ExternalProcess` based on the configured external readers. +3. The entrypoint instantiates resource readers (`ResourceReaders.External`) and module key factories (`ModuleKeyFactories.External`) referencing the `ExternalProcess` and uses them to build an evaluator. +`ResourceReaders.External` and `ModuleKeyFactories.External` function almost identically to the existing `ClientResourceReader` and `ClientModuleKeyFactory` and pkl-server will be updated to reuse these implementations where possible. +4. Evaluation begins. +5. A resource or module read is encountered. +6. Upon first read only: +1.. The `ExternalProcess` is started, which starts the configured child process. +2.. The child process is sent a `Discover(Module|Resource)ReaderRequest` message. +3.. A `DiscoverReadersResponse` message is received (within some timeout). +7. The appropriate `Read*Request` message is sent to the child process, the corresponding `Read*Response` message is awaited. +8. Evaluation continues. +9. Evaluation completes. +10. The `ExternalProcess` is closed, the child process is sent SIGTERM. +11. If the child process has not terminated after some timeout (3 seconds?), it is forcefully stopped via SIGKILL. === Java API New APIs: -* `ExternalResourceReader` - a `ResourceReader` implementation similar (identical?) to `ClientResourceReader`. -* `ExternalModuleKeyFactory` - a `ModuleKeyFactory` implementation similar (identical?) to `ClientModuleKeyFactory`. -* `ExternalModuleKey` - a `ModuleKey` implementation similar (identical?) to `ClientModuleKey`. -* `ExternalReader` - manages the lifecycle of child processes. - ** Explicit `open`/`close` methods to manage child process lifecycle. - ** Opening an `ExternalReader` spawns the subprocess, sets up the `MessageTransport`, sends a `DiscoverReadersRequest` message, and awaits a `DiscoverReadersResponse` response. - ** Methods to procure `ExternalResourceReader` and `ExternalModuleKeyFactory` instances. +* `ResourceReaders.External` - a `ResourceReader` implementation identical to `ClientResourceReader`. +* `ModuleKeyFactories.External` - a `ModuleKeyFactory` implementation similar to `ClientModuleKeyFactory`. +* `ModuleKeys.External` - a `ModuleKey` implementation identical to `ClientModuleKey`. +* `ExternalProcess` - manages the lifecycle of child processes. + ** Explicit `close` methods to manage child process lifecycle. + ** The `ExternalProcess` spawns the subprocess on first access, which sets up the `MessageTransport`, sends the appropriate `Discover*ReaderRequest` message, and awaits the corresponding `Discover*ReaderResponse` response. This proposal requires that the message passing API functionality move out of pkl-server and into pkl-core. The code added to pkl-core will include the new APIs and the core messaging code currently part of pkl-server (`pkl-server/src.main/kotlin/org.pkl.server/Message*.kt`). === Message Passing API -Two new message types will be added: +Four new message types will be added: [source,pkl] ---- -/// Code: 0x2e +/// Code: 0x100 /// Type: Server Request -class DiscoverReadersRequest { +class DiscoverModuleReaderRequest { /// A number identifying this request requestId: Int + + /// The scheme of the resource to discover the spec for + scheme: String } -/// Code: 0x2f +/// Code: 0x101 +/// Type: Server Request +class DiscoverResourceReaderRequest { + /// A number identifying this request + requestId: Int + + /// The scheme of the resource to discover the spec for + scheme: String +} + +/// Code: 0x102 /// Type: Client Response -class DiscoverReadersResponse { +class DiscoverModuleReaderResponse { /// A number identifying this request requestId: Int - /// Register client-side module readers. + /// Client-side module reader spec. /// + /// Null when the external process does not implement the requested scheme. /// [ClientModuleReader] is defined at https://pkl-lang.org/main/current/bindings-specification/message-passing-api.html#create-evaluator-request - clientModuleReaders: Listing? + spec: ClientModuleReader? +} + +/// Code: 0x102 +/// Type: Client Response +class DiscoverResourceReaderResponse { + /// A number identifying this request + requestId: Int - /// Register client-side resource readers. + /// Client-side resource reader spec. /// + /// Null when the external process does not implement the requested scheme. /// [ClientResourceReader] is defined at https://pkl-lang.org/main/current/bindings-specification/message-passing-api.html#create-evaluator-request - clientResourceReaders: Listing? + spec: ClientResourceReader? } ---- -`CreateEvaluatorRequest` will be expanded with an additional property: +`CreateEvaluatorRequest` will be expanded with additional properties: [source,pkl] ---- -externalReaders: Listing? +externalModuleReaders: Mapping? + +externalResourceReaders: Mapping? class ExternalReader { /// May be specified as an absolute path to an executable @@ -151,9 +182,10 @@ class ExternalReader { === CLI -A new `--external-reader` CLI argument will be added to configure external readers. -This argument can be provided multiple times (using `clikt.multiple`) to configure multiple external readers. -The argument may be passed as a space-separated string where the first element becomes `executable` and any remainder becomes `arguments`. +New `--external-resource` and `--external-module` CLI argument will be added to configure external readers. +The arguments can be provided multiple times to configure multiple external readers. +The arguments are passed as `=`-delimited key-value pairs where the key is the reader's URI scheme. +The argument values may be passed as space-separated strings where the first element becomes `executable` and any remainder becomes `arguments`. TBD: It might be best if the argument value is link:https://docs.python.org/3/library/shlex.html#shlex.split[shlex'd] instead of split to support passing arguments to the reader process that contain spaces. @@ -163,7 +195,9 @@ The `EvaluatorSettings` module will be expanded to enable configuring external r [source,pkl] ---- -externalReaders: Listing? +externalModuleReaders: Mapping? + +externalResourceReaders: Mapping? class ExternalReader { /// May be specified as an absolute path to an executable @@ -186,7 +220,8 @@ The `EvaluatorOptions` type will be expanded to include a new property for exter ---- type EvaluatorOptions struct { // ... - ExternalReaders []ExternalReader + ExternalModuleReaders map[string]ExternalReader + ExternalResourceReaders map[string]ExternalReader // ... } @@ -198,7 +233,7 @@ type ExternalReader struct { A new `ExternalReaderRuntime` type will be introduced to implement the child process message passing interface. It makes sense to expand the existing libraries to add this functionality as much of the message passing infrastructure and types for implementing resource and module readers can be reused. -An `ExternalReaderRuntime` is configured with zero or more `ResourceReader` instances and zero or more `ModuleReader`. +An `ExternalReaderRuntime` is configured with zero or more `ResourceReader` instances and zero or more `ModuleReader` instances. When started, the runtime will consume messages from the configured `Reader`, dispatch calls to the configured readers, and send responses to the configured `Writer`. [source,go] @@ -237,7 +272,7 @@ var WithStreams = // ... From a language perspective, this proposal is purely additive. -In the case where newer language bindings configure external readers against an older `pkl` binary, the new `CreateEvaluatorRequest.externalReaders` field will be ignored silently. +In the case where newer language bindings configure external readers against an older `pkl` binary, the new `CreateEvaluatorRequest.external(Module|Resource)Readers` fields will be ignored silently. If module evaluation relies on configured external readers, it will fail accordingly. Any usage of the pkl-server APIs that are moving to pkl-core will break. @@ -245,6 +280,8 @@ It's unlikely there are clients of these APIs outside the apple/pkl repo. == Future directions +* Java library for bindings to support being an external reader client +* Configuration of external readers via `~/.pkl/settings.pkl` * Support for specifying URIs for external reader executables so they may be distributed in Pkl packages. This is potentially very valuable for statically compiled reader binaries, but significantly complicates the implementation. The design, as proposed, does not prohibit implementing this as a future enhancement. @@ -267,7 +304,7 @@ class LDAPQuery { fixed results: Listing = new json.Parser { useMapping = true }.parse( read("\(protocol)://\(host):\(port)/\(baseDN)?\(attributes.join(","))?\(scope)?\(filter)").text - ) + ) as Listing } local queryResults = new LDAPQuery { @@ -282,25 +319,17 @@ username = "john_appleseed" email = queryResults[0]["mail"][0] ---- -* Java library for bindings to support being an external reader client -* To improve CLI ergonomics, could implement additive `--allow-resources`/`--allow-modules` args (current flags replace full list) -* Manage external reader processes separately from EvaluatorImpl lifetime -** Potential large savings in per-evaluator overhead for Java API and Language Binding usage -** Savings for CLI usage (primary use case) would be minimal -** Code is more complicated (need an ExternalReaderManager sort of mechanism tracking unique commands => processes) -** Change could be made as followup work with only changes to Java APIs and internals == Alternatives considered === One shot, per-read subprocesses -Instead of "persistent" reader processes invoked during evaluator initialization. Instead of using the msgpack message-passing API, reader binaries could be invoked with the read URI as a CLI argument and return their result on standard output. This potentially greatly lowers the barrier to entry for implementing external readers, even allowing them to be implemented by shell scripts. This approach does not have a clean way to support globbed reads. To resolve globs, Pkl can require many list modules/resources requests. -It's not clear one-shot reader processes would be invoked differently to distinguish read requests from list requests. +It's not clear how one-shot reader processes could be invoked differently to distinguish read requests from list requests. Multiple invocations would also have potentially significant overhead, especially for readers implemented in interpreted languages. There is definitely value in supporting significantly reduced barrier to reader implementation, especially when globbing is not required. @@ -308,28 +337,9 @@ One way this gap might be closed is with a "shim" reader process that translates [source,text] ---- -$ pkl eval --external-reader 'pkl-cmd ldap=pkl-ldap.sh' --allowed-resources prop:,ldap: +$ pkl eval --external-resource ldap='pkl-cmd ldap=pkl-ldap.sh' username = "john_appleseed" email = "appleseed@example.com" ---- It may even make sense for the `pkl` binary itself to provide this functionality. - -=== Up-front scheme -> reader registration - -Instead of Pkl starting reader subprocesses and discovering supported schemes during evaluator initialization, an alternative approach would be to explicit register this mapping. -This would allow reader processes to be launched on first read instead of during evaluator initialization. -This is more efficient in cases where the reader is not actually needed, but requires a greater amount of up-front configuration, especially when the same reader executable will be used for multiple schemes. - -[source,text] ----- -$ pkl eval --external-resource-reader ldap=pkl-ldap --allowed-resources prop:,ldap: -username = "john_appleseed" -email = "appleseed@example.com" ----- - -This approach raises a few questions: - -* Should declaring an external reader automatically allow reads for its scheme? -Declaring explicit allowed resources should probably disable this behavior. -* What happens when a reader doesn't support the scheme it is declared for? From 939cc48b616c100eb6213f24f15e65ca4e4d34dc Mon Sep 17 00:00:00 2001 From: Josh Date: Tue, 24 Sep 2024 08:37:34 -0700 Subject: [PATCH 08/10] Clarify that process termination happens on evaluator close, Discover->Initialize --- spices/SPICE-0009-external-readers.adoc | 45 +++++++++++++------------ 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/spices/SPICE-0009-external-readers.adoc b/spices/SPICE-0009-external-readers.adoc index bdc2be5..cc8c85c 100644 --- a/spices/SPICE-0009-external-readers.adoc +++ b/spices/SPICE-0009-external-readers.adoc @@ -88,13 +88,14 @@ To avoid terminology confusion with the existing language binding message passin 5. A resource or module read is encountered. 6. Upon first read only: 1.. The `ExternalProcess` is started, which starts the configured child process. -2.. The child process is sent a `Discover(Module|Resource)ReaderRequest` message. -3.. A `DiscoverReadersResponse` message is received (within some timeout). +2.. The child process is sent a `Initialize(Module|Resource)ReaderRequest` message. +3.. A `Initialize(Module|Resource)ReaderResponse` message is received (within some timeout). 7. The appropriate `Read*Request` message is sent to the child process, the corresponding `Read*Response` message is awaited. 8. Evaluation continues. 9. Evaluation completes. -10. The `ExternalProcess` is closed, the child process is sent SIGTERM. -11. If the child process has not terminated after some timeout (3 seconds?), it is forcefully stopped via SIGKILL. +10. The evaluator is closed. +1.. The `ExternalProcess` is closed, the child process is sent SIGTERM. +2.. If the child process has not terminated after some timeout (3 seconds?), it is forcefully stopped via SIGKILL. === Java API @@ -105,7 +106,7 @@ New APIs: * `ModuleKeys.External` - a `ModuleKey` implementation identical to `ClientModuleKey`. * `ExternalProcess` - manages the lifecycle of child processes. ** Explicit `close` methods to manage child process lifecycle. - ** The `ExternalProcess` spawns the subprocess on first access, which sets up the `MessageTransport`, sends the appropriate `Discover*ReaderRequest` message, and awaits the corresponding `Discover*ReaderResponse` response. + ** The `ExternalProcess` spawns the subprocess on first access, which sets up the `MessageTransport`, sends the appropriate `Initialize*ReaderRequest` message, and awaits the corresponding `Initialize*ReaderResponse` response. This proposal requires that the message passing API functionality move out of pkl-server and into pkl-core. The code added to pkl-core will include the new APIs and the core messaging code currently part of pkl-server (`pkl-server/src.main/kotlin/org.pkl.server/Message*.kt`). @@ -118,28 +119,18 @@ Four new message types will be added: ---- /// Code: 0x100 /// Type: Server Request -class DiscoverModuleReaderRequest { - /// A number identifying this request +class InitializeModuleReaderRequest { + /// A number identifying this request. requestId: Int - /// The scheme of the resource to discover the spec for + /// The scheme of the resource to initialize. scheme: String } /// Code: 0x101 -/// Type: Server Request -class DiscoverResourceReaderRequest { - /// A number identifying this request - requestId: Int - - /// The scheme of the resource to discover the spec for - scheme: String -} - -/// Code: 0x102 /// Type: Client Response -class DiscoverModuleReaderResponse { - /// A number identifying this request +class InitializeModuleReaderResponse { + /// A number identifying this request. requestId: Int /// Client-side module reader spec. @@ -149,10 +140,20 @@ class DiscoverModuleReaderResponse { spec: ClientModuleReader? } +/// Code: 0x102 +/// Type: Server Request +class InitializeResourceReaderRequest { + /// A number identifying this request. + requestId: Int + + /// The scheme of the resource to initialize. + scheme: String +} + /// Code: 0x102 /// Type: Client Response -class DiscoverResourceReaderResponse { - /// A number identifying this request +class InitializeResourceReaderResponse { + /// A number identifying this request. requestId: Int /// Client-side resource reader spec. From cd20d2e9e935fda70b1c0152ec5adde2b704335f Mon Sep 17 00:00:00 2001 From: Josh Basch Date: Wed, 2 Oct 2024 16:02:54 -0700 Subject: [PATCH 09/10] Add CloseExternalProcess message, update flow --- spices/SPICE-0009-external-readers.adoc | 71 ++++++++++++++----------- 1 file changed, 39 insertions(+), 32 deletions(-) diff --git a/spices/SPICE-0009-external-readers.adoc b/spices/SPICE-0009-external-readers.adoc index cc8c85c..00e6f7a 100644 --- a/spices/SPICE-0009-external-readers.adoc +++ b/spices/SPICE-0009-external-readers.adoc @@ -80,22 +80,22 @@ To avoid terminology confusion with the existing language binding message passin === Overall Flow -1. User requests an evaluator (via Java API, Message Passing API, or CLI) with an external reader. -2. The entrypoint (client Java code, `CliEvaluator`, or `Server`) instantiates an `ExternalProcess` based on the configured external readers. -3. The entrypoint instantiates resource readers (`ResourceReaders.External`) and module key factories (`ModuleKeyFactories.External`) referencing the `ExternalProcess` and uses them to build an evaluator. +. User requests an evaluator (via Java API, Message Passing API, or CLI) with an external reader. +. The entrypoint (client Java code, `CliEvaluator`, or `Server`) instantiates an `ExternalProcess` based on the configured external readers. +. The entrypoint instantiates resource readers (`ResourceReaders.External`) and module key factories (`ModuleKeyFactories.External`) referencing the `ExternalProcess` and uses them to build an evaluator. `ResourceReaders.External` and `ModuleKeyFactories.External` function almost identically to the existing `ClientResourceReader` and `ClientModuleKeyFactory` and pkl-server will be updated to reuse these implementations where possible. -4. Evaluation begins. -5. A resource or module read is encountered. -6. Upon first read only: -1.. The `ExternalProcess` is started, which starts the configured child process. -2.. The child process is sent a `Initialize(Module|Resource)ReaderRequest` message. -3.. A `Initialize(Module|Resource)ReaderResponse` message is received (within some timeout). -7. The appropriate `Read*Request` message is sent to the child process, the corresponding `Read*Response` message is awaited. -8. Evaluation continues. -9. Evaluation completes. -10. The evaluator is closed. -1.. The `ExternalProcess` is closed, the child process is sent SIGTERM. -2.. If the child process has not terminated after some timeout (3 seconds?), it is forcefully stopped via SIGKILL. +. Evaluation begins. +. A resource or module read is encountered. +. Upon first read only: +.. The `ExternalProcess` is started, which starts the configured child process. +.. The child process is sent a `Initialize(Module|Resource)ReaderRequest` message. +.. A `Initialize(Module|Resource)ReaderResponse` message is received (within some timeout). +. The appropriate `Read*Request` message is sent to the child process, the corresponding `Read*Response` message is awaited. +. Evaluation continues. +. Evaluation completes. +. The evaluator is closed. +.. The `ExternalReader` is closed, the child process is sent the `CloseExternalProcess` message. +.. If the child process has not terminated after the set timeout (3 seconds), it is forcefully stopped via SIGKILL. === Java API @@ -113,7 +113,24 @@ The code added to pkl-core will include the new APIs and the core messaging code === Message Passing API -Four new message types will be added: +`CreateEvaluatorRequest` will be expanded with additional properties: +[source,pkl] +---- +externalModuleReaders: Mapping? + +externalResourceReaders: Mapping? + +class ExternalReader { + /// May be specified as an absolute path to an executable + /// May also be specified as just an executable name, in which case it will be resolved according to the PATH environment variable + executable: String + + /// Command line arguments that will be passed to the reader process + arguments: Listing +} +---- + +Five new message types will be added: [source,pkl] ---- @@ -150,7 +167,7 @@ class InitializeResourceReaderRequest { scheme: String } -/// Code: 0x102 +/// Code: 0x103 /// Type: Client Response class InitializeResourceReaderResponse { /// A number identifying this request. @@ -162,24 +179,14 @@ class InitializeResourceReaderResponse { /// [ClientResourceReader] is defined at https://pkl-lang.org/main/current/bindings-specification/message-passing-api.html#create-evaluator-request spec: ClientResourceReader? } ----- -`CreateEvaluatorRequest` will be expanded with additional properties: -[source,pkl] +/// Code: 0x104 +/// Type: Server One Way +class CloseExternalProcess {} ---- -externalModuleReaders: Mapping? - -externalResourceReaders: Mapping? - -class ExternalReader { - /// May be specified as an absolute path to an executable - /// May also be specified as just an executable name, in which case it will be resolved according to the PATH environment variable - executable: String - /// Command line arguments that will be passed to the reader process - arguments: Listing -} ----- +The `CloseExternalProcess` message exists primarily because different operating systems provide different abilities to gracefully stop child processes. +Using an in-band message for this purpose reduces the need for external reader developers to address OS-specific implementation details. === CLI From 46ba8682f8cb398c2701ccbfee2487aa835ee62a Mon Sep 17 00:00:00 2001 From: Josh Basch Date: Thu, 10 Oct 2024 23:09:51 -0700 Subject: [PATCH 10/10] Add ExternalReaderRuntime Java API --- spices/SPICE-0009-external-readers.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/spices/SPICE-0009-external-readers.adoc b/spices/SPICE-0009-external-readers.adoc index 00e6f7a..5eed12b 100644 --- a/spices/SPICE-0009-external-readers.adoc +++ b/spices/SPICE-0009-external-readers.adoc @@ -107,6 +107,7 @@ New APIs: * `ExternalProcess` - manages the lifecycle of child processes. ** Explicit `close` methods to manage child process lifecycle. ** The `ExternalProcess` spawns the subprocess on first access, which sets up the `MessageTransport`, sends the appropriate `Initialize*ReaderRequest` message, and awaits the corresponding `Initialize*ReaderResponse` response. +* `ExternalReaderRuntime` - implements the client-side workflow for external readers. This proposal requires that the message passing API functionality move out of pkl-server and into pkl-core. The code added to pkl-core will include the new APIs and the core messaging code currently part of pkl-server (`pkl-server/src.main/kotlin/org.pkl.server/Message*.kt`). @@ -288,12 +289,11 @@ It's unlikely there are clients of these APIs outside the apple/pkl repo. == Future directions -* Java library for bindings to support being an external reader client * Configuration of external readers via `~/.pkl/settings.pkl` * Support for specifying URIs for external reader executables so they may be distributed in Pkl packages. This is potentially very valuable for statically compiled reader binaries, but significantly complicates the implementation. The design, as proposed, does not prohibit implementing this as a future enhancement. -This would also make it very convenient to provide friendly, type-safe Pkl APIs for complex reader URI schemes instead of having the "stringly-typed" URI as the primary API, e.g. building on the `ldap:` example: +This would also make it very convenient to bundle reader executables inside packages to provide friendly, type-safe, and self-contained Pkl APIs for complex reader URI schemes instead of having the "stringly-typed" URI as the primary API, e.g. building on the `ldap:` example: + [source,pkl] ----