This repository has been archived by the owner on Jul 3, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 38
Flytekit integration #139
Labels
Milestone
Comments
ramannanda9
added a commit
that referenced
this issue
Nov 15, 2022
See #139. The GraphAdapter treats a node as a PythonFunctionTask and delegate the execution to execute function in PythonFunctionTask. This way hamilton functions can be executed in Flyte.
ramannanda9
added a commit
that referenced
this issue
Nov 17, 2022
See #139. The GraphAdapter treats a node as a PythonFunctionTask, it adds the node to an ImperativeWorkflow in flyte. The execution of workflow is performed during build_output. This way hamilton functions can be executed in Flyte runtime. We end up adding a PandasSeriesTransformer as that is required for using PandasSeries as function outputs and inputs of flight task nodes. rebase
ramannanda9
added a commit
that referenced
this issue
Jan 6, 2023
See #139. The GraphAdapter treats a node as a PythonFunctionTask, it adds the node to an ImperativeWorkflow in flyte. The execution of workflow is performed during build_output. This way hamilton functions can be executed in Flyte runtime. We end up adding a PandasSeriesTransformer as that is required for using PandasSeries as function outputs and inputs of flight task nodes. Any customtype that is not a dataclass, native type or has support already in flyte will raise a ValueError Adds FlyteKitGraphAdapter tests Adds tests to show unsupported types
ramannanda9
added a commit
that referenced
this issue
Jan 6, 2023
See #139. The GraphAdapter treats a node as a PythonFunctionTask, it adds the node to an ImperativeWorkflow in flyte. The execution of workflow is performed during build_output. This way hamilton functions can be executed in Flyte runtime. We end up adding a PandasSeriesTransformer as that is required for using PandasSeries as function outputs and inputs of flight task nodes. Any customtype that is not a dataclass, native type or has support already in flyte will raise a ValueError Adds FlyteKitGraphAdapter tests Adds tests to show unsupported types
We are moving repositories! Please see the new version of this issue at DAGWorks-Inc/hamilton#73. Also, please give us a star/update any of your internal links. Note that everything else (slack community, pypi packages, etc...) will not change at all. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Is your feature request related to a problem? Please describe.
We should be able to create a flyte workflow from Hamilton functions. i.e. recreate:
in a Hamiltonesque way.
Describe the solution you'd like
We should be able to recreate the above by doing something like this:
Describe alternatives you've considered
TBD.
Additional context
Docs:
This feels very similar to https://www.prefect.io/ v2.0 -- so maybe whatever pattern we come up with here would also help provide integration there.
The text was updated successfully, but these errors were encountered: