Skip to content
This repository has been archived by the owner on Jul 3, 2023. It is now read-only.

Flytekit integration #139

Closed
skrawcz opened this issue Jun 22, 2022 · 1 comment
Closed

Flytekit integration #139

skrawcz opened this issue Jun 22, 2022 · 1 comment
Labels
enhancement New feature or request product idea
Milestone

Comments

@skrawcz
Copy link
Collaborator

skrawcz commented Jun 22, 2022

Is your feature request related to a problem? Please describe.
We should be able to create a flyte workflow from Hamilton functions. i.e. recreate:

from flytekit import task, workflow

@task
def sum(x: int, y: int) -> int:
   return x + y

@task
def square(z: int) -> int:
   return z * z

@workflow
def my_workflow(x: int, y: int) -> int:
   return sum(x=square(z=x), y=square(z=y))

print(f"my_workflow output: {my_workflow(x=1, y=2)}")

in a Hamiltonesque way.

Describe the solution you'd like
We should be able to recreate the above by doing something like this:

# my_funcs.py
def square_x(x: int) -> int:
       return x * x

def square_y(y: int) -> int:
       return y * y

def sum_square_x_y(square_x:int, square_y: int) -> int:
       return square_x + square_y
from hamilton import driver
from hamilton.experimental import h_flyte
import my_funcs

fga = h_flyte.FlyteGraphAdapter(...)
dr = driver.Driver({}, my_funcs, adapter=fga)
result = dr.execute(["sum_square_x_y"], inputs={"x": 1, "y": 2})
print(result)

Describe alternatives you've considered
TBD.

Additional context
Docs:

This feels very similar to https://www.prefect.io/ v2.0 -- so maybe whatever pattern we come up with here would also help provide integration there.

@skrawcz skrawcz added enhancement New feature or request product idea labels Jun 22, 2022
@skrawcz skrawcz added this to the July 2022 milestone Jul 6, 2022
ramannanda9 added a commit that referenced this issue Nov 15, 2022
See #139.

The GraphAdapter treats a node as a PythonFunctionTask and delegate the execution to execute function in PythonFunctionTask.
This way hamilton functions can be executed in Flyte.
ramannanda9 added a commit that referenced this issue Nov 17, 2022
See #139.

The GraphAdapter treats a node as a PythonFunctionTask, it adds the node to an ImperativeWorkflow in flyte.
The execution of workflow is performed during build_output.
This way hamilton functions can be executed in Flyte runtime.

We end up adding a PandasSeriesTransformer as that is required for using PandasSeries as function outputs and inputs of flight task nodes.

rebase
ramannanda9 added a commit that referenced this issue Jan 6, 2023
See #139.

The GraphAdapter treats a node as a PythonFunctionTask, it adds the node to an ImperativeWorkflow in flyte.
The execution of workflow is performed during build_output.
This way hamilton functions can be executed in Flyte runtime.

We end up adding a PandasSeriesTransformer as that is required for using PandasSeries as function outputs and inputs of flight task nodes.

Any customtype that is not a dataclass, native type or has support already in flyte will raise a ValueError

Adds FlyteKitGraphAdapter tests

Adds tests to show unsupported types
ramannanda9 added a commit that referenced this issue Jan 6, 2023
See #139.

The GraphAdapter treats a node as a PythonFunctionTask, it adds the node to an ImperativeWorkflow in flyte.
The execution of workflow is performed during build_output.
This way hamilton functions can be executed in Flyte runtime.

We end up adding a PandasSeriesTransformer as that is required for using PandasSeries as function outputs and inputs of flight task nodes.

Any customtype that is not a dataclass, native type or has support already in flyte will raise a ValueError

Adds FlyteKitGraphAdapter tests

Adds tests to show unsupported types
@elijahbenizzy
Copy link
Collaborator

elijahbenizzy commented Feb 26, 2023

We are moving repositories! Please see the new version of this issue at DAGWorks-Inc/hamilton#73. Also, please give us a star/update any of your internal links.

Note that everything else (slack community, pypi packages, etc...) will not change at all.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request product idea
Projects
None yet
Development

No branches or pull requests

2 participants