-
Notifications
You must be signed in to change notification settings - Fork 85
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove the dependency on Metaflow #82
Milestone
Comments
tdruez
added a commit
that referenced
this issue
Feb 4, 2021
Signed-off-by: Thomas Druez <[email protected]>
tdruez
added a commit
that referenced
this issue
Feb 4, 2021
Signed-off-by: Thomas Druez <[email protected]>
@tdruez Does this mean that most of what we are using from Metaflow is the approach to structuring a Pipeline without really using much Metaflow code? |
@mjherzog Yes, we are basing our Pipelines on the Metaflow Flow structure but inherit from its execution approach (through subprocess) that is not great in the ScanCode.io context (asynchronous tasks). |
tdruez
added a commit
that referenced
this issue
Feb 5, 2021
Signed-off-by: Thomas Druez <[email protected]>
tdruez
added a commit
that referenced
this issue
Feb 5, 2021
Signed-off-by: Thomas Druez <[email protected]>
tdruez
added a commit
that referenced
this issue
Feb 5, 2021
Signed-off-by: Thomas Druez <[email protected]>
tdruez
added a commit
that referenced
this issue
Feb 5, 2021
Signed-off-by: Thomas Druez <[email protected]>
tdruez
added a commit
that referenced
this issue
Feb 5, 2021
Signed-off-by: Thomas Druez <[email protected]>
tdruez
added a commit
that referenced
this issue
Feb 8, 2021
* Remove the dependency on Metaflow #82 Signed-off-by: Thomas Druez <[email protected]> * Unify Pipeline subclasses name #82 Signed-off-by: Thomas Druez <[email protected]> * Move logging to the base Pipeline class #82 Signed-off-by: Thomas Druez <[email protected]> * Add support for exception during Pipeline execution #82 Signed-off-by: Thomas Druez <[email protected]> * Add entry in the CHANGELOG #82 Signed-off-by: Thomas Druez <[email protected]> * Update the profiling method to the new log format #82 Signed-off-by: Thomas Druez <[email protected]> * Replace indent function by built-in Python textwrap.indent #82 Signed-off-by: Thomas Druez <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The benefits from using Metaflow are now outweighed by the issues and lack of customization (logging for example) forcing us into complex workarounds.
Since we do not use much of the Metaflow features (we've only used Metaflow for the structure of the Pipelines), we do not have a lot of code that depends on it.
The required code changes will mostly be needed on the Pipelines side.
Those change will also allow us to have more control over the execution of Pipelines (logging, profiling, branching, ...)
It will remove 1 level of indirection in the execution stack since Metaflow is always run as a subprocess call (it cannot be called within Python by design).
The text was updated successfully, but these errors were encountered: