Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

Add Scatter.empty #362

Merged
merged 1 commit into from
Aug 22, 2019
Merged

Add Scatter.empty #362

merged 1 commit into from
Aug 22, 2019

Conversation

nh13
Copy link
Member

@nh13 nh13 commented Aug 22, 2019

so we can have:

Scatter().flatMap { result => if (condition(result)) Scatter(result) else Scatter.empty

Also, see the added tests.

Depends on #360

@nh13 nh13 requested a review from tfenne August 22, 2019 06:35
@nh13
Copy link
Member Author

nh13 commented Aug 22, 2019

Added
1.
Scatter.apply[Result <: Task](task) wraps a Scatter around a single Task for use with flatMap. So we can do scatter.flatMap { item => if (condition(item)) Scatter(toResult(item)) else Scatter.empty. Note that we could implement Scatter.filter as

def filter(f: Source => Boolean): Scatter[Source] = {
  this.flatMap { source => if (f(source)) Scatter(source) else Scatter.empty }
}

Scatter.apply(Iterable) creates a Scatter from an iterable. It was convenient for (1), but also implemented in #359

Copy link
Member

@tfenne tfenne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned in slack, I think you're really redefining the scatter/gather API as a more general "working with collections of tasks" API. Long term I think it's confusing to introduce some of these concepts while maintaining scatter/gather sematics. If we keep pushing in this direction I think we should probably re-write this in terms of a TaskSeq class (which is what Scatter is becoming).

tasks/src/main/scala/dagr/tasks/ScatterGather.scala Outdated Show resolved Hide resolved
tasks/src/main/scala/dagr/tasks/ScatterGather.scala Outdated Show resolved Hide resolved
@codecov-io
Copy link

codecov-io commented Aug 22, 2019

Codecov Report

Merging #362 into master will decrease coverage by 0.07%.
The diff coverage is 50%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #362      +/-   ##
==========================================
- Coverage   92.03%   91.95%   -0.08%     
==========================================
  Files          31       31              
  Lines        1155     1156       +1     
  Branches       66       65       -1     
==========================================
  Hits         1063     1063              
- Misses         92       93       +1
Impacted Files Coverage Δ
...ore/src/main/scala/dagr/core/tasksystem/Task.scala 95.52% <50%> (-1.45%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2b5adf6...b29c32a. Read the comment docs.

@nh13 nh13 changed the base branch from nh_gettasks_none to master August 22, 2019 18:33
@nh13 nh13 merged commit 71136cc into master Aug 22, 2019
@nh13 nh13 deleted the nh_scatter_gather_empty branch August 22, 2019 18:43
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants