-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bugs in optimization #137
Comments
Good catch. Yeah, because shuffle has a target, it's behavior is somewhat I like the idea of having the shuffle be a dependency of the target. Maybe
The user would have to remember to use the output of the shuffle instead of |
Yes, if we can return the final target as the result of the shuffle expr, 2014-04-21 15:22 GMT-04:00 Russell Power [email protected]:
Chen Qi |
The simplest way to solve this question is to change Russell's method seems the most general way to solve the issue. Another idea is to add a variable |
Double checking, it looks like shuffle returns
The behavior of shuffle is tricky, because it doesn't quite fit into the
But I can't say off the top of my head. I'll send a pull request to fix the visit for ndarrayexpr. |
When I try to add optimization into some applications, I find that there are something wrong for map and shuffle fusion. The example is as follows:
In [5]: x = expr.ndarray((10,), dtype=np.float, reduce_fn=np.add)
In [6]: y = expr.ones((10,10))
In [7]: expr.shuffle(y, lambda array, ex: [(extent.index_for_reduction(ex, axis=1), array.fetch(ex).sum(axis=1))], target=x).optimized().force()
Out[7]: DistArrayImpl(id=75153680, shape=(10,), dtype=float64)
In [8]: x.glom()
Out[8]:
array([ 6.92288859e-310, 6.92288859e-310, 6.92288859e-310,
6.92288859e-310, 6.92288859e-310, 6.92288859e-310,
6.92288859e-310, 6.92288859e-310, 6.92288859e-310,
6.92288859e-310])
In [9]: expr.shuffle(y, lambda array, ex: [(extent.index_for_reduction(ex, axis=1), array.fetch(ex).sum(axis=1))], target=x).force()
Out[9]: DistArrayImpl(id=75154064, shape=(10,), dtype=float64)
In [10]: x.glom()
Out[10]: array([ 10., 10., 10., 10., 10., 10., 10., 10., 10., 10.])
I think when we try to set a target for shuffle, we cannot fusion the target create map with this shuffle since we will use this target in the future. Moreover, can we add this shuffle expr as a dependency of expr that uses this target? Then we will not need to force it each time we use this kind of shuffle.
The text was updated successfully, but these errors were encountered: