Dynamic scheduling for parallel_for and parallel_reduce

[mrtupek]

We've found that some of our loops are faster using dynamic scheduling in openmp.

[mhoemmen]

See issue 100:

#100

I'm not sure how to mark issues as duplicates of another issue, or I would do that now :-)

[hcedwar]

Dynamic scheduling options for both RangePolicy and TeamPolicy

[hcedwar]

Adding optional template parameter to policies to specify dynamic scheduling. Policy templates should be switched to variadic arguments.

[crtrott]

How about Kokkos::ScheduleDynamic or Kokkos::Schedule < Kokkos::Dynamic >

[nmhamster]

Prefer Kokkos::Schedule < Kokkos::Dynamic >

[crtrott]

Ok this is the syntax I am implementing right now (starting with the RangePolicy):

ExecutionPolicy< class ... Traits >

where Traits can be any or none of the following

ExecutionSpace (some Kokkos execution space)
WorkTag (some user defined class)
Kokkos::ScheduleKokkos::Dynamic , Kokkos::ScheduleKokkos::Static
Kokkos::ChunkSize
Kokkos::IterationType<INT_TYPE>

deprecated: INT_TYPE (i.e. some raw integer type as a direct template argument)

So someone could really customize an execution policy like this:

RangePolicy<InitialIntegrate, OpenMP, Schedule, ChunkSize<32>, IterationType >

Right now order of arguments doesn't matter, but only one of each type is accepted (it static asserts with a helpful message if you for example try RangePolicy<OpenMP,Serial>).

I am pretty much done with RangePolicy accepting these arguments, but Schedule doesn't do anything yet. The other ones doe the same thing as the equivalent before.
This lives on the experimental branch.

[crtrott]

One more comment: I will strongly advocate for Schedule, and ChunkSize to have only hint character. Otherwise people can do all kinds of crazy things. For example if you are guaranteed a ChunkSize I think it is possible to write a legal Kokkos::parallel_for which is not vectorizable (effectively you know that successive Chunk items will not collide). While that has its advantages, I think the loss in freedom to map iterations to hardware is far worse. So I rather would say those things are hints, and force people to write portable code. Static for example has an issue because on GPUs there is no true static execution since the hardware has a dynamic scheduler build in.

[hcedwar]

Use the term "properties" for this kind of template parameters:
RangePolicy< class ... Properties >
The term "traits" is already in the C++11 meta-programming vernacular.

[crtrott]

Just noticed I had done that already in the actual implementation on experimental.
I also changed ChunkSize to become a runtime argument, so that one could adapt the chunk size to the input deck. I.e.:

Kokkos::RangePolicy<Kokkos::Schedule<Kokkos::Dynamic> >(0,N, Kokkos::ChunkSize(32) )
// This uses all options including TagType in order to call operator(MyCustomTag, int)
Kokkos::RangePolicy<Kokkos::Schedule<Kokkos::Dynamic>, 
                                   Kokkos::IterationType<int>,
                                   Kokkos::OpenMP,
                                   MyCustomTag >(0,N, Kokkos::ChunkSize(32) )

[crtrott]

Would a better word for IterationType maybe be IndexType. It is the integer type the policy will internally use to provide indicies to the functor. This includes stuff like team_rank, league_rank, league_size team_size etc. when we extend the PolicyTraits to be used for the TeamPolicy.

[nmhamster]

I prefer IndexType, seems to me to be more logical.

[crtrott]

I actually think the same.

[crtrott]

This is now in master using setter functions for runtime parameters.