Allow overwritting action by environment

[mryellow]

No description provided.

[mryellow]

Could do it this way around, looking more like OneStepQAgent.

if taken and taken ~= action then
  action = taken
end
self.actions[self.batchIdx] = action

[Kaixhin]

Code looks fine, but what is this trying to achieve? If the chosen action may not be deterministically executed in the environment, the agent should still treat it as if the chosen action were taken, otherwise it will not adapt to the real stochasticity in the application of its action. For an ALE example see Increasing the Action Gap.

[mryellow]

Expert instruction is one application. Can forget whatever the net came up with and learn from experiences which have been overwritten with expert play, or some hand-crafted strategy.

Last time I used it was forcing robot to repeat turning on the spot actions at random intervals when epsilon was below a threshold, to escape corners and get some better experiences again.

[Kaixhin]

Got it - can you add a short 2nd paragraph to the custom docs to make people aware of this modification from the rlenvs API, along with a use-case as above?

[mryellow]

Yeah no problem.

Describing it just now, my method has always been a hack of discarding the agents selection after doing the unneeded work. Could pass a variable back toward agent.observe() much like terminal is used at the start of the loop but defined at the end, something like next_action. Environment doesn't overwrite the current action, but can specify the next. Skipping the forward pass if it's defined.

Doesn't really matter for my robot use-case as I'm throwing away CPU cycles to run at real-time (10Hz) rather than fast as possible, guess it matters when comes to battery drain on hardware.

For real-time expert play, wonder if that next_action approach would introduce a little too much lag, with the teleop really needing to apply for the action to the current cycle.

[mryellow]

lag

Not a lot happens before back to start of loop again. next_action might be as good as "do it 'now' given what I see this frame". It could work either way.

[mryellow]

Closing in favour of #57

Avoids doing work only to then discard it.

[mryellow]

Was thinking of adding the extra return to validation agents, allowing environment to pass back any change during validation also. However nothing is done with the action apart from acting.

Could add the signature so the variable is already there if it's needed in future...

So far prioritised reply hasn't crashed with the code this way around, firing off the observe methods must play into setting up the distribution stratum at some point.

[mryellow]

Could add the offset actionOffset before returning so it doesn't need adding to compare actionTaken ~= action.

edit: Done

[mryellow]

Think I had some additional documentation changes to do on this. Been distracted working on hardware and getting dev environment fixed up, once that's outta the way will get this cleaned up.

[mryellow]

In the readme can you add a line about how this doesn’t affect validation async/non-async?

Will get to finishing this off soonish.

https://gitter.im/Kaixhin/Atari?at=57cbda0bd52261ec345029ba