Softmax

[torms3]

Problem

Currently softmax is applied in the frontend. There is even a separate function run_softmax in the forward.py. This is super ugly. We do have softmax_edges in the backend, but this is nearly useless because we always need to use the softmax in conjunction with the cross-entropy loss when training. Because of this, when running forward.py, it applies softmax if the type of loss is softmax_loss. But what if we want to output the scores (i.e. before applying softmax) instead of the probabilities? There is no way except for manually changing the type of loss. Super ugly.

Solution

1. (quick, easy but not intuitive) Differentiate the behavior of softmax_edges, depending on the phase (TRAIN/TEST). In TRAIN, it's equivalent to dummy_edges. In TEST, it does its job. Remove run_softmax.
2. (general,intuitive but takes time) Implement general flow graph. softmax_loss in the topmost part should be able to handle both TRAIN and TEST.

Priority

This is not urgent.