Remove memory scope and improve memory management

[zachgk]

The MemoryScope reveals a number of shortcomings within the DJL memory
management. While the MemoryScope is deleted, many of them are fixed as part of
this PR.

First, the NDManager.{attach, detach} were renamed to xxxInternal. This is to
differentiate them from the attach and detach methods that are intended to be used.

There are two new concepts in memory management. An NDResource interface was
created to combine the concepts of managed memory that was used in NDArray and
NDList. It could also be used in more classes in the future. This includes the
getManager, attach, and detach operations.

Within the NDManager, it gains a second "management convention". The first
convention of normal resources are added to the manager and then closed when the
manager closes. This works for small numbers of things on the NDArray, but not
when operations transitively create. So, the second convention is a
tempResource. Instead of freeing them when the manager is closed, they are
returned to their original manager. This is used to create a temporary scope, do
operations within it, and then the inputs and return value are returned to the
parent while the intermediate work is cleaned. This also matches the concepts of
ownership/borrowing as well.

Using these, a few additional helper methods were created. There is
NDManager.from(resource) to ease creation of managers based on a resource.
There is also scopeManager.ret(returnValue) to help with returning values
outside of the scopeManager. Lastly, there is a scopeManager.{temp,}AttachAll
to attach a number of resources to a manager within a single call.

Using these improvements, the new method were applied to the old locations where
MemoryScope was used as well as an additional case in NDManagerEx.

Also, the old attach methods were altered to be void. Because the return
values are no longer used anywhere and are not as necessary in the current
scheme, I figured it would simplify things. It also helps for things like
NDList.attach which does not have a single original NDManager when attaching.

[stu1130]

I also don't like attach and detach since it confuses users with attachGradient and detachGradient

[zachgk]

As in the names attach and detach? Yeah, I didn't change that in this PR (it is already long enough), but maybe in a following one. Is there a different name you want to propose? We should probably rename attachGradient too because it's not a great name either.

Also, @stu1130 are we supposed to have a detachGradient method because I don't see it

[frankfliu]

Just replace MemoryScope with NDList should work, doesn't see the need of changing NDManager.

[stu1130]

detachGradient

I think we call stopGradient

[zachgk]

@frankfliu I found it somewhat difficult working with the NDList. The main goal which I saw was that the NDArrays (and NDLists) had to be moved into a new list temporarily and then moved back afterwards. I guess it would be possible to do it by moving them into an NDList, recording all of the previous managers, and then reattaching back to the previous managers manually.

My issue with this is that it seems too low leveled of memory operations. The actions you are taking don't match up very well with intent and it is easy to forget to move all of the NDArrays back. This could be the source for non-stop bugs in the future the same way as segfaults in C.

The second issue lies in the number of items. In the NDManager with tempAttach, there is 1 variable to manage: the new scope. And with a try-with-resources, it closes automatically which is most of the problem. Using pure NDLists, you have three: new scope, NDList of variables to return, and the list of managers to return them to. This increases linearly with the number of new scopes as new variable names users have to create and manage.

[lanking520]

what makes difference on tempAttachInternel comparing to the method attachInternal?

[lanking520]

@zachgk please do a poll on naming

[codecov-io]

Codecov Report

Merging #695 (5ab673e) into master (48cf663) will increase coverage by 0.10%.
The diff coverage is 26.50%.

@@             Coverage Diff              @@
##             master     #695      +/-   ##
============================================
+ Coverage     68.95%   69.06%   +0.10%     
- Complexity     3966     3976      +10     
============================================
  Files           462      461       -1     
  Lines         18634    18632       -2     
  Branches       1998     1996       -2     
============================================
+ Hits          12850    12868      +18     
+ Misses         4753     4748       -5     
+ Partials       1031     1016      -15     

Impacted Files	Coverage Δ	Complexity Δ
api/src/main/java/ai/djl/ndarray/NDArray.java	74.25% <ø> (ø)	89.00 <0.00> (ø)
...i/src/main/java/ai/djl/ndarray/NDArrayAdapter.java	0.00% <ø> (ø)	0.00 <0.00> (ø)
...src/main/java/ai/djl/nn/transformer/BertBlock.java	46.20% <0.00%> (+0.29%)	8.00 <0.00> (ø)
...l/nn/transformer/BertMaskedLanguageModelBlock.java	35.48% <0.00%> (-1.19%)	2.00 <0.00> (ø)
...jl/nn/transformer/BertMaskedLanguageModelLoss.java	17.85% <0.00%> (ø)	1.00 <0.00> (ø)
...va/ai/djl/nn/transformer/BertNextSentenceLoss.java	18.18% <0.00%> (ø)	1.00 <0.00> (ø)
...va/ai/djl/nn/transformer/BertPretrainingBlock.java	42.50% <0.00%> (+1.03%)	2.00 <0.00> (ø)
...ain/java/ai/djl/onnxruntime/engine/OrtNDArray.java	0.00% <0.00%> (ø)	0.00 <0.00> (ø)
...n/java/ai/djl/onnxruntime/engine/OrtNDManager.java	0.00% <0.00%> (ø)	0.00 <0.00> (ø)
...ain/java/ai/djl/paddlepaddle/engine/PpNDArray.java	35.55% <12.50%> (-3.47%)	8.00 <0.00> (ø)
... and 24 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 48cf663...5ab673e. Read the comment docs.