Multi-GPU #2114
Multi-GPU #2114
Conversation
|
Hi! I had tried this branch, and when I tried to using the gpus.bin work on Mnist, it failed. I input command: the error information like this: |
|
That's strange, it seems to be during shutdown when destructors get called. Did it train at all before reaching the end? |
|
it didn't print the log information such as network configuration , training/testing loss, and testing accuracy at the screen. But when I call the nvidia-smi after start the gpu.bin, I can watch it really enable the GPU 0 and GPU 1 to work, the GPU monitor information like this: |
|
GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | |
|
So from the above monitor information, I guess the training should started, but I don't know how the error occured and also how to fixed it. Hope you can give me suggestions. Thanks! |
|
When I attempt to add debug code to watch what's happened for this error: The changed code: } |
|
If it does not print any log, it might be glog config: |
|
It crashes if I try more than 4 cards at a time. Any idea? *** SIGSEGV (@0x30) received by PID 4086 (TID 0x7ffb40f6f720) from PID 48; stack trace: *** To provide more information, on the server, I have 8 GPUs. The training ran fine with with GPUs 0:1:2:3. However, if I used 0:x for x from 4 to 7, I had this error message: "Check failed: access GPU 0 cannot access GPU x". It crashed if I used any GPU from 4 to 7 for training. |
|
I seem to figure out the problem. In line 69 of gpus.cpp, "params[gpu[0]]" should be "params[0]". Similarly, in line 76 of gpus.cpp, "params[device]" should be "params[i]". The segmentation fault was caused by the wrong indexing of the solvers. |
|
@hhuytho |
|
I used the "multi_gpu" branch from cypof. Your problem doesn't seem to relate to the bug I listed. Maybe it has something to do with the peer to peer communication between the GPUs? |
|
@cypof Thanks for this PR! I tried it on 2 K80s (4 GPUs) with the multi_gpu branch, it failed to converge on MNIST data. It runs fine, no crashes. caffe.bin in the same branch converges. I think there is something wrong in gpus.cpp codes. BTW, I used mnist on lmdb, cudnn r1 enabled. Am I missing anything? |
|
New version, should be more stable and faster. The gradients are now combined using a map-reduce scheme, first the GPUs on the same board, then same CPU socket, then to the root. It has not been tested a lot but seems to work OK. The batch size and gradients are scaled, so that multi-gpu should now be exactly equivalent to running on a single one. I tried comparing the weights, and they are equal at least to 10e-5. The difference must be numerical approximations. To make runs reproducible it is necessary to set PREFETCH_COUNT to 1 in data_layer.hpp, otherwise examples might be reordered. I also added a bunch of benchmarking logs that can be enabled through constants in Makefile.config. The code is in caffe.cpp instead of samples, and there is no parallel solver class anymore. Instead it adds a callback on the solver created by caffe.cpp, so it should work with advanced algo like AdaGrad etc., we haven’t tried. I also did some refactoring in solver.cpp, but not changed anything functionally. We haven’t ran all the way on ImageNet yet but early benchmark seems to show using the default batch sizes and settings: AlexNet GoogLeNet I don’t know if we can get much more through data parallelism. 16 bits float transfers might help a bit. I hope hybrid models using #2219 can go further. Larger batches get better GPUs utilisation, but lower SGD convergence. On AlexNet the batch is already 256 and increasing seems hurt convergence a lot. On GoogLeNet going from 32 to 128 should be OK but we have no numbers yet. |
|
If you get an error about nvidia-ml, install the GDK: https://developer.nvidia.com/gpu-deployment-kit, also this currenly only works on CUDA-7. |
|
I think you can get around needing the GDK for your specific case of detecting multiGPU boards using direct device queries through the runtime. Something like: |
| } | ||
| #ifdef BENCHMARK_SOLVER | ||
| LOG(INFO)<< timing.str(); | ||
| #endif | ||
|
|
||
| // Save a snapshot if needed. | ||
| if (param_.snapshot() && (iter_ + 1) % param_.snapshot() == 0) { |
There was a problem hiding this comment.
you might need to add "&& Caffe::root_solver()" here
|
Thanks @cypof for this great PR! I've tried it on 2 and 4 gpus machines, local and ec2, with and without p2p access, on different nets, etc... it seems to be pretty stable so far... The speed up factors also seem to match with your numbers.... |
|
Thanks @thatguymike @sontran for checking this out. I tried the built-in CUDA API before switching to GDK but it seems to return 0 even if the GPUS are on a board. Maybe I was mistaken, I will look again. I hope the GDK will be merged into CUDA, there is lots of useful stuff in there. In terms of architecture, it seems on some machines doing a all-to-all is faster than map-reduce. I'm think it is possible to write a simple code that does both, always reducing on dual-gpu cards then all-to-all. I will probably stop there for single node, unlikely to gain much more this way. |
|
@bhack as long as you can memcpy between two GPUs it should work. For performance you also want to have a memcpyAsync. Not sure what is provided by OpenCL. |
|
Getting scaling performance really requires optimal data transfers. The GDK provides methods to get system topology and information on what GPUs can do direct DMAs. (I think we can replace those with standard CUDA calls, working on that now). lspci/hwloc might be usable to get system topology, but I don't believe there is a standard way in OpenCL to setup and manage P2P DMA transfers between devices. |
|
With respect to needing the GDK, here is work-alike for seeing if 2 ASICs are on the same board. Also, one can parse the output of 'nvidia-smi topo -m' to get more topology information: // Group GPUs by board
for (int i = 0; i < remaining.size(); ++i) {
cudaDeviceProp prop_a;
CUDA_CHECK(cudaGetDeviceProperties(&prop_a, remaining[i]));
for (int j = i + 1; j < remaining.size(); ++j) {
cudaDeviceProp prop_b;
CUDA_CHECK(cudaGetDeviceProperties(&prop_b, remaining[j]));
int same_board = 0;
if(prop_a.isMultiGpuBoard && prop_b.isMultiGpuBoard)
same_board = prop_a.multiGpuBoardGroupID == prop_b.multiGpuBoardGroupID ? 1 : 0;
if (same_board) {
pairs->push_back(DevicePair(remaining[i], remaining[j]));
DLOG(INFO) << "\n\nSame board: " << remaining[i] << ":" << remaining[j] << " - Board ID:" << prop_a.multiGpuBoardGroupID;
remaining.erase(remaining.begin() + j);
}
}
} |
|
I think that p2p will be covered by hsa initiative of hsa foundation chipmakers. |
|
@cypof Could you show GPU topology on 4 gpus machines? |
|
This depends on the general system topology. The code currently tries to align the tree reduction strategy with the system. For example, if you have a dual GPU board, the code attempts to do the first reduction on the board, and then one of those ASICs communicates with one ASIC on the other dual GPU board. If you have 4 independent GPUs, it tries to find P2P pairs, but the current code doesn't go much deeper into the topology trying to find host PCIe bridges (like PLX). But, it will attempt to avoid crossing PCIe root complexes, where P2P doesn't work, until the top of the reduction tree. Basically, the code tries to optimize bandwidth use at each level in the reduction tree. It should do the right thing for most 4 GPU systems. For more complex server systems with PLX bridges or multiple root complex, there is more tuning that needs to be done. However, the primary issue in the code at the moment is that there is a race somewhere and even with fixing the incoming random seed, we get different results run to run. This can lead to things failing to converge. |
|
Sorry for the previous comment, there something not clear, so I add this comment to correct it. I had 4 gpus to run current multi-gpus branch, however I only could run the gpu 0 and 1 successfully, if I try to run the gpu 2 and 3, caffe will fail, the fail information: I feel it out of my expectation, because I can run the gpu 2 and 3 in single gpu mode and also I can run on the gpu 0 and 1 at the same time, I think there should no reason can't run on gpu 2 and gpu 3 at the same time. @sontran |
|
@wkal I see you are running the gpus.bin executable. The code is now in caffe.bin itself so you might be running an old version. Try a make clean, and run caffe train instead. |
|
NVIDIA=CUDA? |
|
@bhack @cypof |
|
The same techniques should generally work. However, there are going to be some issues with getting good performance, notably no standard way to do P2P DMA transfers between devinces and being able to walk the topology of the PCIe setup. Without P2P DMA, you get communication bound VERY quickly having to copy through host. Walking the topology to figure out who can talk to each other at high speed is also part of this. Multi-device support in OpenCL is a little trickier generally to deal with, at least if you need tight communication and synchronization between devices. |
|
Yes, I worked on the AMD extension many years ago and was a member of the committee for many years. ;-) My point was there is no standard way in OpenCL to do this. The memory model is a little tight and you can see some of the issues crop up when using the extension, specifcally around map/unmap and synchronization on buffers between multiple devices and the host. Again, all of the work @cypof and I did on this PR should functionally port to OpenCL. |
|
@thatguymike It is easy to create universal features on hardware where you are the only vendor ;) |
|
@thatguymike |
|
This is very close to merge and our only reservation is how to refactor 346e262 for clarity since the use of member variables and the naming of |
|
Does anyone else have the issue of loss blowing up after few thousand iterations? I trained on ImageNet with 2 GeForce 980 GPUs, initially the losses were normal, decreasing from 7.xx then suddenly it blew up to ~87 after ~20k iterations. The batch size was 128 and the learning rate, momentum, stepsize etc were same as the bvlc_reference_caffenet solver. I had to resume from 10k and it trained better again. This is not a one time thing, it happened to me at least 4-5 times, and it has been reproduced in another machine as well (with Titan Xs). |
|
@sri-25 I haven't seen an issue like this. Just to be sure (since you didn't mention it in your post), the network you're training works on a single GPU with twice the batch size? |
|
See #2870 for my packaging of this PR for merge -- the history preserves authorship and I only addressed my concerns about |
|
@cdoersch Yeah, it trains well on a single GPU (with 2x batch size). FYI, I'm training on ImageNet with bvlc_reference_caffenet solver/train_val. |
| @@ -82,7 +96,7 @@ class GradientBasedSolverTest : public MultiDeviceTest<TypeParam> { | |||
| " value: 1.0 " | |||
| " } " | |||
| " data_filler { " | |||
| " type: 'gaussian' " | |||
| " type: '" << (run.const_data ? "constant" : "gaussian") << "' " | |||
There was a problem hiding this comment.
The targets top needs to always be Gaussian or else the test is ineffective. The constant vs. gaussian condition should apply to the filler for data. Switching this reveals that the CheckAccumulation test is broken since it doesn't set constant data. More ominously, it reveals that all the TestLeastSquares* fail for multi-GPU. Let's follow up on #2870.
|
Closing since this is continued on in #2870. |
|
@thatguymike I replace the tensor RT 2.x with the version 3.x and cudnn 7.x but the caffe Makefile.config can not find the cudnn now . Did you meet that similarity questions? or other suggestion?,thank you |
Uses CUDA peer-to-peer for communication, and parts of #1148. SGD is now synchronous instead of asynchronous, as @longjon showed bandwidth on one box is actually high enough. We haven’t really benchmarked yet, but it seems to work great. It also gets rid of the momentum coordination problem.
The synchronization code needs to hook into the solver, so it is a bit more invasive than before, but still pretty isolated. I refactored solver.cpp to separate the regularization and gradient compute phases so that they can be invoked at different times by the parallel solver.
One thing still missing is the way to compute the actual number of iterations. For now each solver runs as if it was by itself, so the run is going to take as long as without parallelism. I guess we could adapt the solver to run 1/N steps instead. Also the batch size should be experimented with, as now effectively N times larger. On that, would it be more convenient to switch to the number of images to compute progress, instead of iterations, to be independent of batch size?
To try it, run the samples in example/parallel/