Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

在edvardhua/single-pose docker环境里,训练cpm模型,训练的速度特别慢 #132

Open
cncowboy opened this issue Apr 16, 2020 · 1 comment

Comments

@cncowboy
Copy link

cncowboy commented Apr 16, 2020

配置项只改了下面两项,
visible_devices: '0, 1'
multiprocessing_num: 24
因为:我的主机是2块2080ti显卡,cpu是2颗12核
可是我训练速度很慢,cpu, gpu, 内存利用率都很低

如果我只用一张gpu卡,速度反而加快了,

不知道为什么,各位大牛,帮帮忙!谢谢

@cncowboy
Copy link
Author

查看cpu、gpu的使用情况,发现:
1、在gpu开始训练前,只有一个cpu核在预加载数据,利用率近100%
2、在预加载第一批数据后,gpu开始训练,只配了一张卡,利率用近50%,此时cpu全空闲
3、第一批预加载的数据训练完,就进入到cpu加载一个batchsize,然后gpu训练一个batchsize的阶段,此时,cpu,gpu利用率超低,估计不是并行的

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant