Whale：统一多种并行化策略的分布式深度学习框架.pdf

编号：29537

PDF 19页 1.72MB 下载积分：VIP专享

下载报告请您先登录！

Whale：统一多种并行化策略的分布式深度学习框架.pdf

1、A Unified Distributed Training Framework Whale Ang Wang wangang.waalibaba- PAI, Alibaba Cloud 15/12/2020 Motivation 25117340 1500 11000 175000 0 50000 100000 150000 200000 Parameters(M) 8 16 32 80 0 10 20 30 40 50 P4P100V100A100 Memory (GB) 1 Models are getting larger and more complex Larger models

2、lead to better results with lower validation perplexities Model size grows far beyond upgrading of hardware 1 https:/ Models are getting larger Data Parallel(DP) is widely used in distributed training as it is simple and easy to implement. DP is not always optimal for every distributed training work

3、loads. Necessary to find an efficient parallel strategy that can make full use of the resources and speedup the training. Motivation Distribute the training workload with data parallelism Data parallelism becomes less optimal for lots of distributed workloads Motivation Difficult to increase the batch size in a single GPU device due to the limitation of the GPU device memory capacity. Large weight

友情提示

1、下载报告失败解决办法
2、PDF文件下载后，可能会被浏览器默认打开，此种情况可以点击浏览器菜单，保存网页到桌面，就可以正常下载了。
3、本站不支持迅雷下载，请使用电脑自带的IE浏览器，或者360浏览器、谷歌浏览器下载即可。
4、本站报告下载后的文档和图纸-无水印,预览文档经过压缩，下载后原文更清晰。

本文（Whale：统一多种并行化策略的分布式深度学习框架.pdf）为本站（X-iao）主动上传，三个皮匠报告文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若此文所含内容侵犯了您的版权或隐私，请立即通知三个皮匠报告文库（点击联系客服），我们立即给予删除！

温馨提示：如果因为网速或其他原因下载失败请重新下载，重复下载不扣分。