When 27,000 GPUs Make You Miserable
·45 words·1 min
If you were complaining about compute infra, this will make you miserable: ORNL now has “super computer” w/ 27000 V-100s yielding 2 EFLOPs! Can algos leverage this scale? Upto 6000 GPUs things seem meaningful, beyond that not so much (see fig 9 inset): https://arxiv.org/abs/1909.11150