Technical Challenges to Scale Beyond GPT4 to 100K H100s
by Brian Wang from NextBigFuture.com on (#6STCR)
Up until late 2024, no one has been able to massively increase the amount of compute dedicated to a single model beyond the OpenAI GPT 4 model level. This information is from semianalysis and EIA. Google's Gemini Ultra, Nvidia Nemotron 340B, and Meta LLAMA 3 405B had similar or slightly more compute than GPT-4, but ...