Executives in the AI sector are reassessing the traditional approach of using vast amounts of data to train large language models (LLMs). Industry leaders from companies like OpenAI, Meta, and Google are exploring more efficient training methods as concerns grow about the limitations of current scaling practices.
Historically, the assumption has been that more data leads to smarter models. However, experts like Alexandr Wang, CEO of Scale AI, highlight that the industry is now questioning whether this scaling law will continue to hold. Aidan Gomez, CEO of Cohere, describes the current method as 'mindless,' advocating for smaller models that are both cost-effective and efficient.
Richard Socher, former Salesforce executive, suggests an alternative training method that involves translating questions into computer code, which could reduce inaccuracies and enhance model capabilities. While some believe the industry is approaching a scaling wall, Microsoft CTO Kevin Scott argues that significant returns on scale are still possible.
OpenAI's recent release, o1, aims to improve upon existing models by focusing on quantitative questions, although it requires more computational power and is slower than its predecessor, ChatGPT. The ongoing evolution in AI training methods reflects a critical shift in the industry as it seeks to advance toward more intelligent systems.