The relentless march towards full self-driving autonomy rages on, and AI pioneers like Nvidia’s Jim Fan are dropping tantalizing hints about what could be Tesla’s next big breakthrough – integrating powerful multimodal reasoning capabilities into FSD.
According to Jim Fan, a senior research manager leading Nvidia’s embodied AI efforts, Tesla’s future FSD v13 update “will likely be grokking language tokens.” What does that mean exactly? He’s referring to xAI’s unveiled Grok-1.5V model, which can process all kinds of visual data like documents, diagrams and photos alongside text.
The true potential, speculates Jim Fan, is leveraging Grok’s unique skills to solve tricky “edge cases” that frequently stump self-driving systems. By lifting “pixel->action mapping to pixel->language->action,” Grok could empower FSD to “break down complex scenarios, reason with rules and counterfactuals, and explain its decisions” with unprecedented nuance.
Of course, for all of Grok’s multimodal talents, actually integrating and fine-tuning the large model for production automotive use would be no simple feat. Or would it?
Jim Fan posits that “with Tesla’s highly mature data pipeline, it is not hard to label tons of edge cases with high-quality human explanation traces and finetune Grok to be far better than GPT-4V…for multimodal FSD reasoning.”
The tantalizing implication? By leveraging its already unrivaled real-world training dataset, Tesla could rapidly give Grok an “unfair advantage” in practical autonomous driving skills over more generic large language model competitors. An enticing prospect for sure.
There is one potential stumbling block, however. Jim Fan wonders whether an “edge deployed” model like Grok might exceed the compute constraints of Tesla’s on-vehicle hardware. His solution? A “specially trained multi-modal small language model for reasoning” could preserve those advanced skills within FSD’s tighter silicon budget.
Speculation aside, one thing is clear – the race to harmonize cutting-edge multimodal AI with self-driving vehicles has officially begun. And Tesla may have a powerful new weapon waiting in the wings.