Apple has revealed very little about its efforts in the AI space. Because Siri is so unhelpful, the company appears to be lagging behind OpenAI, Microsoft, and Google in AI research—but it seems its efforts are actually more advanced than we might imagine. Let's dig into two papers from the company that have recently come to light.
Making on-device AI processing easier to achieve
Apple is working on a breakthrough technology that would allow iPhones and other Apple devices to run powerful AI models on-device without sacrificing speed or performance. The secret lies in using flash memory—the same memory used to store apps and photos—to run large language models (LLMs) capable of understanding and generating natural language.
LLMs are the backbone of AI chatbots and assistants like Apple's Siri, which can converse with humans and provide useful information and services. However, LLMs consume massive amounts of memory, requiring gigabytes of RAM to function properly. This poses a challenge for mobile devices like the iPhone, which have limited RAM capacity.
To overcome this challenge, Apple's researchers devised new techniques that leverage flash memory—which is more abundant and cheaper than RAM—to store and access LLM data. In a new research paper titled "LLM in a flash: Efficient Large Language Model Inference with Limited Memory," the research team describes two key innovations that make this possible.
The first is windowing. This recycling technique reduces the amount of data that needs to be transferred between flash memory and RAM. Instead of loading new data every time, the LLM reuses portions of data that have already been processed, saving both time and energy.
Then there's "Row-Column Bundling," a read technique that improves the speed of retrieving data from flash memory. Rather than reading one word at a time, the LLM reads larger chunks of data simultaneously, improving efficiency and throughput. The researchers claim this technique allows LLMs of up to twice the available RAM size to run, achieving speedups of 4–5x on CPU and 20–25x on GPU. They also state that the technique is "particularly crucial for deploying advanced LLMs in resource-limited environments, thereby expanding their applicability and accessibility."
This breakthrough carries significant implications for Apple and its users. Future iPhones may be able to run more advanced and sophisticated AI models on-device, without relying on cloud servers or compromising privacy. This could enhance Siri's capabilities, including real-time language translation, better speech recognition, and smarter responses. It could also enable new AI-powered features in photography, augmented reality (AR), and other domains.
HUGS
In another research paper, Apple's researchers propose a new method called HUGS (Human Gaussian Splats) for generating animated 3D avatars from short monocular videos. According to the paper's lead author, Muhammed Kocabas, HUGS can automatically learn to disentangle static scenes and fully animatable human avatars from just 50–100 frames of monocular video.
The technique uses 3D Gaussian splatting to represent both the human and the background scene, and a statistical body model called SMPL to initialize the human model. However, HUGS lets the Gaussians deviate to capture details like clothing and hair.
A new neural deformation module uses linear blend skinning to realistically animate the Gaussians while avoiding artifacts as the avatar's pose changes. HUGS enables novel pose synthesis for humans as well as novel view synthesis for both the human and the scene.
Compared to previous avatar generation methods, HUGS is up to 100x faster for training and rendering, and it surpasses the state of the art in 3D reconstruction quality. The researchers demonstrate that photorealistic results can be achieved with just 30 minutes of optimization on a typical gaming GPU.
Apple's generative AI roadmap
This breakthrough technology opens up new possibilities in virtual try-on, telepresence, and synthetic media. It could also make it possible to create novel 3D scenes using an iPhone's camera. HUGS' speed and efficiency represent a truly impressive achievement for Apple's researchers.
According to Bloomberg, Apple is already working on integrating its generative AI research into Siri. The company held an AI summit in February 2023 to brief employees on its large language model research. Apple is aiming for a smarter version of Siri that is deeply integrated with AI and capable of handling complex queries and tasks. Apple also plans to update how Siri interacts with the Messages app, enabling users to answer complex questions and autocomplete sentences more effectively. Beyond that, Apple is rumored to be adding AI to as many apps as possible.
Apple is also developing its own generative AI model called "Ajax," designed to rival OpenAI's GPT-3 and GPT-4, some of the world's most advanced AI models. Ajax operates with 200 billion parameters, demonstrating significant complexity and capability in language understanding and generation. Known internally as "Apple GPT," Ajax aims to unify machine learning development across Apple, suggesting a broader strategy of deeply integrating AI into Apple's ecosystem.
According to The Information and analyst Jeff Pu, Apple will make several generative AI features available on iPhone and iPad around the second half of 2024, when iOS 18 arrives. This could be a game-changer for Apple and its users, offering a new level of interactivity and engagement with their devices.
Papers