Apple has not unveiled any world-astonishing advanced AI like OpenAI's ChatGPT or Google's Gemini, and persistent criticism suggests that its development lags behind competitors. However, while there have been no official announcements from Apple itself, the company's researchers have been actively publishing research papers, offering glimpses into Apple's aggressive efforts toward innovative AI development. A newly published paper now reports that the company has developed a new AI that allows Siri not only to recognize the content of images, but also to understand with high precision the conversations and background context within such images.

It was recently reported that Apple may incorporate Google's Gemini model into its next iPhone, but the company also appears to be actively researching on-device AI processing, with rumors suggesting it may adopt a hybrid approach combining cloud-based large language models with on-device small models. The proprietary AI that Apple is said to be announcing soon is reportedly the latter—a small model that runs on-device.

Getting a computer program to perform a task based on ambiguous language input from a user, such as saying "this" or "that," is called reference resolution. Since computers cannot interpret images the way humans do, this is a complex problem to solve, but Apple has demonstrated the potential to solve it using AI models.

Apple's newly revealed paper details a system called "ReALM (Reference Resolution As Language Modeling)," which has the potential to enhance the usefulness of voice assistants powered by language models. According to the company, this system may match or even surpass OpenAI's GPT-4.

When speaking to a smart assistant like Siri, users may reference any number of contextual pieces of information to interact with—background tasks, data displayed on screen, and other non-conversational entities. Traditional parsing methods have relied on very large models or reference materials such as images, but Apple has streamlined the approach by converting everything into text.

ReALM can understand references to on-screen entities, such as the "260 Sample Sale" list shown in this mockup, enabling more natural interactions with voice assistants. (Credit: arxiv.org）

According to Apple, the system is capable of taking into account both what is displayed on screen and active tasks, as shown below:

  1. On-screen entities: Entities currently displayed on the user's screen.
  2. Conversational entities: Entities relevant to the conversation. These entities may come from the user's previous turns (for example, when a user says "Call Mom," Mom's contact would be the relevant entity in question), or they may come from the virtual assistant itself (for example, when the agent provides the user with a list of locations or alarms to choose from).
  3. Background entities: For example, an alarm that has started ringing or music playing in the background.

This has the potential to significantly boost the capabilities of Siri, which until now could hardly be described as particularly intelligent.

According to Apple, even the smallest ReALM model achieved performance comparable to GPT-4 while using far fewer parameters. This is reportedly a parameter size that allows for on-device operation. Furthermore, when the number of parameters used in ReALM was increased, it reportedly came to significantly outperform GPT-4.

Why did such efficiency become possible? The reason lies in the fact that GPT-4 relies on image analysis to understand information displayed on screen. Since much of the image training data is built on natural images rather than artificial, code-based web pages filled with text, direct OCR is inefficient in this context.

By converting images into text, ReALM eliminates the need for such advanced image-recognition parameters, making it smaller and more efficient. Apple also states that it avoids hallucination issues by including features such as constraining the decoding process and using simple post-processing.

For example, if you're scrolling through a website and want to call the company, simply saying "call the business" would require Siri to parse what you mean from the context. Siri would "recognize" that there is a phone number displayed as the business number on the page, and would be able to call that number without further prompting from the user.

At WWDC in June, Apple is expected to make a major push announcing its long-rumored, repeatedly foreshadowed AI initiatives. It remains unclear exactly what form this will take, but it is certainly clear that a great deal of internal research is underway as the company works desperately to keep pace with the rapidly evolving AI field.

Paper

References

Research Abstract

Reference resolution is an important problem, one that is essential to understand and successfully handle context of different kinds. This context includes both previous turns and context that pertains to non-conversational entities, such as entities on the user's screen or those running in the background. While LLMs have been shown to be extremely powerful for a variety of tasks, their use in reference resolution, particularly for non-conversational entities, remains underexplored. This paper demonstrates how LLMs can be used to create an extremely effective system to resolve references of various types, by showing how reference resolution can be converted into a language modeling problem, despite involving forms of entities like those on-screen that are not traditionally conducive to being reduced to a text-only modality. We demonstrate large improvements over an existing system with similar functionality across different types of references, with our smallest model obtaining absolute gains of over 5% for on-screen references. We also benchmark against GPT-3.5 and GPT-4, with our smallest model achieving performance comparable to that of GPT-4, and our larger models substantially outperforming it.