Here are all iOS 18 AI features that Apple is about to announce

Apple is currently working on a number of AI features for iOS 18 and iPadOS 18 that it will announce at WWDC 2024. Apple will be using on-device LLM to power these features that will be used to improve Siri, Notes, Messages, Photos, Calendar, and much more.

As per a report by AppleInsider, Apple is working on Project Greymatter, which is the name of the bunch of AI features that will be in iOS 18 and iPadOS 18. Here is a complete breakdown of the new features and how they will improve different parts of the operating systems on iPhone and iPad:

Notifications

A new notification summarization feature internally known as “Greymatter Catch Up” will make it easy for users to get a complete summary of their recent notifications. Apple already has a feature called “Scheduled Summary” which summarizes notifications on iOS and iPadOS and it will be a smarter upgrade to it.

Siri

Siri is expected to get a smarter response framework thanks to a new on-device Ajax LLM, which will also power Apple’s built-in applications. Siri will be able to use user’s data such as contacts, calendar events, locations, and more to provide smarter responses.

The smart voice assistant will also be able to create summaries of lengthy articles, similar to Safari 18, as well as transcribe audio in Notes and Voice Memos apps. While all this processing is going to be done locally with the Ajax LLM, cloud-based processing will be used for computationally complex tasks where needed, while keeping privacy intact. As expected, privacy will be a major focus for Apple when announcing these new features.

Siri’s voice will also be improved further to make it sound more natural, as well as text-to-speed improvements will be added too.

Another addition to Siri will be the cross-device media and TV controls, while will allowing users to use one of their Apple devices to control playback on another one. AppleInsider believes that this feature will be released later in 2024.

Photos

iOS 18, iPadOS 18, and macOS 15 will also get generative AI upgrades for the Photos app. One of the features will be Clean Up, which will allow removing unwanted objects in images with the use of AI – something which is already available in Google Photos.

The company is also working on an application called “Generative Playground” which will use its LLM to allow for the creation and editing of images. This will allow users to generate images and send them via iMessage, as well as create unique and custom emojis. How these custom emojis will work between different platforms like Apple’s operating systems and Windows and Android is yet to be seen.

Notes

The Notes app will potentially get support for text and image generation as well, however, it is yet to be confirmed. What is certain is that Notes will feature support for in-app audio recording and transcription, as well as summarization.

These features will make Notes an even more useful app, specially with the availability of these features on all Apple platforms including iOS and macOS.

New Math Notes will also be available which will allow for complex mathematical notations and integrate with the new and improved Calculator app. Math Notes will not only recognize text in complex mathematical notations anywhere in the operating system, but also offer auto-correct and be able to solve them. Graphing expressions will also be supported with this update.

These features are also expected to be made available to visionOS later this year in the form of improved voice commands, predictive text, smart replies, and more.

What’s missing from this news report is whether Apple will be adding generating AI features to its camera app for improved detection of the real-world. Photos app already has the capability of recognizing plants and different objects, but generative AI could really boost these features.

What differentiates Apple’s AI offerings when compared to competitors like Google Gemini and ChatGPT is that the Cupertino giant is not building a chatbot. It is more focused on providing these AI-based improvements as part of its operating system and apps, which should offer a lot of value to consumers. For chatbot features, users will still have to rely on competitors which are easily available in the App Store.

Apple has already put in a lot of work on AI and machine learning in iOS and macOS in the past, but now it is seemingly playing catchup to competitors when it comes to generative AI. It will be interesting to see how the new features work and how well are they received by users.

Imran Hussain