New intelligence features in iOS 15: Live Text, Visual Look Up, smarter Siri, much more

Apple announced the new iOS 15 update coming to compatible iPhone models in Fall, 2021. The new update is packed with exciting new features, and performance improvements like the upcoming intelligence features, which bring the power of on-device machine learning to enrich users’ experience by adding more ease to organizing photos, sharing information, improving search, and a lot more. Here are all the new intelligence features in iOS 15.

Here are all the new intelligence features on iOS 15

Power by Apple Neural Engine, the enhanced ML capability in iOS 15 will allow iPhone owners to use their smartphones in a variety of new and exciting ways to not only share information but also to search, stay updated, and also more. Here are the new features users will be able to enjoy this Fall.

Photos

Apple redefines the use of photos and makes them more than just memories to cherish by adding new functionality to them with two new features:

Live Text gives new functionality to users’ photos by turning visual information in pictures into text like project planning on a whiteboard, a restaurant’s phone number, and brands printed on T-shirts. Apple Neural Engine and Camera app use on-device intelligence to quickly recognize and copy text, so that users can share the data via messages, save it in the Notes app, call a pizza place to place an order, book an appointment, and much much more.
Visual Look Up feature gives users quick and easy access to information of any object, place, animal, flower, and more in their photos. Simply by swiping up on the image or tapping on its information icon, users will get to learn about or find whatever their heart desire and the moment requires.

Spotlight Search

Spotlight Search will make it easier for users to get more information on a variety of categories in a single search, find photos, download apps, and more.

One quick and “rich” search result to show information of contacts, actors, musicians, movies, and shows.
The entire Photos library will be searched by locations, people, scenes, and the subject of the photos like cars, animals, flowers. In addition, users will be able to share the found images via the messages app.
Search for monuments, events, people, objects, animals, or more from the web.
Download apps from the App Store directly from Spotlight.
Action button for business on the Maps that support App Clips. When those businesses are searched for in Spotlight, the Action buttons will include menu, order, reservation, pricing, delivery, and more.
Access Spotlight by swiping down from the Lock Screen or Notification Center.

Interactive Memories

Photos app gets redesigned Interactive Memories with new immersive UI, mixes, personalization options, and more to give a comfortable and more personal feel.

Memories get a redesigned user interface:
- New animated cards with smart titles, multiple image collages, and animation and transition styles.
- Simply tap to pause, replay, skip, and more while the music keeps on playing in the background and the timing automatically adjusts to keep the transition on the beat.
- In addition, “change the song or Memory look or add or remove photos, and the adjustment happens in real-time, without the need for the movie to recompile.”
- New browser-view not only gives a bird’s eye view of all the content in memory but also allows users to add, remove, or change sections of memory.
New Memory types include international holidays, trends over time, child-focused, and improved pet memories which can distinguish between dogs and cats.
Twelve new Memory looks adjust the color tone to apply mood by analyzing each photo and video.
Apple Music subscribers can add any song from the vast music library on the service to their memories. Apple also offers deeper integration between Photos and Music apps, by showing personalized song suggestions to subscribers of the music streaming service based on the latest trends in music and those in their photos and videos. The song suggestions are done on the device for users’ privacy.
New Memory mixes will allow users to select songs that suit the occasion in their memories.
Memories Watch next feature suggests other memories to watch after one finishes.

Health app

Health app updates provide new ways to share preferred data with family, friends, or health consultants vitals, health health, activity Medical ID, and more to keep a collect track of users’ health.

The new Health data notifications will also be received by people with who users chose to share their health data. Notifications of changes in users’ health like sleep patterns, heart rates will be shared.
Users will be able to share the health trends and discuss them on text via messages app.
Trends on the Health app give a holistic summary of a health metric’s progression or regression over time.
The new Walking Steadiness health metric on the iPhone gives users an assessment of their risk of falling. Custom algorithms analyze users’ balance, strength, and gait to gauge the low or high intensity of their walking steadiness. Users at risk of falling are offered curated exercise to improve their walking steadiness.
Lab enhancements offer a detailed description of lab results for better understanding including lab highlights in the health summary to know if the test scores are in range. Users will also be able to pin the most important labs on top.
Save COVID-19 test results in the Health app by scanning the QR code from healthcare providers.
Track Blood glucose levels and review the data on interactive charts.

Privacy updates

Apple has beefed up privacy in iOS 15 by bringing more transparency to data collection and how it is used by developers.

The new Apple Privacy Report gives users a detailed summary of how their opt-in tracking data was used and who was it shared with.
Mail Privacy Protection is a new protection layer on the iPhone that prevents email senders from gaining access to users’ IP addresses to track their Mail activity.

“If you choose to turn it on, it hides your IP address so senders can’t link it to your other online activity or determine your location. And it prevents senders from seeing if you’ve opened their email.”

Smarter Siri

Siri is smarter, faster, and more secure in iOS 15 because Apple Neural Engine keeps all the communication stays on the device.

High-quality on-device speech recognition capability comes with a personalized experience by offering Siri on-device learning to know users most interacted with contacts, frequently used words, searched and read topics.
Siri comes with offline support to carry out requests without an internet connection like settings alarms, making phone calls, sending messages, controlling audio playback, adjusting Settings, and launching apps.
Siri will be able to intelligently share items on screen like songs from Apple Music, photos, web pages, locations, and much more. In addition, Siri will also send messages or call the onscreen contacts.

“For example, just say “Send this to Vivek” and Siri will send it. If the item cannot be shared, Siri will offer to send a screenshot instead.”

For example, if you’re looking at a contact in the Contacts app, a conversation with someone in Messages, or a notification of a message or missed call from someone, you can say “Message them I’m on my way” and Siri will send it to the appropriate contact.

4. Announce Notifications features allows Siri to automatically read out important notifications on AirPods. In addition, Siri will also announce messages in CarPlay. Users will able be able to enable announce Notifications for each app in the Settings.

5. Siri can smartly maintain context to understand a variety of intent in commands like after asking“Is Glacier National Park still open?” you could ask “How long does it take to get there?” and Siri will make the connection. Siri can also smartly control HomeKit at specific timings. For example, say “Hey Siri, turn off my bedroom lights at 7 p.m.” or “Hey Siri, turn off all the lights when I leave.”

6. Siri new neural text-to-speech voices now support more languages: “Swedish (Sweden), Danish (Denmark), Norwegian (Norway), and Finnish (Finland).” In addition, it also supports mixed English and 9 more Indian languages: Hindi, Telugu, Kannada, Marathi, Tamil, Bengali, Gujarati, Malayalam, and Punjabi.”