Apple’s AI-based “Audio Intelligence” technology for the Apple Watch is currently rolling out in beta, initially to developers in the iOS 27.3 update, after first being announced at the company’s Surprise and Shine hardware event in September. The technology includes a set of four optional AI features that allow Apple Watch Series 12 wearers to detect important sounds, identify songs, catch up on conversations and take notes.
The sound recognition feature will help deaf and hard of hearing users identify important sounds such as sirens, alarms or doorbells from their watch, while the song recognition feature leverages Apple’s Shazam app to detect music playing in the background and identify the song name and artist.
However, the most potentially controversial features are those that listen to human conversations: Live Rewind and Siri Recaps.
Given the backlash to AI recording devices, like Meta’s AI Glasses and other AI wearables, Apple has attempted to create a space where features can be useful to Apple Watch owners, while still respecting the privacy of those around them.
With Live Rewind, the idea is to let you go back in time to see what someone just said, in case you missed it. For example, if someone was giving you verbal instructions or reading a long list of special dishes at a restaurant, you could double-tap the watch’s digital crown to see what was said in the last 15 seconds as a snippet of text displayed on the watch face. You can either view this snippet, then do nothing, and it will disappear forever. You can also save it to the standalone Siri app that comes with iOS 27 for later access, which can be useful if you’re trying to remember someone’s name or take a quick note of a quick presentation.
Apple says users will also be able to tap the “Ask Siri” option that appears below the snippet of text if they have questions about something that was said, like a technical term they don’t understand, or perhaps a sauce option the server mentioned that they don’t know about, among other things.
When you double-tap the digital crown, a chime sounds and an animation appears on the watch. This is intended to signal to people, through the combination of gesture, sound and visual display, that you have recorded something that was said.
These clips will be automatically deleted after 30 seconds if you don’t save them.
The other option, Siri Recaps, takes this ambient listening technology even further by allowing Apple Watch owners to take high-level notes about their conversations.
It’s important to note that Apple says it doesn’t identify speakers, record raw audio, or provide verbatim transcriptions. Instead, the feature summarizes a conversation with a short sentence and a few bullet points that identify key points. For example, depending on the topic, bullet points may include things like important dates or deadlines, information about pricing or payment plans, and other factual information.
Apple suggested that this feature could be used in scenarios like work meetings, parent-teacher conferences, personal training sessions, or other places where you may need to remember tasks you need to complete or things that were said. Essentially, it’s meant to mimic a user’s ability to take their own notes, without adding emotion or tone.
This can be especially helpful when you are unable to write your own notes, either because your hands are full or you are trying to focus on what is being said at the time. Recaps won’t record conversations when you’re listening to music through your headphones or using their noise-canceling features.
At the end of the day, you’ll be prompted to review your Siri summaries saved when the feature is enabled, and then choose which ones, if any, you want to save. These notes can be saved to your Notes app or even exported to your Journal app, if they are more personal in nature. If you do nothing, the recaps will be automatically deleted within seven days.
This reminder to check the Siri summaries you’ve saved also appears when you unlock your Apple Watch for the first time in the morning.
Users can choose to enable Siri summaries manually or set up a schedule, such as having the summary only happen during work or ensuring they never run at night. Even though they run on a schedule, you can still turn off the summary feature at any time from Control Center.
Apple notes that Siri Recaps was designed to omit sensitive information such as government-assigned identifiers, financial and authentication data, certain personal identifiers, and any language that could promote self-harm.
The Audio Intelligence features in Apple Watch Series 12 and Apple Watch Ultra 4 are powered by a combination of Apple hardware, software and services. This includes the S11 chip, built-in microphone, and Apple Intelligence models that work on the iPhone, as well as Apple’s Private Cloud Compute.
The always-on processor includes a secure enclave – a memory space protected by hardware and not accessed by the rest of the operating system – where a dedicated audio buffer processes raw audio. This allows audio to be processed without it actually being accessible to anyone, including Apple. Audio is only accessible to the paired iPhone via a secure pairing mechanism between it and the watch.
To work, lightweight AI models on the Apple Watch’s S11 chip detect whether there is a conversation the wearer is part of, which is then transferred to the iPhone’s secure enclave. On-device processing on the iPhone turns the audio into condensed text, that is, the text of the conversation without the filler words. This condensed text is then encrypted and sent to Apple’s Private Cloud Compute (PCC) to create the conversation summary that is sent back to the phone. (The summary and any summaries used to create it are deleted from the PCC immediately afterwards).
The summaries you can then access on your iPhone are only accessible to you, not Apple. And they’re encrypted when synced between your multiple Apple devices.
Similarly, Apple Watch transfers the audio to the iPhone for processing to create the 15-second clips for Live Rewind.
Apple says none of these features should impact the battery life of your watch or phone. The features will then be deployed in public beta for broader testing before becoming available to everyone.
When you purchase through links in our articles, we may earn a small commission. This does not affect our editorial independence.
Gn tech
