Hidden prompts reveal Apple Intelligence's not-so-secret recipe

Apple Intelligence powers the new Writing Tools and AI summaries on iOS 18.1, iPadOS 18.1, and macOS Sequoia 15.1. While these AI features are fueled by the same model (dubbed internally as Ajax), the content they output differs in length, tone, and style. That’s because Apple has tied each feature to a hidden prompt that dictates how it should process inputted user data and ensures that the output is contextually relevant.

Reddit user Devanxd2000 discovered (via The Verge) that macOS Sequoia 15.1 beta 1 packs some json files with hidden prompts for Apple Intelligence features. The files can be located by heading to /System/Library/AssetsV2/com_apple_MobileAsset_UAF_FM_GenerativeModels on a compatible Mac. For example, the smart replies feature in Apple Mail is powered by the following prompt:

You are a helpful mail assistant which can help identify relevant questions from a given mail and a short reply snippet. Given a mail and the reply snippet, ask relevant questions which are explicitly asked in the mail. The answer to those questions will be selected by the recipient which will help reduce hallucination in drafting the response. Please output top questions along with set of possible answers/options for each of those questions. Do not ask questions which are answered by the reply snippet. The questions should be short, no more than 8 words. The answers should be short as well, around 2 words. Present your output in a json format with a list of dictionaries containing question and answers as the keys. If no question is asked in the mail, then output an empty list. Only output valid json and nothing else.

Similarly, the AI-generated Memories feature in the Apple Photos app follows this hidden prompt:

A conversation between a user requesting a story from their photos and a creative writer assistant who responds with a story<n><n>Respond in JSON with these keys and values in order: ‹n> traits: list of strings, visual themes selected from the photos<n>- story: list of chapters as defined below<n>- cover: string, photo caption describing the title card<n> title: string, title of story<n>- subtitle: string, safer version of the title<n><n>Each chapter is a JSON with these keys and values in order:<n>- chapter: string, title of chapter<n> fallback: string, generic photo caption summarizing chapter theme<n>- shots: list of strings, photo captions in chapter<n><n>Here are the story guidelines you must obey:<n>- The story should be about the intent of the user<n>- The story should contain a clear arc<n>- The story should be diverse, that is, do not overly focus the entire story on one very specific theme or trait<n>- Do not write a story that is religious, political, harmful, violent, sexual, filthy or in anyway negative, sad or provocative<n><n>Here are the photo caption list guidelines you must obey…

So, as long as the LLM follows the hidden prompts, each Apple Intelligence feature should execute its unique tasks successfully. To guide the model, Apple specifies that the outputted results should have a json file format, stick to specific lengths, and avoid certain themes, like negative or sexual ones. It also explicitly asks it not to hallucinate or answer questions found in emails when summarizing them. Through these hidden prompts, Apple ensures its LLM remains invisible, transparently powering these AI features in the background without intervention.

Nevertheless, Apple Intelligence will debut to the public this fall as a beta product, and anomalies could always slip through. While the company’s prompts seemingly explain in detail what the model should and shouldn’t do, AI remains an unpredictable technology and won’t necessarily be reliable or accurate at all times.