Apple co-created an AI tool that can perform complex image edits based on text prompts

Apple and researchers from the University of California, Santa Barbara, have co-created an AI tool that’s capable of performing image edits based on text prompts (via Venture Beat).

Called “MGIE,” the AI was presented in a paper at the International Conference on Learning Representations 2024. It’s a multimodal large language model, like Google Gemini, that can edit images much like you would do on Photoshop. Only here, you can express your thoughts in text and the AI will do all the editing work for you.

Say you have an image of a Pizza. You can tell MGIE to “make it more healthy,” and it will add healthier toppings to the pie in the image. Apple’s co-authored paper also presents other edit use cases where you can remove objects from images, change colors, and enhance lighting and other details of an image. It can even turn a forest path into a beach, change the background of photos, create artistic sketches, and much more. Think of Google’s Magic Editor on steroids. You can view examples of MGIE’s editing capabilities here.

“MGIE consists of an MLLM (Multimodal Large Language Model) and a diffusion model. The MLLM learns to derive concise, expressive instructions and offers explicit visual-related guidance. The diffusion model is jointly updated and performs image editing,” the paper explains.

There’s no telling how Apple plans to use these learnings on actual consumer-facing image editing tools. We do know that the company is working on generative AI features for its platforms. It’s possible we might see AI-based editing tools on the new iPhone 16 series. Although we presume MGIE’s extensive editing capabilities might need a healthy amount of processing, so Apple might introduce a toned-down version of the AI if and when it’s applied on iPhones.

If you’re interested in trying out MGIE, you can check out a demo hosted here.