InstructPix2Pix: Edit Images Using Simple Text Commands

Modifying images just got even easier. InstructPix2Pix lets you make changes to an image via natural language text instructions. Give it a try for free!

Howdy, partners!

As you know, what fascinates me most about text-to-image AI is that it lets complete art amateurs like myself visualize ideas just by typing requests.

But if you ever want to edit an AI-generated picture, you’ll need access to (and knowledge of) some post-processing tools. It’s not like you can edit an image in the same way you first generated it, just by asking.

Or…can you?

*plot twist*

You absolutely can!

Only two months ago, I told you about a tool that lets you replace objects in images with other objects using simple text prompts.

Well, that’s old hat. AI progress can’t be stopped, so we now have an advanced tool that lets you do even more.

It’s called InstructPix2Pix. (Just rolls off the tongue, doesn’t it?)

Let’s see what it does and how you can play with it.

What is InstructPix2Pix?

InstructPix2Pix is a new method of manipulating images.

It combines the comprehension skills of large language models with the art creation skills of Stable Diffusion.

The result? You can edit images just by telling AI what you want to happen:

Here’s a nice quick video to showcase its capabilities, even though we’ll be using a different interface in the post:

Update 01-02-2023: There’s now a simple chat interface with a limited feature set that works just like the above video. It’s called Paint By Text. Enjoy.

How can you test InstructPix2Pix?

Want to try it for yourself?

Good news: There’s nothing stopping you from doing so right the hell now!

There are at least three ways to check out InstructPix2Pix, and they are all completely free.

1. Quick & dirty: Hugging Face

The quickest way to get started is via this Hugging Face demo page.

It’s honestly pretty self-explanatory:

1. Drop (or upload) your image into the designated spot:

2. Add your instructions in the “Edit instruction” field:

3. Then hit the “Generate” button to the left

You can tweak the relative strengths of the image and the text prompt, but default settings should be fine to start with. Here’s what I got when asking to switch this awesome Pixabay image to a daylight setting:

Pretty impressive!

The main downside of the Hugging Face demo site is that there’s usually a long(ish) queue and it can also be offline at random times.

2. Slightly nerdy: Google Colab

If you’re not afraid of warning messages from Google, there’s a more reliable Google Colab option, which also implements changes faster.

Don’t panic!

It may look overwhelming at first, but here’s all it takes to get it working:

1. Click “Connect” in the top right and wait until it says “Connected”:

2. In the “Runtime” dropdown, select “Run all”
(or click Ctrl+F9 if you’re feeling fancy):

3. If this scary box comes up, click “Run anyway”:

4. Wait (for about a minute or so, until all the lines of code finish running).

5. Scroll down to the “Gradio Demo” section:

6. You can now upload and work with images directly in the Colab, or…

7. Click the provided “Public URL” link to open the GUI in a new tab/window

The rest of the interface is very similar to the Hugging Face demo, so you should know exactly what to do.

Have fun!

3. Safe & sturdy: PlaygroundAI

AI art site PlaygroundAI recently incorporated InstructPix2Pix into the platform, so you can run it directly in their interface.

You’ll just need to sign up with your Google account to access it.

After you’ve signed up, here’s what you do:

1. Click “Create” in the top right

2. Click the blue “+IMPORT IMAGE” at the top (to the left of “Prompt”):

3. Upload your image

4 Add your instructions into the “Edit Instruction” box

5 Click the “Generate” button at the bottom:

Poof…magic!

Again, there are different settings you can play with, but starting with the default ones should work just fine.

Limitations of InstructPix2Pix

InstructPix2Pix can do a lot, but it’s not flawless. The authors themselves mention the following “failure cases”:

In addition, I also came across the following limitations:

1. Can’t make several changes in one go
   If you e.g. want to change the season and turn a tree into a bush, you’ll want to split those instructions into two separate commands. It’s hard for AI to parse both of them in a single prompt.

2. Hard to manipulate similar elements

   If there are multiple elements of the same type in the picture, it’s hard to explain to the model exactly which one of them you want changed.

Still, for something that can make such dramatic changes with so little effort, InstructPix2Pix is already an impressive sneak peek at what’s to come.

Over to you…

Have you tried InstructPix2Pix yet? Do you know of an alternative tool that does something similar? Or do you have beginner-friendly hacks to make simple image alterations using Photoshop, GIMP, etc.?

I’m always happy to discover new tools and methods, so feel free to send me an email or leave a comment below.