From Sketch to Reality: Let’s Test Drive Adobe’s "Structure Reference"
How does it work? What can you use it for? How many questions can fit in a subheader?
Happy Thursday, structural sketchers,
In February 2023, we got access to a free ControlNet space on Hugging Face. This allowed complete beginners to use ControlNet to condition Stable Diffusion on an input image. The resulting picture would then retain the characters’ poses, locations of objects, and so on.
It’s pretty damn neat, but the Hugging Face implementation was slow and buggy.
Last week, Adobe Firefly implemented a strikingly similar feature called Structure Reference.
Now it’s even easier to turn a starting image or sketch into any scene you want.
Let’s take a look at how it works and what you can do with it.
What is Adobe’s Structure Reference?
Structure Reference works a bit like the starting image in Stable Diffusion or an image prompt in Midjourney.
But unlike them, Adobe doesn’t simply take inspiration from the source image. Structure Reference respects the position of objects and the overall image structur—oh, I get it now!
Say we take this cute dragon:
Then we use it as an image prompt in Midjourney with the following prompt:
Giant blue dog sitting on top of a mountain
Midjourney might generate a result like this:
You can see how Midjourney borrows stylistic elements from the original but still does largely its own thing.
Now compare that to Structure Reference in Adobe Firefly with the same prompt:
Adobe is much more literal, tracing the outline of our subject almost perfectly.
“But wait a second…the Firefly version sucks!” you scream and shake your fist at the screen. “Midjourney’s oversized cybernetic blue dog is much more realistic. Adobe’s looks like someone slapped a shitty Halloween costume onto a dragon!”
I hear you. I’m all for anatomically correct cybernetic dogs.
Two things:
This rigid precision has many cool applications, as we’ll see in a moment.
You can adjust the strength of the reference image to give Adobe more freedom.
This segues nicely into my next section. How convenient!
How to use the Structure Reference
Let’s start at the beginning, which feels like a perfectly reasonable place to start.
(If you’ve used Adobe Firefly before, you can jump to Step #2, you rebel.)
Step #1: Create an Adobe Firefly account
Head to firefly.adobe.com, type in a text prompt, and hit the big blue “Generate” button next to the prompt box:
If you’re not signed in, this will bring up a “Create account” box:
Sign up with your preferred account or email, and you’re ready for the next step.
Step #2: Pick an existing image to bring up the editor
Oddly enough, Adobe Firefly only shows you the full editor after you’ve generated your first image or selected one from the community.1
We’ll do the latter to speed things up. Scroll down to “Get inspired by the community” and click on any image:
You’ll be taken to the editor and should see a screen like this:
Step #3: Clean up the settings
These affect your image, so you’ll want to adjust them to your needs.
1. General settings
Pick your preferred aspect ratio:
2. Content Type
Here you can tell Adobe whether you’re after a photo or art. I recommend simply setting this to “Auto” to begin with:
3. Prompt and Styles
We’ll skip the Structure tab for now and head straight to the “Prompt” box filled with questionable modifiers:
Hit the “Clear” button to remove the preset styles, then delete the text:
Aaaah, pristine bliss!
Step #4: Upload your reference image
Now let’s go back to the left-hand Structure tab and upload our reference image:
I’ll be sticking with our yellow dragon for now:
Step #5: Adjust the reference strength
Once you upload a reference image, you can decide how strictly Adobe should follow it:
My first blue dog example used the highest strength setting. The default is in the middle.
Here’s a comparison at low, medium, and high strength settings with the same reference image and prompt:
Adobe takes much more structural liberty at the lowest strength, borrowing little from the reference image. You’ll want to play with this slider based on your needs.
Step #6: Generate your new image
Now you’re ready to give Adobe a new prompt and see what it does in combination with the reference image.
I consulted my daughter on this, and you’ll never guess what creature she picked:
Pastel painting of a rainbow unicorn
I kept the “Strength” slider in the middle and gave it a go:
Here’s how Adobe handled it:
It’s alive!
What can you use Structure Reference for?
So what’s the point of this feature?
Is it just a fancy toy to briefly impress your kids with?
Absolutely!
But you can do all sorts of other, marginally more useful things.
For instance…
1. Reimagine your living space
Say this is your living room:
First, can I live with you?
Second, check this out:
Thanks to Structure Reference, you can reimagine your space as a
Rustic cottage with a wooden fireplace and green curtains
or a
Futuristic neon command room on a spaceship
Or anything else for that matter.
Granted, Firefly keeps all items intact and simply re-skins them, but you can always remove any unwanted furniture or decorations before uploading the image to Adobe. (Use an AI-powered tool like cleanup.pictures or Adobe Firefly’s own Generative Fill.)
2. Turn sketches into mockups
Say you’re designing a wasp-shaped drone because you haven’t watched Black Mirror’s “Hated In The Nation” and somehow think that’s a great idea.
You feed Firefly the following sketch…
…then you ask it for a “Product shot of a cybernetic metal wasp with laser eyes.”
Congratulations, you have officially doomed us all:
3. Get fashion inspiration
Say you are this stylish stock photo man:
First, can I live with you?
Second, let’s say you’re curious about how you’d look in a green turtleneck sweater and a purple fedora because you’re the kind of maverick who skips straight to Step #2 in an online step-by-step guide.
You share your image with Firefly and add this prompt:
Black man wearing a green turtleneck sweater and a purple fedora2
On the minus side, Adobe has changed you into a completely different person.
On the plus side, you are now convinced that you might just pull off this purple-fedora-and-green-turtleneck look.
4. Control your visual storytelling
Structure Reference makes it very easy to arrange your visual narratives and pose your characters.
For instance, you can take this archival photo of the first person to ever catch the Sun…
…then mix it with the following prompt:
Wizard casting a blue spell at a dragon
In the end, you’ll get something like this:
If you’re feeling fancy, you might even throw it into a text-to-video tool like Pika to add a bit of animation and sound effects:
But you can of course do better than finding off-the-shelf images.
Here are some ideas:
Pose for photos yourself and use them as reference images
Draw rough reference sketches if you’re artistically inclined
Use sites like JustSketchMe or PoseMyArt to layout your scene
5. Remix stuff for fun
There’s nothing wrong with simply goofing around a bit.
Why not take an overused “Distracted Boyfriend” meme…
…and turn it into “Cute Pixar aliens on another planet”:
I take it all back.
This is nightmare fuel.
Have fun trying to sleep tonight.
Over to you…
I’m sure there are many practical applications I didn’t cover. Let me hear what other uses you see for Structure Reference.
If you’re already experimenting with Structure Reference, I’d love to hear about your experience.
Leave a comment or shoot me an email at whytryai@substack.com.
You can also message me directly:
If you know of a way to circumvent this, please let me know so I can update the post.
Firefly seems to suffer from biases similar to other AI models, so it insisted on giving our man white skin without the “black” modifier.
"Reference strength" is such an important, simple concept. I love that solution.