Good Morning from my Robotics Lab! This is Shadow_8472 and I\u2019ve made up my mind on StableSwarmUI as a replacement to A1111. Let\u2019s get started!<\/p>\n\n\n\n
Generative AI (Artificial Intelligence) is the technology buzz word of the decade so far thanks to open sourced models. Automatic1111 has an extensive community library, but ComfyUI\u2019s flexibility may yet challenge it as the next favorite. While not yet polished to A1111\u2019s visual aesthetic, a total AI noob should find StableSwarmUI navigable while letting him\/her peek at Comfy beneath.<\/p>\n\n\n\n
Learning ComfyUI Basics<\/p>\n\n\n\n
I\u2019m taking that peek\u2026 ComfyUI looks like boxes and spaghetti. The correct term is \u201cworkflow.\u201d Each node represents some unit of work similar to any other UI. The power of Comfy is the ability to arbitrarily link and re-arrange nodes. Once my first impression \u2013intimidation\u2013 wore off, I found grouping familiar options by node and color coding their connections made the basic workflow more intuitive while highlighting my gaps in understanding of the Stable Diffusion process.<\/p>\n\n\n\n
Let\u2019s define some terms before continuing. Be warned: I\u2019m still working on my intuition, so don\u2019t quote me on this.<\/p>\n\n\n\n
The basic Stable Diffusion workflow starts with an empty Latent Image node defining height, width, and batch size. Alongside this, a model or checkpoint is loaded. CLIP Text Encode nodes are used to enter prompts (typically both positive and negative). A KSampler node does the heavy lifting, combining everything into a low-resolution preview based off the latent image (if enabled). Finally, a VAE decoder node turns your latent image into a normal picture.<\/p>\n\n\n\n
While I\u2019m still developing an intuition for how a latent space works, I\u2019m imagining a tent held up by a number of poles defining its shape. You are free to interpolate between these points, but quirks can arise when concepts bleed into each other: like how you\u2019d tend to imagine bald people as male.<\/p>\n\n\n\n
ControlNet<\/p>\n\n\n\n
The next process I wish to demystify to myself is ControlNet. A second model is loaded to extract information from an existing image. This information is then applied to your positive prompt. (Let me know if you get any interesting results conditioning negative prompts.) Add in a second or more ControlNets, and combining them presents its own artistic opportunity.<\/p>\n\n\n\n
For this exercise, I used a picture I made during my first attempt at Stable Diffusion: a buff angel with a glowing sword. As a challenge to myself, I redid it with SDXL (Stable Diffusion eXtra Large). I used matching ControlNet models for Canny and OpenPose. Some attempts came up with details I liked and tried to keep. I added the SDXL refiner model to try fix his sword hand. It didn\u2019t work, but in the end, I had made a generation I liked with a few golden armor pieces and a red, white, and blue \u201c(kilt:1.4).\u201d Happy 4th of July!<\/p>\n\n\n\n
Practical Application<\/p>\n\n\n\n
A recent event has inspired me to try making a landscape picture with a pair of mason jars \u2013one full of gold coins, and the other empty\u2013 both on a wooden table in front of a recognizable background. It\u2019s a bit complex to generate straight out of text, but it shouldn\u2019t be too hard with regional conditioning, right?<\/p>\n\n\n\n
Impossible. Even if my background came out true, I\u2019d still want the mason jars to match, which didn\u2019t happen. This would have been end of the line of the if I were limiting myself to A1111 without researching additional plugins for my already confusing-to-manage cocktail. With Comfy, My basic idea is to generate a jar and generate another filled jar based off it, then generate them together in front of my background.<\/p>\n\n\n\n
Again: easier said than done. Generating the initial mason jar was simple. I even arranged it into a tidy group. From there, I made a node group for ControlNet Canny and learned about Latent Composite \u2013 both of which allowed me to consistently put the same jar into a scene twice (once I figured out my dimensions and offsets), but filling\/emptying one jar\u2019s gold proved tricky. \u201cFilling\u201d it only ever gave me a quarter jar of coins (limited by the table visible through the glass), and emptying it left the glass surface horribly deformed. What\u2019s more is that the coins I did get would often morph into something else \u2013such as maple syrup\u2013 with too high of a denoise in the KSampler. On the other hand, too low a value, and the halves of the image don\u2019t fuse. I even had coins wind up in the wrong jar with an otherwise clean workflow.<\/p>\n\n\n\n
Even though I got a head start on this project, I must lay it down here, incomplete. I have seen backgrounds removed properly with masking, so I\u2019ll be exploring that when I come back.<\/p>\n\n\n\n
Takeaway<\/p>\n\n\n\n
ComfyUI looks scary, but a clean workflow is its own work of art. Comfy\u2019s path to mastery is clearer than A1111. Even if you stick to basics, StableSwarmUI has simpler interfaces \u2013 a simple prompt and an \u201cunpolished A1111-esk\u201d front panel for loaded pre-made workflows.<\/p>\n\n\n\n
Final Question<\/p>\n\n\n\n
I\u2019m probably being too hard on myself compositing glass in-workflow. Let me know what you think. What tips and tricks might you know for advanced AI composition? I look forward to hearing from you in the comments below or on my Socials<\/a>!<\/p>\n","protected":false},"excerpt":{"rendered":" Good Morning from my Robotics Lab! This is Shadow_8472 and I\u2019ve made up my mind on StableSwarmUI as a replacement to A1111. Let\u2019s get started! Generative AI (Artificial Intelligence) is the technology buzz word of the decade so far thanks to open sourced models. Automatic1111 has an extensive community library, but ComfyUI\u2019s flexibility may yet … <\/p>\n