If you would prefer a full video tutorial on this with explanation of each model, examples of what control net does, as well as an installation tutorial check out my full guide on Control Net on YouTube below:
This guide is assuming that you already have stable diffusion installed. If you don't have stable diffusion, watch this video to install it:
I recommend grabbing some custom v1.5 model files from this site as v2.1 dosn't seem to work with the MikuBill version. This version GitHub - lllyasviel/ControlNet: Let us control diffusion models! of Control Net works with Stable Diffusion but it only have 4 contributors.
Step 1. To save time and to show you what your committing too, you need to know about the size of the models and start the download for them as it will take a while.
Each model is 5.71 GB each and you click on the down arrow to start downloading the model.
The files will go into your Control Net extensions/model folder, but you won't have this until you install the extension which will be the next step. (more on this later in the tutorial)
A quick description of each model is:
Canny - Edge detection model for straight lines great for anime
Depth - Depth Mapping (creates a mist for 3d awareness that you can transfer style over)
HED - Mixture of Canny and Depth (good for sketches, humans, and things with fine details)
MLSD - Good for architecture houses and straight lines
normal - Bump mapping, texture mapping, or height mapping. Creates a 3d models where light will refract off the object
openpose - stick figure model, great for just the pose but all hands and legs must be showing on subject
Scribble - Similar to Canny but scribbles instead of makes sharp lines
Seg - segmentation maps similar to Nvidia Canvas but img2img, objects and areas are separated or segmented by color gradient maps.
See github for more details
Step 2. Copy the GitHub address.
Copy one of the 2 links above. I use Mikubill's because they have more developers, but it doesn't have V2.1 support. I don't use V2.1.
This is the link I'm using, feel free to copy and paste this one: https://github.com/Mikubill/sd-webui-controlnet
Open Stable Diffusion and go to Install from URL:
Copy and paste the address from earlier : https://github.com/Mikubill/sd-webui-controlnet
into the "URL for ext git repository and then click install. This will only take 3 seconds to install.
Click on the installed tab and then click on Apply and Restart UI
Step 3. Now Click on settings which is right next to the extension tab (above)
In settings click on Control Net
In Control Net > Settings > Mutli ControlNet: Max Models amount (requires restart)
increase the number of control models by clicking to the right of the slider. Three models is plenty.
Congrats!!! you now have control net and multi control net but you will have to reload your UI and then restart your stable diffusion webui-user.bat file for it to work.
NOT SO FASSSSTTT!
Where is it? and what about the models???? Good Question.
Control net is found in the Text2img and img2img tabs near the bottom:
Click on the arrow to the right of Control Net to see options.
Starting at the top you will notice you have control model -0 -1 and -3 those are the multiple tabs of control net.
The drop image here section is self explanatory and you drop an image there.
The enable button will have to ticked for this tool to work and this is a great feature because when your working with multiple tabs you can just unclick the ones you choose not to use anymore.
Invert Input Color and RGB to BGR are just color reversal buttons
Low VRAM is great if you have less than 4-8GB of VRAM, it really depends on the resolution and steps you choose.
Guess mode will try to guess the prompt if you leave it blank but it produces some nice results when you turn it on so I would experiment with it on and off. +
Preprocessor and Model Should always match, or you won't have control in your control net.
Weight determines the effect of the tool, anything over 1 is extreme and below is mild.
Resize mode should probably just be left to Scale to Fit for best results:
Canvas height and width has nothing to do with your picture, this creates a canvas you can use the scribble model with when you click on create blank canvas. The height and width will be used with your normal stable diffusion settings at the top.
You will notice that you models will not look like mine and it will be completely blank.
Drag the models you downloaded in step 1. into your extensions folder for this tool
My extensions folder is C:\Users\PcUser\Documents\Stable Diffusion\stable-diffusion-webui\extensions\sd-webui-controlnet\models
Now all your models should show up. If they don't, click on the trefresh button.
Refresh button:
Now just put something in the prompt and click generate as you normally would with the control net set up below and your good to go. Watch my tutorial for more details and examples.
Additional TIP:
If you want additional tools like open pose and depth map models keep reading.
Depth Map Tools install - Repeat step 2. but this this time use this link https://github.com/jexom/sd-webui-depth-lib to install from URL to get depth mapping tools. This tool will allow you to fix hands which is my next tutorial.
Open Pose tools - this will allow you to make your own poses by manipulating stick figures / open pose models inside stable diffusion. Just repeat step 2 but copy and pasting a URL into the extensions install from URL but use this link this time: https://github.com/fkunn1326/openpose-editor
You have now installed Control Net, Enabled Multi Control Net, Open Pose Editor, and Depth Map tools to fix those ugly hands.
I am making a video on the depth map tools right now and it should be available in a day or two but in the meantime watch my control net install video because it will show you what control net can do for you and give you examples of what ever of the models can provide you.
Check out my youtube video to see the many examples and if you like Ai Art and AI tools - voice cloning, Chat GPT, GPT4, Google Bard, etc... check out my channel.
Need to get some poses for your Control net?
Try PoseMy.Art . You immediately get started when you get to the site, and you have charter you can animate and move around for free.
Do you just want access to 1000's of poses and to lazy to make one yourself? Try POSEMANIACS - Royalty free 3d pose reference for all artists
Comments