> ## Documentation Index > Fetch the complete documentation index at: https://docs.shaktistudio.ai/llms.txt > Use this file to discover all available pages before exploring further. # Multi Control Net FluxPipeline provides support for multiple image generation pipelines with and without controlnets, including text-to-image (txt2img), image-to-image (img2img), and inpainting. ## Important Note Ensure that a volume mount is added to the deployment, as all images generated are dumped inside `/data/outputs` directory in the container. *** ## Model Optimization Configuration ### Optimization Settings For optimization, under the optimization config, use: ```json theme={null} "optimisations": { "attention_caching": { "type": "auto", "enabled": true, "extra_params": { "threshold": 0.1 } } } ``` * Higher threshold values result in greater speed gains but may degrade image generation accuracy. * We recommend a threshold of 0.1, which can provide up to a 40% speed improvement during inference while maintaining reasonable quality. ## Pipeline Settings For optimization, under the optimization config, use: * Multiple ControlNet models can be added under the controlnets section. * Each ControlNet model requires a name, source, and authentication details if needed. ```json theme={null} { "type": "flux", "loras": [], "lora_repo": { "path": "", "type": "", "secret": { "type": "" }, "ownership": "" }, "pipelines": [ "txt2img" ], "controlnets": [ { "name": "canny", "source": { "path": "InstantX/FLUX.1-dev-Controlnet-Canny", "type": "hf", "secret": { "type": "hf", "token": "" } } }, { "name": "depth", "source": { "path": "InstantX/FLUX.1-dev-Controlnet-Depth", "type": "hf", "secret": { "type": "hf", "token": "" } } } ], "model_choice": { "flux_type": "flux" }, "custom_pipeline_config": [], "custom_pipeline_resources": "" } ``` Here are some key pointers for understanding and structuring controlnet requests: ## Understanding ControlNet Parameters ### **ControlNet Name Convention:** * The parameters follow a structured pattern: ``` __control_image _weightage ``` * Example for Canny: ``` "canny_control_image" - The input image processed with the **Canny edge detection** model. "canny_weightage" - Defines the influence of the **Canny edge map** on the final image generation. ``` * Example for Depth: ``` "depth_control_image" → The input image processed with the `Depth estimation` model. "depth_weightage" → Determines how strongly the depth control image impacts the generation. ``` ### **Extensibility for Multiple ControlNets:** * This pattern allows easy extension to additional ControlNet models in a structured way. * If you add a new ControlNet (e.g., OpenPose), you'd include: ```json theme={null} "openpose_control_image": "URL_to_openpose_image", "openpose_weightage": 0.5 ``` ### **How Weightage Works::** * Each weightage parameter (canny\_weightage, depth\_weightage, etc.) determines the degree of influence that specific ControlNet has on the final image. * Higher values make the model adhere more strictly to the control image, potentially sacrificing flexibility. * Lower values allow more artistic freedom but reduce adherence to structured inputs. ### **Combining Multiple ControlNets:** * You can combine multiple ControlNets in a single request to layer different structural constraints. * In this example: * `Canny edge detection` helps maintain sharp edges in the image. * `Depth estimation` preserves 3D structural information. * By adjusting the weightages, you can balance between these two influences. ### **Generalized Pattern for Other ControlNets:** ```json theme={null} "_control_image": "", "_weightage": ``` * Example with Pose and Normal Map: ```json theme={null} "pose_control_image": "URL_to_pose_estimation_image", "pose_weightage": 0.5, "normal_control_image": "URL_to_normal_map_image", "normal_weightage": 0.3 ``` ## Supported Pipelines 1. txt2img - Generates an image from text input. 2. txt2img\_controlnet - Generates an image from text input with controlnet support. 3. img2img - Generates an image based on an input image and a given prompt. 4. img2img\_controlnet - Generates an image based on an input image and a given prompt with controlnet support. 5. inpaint - Modifies specific regions of an image based on a mask and a given prompt. 6. inpaint\_controlnet - Modifies specific regions of an image based on a mask and a given prompt with controlnet support. ## Example Requests ## txt2img ```json theme={null} { "prompt": "A girl in city, 25 years old, cool, futuristic ", "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "seed": 2064977189, "guidance_scale": 4.5, "strength": 0.8, "scheduler": "EULER-A", "model_type": "txt2img" } ``` ## txt2img\_controlnet ```json theme={null} { "prompt": "A girl in city, 25 years old, cool, futuristic ", "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "seed": 2064977189, "guidance_scale": 4.5, "canny_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny/resolve/main/canny.jpg", "canny_weightage": 0.4, "depth_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Depth/resolve/main/depth.jpg", "depth_weightage": 0.4, "strength": 0.8, "scheduler": "EULER-A", "model_type": "txt2img_controlnet" } ``` ## img2img ```json theme={null} { "prompt": "cat wizard, gandalf, lord of the rings, detailed, fantasy, cute, adorable, Pixar, Disney, 8k ", "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "image": "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg", "seed": 89395930, "guidance_scale": 7.0, "strength": 0.5, "scheduler": "EULER-A", "model_type": "img2img" } ``` ## img2img\_controlnet ```json theme={null} { "prompt": "cat wizard, gandalf, lord of the rings, detailed, fantasy, cute, adorable, Pixar, Disney, 8k ", "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "image": "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg", "canny_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny/resolve/main/canny.jpg", "canny_weightage": 0.4, "depth_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Depth/resolve/main/depth.jpg", "depth_weightage": 0.4, "seed": 89395930, "guidance_scale": 7.0, "strength": 0.5, "scheduler": "EULER-A", "model_type": "img2img_controlnet" } ``` ## inpaint ```json theme={null} { "prompt": "Face of a yellow cat, high resolution, sitting on a park bench ", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png", "mask_image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png", "seed": 89395930, "guidance_scale": 7.0, "strength": 0.5, "scheduler": "EULER-A", "clip_skip": 0, "use_foocus": true, "model_type": "inpaint" } ``` ## inpaint\_controlnet ```json theme={null} { "prompt": "Face of a yellow cat, high resolution, sitting on a park bench ", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png", "mask_image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png", "canny_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny/resolve/main/canny.jpg", "canny_weightage": 0.4, "depth_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Depth/resolve/main/depth.jpg", "depth_weightage": 0.4, "seed": 89395930, "guidance_scale": 7.0, "strength": 0.5, "scheduler": "EULER-A", "clip_skip": 0, "use_foocus": true, "model_type": "inpaint_controlnet" } ``` ## Example Response ```json theme={null} { "response_id": "afbc439946a44d98bb8062c8b36ec16d", "inference_time_taken": 6.336474418640137, "lora_time": 1.8092551231384277, "total_time_taken": 7.176232099533081, "request_id": "6b060ab415d84117b7b6403d622414f5", "error": null } ``` ## Key Notes * Ensure volume mounting in deployment for image storage. * ControlNet models are `not loaded by default`. * Supports `multiple pipelines` for text-to-image, image-to-image, and inpainting.