Summary of CMIS 4140 Control Net Prompting and Intro to LoRA Training 06/17/25

Summary of "CMIS 4140 Control Net Prompting and Intro to LoRA Training 06/17/25"

---

Main Ideas and Concepts

• Course Context and Introduction
  • This is the third class of CMIS 4140.
  • The instructor missed the previous class due to a project with the Atlanta Braves, hinting at future student collaboration opportunities.
  • The class focuses on catch-up and deepening understanding of ControlNet and LoRA training within Comfy UI and Flux.
  • These concepts apply broadly to stable diffusion workflows, not just Flux.
• ControlNet Overview
  • ControlNet is used to guide AI image generation by using reference images processed into depth maps, poses, line art (Canny edges), etc.
  • It helps control the output for consistent characters, backgrounds, poses, and art styles.
  • Different preprocessors (OpenPose, DW Preprocessor) extract pose, face, and hand data.
  • Multiple ControlNet modules can be chained, but overusing them can degrade image quality.
  • Depth maps are preferred for maintaining pose accuracy and quality.
  • Canny edges provide line art control, useful for artists who sketch.
  • ControlNet strength settings influence how strictly the AI follows the reference.
• LoRA (Low-Rank Adaptation) Models
  • LoRAs are custom-trained models that act as creative building blocks to represent characters, styles, objects, or locations.
  • The instructor provided example LoRAs of famous people and a product (Duke’s Mayonnaise) for academic use only (not commercial).
  • LoRAs enable consistent character generation and style control.
  • LoRAs can be blended or stacked but should be limited to 3-4 for best results.
  • Training LoRAs involves sourcing 26-41 handpicked images showing varied poses, expressions, lighting, and framing.
  • Using CG characters or real photos is acceptable; the instructor recommends clean, legal datasets.
  • Overtraining (too many images or too many epochs) can cause unnatural, overly CG-like results.
  • LoRAs trained on fewer, high-quality images tend to yield better, more natural results.
• Training LoRAs with Flux Gym on RunPod
  • Flux Gym is a tool/environment for training LoRAs.
  • RunPod cloud service is used to deploy GPU instances (e.g., Nvidia 4090) for training.
  • The process involves:
    • Preparing and renaming images (using Bulk Rename Utility for batch renaming).
    • Uploading images and captions to Flux Gym.
    • Setting training parameters:
      • Learning rate: 5e-4
      • Save every epoch (set to 1)
      • Network dimension (LoRA rank): 16 (default is 4)
      • Enable bucket (for multi-resolution images)
      • Minimum SNR gamma: 5
      • Multi-res noise discount: 0.3
      • Multi-res noise iterations: 6
      • Noise offset: 0.1
      • Train batch size: 2 (for 4090 GPU)
      • Max epochs: 8 (recommended to avoid overtraining)
    • Starting the training and monitoring progress.
    • Downloading trained LoRA files (typically epochs 6 and 7 are optimal).
  • Training can be done locally or on cloud pods; pods must remain active during training.
  • The instructor emphasizes the importance of prompt engineering alongside LoRA training for best results.
• Prompt Engineering and Workflow Tips
  • Use trigger words to call specific LoRAs in prompts (e.g., “Freeman” for Morgan Freeman LoRA).
  • Structure prompts with a cue for shot type, character description, environment, and realism modifiers.
  • Use tools like ChatGPT to generate or refine prompts.
  • Seed searching (testing different random seeds) is necessary to find the best output.
  • ControlNet can be toggled on/off to balance pose control vs. creative freedom.
  • Resolution settings (720p vs 1080p) impact speed and detail.
  • Steps (20-28) control the quality and training detail.
• Ethics and Usage
  • Provided LoRAs of celebrities are for academic use only, not commercial.
  • Students are encouraged to create their own LoRAs from legal, personally sourced images.
  • LoRAs can represent people, places, art styles, or objects.
  • The instructor stresses ethical considerations and responsibility when using AI-generated likenesses.
• Industry and Career Perspective
  • AI tools like ControlNet and LoRA represent a new frontier in digital art and content creation.
  • Mastery of these tools can provide a competitive edge in the chaotic creative industry.
  • The instructor shares personal motivation to democratize access to advanced creative tools at an affordable price.
  • Emphasizes the

Category

Educational

Video