diffusers
157c9011 - Add BLIP Diffusion (#4388)

Comment changes are shownComment changes are hidden
Commit
1 year ago
Add BLIP Diffusion (#4388) * Add BLIP Diffusion skeleton * Add other model components * Add BLIP2, need to change it for now * Fix pipeline imports * Load pretrained ViT * Make qformer fwd pass same * Replicate fwd passes * Fix device bug * Add accelerate functions * Remove extra functions from Blip2 * Minor bug * Integrate initial review changes * Refactoring * Refactoring * Refactor * Add controlnet * Refactor * Update conversion script * Add image processor * Shift postprocessing to ImageProcessor * Refactor * Fix device * Add fast tests * Update conversion script * Fix checkpoint conversion script * Integrate review changes * Integrate reivew changes * Remove unused functions from test * Reuse HF image processor in Cond image * Create new BlipImageProcessor based on transfomers * Fix image preprocessor * Minor * Minor * Add canny preprocessing * Fix controlnet preprocessing * Fix blip diffusion test * Add controlnet test * Add initial doc strings * Integrate review changes * Refactor * Update examples * Remove DDIM comments * Add copied from for prepare_latents * Add type anotations * Add docstrings * Do black formatting * Add batch support * Make tests pass * Make controlnet tests pass * Black formatting * Fix progress bar * Fix some licensing comments * Fix imports * Refactor controlnet * Make tests faster * Edit examples * Black formatting/Ruff * Add doc * Minor Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Move controlnet pipeline * Make tests faster * Fix imports * Fix formatting * Fix make errors * Fix make errors * Minor * Add suggested doc changes Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * Edit docs * Fix 16 bit loading * Update examples * Edit toctree * Update docs/source/en/api/pipelines/blip_diffusion.md Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * Minor * Add tips * Edit examples * Update model paths --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Author
Parents
  • docs/source/en
    • File
      _toctree.yml
    • api/pipelines
      • File
        blip_diffusion.md
  • scripts
    • File
      convert_blipdiffusion_to_diffusers.py
  • src/diffusers
    • File
      __init__.py
    • pipelines
      • File
        __init__.py
      • blip_diffusion
        • File
          __init__.py
        • File
          blip_image_processing.py
        • File
          modeling_blip2.py
        • File
          modeling_ctx_clip.py
        • File
          pipeline_blip_diffusion.py
      • controlnet
        • File
          __init__.py
        • File
          pipeline_controlnet_blip_diffusion.py
    • utils
      • File
        dummy_pt_objects.py
  • tests/pipelines
    • blipdiffusion
      • File
        __init__.py
      • File
        test_blipdiffusion.py
    • controlnet
      • File
        test_controlnet_blip_diffusion.py