diffusers
Feature flux controlnet img2img and inpaint pipeline
#9408
Merged
Overview
Commits
16
Changes
View On GitHub

Feature flux controlnet img2img and inpaint pipeline #9408

yiyixuxu merged 16 commits into huggingface:main from ighoshsubho:feature/flux_controlnet_image_inpaint
ighoshsubho
ighoshsubho248 days agoπŸš€ 2

What does this PR do?

Fixes #9402 and #9158

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

ighoshsubho Implemented FLUX controlnet support to Img2Img pipeline
070cc7cf
ighoshsubho Init remove inpainting
b7e7da32
ighoshsubho
ighoshsubho247 days ago

Hey @a-r-r-o-w can you suggest if the flux img2img controlnet pipeline needs any fix or if this architecture is fine to go with.

a-r-r-o-w
a-r-r-o-w commented on 2024-09-11
Conversation is marked as resolved
Show resolved
src/diffusers/pipelines/flux/pipeline_flux_controlnet_image_to_image.py
80 timesteps = scheduler.timesteps
81 return timesteps, num_inference_steps
82
83
class FluxControlNetImg2ImgPipeline(FluxControlNetPipeline):
a-r-r-o-w247 days ago

We don't derive from pipelines. Instead, we use a # Copied from mechanism to ensure that methods required across different pipelines remain consistent in implementation. You could refer to some of our other Controlnet img2img pipelines for an example on this. Here's the Diffusers philosophy guide which might find insightful

Here, you could derive from the base DiffusionPipeline and FluxLoraLoaderMixin classes. I think the FromSingleFileMixin is also applicable here as used in FluxControlNetPipeline. After that you can copy all the relevant methods you need.

ighoshsubho247 days agoπŸ‘ 1

sure will make an update for this

ighoshsubho Flux controlnet img2img and inpaint pipeline
f80d17c6
ighoshsubho ighoshsubho changed the title Feature flux controlnet image to image pipeline Feature flux controlnet img2img and inpaint pipeline 247 days ago
ighoshsubho Merge branch 'main' into feature/flux_controlnet_image_inpaint
ba3f1775
ighoshsubho ighoshsubho marked this pull request as ready for review 247 days ago
yiyixuxu
yiyixuxu approved these changes on 2024-09-11
yiyixuxu246 days ago❀ 1

thanks!

Conversation is marked as resolved
Show resolved
src/diffusers/pipelines/flux/pipeline_flux_controlnet_image_to_image.py
518
519 return latents
520
521
 def prepare_latents(
yiyixuxu246 days ago

this should be same as img2img, can we add a copied from?

ighoshsubho246 days ago

will do

ighoshsubho246 days ago

fixed

yiyixuxu
yiyixuxu246 days agoπŸ‘ 1

awesome! thank you!
can you share example script + result from these two pipelines?

yiyixuxu
yiyixuxu246 days agoπŸ‘ 1

can you run make style and fix copies so CI can pass?

ighoshsubho
ighoshsubho246 days ago

awesome! thank you!
can you share example script + result from these two pipelines?

Sure I will!

ighoshsubho Merge branch 'huggingface:main' into feature/flux_controlnet_image_in…
72ac92a6
ighoshsubho style and quality enforced
d71ef15a
ighoshsubho doc string added for controlnet flux inpaint and img2img pipelines, a…
d55b5cb1
ighoshsubho added example usecases in inpaint and img2img pipeline
a30ca655
ighoshsubho
ighoshsubho246 days ago

awesome! thank you! can you share example script + result from these two pipelines?

awesome! thank you! can you share example script + result from these two pipelines?

Here is example for img2img -

diffusers/src/diffusers/pipelines/flux/pipeline_flux_controlnet_image_to_image.py

Line 42 in a30ca65

Examples:

Here is example for inpaint -

diffusers/src/diffusers/pipelines/flux/pipeline_flux_controlnet_inpainting.py

Line 43 in a30ca65

Examples:

ighoshsubho
ighoshsubho245 days ago

@yiyixuxu any updates, are we good to go or does it require further changes

ighoshsubho make fix copies added
b29d96ff
ighoshsubho Merge branch 'main' into feature/flux_controlnet_image_inpaint
21862e2a
yiyixuxu
yiyixuxu244 days agoπŸ‘ 1

can we look into the failing fast tests?

a-r-r-o-w
a-r-r-o-w commented on 2024-09-13
a-r-r-o-w244 days agoπŸ‘ 2

Thank you, the PR looks very clean and everything seems right! Great work @ighoshsubho

Just some minor nits from me. Could you also add these pipelines to the flux docs and address the failing tests?

Btw @yiyixuxu, do we need the slow integration tests here? I think we don't have them on derived pipelines that are made from two or more components (such as Flux + ImageToImage, or AnimateDiff + VideoToVideo)

Conversation is marked as resolved
Show resolved
src/diffusers/pipelines/flux/pipeline_flux_controlnet_inpainting.py
175 return timesteps, num_inference_steps
176
177
178
class FluxControlNetInpaintPipeline(DiffusionPipeline, FluxLoraLoaderMixin, FromSingleFileMixin):
a-r-r-o-w244 days ago

Could you add the docstring here similar to this?

ighoshsubho244 days ago

Fixed now

Conversation is marked as resolved
Show resolved
src/diffusers/pipelines/flux/pipeline_flux_controlnet_image_to_image.py
167 return timesteps, num_inference_steps
168
169
170
class FluxControlNetImg2ImgPipeline(DiffusionPipeline, FluxLoraLoaderMixin, FromSingleFileMixin):
a-r-r-o-w244 days ago

Docstring here too

ighoshsubho244 days ago

Fixed now

Conversation is marked as resolved
Show resolved
src/diffusers/pipelines/flux/pipeline_flux_controlnet_image_to_image.py
679 is True, otherwise a `tuple`. When returning a tuple, the first element is a list with the generated
680 images.
681
682
 Examples:
a-r-r-o-w244 days ago

Usually, we have the Examples before the Returns. Let us know if you saw a difference somewhere

ighoshsubho244 days ago

Fixed

Conversation is marked as resolved
Show resolved
src/diffusers/pipelines/flux/pipeline_flux_controlnet_inpainting.py
793 is True, otherwise a `tuple`. When returning a tuple, the first element is a list with the generated
794 images.
795
796
 Examples:
a-r-r-o-w244 days ago

Same comment on Examples being before Returns

ighoshsubho244 days ago

This is fixed too

ighoshsubho
ighoshsubho244 days ago (edited 243 days ago)

Thank you, the PR looks very clean and everything seems right! Great work @ighoshsubho

Just some minor nits from me. Could you also add these pipelines to the flux docs and address the failing tests?

Btw @yiyixuxu, do we need the slow integration tests here? I think we don't have them on derived pipelines that are made from two or more components (such as Flux + ImageToImage, or AnimateDiff + VideoToVideo)

will address the tests and minor fixes

quick update, @a-r-r-o-w and @yiyixuxu I found a small bug in my inpaint controlnet pipeline for which the tests are failing will address this by today

ighoshsubho docs added for img2img and inpaint, also added docs to pipelines
e543d3a5
a-r-r-o-w
a-r-r-o-w commented on 2024-09-15
a-r-r-o-w243 days ago

Just two more change requests

Conversation is marked as resolved
Show resolved
docs/source/en/api/pipelines/flux.md
160162
161163## FluxPipeline
162164
163[[autodoc]] FluxPipeline
164 - all
165 - __call__
165
[[autodoc]] FluxPipeline - all - **call**
a-r-r-o-w243 days ago

Not sure why these were changed, please revert to keep the scope of this PR limited

ighoshsubho243 days ago

sure will do

Conversation is marked as resolved
Show resolved
docs/source/en/api/pipelines/flux.md
174172
175[[autodoc]] FluxInpaintPipeline
176 - all
177 - __call__
173[[autodoc]] FluxInpaintPipeline - all - **call**
174
175
## Flux ControlNet Inpaint Pipeline
a-r-r-o-w243 days ago (edited 243 days ago)

We don't need to provide an example here. autodoc will take the example from the EXAMPLE_DOC_STRING defined at the top of the pipeline file and show the relevant usage here.

ighoshsubho243 days ago

sure

ighoshsubho Fix tests and minor bugs
245f97bc
ighoshsubho Merge branch 'main' into feature/flux_controlnet_image_inpaint
30be6075
ighoshsubho fix flux docs
b466a7a4
ighoshsubho Flux tests fix
833d3488
ighoshsubho
ighoshsubho242 days ago

@yiyixuxu and @a-r-r-o-w fixed the minor bugs and tests should pass now

mrtpk
mrtpk241 days ago❀ 4

Can't wait for this awesomeness!!!!

ighoshsubho Merge branch 'main' into feature/flux_controlnet_image_inpaint
b385dcf7
HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev240 days ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

yiyixuxu yiyixuxu merged bb1b0fa1 into main 240 days ago
yiyixuxu
yiyixuxu240 days agoπŸŽ‰ 3

thank you!

mrtpk
mrtpk240 days ago❀ 1

Thank you @ighoshsubho

DoiryCool
DoiryCool207 days ago

How can I use other controlnets, such as XLabs-AI/flux-controlnet-canny-v3? The canny model in the example exceeds my vmem

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum