diffusers
[feat] Add ostris / Flex.2-preview pipeline
#11485
Open

[feat] Add ostris / Flex.2-preview pipeline #11485

victolee0 wants to merge 13 commits into huggingface:main from victolee0:feat/flex.2
victolee0
victolee085 days ago (edited 85 days ago)👍 1❤ 2🚀 1

What does this PR do?

Fixes #11394 (issue)

Before submitting

Who can review?

@yiyixuxu @asomoza

Example code

import torch
from diffusers import Flex2Pipeline
from diffusers.utils import load_image

name_or_path = "ostris/Flex.2-preview"

inpaint_image = load_image("https://ostris.com/wp-content/uploads/2025/04/dog.jpg")
inpaint_mask = load_image("https://ostris.com/wp-content/uploads/2025/04/dog_mask.jpg")
control_image = load_image("https://ostris.com/wp-content/uploads/2025/04/dog_depth.jpg")

dtype = torch.bfloat16

pipe = Flex2Pipeline.from_pretrained(
     name_or_path,
     torch_dtype=dtype
 ).to("cuda")

image = pipe(
     prompt="A white friendly robotic dog sitting on a bench",
     inpaint_image=inpaint_image,
     inpaint_mask=inpaint_mask,
     control_image=control_image,
     control_strength=0.5,
     control_stop=0.33,
     height=1024,
     width=1024,
     guidance_scale=3.5,
     num_inference_steps=50,
     generator=torch.Generator("cpu").manual_seed(42)
 ).images[0]
image.save(f"robot_dog.png")

Result:

robot_dog

Official Example:

image

feat: add flex.2 pipeline
d462ad32
add: __init__
5de7cdc2
fix
2f6fda3e
fix transformer model input
6b36666f
fix: remove unuse args in __init__
100e8a13
fix: fix typo in __call__
3b2d0bf7
docs: fix flex docs
88428b78
docs: fix flex docs
fff055ad
refactor: remove constant variable
291fb843
fix: typo in docstring
db3cc522
fix: remove print
6283f93c
test: add test for flex
dcd4563f
fix: for test
46d62afb
victolee0 victolee0 changed the title [WIP] Add ostris / Flex.2-preview pipeline [feat] Add ostris / Flex.2-preview pipeline 85 days ago
tin2tin
tin2tin85 days ago

Very cool. Haven't tested it yet. Does it automatically detect the type of control image is input (ex. OpenPose, canny or depth)?

victolee0
victolee085 days ago

@tin2tin No, you must manually select the correct type that matches your specific input image.

tin2tin
tin2tin85 days ago

How/where do you set that? In the ex. It's depth, but it could have been ex. Canny as well?

nitinmukesh
nitinmukesh85 days ago (edited 85 days ago)

@tin2tin

It's very complex, took me 5 days. I just finished integrating, you can refer my repo for code tomorrow but it's custom pipeline based and you can replace with diffusers pipeline.

image


image

nitinmukesh
nitinmukesh85 days ago (edited 85 days ago)

How/where do you set that? In the ex. It's depth, but it could have been ex. Canny as well?

You don't have to specify that, it automatically determines it.
control_image can have Line, Pose, Depth.

victolee0
victolee085 days ago👍 1

Oh, I misunderstood the question. sry

victolee0
victolee082 days ago
asomoza
asomoza82 days ago (edited 82 days ago)

Hi @victolee0 and thanks for your contribution, I really like what @jaretburkett and Flex wants to do but the current model is an Early access preview release which means that we should probably wait for the full release.

Also I see two more issues with this PR:

  • This pipeline is the custom pipeline that the author published with the model right? If it is, then the priority for adding this goes to the model author unless they don't want to.
  • Flex uses Flux as a base (with some modifications) and probably it's just going to be just this pipeline which I'm not sure that needs a whole directory and maybe it can go in the Flux one. Not sure about this and also we're going to have the same problem with chroma. What do you think @yiyixuxu?
victolee0
victolee060 days ago

Thanks for the feedback! I reached out to the original model author about this PR, but haven't received a response for about two weeks. Given that the priority should go to the model author and considering the early access status you mentioned, I think it would be best to close this PR for now.
If the original author decides they don't want to handle the integration in the future, I'd be happy to revisit this with the proper approach.
Thanks again for taking the time to review this!

@asomoza

Login to write a write a comment.

Login via GitHub

Reviewers
No reviews
Assignees
No one assigned
Labels
Milestone