PR #2801 Add YuE (music gen) from fal.ai

Wauplin205 days ago (edited 203 days ago)❤ 1

Will keep it as draft for now. I mostly wanted to see how that integration would work given that it doesn't use the same parameters as we have for text_to_speech (has an extra "genres" parameters). For now I added it to the signature + docstring but it's not compliant with https://github.com/huggingface/huggingface.js/blob/main/packages/tasks/src/tasks/text-to-speech/spec/input.json.

Another possibility is to add a text-to-audio or text-to-music but that seems premature given it's a single model, single provider and still experimental. Will have to wait more example before an harmonization.

Working code example:

import os
from huggingface_hub import InferenceClient

lyrics = """
[verse]
In the town where I was born
Lived a man who sailed to sea
And he told us of his life
In the land of submarines
So we sailed on to the sun
'Til we found a sea of green
And we lived beneath the waves
In our yellow submarine

[chorus]
We all live in a yellow submarine
Yellow submarine, yellow submarine
We all live in a yellow submarine
Yellow submarine, yellow submarine
"""
genres = "pavarotti-style tenor voice"

t0 = time.time()
client = InferenceClient(
    provider="fal-ai",
    model="m-a-p/YuE-s1-7B-anneal-en-cot",
    api_key=os.getenv("FAL_AI_KEY"),
)
audio = client.text_to_speech(lyrics, extra_parameters={"genres": genres})
with open("output.mp3", "wb") as f:
    f.write(audio)

DnwXm1UGLGJZGPPFGbnL.mp4

(what a masterpiece, I know)

Process took 594s (~10min) to run on fal.ai. This means it will fail if we go through HF proxy as there is a 60s timeout. For now only works when passing a FAL_AI key directly. This can be fixed using async/queuing mechanism but out of scope for this PR.

YuE first test (fal.ai provider)

35b9158f

HuggingFaceDocBuilderDev205 days ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Merge branch 'main' into fal-text-to-speech

86988175

hanouticelina204 days ago

maybe genres can be passed inside extra_parameters when #2812 is merged.

Wauplin203 days ago

Waiting for it! 😄

hanouticelina203 days ago🎉 1

here you go : 07e1adb 😄

Merge branch 'main' into fal-text-to-speech

853da7f1

pass genres as extra parameter

f0c8897d

docstring

16d91538

Wauplin marked this pull request as ready for review 203 days ago

things

add270c8

Wauplin commented on 2025-01-31

src/huggingface_hub/inference/_providers/fal_ai.py

	156		def _prepare_payload(self, inputs: Any, parameters: Dict[str, Any]) -> Dict[str, Any]:
	157		parameters = {k: v for k, v in parameters.items() if v is not None}
	158		return {
	159		"lyrics": inputs,

Wauplin203 days ago

I wonder in the future how we will distinguish between "lyrics" (for music-gen) and "text" (for text-to-speech). For now there's only 1 model so let's do it this way.

hanouticelina203 days ago

yes, same issue with replicate for this task 😕

Wauplin requested a review from

hanouticelina 203 days ago

Wauplin changed the title ~~YuE first test (fal.ai provider)~~ Add YuE (music gen) from fal.ai 203 days ago

hanouticelina approved these changes on 2025-01-31

hanouticelina203 days ago

thank you!

Wauplin merged 63aada50 into main 203 days ago

Wauplin deleted the fal-text-to-speech branch 203 days ago

huggingface_hub
Add YuE (music gen) from fal.ai
#2801

Merged

Add YuE (music gen) from fal.ai #2801

huggingface_hub Add YuE (music gen) from fal.ai #2801 Merged

Add YuE (music gen) from fal.ai #2801

huggingface_hub
Add YuE (music gen) from fal.ai
#2801

Merged