llama.cpp
34ff2e04 - mtmd: drama3 voice cloning (#32586)

Commit
9 hours ago
mtmd: drama3 voice cloning (#32586) * mtmd: drama3 voice cloning The Fish S2 codec encoder (fish_s2_spkenc) now also serves drama3: - conversion: drama3 writes the codec encoder and the audio encoder KV, shared with fish_s2 - fish_s2_spkenc: outputs the codes of the clip (gen-side reference) and, in a drama3_gen mmproj, builds the drama3 backbone rows (E_lm[semantic] + E_cb[1][code 1]) / sqrt(2) - drama3_gen tower: teacher forcing when the caller gives the codes of the next frame - drama3 pipeline: reference-bracket-v1 prompt with one reference (vLLM-Omni fish_omni), the reference frames go through the backbone as codec rows and through the tower with their codes - mtmd_gen_state_get_ref_codes(): read-only access to the reference codes of a gen state Claude-Session: https://claude.ai/code/session_01PpX3DmaxNSmJbxGaxnUBrX * fix correctness * clean up * rm redundant code * fish_s2: rm unused import --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Author
Parents
Loading