PR #12069 [Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM

DarkLight1337 assigned

DarkLight1337 1 year ago

refactor for images

f78ad12a

supprot image embedding for minicpmv

95230b95

[Bugfix][SpecDecode] Adjust Eagle model architecture to align with in…

42ffb1b2

[Bugfix] fused_experts_impl wrong compute type for float32 (#11921)

43ff2e9e

[CI/Build] Move model-specific multi-modal processing tests (#11934)

0ec99742

[Doc] Basic guide for writing unit tests for new models (#11951)

b4a90946

[Bugfix] Fix RobertaModel loading (#11940)

ac291981

[Model] Add cogagent model support vLLM (#11742)

286107f3

[V1] Avoid sending text prompt to core engine (#11963)

535e120b

[CI/Build] Add markdown linter (#11857)

925562bb

[Model] Initialize support for Deepseek-VL2 models (#11578)

936b3067

[Hardware][CPU] Multi-LoRA implementation for the CPU backend (#11100)

141151f2

[Hardware][TPU] workaround fix for MoE on TPU (#11764)

eac78115

[V1][Core][1/n] Logging and Metrics (#11962)

e2518667

[Model] Support GGUF models newly added in `transformers` 4.46.0 (#9685)

e46c06bf

[V1] [2/n] Logging and Metrics - `OutputProcessor` Abstraction (#11973)

d12c0de4

[MISC] fix typo in kv transfer send recv test (#11983)

e459c90f

[Bug] Fix usage of `.transpose()` and `.view()` consecutively. (#11979)

93a78bae

[CI][Spec Decode] fix: broken test for EAGLE model (#11972)

dd2f6271

[Misc] Fix Deepseek V2 fp8 kv-scale remapping (#11947)

570e067c

[Misc]Minor Changes about Worker (#11555)

eaccb74b

[platform] add ray_device_key (#11948)

7adb4a03

Fix Max Token ID for Qwen-VL-Chat (#11980)

a014ddd3

[Kernel] unified_attention for Attention.forward (#11967)

cedf6cc5

[Doc][V1] Update model implementation guide for V1 support (#11998)

a1f053f3

[Doc] Organise installation documentation into categories and tabs (#…

651ee498

[platform] add device_control env var (#12009)

adc0b547

[Platform] Move get_punica_wrapper() function to Platform (#11516)

1fa0b25e

bugfix: Fix signature mismatch in benchmark's `get_tokenizer` functio…

e55869e2

[Doc] Fix build from source and installation link in README.md (#12013)

7f2aa685

[Bugfix] Fix deepseekv3 gate bias error (#12002)

a1f08148

[Docs] Add Sky Computing Lab to project intro (#12019)

0ca468e2

[HPU][Bugfix] set_forward_context and CI test execution (#12014)

6bec0d04

[Doc] Update Quantization Hardware Support Documentation (#12025)

0badf142

[HPU][misc] add comments for explanation (#12034)

c6a5060b

[Bugfix] Fix various bugs in multi-modal processor (#12031)

055a2b7c

[Kernel] Revert the API change of Attention.forward (#12038)

941a5d5c

[Platform] Add output for Attention Backend (#11981)

3a05c492

[Bugfix][Kernel] Give unique name to BlockSparseFlashAttention (#12040)

87a687bc

Explain where the engine args go when using Docker (#12041)

3183e6ac

[Doc]: Update the Json Example of the `Engine Arguments` document (#1…

cc9cde52

[Misc] Merge bitsandbytes_stacked_params_mapping and packed_modules_…

58d45cd2

[Kernel] Support MulAndSilu (#11624)

bb13b8a6

[HPU][Bugfix] Don't use /dev/accel/accel0 for HPU autodetection in se…

1bba3f63

[Platform] move current_memory_usage() into platform (#11369)

ef22c6cf

[V1][BugFix] Fix edge case in VLM scheduling (#12065)

94adbff1

[Misc] Add multipstep chunked-prefill support for FlashInfer (#10467)

654f5d7f

[core] Turn off GPU communication overlap for Ray executor (#12051)

8146c68b

[core] platform agnostic executor via collective_rpc (#11256)

59e5cf4c

merge main

920038b3

video embedding supports

6f6d2eb0

update support for minicpmo on images and videos

364bca1b

audio language

c2d8dbb8

audio embedding inputs

1ba77eb4

format

1c6f7d84

HwwwwwwwH force pushed to 1c6f7d84 1 year ago

mergify added documentation

mergify added ci/build

mergify added frontend

mergify added needs-rebase

merge main x

26d40a57

merge main

24d9a809

mergify removed needs-rebase

Merge branch 'main' of https://github.com/vllm-project/vllm into mini…

29774db0

docs/server-chat-utils/tests for minicpmo

6c409c5e

Merge branch 'minicpmv-refactor' of github.com:HwwwwwwwH/vllm into mi…

ee2f7da8

HwwwwwwwH marked this pull request as ready for review 1 year ago

HwwwwwwwH requested a review from

DarkLight1337 1 year ago

HwwwwwwwH requested a review from

ywang96 1 year ago

DarkLight1337 commented on 2025-01-23

Update docs/source/models/supported_models.md

42e7e782

Update tests/models/decoder_only/vision_language/test_models.py

6c0a6863

format

c15228b8

Merge branch 'minicpmv-refactor' of github.com:HwwwwwwwH/vllm into mi…

c51026de

DarkLight1337 commented on 2025-01-23

split minicpmo in a separate file

ac26f599

format

8b0cbf7c

DarkLight1337 commented on 2025-01-23

Update vllm/model_executor/models/minicpmo.py

428ae5aa

add hints

edfac98d

format

4ed8b116

DarkLight1337 commented on 2025-01-24

clean unnecessary logic of WhisperEncoder

b44085ee

format

763c5784

DarkLight1337 commented on 2025-01-24

Update vllm/model_executor/models/minicpmo.py

cd684848

add torchaudio for test

1e47208d

add annotations

781d1c38

format

f0b0270c

ywang96 assigned

ywang96 1 year ago

Merge remote-tracking branch 'upstream/main' into minicpmv-refactor

ed1dd9eb

enable MiniCPMV-MiniCPMO for cache

6d5978a3

Merge branch 'minicpmv-refactor' of github.com:HwwwwwwwH/vllm into mi…

3bb67f88

add multimodal tests for minicpmv

25d86ce7

format

bec9a733

custom_hf_runner for minicpmo

2120dd6e

Merge branch 'minicpmv-refactor' of github.com:HwwwwwwwH/vllm into mi…

0fd43479

format

6d2f4e41

pass all tests

fac61ebf

format / pass all tests

6037606d

DarkLight1337 commented on 2025-01-27

fix num_slices bug

b6f24f70

Merge branch 'minicpmv-refactor' of github.com:HwwwwwwwH/vllm into mi…

e439d3a7

Isotr0py commented on 2025-01-27

add examples

0f67ac91

add examples and format tests

eab479fe

format

05a0ef81

Update tests/models/decoder_only/vision_language/vlm_utils/model_util…

6650450b

Update vllm/model_executor/models/minicpmv.py

8f5b0690

Update vllm/model_executor/models/minicpmv.py

de0b55f6

Update vllm/model_executor/models/minicpmv.py

ad528591

Update vllm/model_executor/models/minicpmv.py

c5b912d8

Update vllm/model_executor/models/minicpmo.py

00e9e5a4

alphabet

49ea11e3

add annotations

595c6791

Merge branch 'minicpmv-refactor' of github.com:HwwwwwwwH/vllm into mi…

061596f8

add torchaudio dependency

26d4b2bd

format

5867171d

torchaudio

bed7843f

fix minicpmo_patch_hf_runner

715bd9fd

fix slice bug

cf4788fb

Merge branch 'main' into minicpmv-refactor

53c679e7

format

3127a6b6

ywang96 added ready

test model register

290795b2

delete minicpmv2.5 in test_common

d9dedd70

add dependencies of minicpmo audio tests

f6d5cfa9

format

da2ddd3b

add vocos in requirements_test.in

4222899b

Merge branch 'minicpmv-refactor' of github.com:HwwwwwwwH/vllm into mi…

26ebc7cf

alphabet in example file and server

2e93896c

Merge branch 'main' into minicpmv-refactor

0dfa513c

DarkLight1337 approved these changes on 2025-01-28

mergify added needs-rebase

Merge branch 'main' into minicpmv-refactor

dadd0300

mergify removed needs-rebase

ywang96 approved these changes on 2025-01-29

mergify added needs-rebase

merge main && fix conflict

f5a188a3

delete vocos in setup.py

8216fd5f

mergify removed needs-rebase

DarkLight1337 commented on 2025-01-29

update docs

4cfd785c

DarkLight1337 enabled auto-merge (squash) 1 year ago

DarkLight1337 merged d93bf4da into main 1 year ago

vllm
[Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM
#12069

Merged

[Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM #12069

vllm [Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM #12069 Merged

[Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM #12069

vllm
[Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM
#12069

Merged