add support for jev like models #49566
ArthurZucker
wants to merge
76 commits into
main
from
add-jev-models
Cleanup and huge simplification to an alternate approach
7fc2116a
Cover all control tokens, not just special tokens
83a6d2b3
Comment cleanup
209fb6ce
Comment cleanup
3a1a7d2d
Refactor everything out into chat_template_utils, couple of cleanups
80064e77
Add FIXME
3f9df953
Merge remote-tracking branch 'origin/sanitize_chat_inputs' into add-j…
45679330
Add the decision pipeline: chat templates with marks, decision heads …
4fb24b84
Ignore the decision objects in the docstring checker
54ded2e6
Decision heads of the most downloaded model of each family, with thei…
3b6a517d
Decision heads as models named after the first model that used them, …
ffa7f5d7
Decision requests go through JSON once; docs of the jev and kev heads
474b4962
Merge branch 'main' into sanitize_chat_inputs
b9e16e18
Calibrated temperature and noul band as decision processors, out of t…
022ad0c9
Merge branch 'sanitize_chat_inputs' into add-jev-models
95db58d7
No adapter loading in the decision pipeline: decision LoRAs live at t…
2c1c8022
Unbloat the decision code; clef in modular on MBart, von with the mas…
d27afd9a
Decision head tester mixin; integration tests of the jev, kev and cle…
3c84d737
Forward subfolder to the adapter lookup of AutoModel; expected answer…
785af6fd
von on the decision head tester mixin, integration test of von
07cdef61
Decision code in a decision/ folder like generation/; expected answer…
77b1cdcd
No single-letter names in the decision code
3fbb46c2
Decision pipeline: no head lookup for in-memory models
5420db67
Unbloat von: marks as tensors, no Python loops
252d15cb
Unbloat kev head, template and tests
7e286fc2
No single-letter names
d23e65af
Let templates that escape control tokens skip sanitizing
5ddd5ff6
Clef: conversion mapping instead of a conversion script (WIP)
2842215c
Clef: WIP split into submodules
81f96a73
Base automatically changed from
sanitize_chat_inputs
to
main
7 days ago
Clef tests: new config keys, backbone width distinct from head width
97b2e39e
Merge branch 'main' into add-jev-models
f185a6f5
Drop decision/utils.py: the pipeline applies the head
49038def
Unbloat decision processing and tests; sanitize escaped sentinels and…
b24f33e2
Unbloat von and jev heads
ed18f839
Unbloat clef: simpler spans and routing, same 3 submodules
8ca239d0
Document prompt and lm_head once in auto_docstring
7a5e28da
Decision heads take tensors: option counts and type ids come from the…
b12b997f
better input checks :)
c53f7b8c
better type hinting
4335f5dd
type hint + QOL
e5073f28
cache question rendering / checks
457f0d91
updates
cab3b7bf
add image support
7c0ea55d
processor sanitization
378559bb
pass
0fb7e3e4
arbitrary types
6842f172
updates
22f742bb
Decision heads run their backbone; sanitize marked text in the tokenizer
ed4917a5
Load sub-models without weights in the checkpoint from the repo of th…
1b936d9a
Heads build their backbone from a required backbone_config; root conv…
b348601f
Match excluded sub-model prefixes with one compiled regex
e081eb3b
better
62a629ea
more model support?
cb4a4235
update
9b3e6454
Test the pplx-decider and d3 heads; document them in JEV
9f75041a
Test the d3 head
509bf9a4
Keep the readout of JEV heads in float32 from their config, as d3 run…
2fdb4c97
Convert causal and text-only code readouts and merged JEV checkpoints…
002b0773
JEV heads load in the dtype of their backbone; test JEV-35B
fee155ee
Merge branch 'main' into add-jev-models
2001517d
Root conversions only skip a base model of another type; hint at clos…
f3b2b310
Load the adapter of a sub-model read from another repo; convert the L…
7c66228c
Merge remote-tracking branch 'origin/add-jev-models' into add-jev-models
1cbb10f2
Merge branch 'main' into add-jev-models
2bf82476
keep_in_fp32_modules is a common config field; heads call their sub-m…
fc02b75b
Merge remote-tracking branch 'origin/add-jev-models' into add-jev-models
d1eee8ee
Merge branch 'main' into add-jev-models
74be39ab
Exclude a foreign base model only when it is a PreTrainedModel; allow…
73eed592
Merge remote-tracking branch 'origin/add-jev-models' into add-jev-models
a7705f03
Merge branch 'main' into add-jev-models
a8d5f84a
t checkout main
495a7af7
cleanup
631cdc5d
....
7d8ca57d
document the main contribution better
92b46a8f
update
b4b30fa5
better names and co
bb3f5758
Reviewers
Assignees
No one assigned
Labels
New model
Milestone
No milestone
Login to write a write a comment.
Login via GitHub