Adding FlaxNoRepeatNGramLogitsProcessor #29677
ArthurZucker
merged
14 commits into
huggingface:main
from
giganttheo:feature/ngram_blocking_flax
fix issue with logit processor in beam search in Flax
09b70d7e
adding FlaxNoRepeatNGramLogitsProcessor class + unit test
037bcc25
style correction and code verification
e63313c1
add FlaxNoRepeatNGramLogitsProcessor to the test_processor_list and t…
2a8ef7be
fix an issue where ngrams are banned only if they appear ==1 time + u…
022b83e7
replace non-jit compatible masking of ngrams that are not yet generat…
b33c4d02
Revert "fix issue with logit processor in beam search in Flax"
1ee47f45
add FlaxNoRepeatNGramLogitsProcessor to _get_logits_processor
b410517c
change the method of casting to boolean of banned tokens indices
26900e01
fix code style
8a6828c7
gante
approved these changes
on 2024-03-20
remove some useless operations + significantly faster computation of …
6c4a9e25
remove useless loop iterations
9a5c610a
set some variables that were calculated and used multiple times
1b99d0c2
amyeroberts
approved these changes
on 2024-03-28
fix format
382b52d8
Login to write a write a comment.
Login via GitHub