Fix n^2 loop in tokenization (#254)

Commit

3 years ago

Fix n^2 loop in tokenization (#254) This causes long prompts to parse very slowly.