jaime-m-p 3b38d48609 Per token attributes (#7685) ...

* Add per token attributes enum
* Using phi-3 for testing 'rstrip'
* Using jina-v2 for testing 'lstrip'
* Brute force test for 'lstrip' and 'rstrip'
* Implement 'rstrip' and 'lstrip'
* Update phi-3 GGUF file (obsolete since 917dc8c)
* Replace llama_token_type with llama_token_attribs

2024-06-04 09:17:17 +02:00

709 KiB

Raw Permalink History

View Raw