Tokenization issue
#2
by sh0416 - opened
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/incoder-1B")
assert "from ." == tokenizer.decode(tokenizer("from .")["input_ids"], skip_special_tokens=True, cleanup_tokenization_spaces=False)
Raise an assertion error. I suspect that the encoding process remove space.. How to handle it?
Oh, there is a typo in option.. clean_up_tokenization_spaces=False works great. Sorry for the confusion.
sh0416 changed discussion status to closed