tokenizer

#24

by ulasarikaya - opened Dec 26, 2024

there isn't a specific ModernBertTokenizer like there is for DistilBert (transformers.DistilBertTokenizer).

could anyone clarify if there's an equivalent tokenizer available for ModernBert, or do I have to use a generic one like AutoTokenizer?

thanks

•

It uses the PreTrainedTokenizerFast tokenizer. You can see this in tokenizer_config.json. Auto tokenizer will choose that tokenizer class.

how to use it with BertTokenizerFast?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment