jSrSSKrSSKrSSKrSSKJr SSKJr SSKJ r SSK J r SSK J r SSKJr SSKJrJr SS KJr SS KJr SS KJr SS KJrJr SS KJrJ r J!r!J"r" SSK#J$r$ SSK%J&r& SSK'J(r( SSK)J*r* SSK+J,r, SSK-J.r.J/r/J0r0J1r1J2r2J3r3J4r4 SSK5J6r6J7r7J8r8J9r9 \9Rt"\;5rSr?Sr@SrASrB\.S- r.\\ \!\"S.rC\=\@S.rD\7"\.5"SS \155rE\ErFg)!z Tokenization classes for fast tokenizers (provided by HuggingFace's tokenizers library). For slow (python) tokenizers see tokenization_utils.py N) defaultdict)Iterable)copyfile)Any)is_offline_mode) AddedToken processors)Encoding) Tokenizer)Decoder)BPEUnigram) BpeTrainerUnigramTrainerWordLevelTrainerWordPieceTrainer) cached_file) SpmConverter)convert_gguf_tokenizer)is_tekken_vocab_filename)load_gguf_checkpoint)INIT_TOKENIZER_DOCSTRING BatchEncodingPreTokenizedInputPreTrainedTokenizerBase TextInputTruncationStrategygenerate_merges)PaddingStrategyadd_end_docstringshf_apiloggingztokenizer.jsonzspecial_tokens_map.jsonztokenizer_config.jsonztokenizer.modelzadded_tokens.jsonztiktoken.modelu tokenizer_object ([`tokenizers.Tokenizer`]): A [`tokenizers.Tokenizer`] object from 🤗 tokenizers to instantiate from. See [Using tokenizers from 🤗 tokenizers](../fast_tokenizers) for more information. tokenizer_file ([`str`]): A path to a local JSON file representing a previously serialized [`tokenizers.Tokenizer`] object from 🤗 tokenizers. )r r WordLevel WordPiece)tokenizer_file vocab_filec)0^\rSrSrSr\rSrSr\ SUSj5r \ S\ S\ \ \ 4S\ \ \ 44S j5r\ S\ S\ \ \ 4S\ \ \ 44S j5rU4S jr\S\4S j5r\S\4S j5rSVS\ S\ S-S\\ 4SjjrSWS\ \R0-S\S-S\ S-S\S\ S-S\\ S44 U4SjjjrSr\S5r\S5r\R:S5r\R:S5rSr\S\4Sj5r S\ \ \44Sjr!\S\ \ \44Sj5r"\S\ \ \44Sj5r#\S\ \\$44S j5r%\#r&\%r'S\ \ \44S!jr(S\4S"jr)S\4S#jr*\S\+4S$j5r,\S\-4S%j5r.SXS'\/S(\S-S)\S-S*\S+\S,\S-\S.\S\\ \ \ 4\0\/44S/jjr1S0\ S\4S1jr2S2\S\ S-4S3jr3SUS4\0\ \$-S\4S5jjr4SUS6\S\4S7jjr5SUS8\\0\-S9\S\ \0\ -4S:jjr6SYS;\ S6\ S-S<\S\0\ 4S=jjr7S>\8S?\9S@\SA\SB\S-SC\ S-4 SDjr:SS&\8Rv\9RxSSESSSSSSSSSSS&S4S;\=\>-\0\=-\0\>-SF\=\>-\0\=-\0\>-S-S<\S>\8S?\9S@\S-SA\SG\SB\S-SC\ S-SH\S-S(\S-S)\S-S*\S+\S,\S-\S.\SI\S-S\?4(SJjjr@SK\0\ S\ 4SLjrASZSM\\0\-S9\SN\S-S\ 4SOjjrBS[S\ \R0-SP\\ S4S\S-S\ S-S\\ S44 SQjjrCS\SRjrD\ S]SSj5rESTrFU=rG$)^TokenizersBackendWa5 Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). NFc ^[U5nURSS5nUb_[RR U5(a;U[ LdSUR ;dU(a[R"U5US'U$UGb[RR U5(Ga[USS9n[R"U5nSSS5 WRS05RS5nUS ;a][U5n[US5n 0U S 'US :Xa/U S 'XS'/US '[R"[R"U55n O[R"U5n U RUS'U R US'U R"US'U R"bU R"US'U R bU R US'URS5n U (awU RSS5S:XaU Sn O[%U [&5(dU /n U H=n U RS5S:XdMSU ;dM"SSKn U R+U S5US' O URS05RS S5nUR,c0[%U[&5(a['[/[0U55nGO0UR,R2S:XaV[%U[&5(a@U(a9[%US[&[045(aUVs/sHn[1U5PM nnOUR,R2S:Xa"[5U5VVs0sH unnUU_M nnnOUR,R2S :XdUR,R2S:XaP[%U[&5(a;[5U5VVs0sH#unn[%U[&5(aUSOUU_M% nnnXS '[7USS5nS URS05;anU(agUR2S :XaWUSS nUVs/sH=n[%U[85(a[1UR;S55O [1U5PM? nnUUS 'U$URS5nURS5nURS 5nURS 5n[%U[85(aW[RR U5(a3[=U5(a#S S!KJ n U"U5nURC5US'U$[%U[85(a[RR U5(afURES"5(aP[RRGU5[H:waURKUU5nU$URSUU5nU$UcC[%U[85(a.[RR U5(a UUS 'US nUcC[%U[85(a.[RR U5(a UUS 'US nUcUR,bUR,R2S :Xa[%U[5(akS&[T[VS'[&[84U4S(jjm/S)Qn[Y5nUH&nUU;dM UR[T"UU/55 M( []UUS*9nUUS 'U$!,(df  GNJ=fs snfs snnfs snnfs snf![La'n[NRQS#US$US%35 SnAGNSnAff=f)+z Build a `tokenizers.Tokenizer` backend from the available serialization files (tokenizer.json, sentencepiece models, tekken.json, vocab/merges). r&N__init__tokenizer_objectutf-8encodingmodeltype)Nrvocabr merges added_tokenspost_processortokenizer_paddingtokenizer_truncation_json_truncation _json_padding normalizerSequence normalizers Precompiledprecompiled_charsmapr_spm_precompiled_charsmaprr$r% r' merges_filer)MistralConverter.modelz+Could not extract SentencePiece model from z$ using sentencepiece library due to z%. Falling back to TikToken extractor.valuesreturnc>/nUHVnUcM[U[[45(aURT"U55 M<UR [ U55 MX U$N) isinstancelisttupleextendappendstr)rE collectedval_iter_special_tokenss ڸC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\transformers/tokenization_utils_tokenizers.pyrQHTokenizersBackend.convert_to_native_format.._iter_special_tokenssY') !C{ !#e}55!(()=c)BC!((S2 "! ) pad_token unk_token bos_token eos_token sep_token cls_token mask_tokenadditional_special_tokensextra_special_tokens) skip_tokens)/dictpopospathisfiler)__dict__ TokenizerFast from_fileopenjsonloadgetfrom_strdumpsr6padding truncationrIrJbase64 b64decoder1maprK__name__ enumerategetattrrNsplitrintegrations.mistral.tokenizerrC convertedendswithbasenameTIKTOKEN_LEGACY_NAME_convert_from_sentencepiece Exceptionloggerwarning_convert_from_tiktokenrrsetupdater)clstrust_remote_codekwargs local_kwargsfast_tokenizer_filetokenizer_handletokenizer_json model_typeminimal_tokenizer_json minimal_model tok_from_filenormalizer_configr;ror3itemitokenr4merger'rBrC converterespecial_tokens_keysr^keyrQs @rRconvert_to_native_format*TokenizersBackend.convert_to_native_formatisF| *../?F +233))Zs||-KO`/CY]gCg%DJDTDT&'=>E %@A#4#&&w377FEyy eT** UE!23E##y0eT**uE!HtUZm9\9\5:;UTU4[UE;##{22;E2BC2Bha2BC##u, 0B0Bk0QeT**_hin_op_oS[STV[E4)@)@U1XeQN_oEp$) ! gt4J>--gr:: zObObfkOk'0:kqrkqbgZs5K5K% C 01QVW\Q]]kqr)/ X& !%%l3 "&&}5   )!!(+ j# & &277>>*+E+EJbcmJnJn H(4I/8/B/B/DL+ ,  j# & &277>>*+E+E*J]J]^fJgJgww +/CC#&#B#B:|#\L''55j,OL  =Z C88RWW^^J=W=W$.L ! )E >jc::rww~~k?Z?Z%0L "!(+F >cii3 8J8Je8SXbchjnXoXo !Xc] !tCy ! # %(EK*,&&&';\#=N)r'r]r-)convert_slow_tokenizerrrjitemsintrIr_rNrJkeyslenload_tiktoken_bpemaxrangerrMrw)rr'rrtoken_idrrr]base_vocab_sizemax_added_token_idindex new_extrasset_new_extras special_tokenrs rRr(TokenizersBackend._convert_from_tiktokens > %1$4$45K$L$RPR#Y#Y#[0 #[ Mz%/F/F5+CPUJ V#[ 0 +7*:*:;Q*R*XVX *D 1 1#'(<(A(A(C#D /AA*MNBVS!5!:!:!<=[\  0#?4JKKE%((2CE7"0MNK !_N!5  6%%m4"6$. %g +4+>+>+@ '(90 s ;E4 E:cjSSKJn U"U5nUR"UR40UD6nSSKJn UR UR 5nUb#[US5(aUR"S0UD6n[US5(aUR"S0UD6nS U;GaU[LdS UR;GaURS S5nURS S5n UR S 5=(d 0n UbU (aUR!5V V s0sHupX_M n n n U R!5HZup[#U 5n [%U5nU R U 5nU(dM5X:wdM<X;dMCURU5X'XU 'M\ [&R("UR*UU S9nUbUUS 'UR*R,nUR.S:a%UR1SUR2=(d S5 UR4S:a%UR1SUR6=(d S5 UR8S:a%UR1SUR:=(d S5 U$![a1n[RSUR SUS35 SnAGN3SnAff=fs sn n f)Nr)SentencePieceExtractor)SLOW_TO_FAST_CONVERTERSconvert_from_spmz,Could not reorder vocab using converter for z due to z/. Falling back to raw SentencePiece extraction.convert_from_spm_modelr-r,r3r4r)protor3r4rrWrXrVz)rrextractr1rrjrrhasattrrr|r}r~rr)rdr`rrrNrbuild_tokenizer_from_spm_protor trainer_specbos_id setdefault bos_pieceeos_id eos_pieceunk_id unk_piece)rr'rr extractorrconverter_classrr3r4rrr id_to_token new_token current_tokenr- proto_specs rRr{-TokenizersBackend._convert_from_sentencepiece4sB+:6  ((ClC   G599#,,GO*wHZ/[/[.??O,O 30 1 155E EL \ 1s>O7OS]eheqeqSq $$Wd3E!%%h5F$0#3#34J#K#Qr  %9FKkkmTm?5xm T+?+E+E+G'H"8}H #II$/OOH$=M$})C H^+099]+C(09H- ,H ,JJoo    +3C /0'__99 $$) ++K9M9M9VQVW$$) ++K9M9M9WQWX$$) ++K9M9M9XQXYW  NN>s||nHUVTWXGH   "UsAI1J/1 J,;&J''J,c >URSS5nURSS5nURSS5 URSS5nURSS5nURSS5nURS05nURSS 5n URS 5n URS 5n URS 5n Sn Ub[R"U5n GOUb<[R R U5(a[R"U5n GOUbS SK J nJ nJ n [URSS5U40UD6nU"U5unnUSU;a U"U5unnnO[U5nUSSnUSnUSn[UU5un nUR!U5 [#U5S:aUR!U5 GOUR$cU bU bT['U [(5(aU O&[+U 5VVVs0sH ununnUU_M snnnn[[-UU SSS95n O['U [(5(a[[-U /SSS95n Ou['U [.5(aGU (a@['U S[0[.45(a"[[3XRSS5S95n OUR$c [5S5eUc4Uc1UR$c$UR7SS5 UR7SS5 U bXlUR$c [5S 5eURS!S5=(d UR$R8=(d UnUbqUR$R:"S;0UD6 UR7S"US"5 UR7S#US$5 UR7S%US%5 UR7S&US'5 OUR$R=5 URS(S5=(d UR$R>=(d UnUbUR$R@"S;0UD6 UR7S)US)5 UR7S*US+5 UR7S,US$5 UR7S"US-5 UR7S.US.5 S/U;aS0US/'S1U;=(d S2U;nURS1S 5Ul!URS2S 5Ul"URS3S5=n(aUUR$l#U=(d UR$RFSLUl$[JT*U]"S;0UD6 U bXl'Xl(URRUR$l*URVV s1sHn [Y[[U 55iM n!n []UR_5S4S59V"V s/sH"un"n [Y[[U 55U!;dM U PM$ n#n"n [/UR`Rc55U#V s/sHn [eU 5PM sn -n$URfRi5H2n%U%cM[eU%5U$;dMU%U#;dM!U#RkU%5 M4 URlH-n [eU 5U$;dMU U#;dMU#RkU 5 M/ [#U#5S:a/n&URfRi5V's/sHn'U'(dM [eU'5PM n(n'U#Hpn ['U [d5(a [oU SS69n O<['U [n5(a'U Rp(d[eU 5U(;aSU l8U&RkU 5 Mr U&(aURsU&5 UR$Ru5n)U)S7:a[yUR$S8S5blURSS5 URz"UR$UR|RSS54UR|URS9S5S:.UD6UlURH=(d UR$RFSLUl$URH(aUR5 ggs snnnfs sn fs sn n"fs sn fs sn'f![va Sn)GNf=f),TokenizersBackend.__init__..s STUVSWrTr)speciali pre_tokenizerfix_mistral_regex) init_kwargsrr)@r`rjcopydeepcopyrarbrcrerfintegrations.ggufrrrrrrrr _tokenizerrIr_rsr rJrKr ValueErrorrrnenable_truncation no_truncationrmenable_padding_add_bos_token_add_eos_tokenr6_should_update_post_processorsuperr,r'rsplit_special_tokensencode_special_tokensrhashreprsortedradded_tokens_encoderrrN_special_tokens_maprErM_extra_special_tokensrr add_tokensget_vocab_sizeNotImplementedErrorrt_patch_mistral_regexrupdate_post_processor)+selfargsrr9r:r-rrrrr'r3r4fast_tokenizerrrr gguf_pathmetadata_ architecturetokenizer_dictr gguf_paramadditional_kwargsrw vocab_dict _truncation_paddingexplicit_bos_eos_in_kwargsr6radded_tokens_decoder_hashr tokens_to_addencoderspecial_token_valuetokenstall_named_tokens vocab_size __class__s+ rRr,TokenizersBackend.__init__ps"::&8$? ?D9   .5!::&8$?JJ{D1 $jj)94@%zz*@"E!::&8%@ZZ -  7#H%  '!]]+; n.>1)< )(3LA !+K!8#-.@#A 0F|Uc0d -N- MM* +$%) /0 __ $):!&0&=&=UZcdiZjCkZjYQPVQRTUAqDZjCk !.sF]ako/p!qE4((!.srTXbf/g!hE4((Uz%(UTXM7Z7Z!.wU::V^`aKb/c!d __ $r   &+;+CH_   k5 1   k6 2  %,O ?? "RS Sjj!7>p$//B\B\p`p  " OO - - < <   lK ,E F   / J   h H(= >   3[5L M OO ) ) +::148dDOO ?   2H=Q4R S F " ,F9 %4%>%[/U[B["$jj%@$jj%@#ZZ(8$? ?> ?-;DOO *-G-q4??KiKimqKq* "6"  !(O 0040I0I-DHD]D]$^D]5T$u+%6D]!$^!'';'A'A'C X X uDK (AA  X  t005578Ta;bTa5CJTa;bb$(#;#;#B#B#D "*&'w6;NVc;c$$%89 $E//E5z(U--G$$U+0 }  !F040H0H0O0O0QW0Q1UVA0Q W&eS))&ud;Ez22 ==SZ;K-K(,  e$'' 779J  74??OT#R#^ JJ{D )"77  $$^T:!,,"(**-@$"G   DO  . . X$//2P2PTX2X *  - -  & & ( .oDl~%_ US;a[SU<S35eUS:XaSSKJn 0nU(a5[RR U5(aUR U5nU"X5n U(aURSS5n URS [U5R[RR5S 5n [5R"U 4S S 0UD6Rn URUU UU URS 5S9 U $[ T U]D"U4UUUS.UD6$)av Save the full tokenizer state. Extends [`~tokenization_utils_base.PreTrainedTokenizerBase.save_pretrained`] with `save_format` support, for tokenizers converted from a native vocabulary file (e.g. Mistral's `tekken.json`). See the base method for `legacy_format`, `filename_prefix`, and `push_to_hub`. Args: save_format (`str`, *optional*): `"mistral"` to save as a native `tekken.json` by copying the original file (requires the original tekken vocabulary file to be available). The name is normalized to `tekken.json` on save, and the instantiated tokenizer must still match it exactly. `"hf"` or `None` for the default HuggingFace format. kwargs (`dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. Returns: A tuple of `str`: The files saved. )NhfmistralzUnknown save_format=z$. Supported values: 'hf', 'mistral'.r8r)save_tekken_formatcommit_messageNrepo_idexist_okTr) working_dirr;files_timestampsr:r)r3r(r4)rrvr9rarbr+_get_files_timestampsr`rNrusepr" create_repor;_upload_modified_filesrjrsave_pretrained) rr'r3r(r4r5rr9r? save_filesr:r;rs rRrD!TokenizersBackend.save_pretrained9s+> 5 53K?Bfgh h ) # J " rww}}^<<#'#=#=n#M +DAJ!',. dhijdkrTrrrrrrvks rRr&TokenizersBackend.added_tokens_encoders? *00I0I0O0O0QWk)lm)l 1 )lmmmAc6URR5$)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `dict[str, int]`: The added tokens. )rget_added_tokens_decoderr s rRr&TokenizersBackend.added_tokens_decoders7799rTc[URR5SS9VVs0sHupURU_M snn$s snnf)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `dict[str, int]`: The added tokens. c US$rrrps rRr3TokenizersBackend.get_added_vocab..rrrTrrsrts rRget_added_vocab!TokenizersBackend.get_added_vocabs?*00I0I0O0O0QWk)lm)l 1 )lmmmrxcg)z> Returns True, to avoid expensive `assert tokenizer` gotchas. Trr s rR__bool__TokenizersBackend.__bool__srTc4URRSS9$)z4 Size of the full vocabulary with the added tokens. Trergr s rR__len__TokenizersBackend.__len__s---EErTcUR$)zS `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )rr s rRbackend_tokenizer#TokenizersBackend.backend_tokenizers rTc.URR$)zE `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )rdecoderr s rRrTokenizersBackend.decoder s &&&rTTr0return_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosec UcSUR;nUcSUR;nU(aURbU/UR-n OU/n [[5n U Hn U SR U R 5 U(aU SR U R 5 U(aU SR U R5 U(aU SR U R5 U(aU SR U R5 U(dMU SR [U R 55 M X4$)ar Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) token_type_idsattention_mask input_idsspecial_tokens_maskoffset_mappingr) model_input_names overflowingrrJrMidstype_idsrroffsetsr) rr0rrrrrrr encodings encoding_dictrs rR_convert_encoding#TokenizersBackend._convert_encodings( ! ($48N8N$N ! ($48N8N$N ! $)=)=)I! X%9%99I! I#D) A + & - -aee 4$./66qzzB$./66q7G7GH)34;;A This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )rnum_special_tokens_to_add)rrIs rRr+TokenizersBackend.num_special_tokens_to_addOs&88>>rTrskip_special_tokensc@[U[5(aURRU5$/nU(a[ UR 5O [ 5nUH?n[U5nXT;aMUR URRU55 MA U$)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `list[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `list[str]`: The decoded token(s). )rIrrrrall_special_idsrM)rrrr ids_to_skiprs rRconvert_ids_to_tokens'TokenizersBackend.convert_ids_to_tokensds c3  ??..s3 33Fc$../CE EJE# MM$//55e< =   rTtextrc HUR"SXUS.UD6R5$)N)r text_pairrr) _encode_plusr)rrrIrrs rRtokenizeTokenizersBackend.tokenize~s(  ldOaleklssuurTpadding_strategyrrrrrcURRnURRnU[R:XaUbURR 5 OdUUUR URS.n UcSn O"U V s0sHoURU S5_M n n X:waURR"S0U D6 U[R:XaUbURR5 ggU[R:XaUOSn U UbUO URURUR UR"US.n X:waURR$"S0U D6 ggs sn f)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. N)rrrr)rrpad_idrUrrr)rrnrmrDO_NOT_TRUNCATErr\rrjrr DO_NOT_PAD no_padding MAX_LENGTHr pad_token_idrUrr) rrrrrrrrrtargetcurrentrvrs rRset_truncation_and_padding,TokenizersBackend.set_truncation_and_paddingsRBoo00 ??** "4"D"D D&--/) /55!11 F"@FG1kooa66G 11;F; 99 9#**,$$47Q7Q#QZW[F -9-E\4K\K\++!^^#55&8 F!..88"%HsE'rris_split_into_wordsreturn_tensorsrc vSnU"U5(d [S5eUbU"U5(d [S5eU(a@[U[[45=(a" U=(a [US[[45nO[U[[45nU(ay[U[5(a [ S5eUb<[ U5[ U5:wa$[S[ U5S[ U5S35eUb[[X55OUnOU(aX4/OU/n[U[[45(d[ S[U5S 35eURUUUUU U S 9 Uc URnURRU:waUURl URRUUUS 9nUVs/sHnURUU U UUUUUS 9PM nn0nUSSH.nUVVVs/sHunnUUHnUPM M n nnnU UU'M0 UVVVs/sHunnUHnUPM M n!nnnU(a4/n"[U5Hun#un$nU"U#/[ U$S 5-- n"M U"US'US Hn%UR!U%UU5 M [#UU!U S9n&U(dqU cnU(dg[#U&R%5VV's0sH5unn'U[ U'5S:a[U'S[5(aU'SOU'_M7 sn'nU&R&5n&U&$s snfs snnnfs snnnfs sn'nf)Nc[U[5(ag[U[[45(a[ U5S:Xag[US[5(ag[US[[45(a[ US5S:Xd[USS[5(ag[USS[[45(a4[ USS5S:H=(d [USSS[5$ggg)NTrF)rIrNrJrKr)rs rR_is_valid_text_input._is_valid_text_inputs!S!!Ae}--q6Q;!c**!tUm441Q4yA~AaDGS)A)A##AaDGdE];;"1Q47|q0OJqtAwqz34OO$ rTztext input must be of type `str` (single example), `list[str]` (batch or single pretokenized example) or `list[list[str]]` (batch of pretokenized examples) or `list[tuple[list[str], list[str]]]` (batch of pretokenized sequence pairs).rzdwhen tokenizing batches of text, `text_pair` must be a list or tuple with the same length as `text`.zbatch length of `text`: z- does not match batch length of `text_pair`: .z:batch_text_or_text_pairs has to be a list or a tuple (got ))rrrrrr)ris_pretokenized)r0rrrrrrrroverflow_to_sample_mapping) tensor_type)rrIrJrKrN TypeErrorrzipr2rrrr encode_batchrrs&_eventual_warn_about_too_long_sequencerrr)(rrrrrrrrrrrrrrrrrrrrrr is_batchedbatch_text_or_text_pairsrr0tokens_and_encodingssanitized_tokensrrr rstacksanitized_encodingsrrtoksrbatched_outputr\s( rRrTokenizersBackend._encode_pluss0 ($D))W   )=i)H)HW  #D4-8hThjQUVWQX[_afZgFhJ#D4-8J )S))$Tc)n)D .s4yk:I'q*FOEZtC,@'A`d $?H(9':dV $2UDMBBLTRjMkLllmn  ''- 3!1% (  '#'#<#< ?? 0 04H H4HDOO 1OO00 $1/1 $& &  " "!&;&;*C+E'=+ # &  '*1-C&:N&:74DIqQIQ&:EN$) S !.1ES0DWQdqdq0DS %)+ & )*> ? 9D!*qcC[8I4J.JJ*!@=W 9 :)+6I  7 7 :w W7''79LZhin4=V*'5&:&:&<&< Uc%j1nE!Hd9S9S%(Y^^&<(( NW "OS"s L"7L' &L.[SUSS55(a![RR5US'[ USSnU"S(X-S.UD6nU R#XUS9 U Gb@[R"U R55nSU ;aU SHnU SUSnUb"UVs/sHnUR%UU5PM nnUU SUS'UH"nU R'U5nUbM[ S 5e UVs/sHnU R'U5PM snU SUS!'M S"HGnUU ;dM U UunnUb UU;aUUnU R'U5nUc [ S 5eUU/U U'MI U US'[ R"[R"U55n UR(R+5n[,R.Hn[1UU5cM[1UU5nUb UU;aUUnUR2R%US5n[5U[5(a;[UUR6UR8UR:UR<S#S$9UU'MUUU'M UR>(aUR>R+5O/nUbURU5 [AU5S :aUUS%'XS&'URB"S(0UD6$s snfs snf![DaGnS'[GU5;a2UR S&S5 URB"S(0UD6nU UlUsSnA$eSnAff=f))u Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `list[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. r5r6Nr1r2r r3r4rrrg)r$r%z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece.rVridrcontinuing_subword_prefixend_of_word_suffixr ByteLevelr< pretokenizersc32# UH nUSS:Hv M g7f)r2rNr).0 pretokenizers rR .s!(X !(K7(Xsinitial_alphabet)rrJ)rtrainerrJrzQAttempted to set a token in the post processor that does not exist in the mappingr)rrAT) single_wordlstriprstripr`rr]r-z7multiple values for keyword argument 'tokenizer_object'r)$rhloadsrto_strr`rrerkrlrMrrLanypre_tokenizers_fastralphabetMODEL_TO_TRAINER_MAPPINGtrain_from_iteratorrjrrrrSPECIAL_TOKENS_ATTRIBUTESrtrrIrrrr`r]rrrrN)r text_iteratorrrnew_special_tokensspecial_tokens_maprrr5r6rVrrrJ added_tokenrr  trainer_classrtrained_tokenizer_jsonrrrrrspecial_token_fullr]r new_tokenizers rRtrain_new_from_iterator)TokenizersBackend.train_new_from_iteratorsDDOO$:$:$<=%)).9 '++,<= ' "6 *e 3/1N7 #G ,02N7 #H - G $V , 9g&x0<'0:*73G4D3Ew'0 G $V ,0J J/1N7 #G ,Mn]dNeflNmMno>>   *~g66w' 48JJ3EU\F]^iFj3kN7 #K 0!**4::n+EF 'K!ooi6Gd+Ag&v.);G!-+i2HL^2^); 1)*:;C+,<=cB8LF)5TZ![TZ5"4"8"8"FTZ![FLN#34S9(C!'#,#8#8#?#+", s#"(ouCuntejIDYDYZ_D`ntCuN#34S9%@<"0  N2-m>+F+ +{"\Dvj HCPQFR -t4 $ 8 8 +4 ($$ s04T' T,T11 V;;U=6V<U==Vc v^^SSKmSSKJn SSKJn SSKJn Jm U "SS9S[S [4UU4S jj5nU(d[5(aS nUGbRU(dU(GdCU"U5(Ga5U "US UUUS S US9nS nUb[USS9n[R"U5nSSS5 WRS5nURS5nU(a7U RU5U RS5:aU(a UbUS;aU$O-U(a&U RU5U RS5:aU$S nU(dU(Gd\U"U5(GaNU(aSU;a[!USUS5 U c:[#USS 5(d([!USS 5 [$R'SUS35 U$U S Ld[#USS 5(a[!USS 5 SSKnUR*R-UR/S5SS9nUR0n[3UUR*R45(aUUR0S'U$[3UUR*R65(aUR*R9S S S9nUR*R5UU/5UlU$!,(df  GN =f)a6 Patches mistral related tokenizers with incorrect regex if detected 1) Local file with an associated config saved next to it >> Model type one of the mistral models (on older versions) 2) Remote models on the hub from official mistral models >> Tags including `base_model:.*mistralai` rN) lru_cache)version)rr")maxsizemodel_idrFc>T"5RU5nURb1TRSSR UR55(agg![a gf=f)NFzbase_model:.*mistralairT) model_infor|tagssearchr-)rr1r"res rRis_base_mistral?TokenizersBackend._patch_mistral_regex..is_base_mistrals_ ++H5zz%995rwwuzz7JKK   sA A%$A%Tz config.jsonF) cache_dirrlocal_files_only%_raise_exceptions_for_missing_entries'_raise_exceptions_for_connection_errors _commit_hashr.r/transformers_versionrz5.0.0)r8mistral3voxtral ministralpixtralrz$The tokenizer you are loading from 'a' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.z[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+isolated)patternbehavior)r use_regex)r" functoolsr packagingrtransformers.utils.hubrr"rNboolrrgrhrirjparsesetattrrtr}r~rpre_tokenizersSplitRegexrrIr< Metaspacer)rrpretrained_model_name_or_pathrr%r&r)is_localrrrrrrr# _config_filemistral_config_detectedf_configr*transformers_model_typersplit_pretokenizercurrent_pretokenizerr"r"s @@rRr&TokenizersBackend._patch_mistral_regexns* '%> 3  c d    00H ( 4 X/:W*X*X&-#!16;8=) L', #',9Q"iilG:'.{{3I'J$*1++l*C' (GMM:N,ORYR_R_`gRh,h 3?3  )()gmm?\>]^eeH?'$.')EXZ_2`2`I':DA%)3)B)B)H)H * 0 0s!", *I*& ,5+B+B(!"6 8Q8Q8Z8Z[[5G //2"&&:J!VZ[^`c[cVd!!F9S9SRUX9[_`ceh`h[i99vh)T    !c !C$J !Z_`cZd ! &*&*!"& B bkk)B d{B t B  B 4Z B  sCxB B H 46666%%%%)@GCGG A4S>A tCH~  nd38nnn:d3 ?&;::10nc3hn$ FF = '''.2-1*/+0',#-(-( $d{-( $d{ -( $( -( %) -(!%-(-(-( tCH~tL11 2-(^ 7#7#*76d3+;&<6WZ6 ?d?s?*tCyt`cfjknfo`o4vSvd vtvjnorjsvI9)I90I9 I9  I9  $J I9DjI9\gk#',;,F,F2D2T2T!%$))-#'&*-1-1*/+0',#,0)X++d9o=EV@WWX004 ?BTJ[E\\_ccX! X * X 0 X$JXX"X $JXDjXt X $d{X $d{X$(X %)!X"!%#X$%X&'X(#Tk)X, -Xt tCy S %*48 )c?)")'+Tk ) )^&*&* bkk)#s(Od{  t  sCx * CJ BBrTr))GrJrrhra collectionsrcollections.abcrshutilrtypingrtokenizers.pre_tokenizersr9r huggingface_hubrrrr r rRr retokenizers.decodersr rQtokenizers.modelsr rtokenizers.trainersrrrrr5rrrintegrations.ggmlrintegrations.mistral.constantsrmodeling_gguf_pytorch_utilsrtokenization_utils_baserrrrrrrutilsr r!r"r# get_loggerrrr}rSPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILETIKTOKEN_VOCAB_FILEADDED_TOKENS_FILErzr r.r)PreTrainedTokenizerFastrrTrRris  #$7+-/16*^^.05D=HG   H %"3/'(( !! (6EXY,-Y/Y.Yz,,rT