j\ SrSSKrSSKrSSKrSSKJr SSKJr SSKJ r SSK J r J r J r JrJrJr SSKJr SS KJr SS KJr SS KJrJrJr SS KJrJr SS KJr SSK J!r!J"r" \"5(aSSK#J$r$ \RJ"\&5r'S\(\RR-S\(S-S\(S-4Sjr*S\(\RR-S\(S\(S-4Sjr+S,S\(\RR-S\,S-S\-\,\(S-44Sjjr.SSSSS.r/S\0S\-\1\(\244S jr3"S!S"5r4S#\S\(4S$jr5S#\S%\(SS4S&jr6S#\S'\(\RR-S\-\(S(44S)jr7S,S*\(S\(S-S\4S+jjr8g)-zOConversion between Mistral tekken tokenizers and HuggingFace tokenizer formats.N) lru_cache)Path)copyfile) AddedTokenRegex Tokenizerdecoderspre_tokenizers processors)BPE)bytes_to_unicode)TokenizersBackend) cached_fileloggingrequires_backends)CHAT_TEMPLATE_FILE#LEGACY_PROCESSOR_CHAT_TEMPLATE_FILE)is_mistral_common_available)TEKKEN_VOCAB_FILEis_tekken_vocab_filename) Tekkenizer tekken_file chat_templatereturncUbU$[U5RnU[- nUR5(aUR SS9$U[ - nUR5(a.[ USS9n[R"U5nSSS5 US$[5(aSSK J n U"U5$g!,(df  WS$=f![a&n[RSUSUS 35 SnAgSnAff=f) uLResolve the chat template to attach during tekken to HF conversion. Applies a fixed precedence order: 1. `chat_template` argument (if not `None`) — returned unchanged. 2. Sibling `chat_template.jinja` file in the same directory as *tekken_file*. 3. Sibling `chat_template.json` file — value of its `"chat_template"` key. 4. Automatic generation via `mistral_common.integrations.chat_templates` (lazy import). 5. `None` if none of the above succeed. Args: tekken_file (`str` or `os.PathLike`): Path to the `tekken.json` file. chat_template (`str` or `None`): Explicit chat template string. Only `None` triggers the lookup cascade; an empty string is returned as-is. Returns: Resolved chat template string, or `None`. Raises: KeyError: If `chat_template.json` exists but does not contain a `"chat_template"` key. Nutf-8encodingrr)"convert_tokenizer_to_chat_templatez'Failed to generate chat template from 'z': z#. Falling back to no chat template.)rparentris_file read_textropenjsonloadr9mistral_common.integrations.chat_templates.chat_templatesr! Exceptionlogger warning_once) rrr" jinja_path json_pathfdatar!excs ڹC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\transformers/integrations/mistral/tokenizer.py_resolve_chat_templater2*s.  +  % %F,,J##W#55<USLagURSS5 URSS5 USLaH[5(d [S5e[U[40UD6nUc[ S[S US 35eSU4$[5(dg[U[40UD6nUSLU4$) uhResolve whether to use `MistralCommonBackend` for tokenization. Probes for `tekken.json` in the checkpoint directory and checks whether `mistral-common` is installed to determine the appropriate tokenizer backend. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained model hosted on huggingface.co. - a path to a *directory* containing model files. mistral_format (`bool`, *optional*): Tri-state control for tokenizer backend selection: - `True` — force `MistralCommonBackend` (raises `ImportError` if `mistral-common` is not installed, raises `OSError` if `tekken.json` is not found). - `False` — force standard HuggingFace tokenizer. - `None` — auto-detect: selects native (`MistralCommonBackend`) when `mistral-common` is installed and a `tekken.json` is found, regardless of whether HF-format files are also present. Otherwise falls back to the standard HuggingFace tokenizer. **cache_kwargs: Forwarded to [`~utils.cached_file`] (e.g. `cache_dir`, `force_download`, `local_files_only`, `revision`, `token`). Returns: `tuple[bool, str | None]`: A tuple of `(use_mistral_format, tekken_file_path)` where `use_mistral_format` indicates whether `MistralCommonBackend` should be used, and `tekken_file_path` is the resolved path to `tekken.json` (or `None`). Raises: ImportError: If `mistral_format=True` and `mistral-common` is not installed. OSError: If `mistral_format=True` and `tekken.json` cannot be found. F)FNr6Nr7TzZmistral_format=True requires `mistral-common`. Install it with: pip install mistral-commonz Cannot find 'z' at 'zH'. Set `mistral_format=False` to use standard HuggingFace files instead.)popr ImportErrorr9rOSError)r3r;r8rs r1resolve_mistral_formatr@sNE*,,l  ""?ARcVbc   12&9V8WXXX k"" ' ( (;=N_R^_K t #[ 11r:zzzz) bos_token eos_token pad_token unk_tokenrawc>USnSnURS5nUc;[[S/5 [[SU5n[ [R 5n[SU55nU[ [[U555:wa[SU35e[SU55VVs/sHupVUPM nnnURS 5nUc [U5n[[U5U5V s/sHoRU S 9PM n n Xz-U4$s snnfs sn f) aDerive the expected special-token strings and special-token count from a parsed tekken.json. Single source of truth for the special-token layout, shared by `MistralConverter._parse_tekken_file` (to build the vocab) and `_check_tekken_vocab_unchanged` (to compare against a live tokenizer), so the two cannot silently drift apart. Args: raw (`dict`): A parsed tekken.json, i.e. the result of `json.load` on the file. Returns: `tuple[list[str], int]`: The special-token strings, ordered by rank (the real entries from *raw* followed by any `` filler needed to reach `num_special_tokens`), and the resolved `num_special_tokens`. Raises: ImportError: If *raw* has no top-level `"special_tokens"` key (old tekken format) and `mistral-common` is not installed. configzspecial_tokenszmistral-commonSPECIAL_TOKEN_TEMPLATEc3*# UH oSv M g7f)rankN.0entrys r1 *_derive_tekken_specials..sL7Ke&M7KszGtekken.json special_tokens ranks must be contiguous starting at 0, got c 3d# UH&nUS[[USSUS554v M( g7f)rK token_strvalueN)strgetattrrMs r1rPrQs8# -6]Ck(:GU;EW XY Z-s.0default_num_special_tokens)id) getrMistralConverterrVrlistDEPRECATED_SPECIAL_TOKENSsortedrangelen ValueErrorformat) rErGfiller_templatespecial_tokens_dictsdeclared_ranks_rSspecial_stringsnum_special_tokensifiller_stringss r1_derive_tekken_specialsrjs<(]F&O77#34#*-=,>?!*.FX#J$H$HIL7KLLNeC$7899bcqbrstt## -#   LA   $@A! 1 )*;+;?OD)#E)  @ 0)B&"'#) =0 /%qPNFs#D?E#E4E.E!? Erc^^ [5m [S[S[4U 4Sjj5n0n/nTR 5HupEXRU"U5'[ U5S:XaM /n[ S[ U55H8nUSUXGSpUT;dMU T;dMX-T;dM%URXU45 M: [UU4SjS9nURU5 M [USS9nUV s/sHo"U S 5U"U S54PM n n X+4$s sn f) zMExtract a unicode vocab and ordered BPE merge list from byte-level BPE ranks.brc >SRURS5Vs/sHnT[U5PM sn5$s snf)Nzlatin-1)joindecodeord)rchar byte_encoders r1token_bytes_to_string?MistralConverter._extract_merges..token_bytes_to_string;s877@ST@SLT3@STU UTs?rNc$>TUSTUS4$)NrrrL)xrs r12MistralConverter._extract_merges..Ks1Q4)AaD/0Rr:)keyc US$)NrL)vals r1rrNsAr:r) rrbytesrUr~r_r^appendr]extend) rrrx all_mergesrrKlocalindexpiece_lpiece_rrrrs ` @r1r} MistralConverter._extract_merges6s,()  VU Vs V  V!#57 $??,KE26'. /5zQEq#e*-#(%=%-i'Gy,@gFW\eEeLL'D!9:.5&RSE   e $-J,>? \fg\fUX(Q02GA2OP\fg}hs#Dc[[URURSS95n[ UR S5(aSUR lU$)zKBuild a raw `tokenizers.Tokenizer` with BPE model (no pre/post-processing).F)fuse_unk ignore_mergesT)rr rrhasattrmodelrrp tokenizers r1rMistralConverter.tokenizerRsEc$"9"94;S;S^cde 9??O 4 4,0IOO )r:cUR5n[R"[R"[ UR 5SSS9[R "URSS9/5Ul[R "5Ul URUR5 [R "SS9UlU$)zOBuild a fully configured `tokenizers.Tokenizer` with pre-tokenizer and decoder.isolatedF)behaviorinvert)rm use_regex) trim_offsets)rr SequenceSplitrru ByteLevelrm pre_tokenizerr decoderadd_special_tokensrr post_processorrs r1 convertedMistralConverter.convertedYsNN$ "0"9"9$$U4<<%8:V[\(($:O:O[`a #  %..0 $$T%C%CD#-#7#7U#K  r:)rrrmrru)F)__name__ __module__ __qualname____firstlineno____doc__rUboolrrro staticmethoddictrinttupler[r}rrr__static_attributes__rLr:r1rZrZsZ >3 >$ >%*S%*D%*T%*N4s #3d38ndSXY\^aYaSbNc>c8d699r:rZrc URRS5=(d [USS5n[U[5(a4[ U5(a$[ RRU5(d [S5eU$)aLocate the on-disk `tekken.json` a tokenizer was built from. Called once by `save_tekken_format`, which passes the resolved path on to `_check_tekken_vocab_unchanged` to know what to diff against. Args: tokenizer (`TokenizersBackend`): The tokenizer to resolve a source file for. Must have been loaded or converted from a native Mistral tekken vocabulary file, i.e. its `vocab_file` must still point at an existing file whose name identifies it as tekken (see `is_tekken_vocab_filename`). Returns: `str`: The validated path to the tokenizer's source tekken vocabulary file. Raises: OSError: If the tokenizer's `vocab_file` does not resolve to an existing file identified as tekken (see `is_tekken_vocab_filename`). rlNzCannot save in 'mistral' format: the original tekken.json is not available. Load a native checkpoint (that still contains tekken.json) or install mistral-common to use MistralCommonBackend.) init_kwargsrYrV isinstancerUrospathisfiler?)rrls r1_resolve_tekken_sourcerjss&&&**<8bGI|]a? **%9:"!#g,/2DDi.8A8V8V8]8]8_`8_u]]8_`4NO3KLN'>MN J   LL45F4GH I  LL<^rsV YY!6><<P=B  B   H %7bkk(97#PT*7Y\_cYc7t#&#4 4Z :#'C2#&#4C24KC2 4t  C2N  1@1@%S 3*?1@hffR&7C:> ->> C> TX> B*"3*S2;;EV*[`adfiai[j*^!%:r: