jqSrSSKrSSKrSSKrSSKrSSKrSSKJrJ r J r SSK J r \ "\ 5rSrSrSrSr\ "S S 55r"S S \5r"S S\5rSrg)z[Encoding categorical data as integers Authors * Samuele Cornell 2020 * Aku Rouhe 2020 N)mark_as_loader mark_as_saverregister_checkpoint_hooks) get_loggerzzcH\rSrSrSrSrSrS'SjrSrSr \ S5r S(S jr S(S jr S)S jr//S S 04SjrSrSrSrSr\4SjrSrSrS*SjrS*SjrS*SjrS*SjrSrSr\S5rSr \!S(Sj5r"Sr#S r$S!r%S"r&S#r'\(S$5r)\(S%5r*S&r+g )+CategoricalEncoder a Encode labels of a discrete set. Used for encoding, e.g., speaker identities in speaker recognition. Given a collection of hashables (e.g a strings) it encodes every unique item to an integer value: ["spk0", "spk1"] --> [0, 1] Internally the correspondence between each label to its index is handled by two dictionaries: lab2ind and ind2lab. The label integer encoding can be generated automatically from a SpeechBrain DynamicItemDataset by specifying the desired entry (e.g., spkid) in the annotation and calling update_from_didataset method: >>> from speechbrain.dataio.encoder import CategoricalEncoder >>> from speechbrain.dataio.dataset import DynamicItemDataset >>> dataset = { ... "ex_{}".format(x): {"spkid": "spk{}".format(x)} for x in range(20) ... } >>> dataset = DynamicItemDataset(dataset) >>> encoder = CategoricalEncoder() >>> encoder.update_from_didataset(dataset, "spkid") >>> assert len(encoder) == len( ... dataset ... ) # different speaker for each utterance However can also be updated from an iterable: >>> from speechbrain.dataio.encoder import CategoricalEncoder >>> from speechbrain.dataio.dataset import DynamicItemDataset >>> dataset = ["spk{}".format(x) for x in range(20)] >>> encoder = CategoricalEncoder() >>> encoder.update_from_iterable(dataset) >>> assert len(encoder) == len(dataset) Note ---- In both methods it can be specified it the single element in the iterable or in the dataset should be treated as a sequence or not (default False). If it is a sequence each element in the sequence will be encoded. >>> from speechbrain.dataio.encoder import CategoricalEncoder >>> from speechbrain.dataio.dataset import DynamicItemDataset >>> dataset = [[x + 1, x + 2] for x in range(20)] >>> encoder = CategoricalEncoder() >>> encoder.ignore_len() >>> encoder.update_from_iterable(dataset, sequence_input=True) >>> assert len(encoder) == 21 # there are only 21 unique elements 1-21 This class offers 4 different methods to explicitly add a label in the internal dicts: add_label, ensure_label, insert_label, enforce_label. add_label and insert_label will raise an error if it is already present in the internal dicts. insert_label, enforce_label allow also to specify the integer value to which the desired label is encoded. Encoding can be performed using 4 different methods: encode_label, encode_sequence, encode_label_torch and encode_sequence_torch. encode_label operate on single labels and simply returns the corresponding integer encoding: >>> from speechbrain.dataio.encoder import CategoricalEncoder >>> from speechbrain.dataio.dataset import DynamicItemDataset >>> dataset = ["spk{}".format(x) for x in range(20)] >>> encoder.update_from_iterable(dataset) >>> 22 >>> encode_sequence on sequences of labels: >>> encoder.encode_sequence(["spk1", "spk19"]) [22, 40] >>> encode_label_torch and encode_sequence_torch return torch tensors >>> encoder.encode_sequence_torch(["spk1", "spk19"]) tensor([22, 40]) >>> Decoding can be performed using decode_torch and decode_ndim methods. >>> encoded = encoder.encode_sequence_torch(["spk1", "spk19"]) >>> encoder.decode_torch(encoded) ['spk1', 'spk19'] >>> decode_ndim is used for multidimensional list or pytorch tensors >>> encoded = encoded.unsqueeze(0).repeat(3, 1) >>> encoder.decode_torch(encoded) [['spk1', 'spk19'], ['spk1', 'spk19'], ['spk1', 'spk19']] >>> In some applications, it can happen that during testing a label which has not been encountered during training is encountered. To handle this out-of-vocabulary problem add_unk can be used. Every out-of-vocab label is mapped to this special label and its corresponding integer encoding. >>> import torch >>> try: ... encoder.encode_label("spk42") ... except KeyError: ... print("spk42 is not in the encoder this raises an error!") spk42 is not in the encoder this raises an error! >>> encoder.add_unk() 41 >>> encoder.encode_label("spk42") 41 >>> returns the encoding This class offers also methods to save and load the internal mappings between labels and tokens using: save and load methods as well as load_or_create. z => z================ c N0Ul0UlXlURU5 gN)lab2indind2labstarting_indexhandle_special_labels)selfrspecial_labelss ڨC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\speechbrain/dataio/encoder.py__init__CategoricalEncoder.__init__s&  , "">2c:SU;aURUS5 gg)z)Handles special labels such as unk_label. unk_labelN)add_unk)rrs rr(CategoricalEncoder.handle_special_labelss . ( LL 4 5 )rc,[UR5$r )lenrrs r__len__CategoricalEncoder.__len__s4<<  rc6U"5nURU5 U$)z,Recreate a previously saved encoder directly)load)clspathobjs r from_savedCategoricalEncoder.from_savedse  rFcU(a [RRU5nO [U5nUHnUR U5 M g)zUpdate from iterator Arguments --------- iterable : iterable Input sequence on which to operate. sequence_input : bool Whether iterable yields sequences of labels or individual labels directly. (default False) N) itertoolschain from_iterableiter ensure_label)riterablesequence_inputlabel_iteratorlabels rupdate_from_iterable'CategoricalEncoder.update_from_iterables< &__::8DN!(^N#E   e $$rc^URT/5 URU4SjU5US9 SSS5 g!,(df  g=f)aQUpdate from DynamicItemDataset. Arguments --------- didataset : DynamicItemDataset Dataset on which to operate. output_key : str Key in the dataset (in data or a dynamic item) to encode. sequence_input : bool Whether the data yielded with the specified key consists of sequences of labels or individual labels directly. c3,># UH oTv M g7fr ).0 data_point output_keys r ;CategoricalEncoder.update_from_didataset..sD)JJ')s)r/N)output_keys_asr2)r didatasetr9r/s ` rupdate_from_didataset(CategoricalEncoder.update_from_didatasets? % %zl 3  % %D)D- & 4 3 3s 8 ANc|UR(a/URRn[R SUS35 U(a [ R RU5nO [U5n[R"U5nURU5HupX:a U$URU5 M U$)abProduce label mapping from iterable based on label counts Used to limit label set size. Arguments --------- iterable : iterable Input sequence on which to operate. sequence_input : bool Whether iterable yields sequences of labels or individual labels directly. False by default. n_most_common : int, None Take at most this many labels as the label set, keeping the most common ones. If None (as by default), take all. min_count : int Don't take labels if they appear less than this many times. Returns ------- collections.Counter The counts of the different labels (unfiltered). z+Limited_labelset_from_iterable called, but zx is not empty. The new labels will be added, i.e. won't overwrite. This is normal if there is e.g. an unk label already.) r __class____name__loggerinfor)r*r+r, collectionsCounter most_common add_label) rr.r/ n_most_common min_countclsnamer0countsr1counts rlimited_labelset_from_iterable1CategoricalEncoder.limited_labelset_from_iterables2 <<nn--G KKi HH  &__::8DN!(^N$$^4"..}=LE   NN5 ! >  rcD[RRR5(a{UR U5(deUHnUR Xt5 M UH#nUc [ S5eURXU5 M% URU5 URU5 [RRR5 URU5 g![RRR5 URU5 f=f)zConvenient syntax for creating the encoder conditionally This pattern would be repeated in so many experiments that we decided to add a convenient shortcut for it here. The current version is multi-gpu (DDP) safe. Nz,Provide an output_key for DynamicItemDataset) sbutils distributedif_main_processload_if_possibler2 ValueErrorr>rsave ddp_barrierr") rr$from_iterablesfrom_didatasetsr/r9rr.r=s rload_or_create!CategoricalEncoder.load_or_creates xx##3355,,T22$211(K%3%4 %-", N#22%> &5..~>IIdO HH , , . IIdO HH , , . IIdOs B(C$$;DcXR;a$URRn[SU35eUR 5nX0RU'XR U'U$)a<Add new label to the encoder, at the next free position. Arguments --------- label : hashable Most often labels are str, but anything that can act as dict key is supported. Note that default save/load only supports Python literals. Returns ------- int The index that was used to encode this label. Label already present in )rrArBKeyError _next_indexr)rr1rKindexs rrHCategoricalEncoder.add_label s[ LL nn--G6wi@A A  "# U# U rc`XR;aURU$URU5$)a-Add a label if it is not already present. Arguments --------- label : hashable Most often labels are str, but anything that can act as dict key is supported. Note that default save/load only supports Python literals. Returns ------- int The index that was used to encode this label. )rrH)rr1s rr-CategoricalEncoder.ensure_label7s, LL <<& &>>%( (rcXR;a$URRn[SU35eUR X5 g)asAdd a new label, forcing its index to a specific value. If a label already has the specified index, it is moved to the end of the mapping. Arguments --------- label : hashable Most often labels are str, but anything that can act as dict key is supported. Note that default save/load only supports Python literals. index : int The specific index to use. r^N)rrArBr_ enforce_label)rr1rarKs r insert_labelCategoricalEncoder.insert_labelKs> LL nn--G6wi@A A   u ,rc [U5nXR;a-X RU:XagURURU X R;aURUnSnOSnX RU'XRU'U(a^[R S[ W5SUS[ U5S35 UR 5nXPRU'X0RU'gg)zMake sure label is present and encoded to a particular index. If the label is present but encoded to some other index, it is moved to the given index. If there is already another label at the given index, that label is moved to the next free position. NTFz Moving label z from index z , because z was put at its place.)intrrrCrDreprr`)rr1ra saved_label moving_other new_indexs rrf CategoricalEncoder.enforce_label`sE  LL  U++LLe!45 LL ,,u-KL L# U# U  KK[ 12,'DK=0FH ((*I(1LL %&1LL # rc0XlURU5$)aAdd label for unknown tokens (out-of-vocab). When asked to encode unknown labels, they can be mapped to this. Arguments --------- unk_label : hashable, optional Most often labels are str, but anything that can act as dict key is supported. Note that default save/load only supports Python literals. Default: . This can be None, as well! Returns ------- int The index that was used to encode this. )rrH)rrs rrCategoricalEncoder.add_unks"#~~i((rchURnXR;aUS- nXR;aMU$)z'The index to use for the next new label)rr)rras rr`CategoricalEncoder._next_indexs1##||# QJE||# rc [URR55nURU;=(a" [ S[ USSUSS555$)zCheck that the set of indices doesn't have gaps For example: If starting index = 1 Continuous: [1,2,3,4] Continuous: [0,1,2] Non-continuous: [2,3,4] Non-continuous: [1,2,4] Returns ------- bool True if continuous. c34# UHupX!- S:Hv M g7f)rsNr6)r7ijs rr:3CategoricalEncoder.is_continuous..s6 #A41AEQJ#AsNrs)sortedrkeysrallzip)rindicess r is_continuous CategoricalEncoder.is_continuoussZ"**,-""g- #6 #&ws|WQR[#A6 3  rc~UR5 URU$![a [US5(a"U(aURURs$[US5(aU(d[SUS35e[US5(dU(a[SUS35e[SUS35ef=f)aAEncode label to int Arguments --------- label : hashable Label to encode, must exist in the mapping. allow_unk : bool If given, that label is not in the label set AND unk_label has been added with add_unk(), allows encoding to unk_label's index. Returns ------- int Corresponding encoded int value. rzUnknown label z=, and explicitly disallowed the use of the existing unk-labelzCannot encode unknown label zN. You have not called add_unk() to add a special unk-label for unknown labels.z+Couldn't and wouldn't encode unknown label .) _assert_lenrr_hasattrrrr1 allow_unks r encode_labelCategoricalEncoder.encode_labels"  <<& & t[))i||DNN33{++I$UG,CCT;//I25':44 A%J s!;B<AB<cN[R"URX5/5$)arEncode label to torch.LongTensor. Arguments --------- label : hashable Label to encode, must exist in the mapping. allow_unk : bool If given, that label is not in the label set AND unk_label has been added with add_unk(), allows encoding to unk_label's index. Returns ------- torch.LongTensor Corresponding encoded int value. Tensor shape [1]. torch LongTensorrrs rencode_label_torch%CategoricalEncoder.encode_label_torchs#$!2!25!D EFFrcnUR5 UVs/sHo0RX25PM sn$s snf)aSEncode a sequence of labels to list Arguments --------- sequence : iterable Labels to encode, must exist in the mapping. allow_unk : bool If given, that label is not in the label set AND unk_label has been added with add_unk(), allows encoding to unk_label's index. Returns ------- list Corresponding integer labels. )rrrsequencerr1s rencode_sequence"CategoricalEncoder.encode_sequences1" AIJ!!%3JJJs2c v[R"UVs/sHo0RX25PM sn5$s snf)aEncode a sequence of labels to torch.LongTensor Arguments --------- sequence : iterable Labels to encode, must exist in the mapping. allow_unk : bool If given, that label is not in the label set AND unk_label has been added with add_unk(), allows encoding to unk_label's index. Returns ------- torch.LongTensor Corresponding integer labels. Tensor shape [len(sequence)]. rrs rencode_sequence_torch(CategoricalEncoder.encode_sequence_torchs5$>F GhU  u 0h G  Gs6cUR5 /nURS:Xa2UH*nURUR[ U55 M, U$UH#nURUR U55 M% U$)aNDecodes an arbitrarily nested torch.Tensor to a list of labels. Provided separately because Torch provides clearer introspection, and so doesn't require try-except. Arguments --------- x : torch.Tensor Torch tensor of some integer dtype (Long, int) and any shape to decode. Returns ------- list list of original labels rs)rndimappendrrj decode_torch)rxdecodedelement subtensors rrCategoricalEncoder.decode_torchsr"  66Q;t||CL9:  t00;<rcUR5 /nUH#nURURU55 M% U$![a UR[ U5s$f=f)a|Decodes an arbitrarily nested iterable to a list of labels. This works for essentially any pythonic iterable (including torch), and also single elements. Arguments --------- x : Any Python list or other iterable or torch.Tensor or a single integer element Returns ------- list, Any ndim list of original labels, or if input was single element, output will be, too. )rr decode_ndim TypeErrorrrj)rrrrs rrCategoricalEncoder.decode_ndim5sb"  (G t// :;N (<<A' ' (s,?"A$#A$c\UR5nURXRU5 g)aSave the categorical encoding for later use and recovery Saving uses a Python literal format, which supports things like tuple labels, but is considered safe to load (unlike e.g. pickle). Arguments --------- path : str, Path Where to save. Will overwrite. N) _get_extras _save_literalr)rr$extrass rrWCategoricalEncoder.savePs&!!# 4v6rcUR(a/URRn[R SUS35 UR U5up4nX0lX@lURU5 [RSU35 g)zLoads from the given path. CategoricalEncoder uses a Python literal format, which supports things like tuple labels, but is considered safe to load (unlike e.g. pickle). Arguments --------- path : str, Path Where to load from. zLoad called, but zs is not empty. Loaded data will overwrite everything. This is normal if there is e.g. an unk label defined at init.z!Loaded categorical encoding from N) rrArBrCrD _load_literalr _set_extrasdebug)rr$rKrrrs rr"CategoricalEncoder.load_s| <<nn--G KK#G9-PP  $(#5#5d#; &     8?@rcAURU5 g![a [RSUS35 g[[ 4a [RSUS35 gf=f)azLoads if possible, returns a bool indicating if loaded or not. Arguments --------- path : str, Path Where to load from. end_of_epoch : bool Whether the checkpoint was end-of-epoch or not. Returns ------- bool : If load was successful. Example ------- >>> encoding_file = getfixture("tmpdir") / "encoding.txt" >>> encoder = CategoricalEncoder() >>> # The idea is in an experiment script to have something like this: >>> if not encoder.load_if_possible(encoding_file): ... encoder.update_from_iterable("abcd") ... encoder.save(encoding_file) >>> # So the first time you run the experiment, the encoding is created. >>> # However, later, the encoding exists: >>> encoder = CategoricalEncoder() >>> encoder.expect_len(4) >>> if not encoder.load_if_possible(encoding_file): ... assert False # We won't get here! >>> encoder.decode_ndim(range(4)) ['a', 'b', 'c', 'd'] z%Would load categorical encoding from z, but file doesn't exist yet.FzH, and file existed but seems to be corrupted or otherwise couldn't load.T)r"FileNotFoundErrorrCrrV SyntaxError)rr$ end_of_epochs rrU#CategoricalEncoder.load_if_possiblexsB   IIdO!  LL7v>.. K(  LL7v>YY   s#A%(A%$A%cXlg)aeSpecify the expected category count. If the category count observed during encoding/decoding does NOT match this, an error will be raised. This can prove useful to detect bugs in scenarios where the encoder is dynamically built using a dataset, but downstream code expects a specific category count (and may silently break otherwise). This can be called anytime and the category count check will only be performed during an actual encoding/decoding task. Arguments --------- expected_len : int The expected final category count, i.e. `len(encoder)`. Example ------- >>> encoder = CategoricalEncoder() >>> encoder.update_from_iterable("abcd") >>> encoder.expect_len(3) >>> encoder.encode_label("a") Traceback (most recent call last): ... RuntimeError: .expect_len(3) was called, but 4 categories found >>> encoder.expect_len(4) >>> encoder.encode_label("a") 0 N expected_len)rrs r expect_lenCategoricalEncoder.expect_lens :)rcSUlg)zSpecifies that category count shall be ignored at encoding/decoding time. Effectively inhibits the ".expect_len was never called" warning. Prefer :py:meth:`~CategoricalEncoder.expect_len` when the category count is known.Nrrs r ignore_lenCategoricalEncoder.ignore_lens !rc@[US5(aEURcg[U5nXR:wa[SURSUS35eg[R UR RS[U5S35 UR5 g)zIf `expect_len` was called, then check if len(self) matches the expected value. If it does not, raise a RuntimeError. If neither `expect_len` or `ignore_len` were ever called, warn once.rNz .expect_len(z) was called, but z categories foundz9.expect_len was never called: assuming category count of z to be correct! Sanity check your encoder using `.expect_len`. Ensure that downstream code also uses the correct size. If you are sure this does not apply to you, use `.ignore_len`.) rrr RuntimeErrorrC warning_oncerArBr)rreal_lens rrCategoricalEncoder._assert_lens 4 ( (  (4yH,,,""4#4#4"56#*$57-   >>**+,..1$i[9QQ  OO  rcbSUR0n[US5(aURUS'U$)zjOverride this to provide any additional things to save Call super()._get_extras() to get the base extras rr)rrrrrs rrCategoricalEncoder._get_extrass4 #D$7$78 4 % %"&..F;  rc8SU;a USUlUSUlg)zfOverride this to e.g. load any extras needed Call super()._set_extras(extras) to set the base extras rrN)rrrs rrCategoricalEncoder._set_extrass& & #K0DN$%56rc[USSS9nUR5H?upEUR[U5[R -[ U5-S-5 MA UR[R5 UR5H?upgUR[U5[R -[U5-S-5 MA UR5 SSS5 g!,(df  g=f)z+Save which is compatible with _load_literalwutf-8encoding N) openitemswriterkr VALUE_SEPARATORstrEXTRAS_SEPARATORflush)r$rrfr1indkeyvalues rr CategoricalEncoder._save_literals$g .!%mmo K(889#h. GG&77 8$lln I(8895k"- GGI!/ . .s CC++ C9c0n0n0n[USS9nUHpnU[R:Xa O[UR5R [R SS9upg[ U5n[R"U5nXqU'XU'Mr UHanUR5R [R SS9up[R"U 5n [R"U 5n XU 'Mc SSS5 XU4$!,(df  N=f)ztLoad which supports Python literals as keys. This is considered safe for user input, as well (unlike e.g. pickle). rrrs)maxsplitN) rr rstripsplitrrjast literal_eval) r$rrrrlineliteralrr1 literal_key literal_valuerrs rr CategoricalEncoder._load_literals  $ )Q->>>#zz|11&66 2  #h((1!$$ -1ZZ\-?-?&66.@.* &&{3((7#s *('')* )s CC;; D )rrrrr)r)F)FNrsT),rB __module__ __qualname____firstlineno____doc__rrrrr classmethodr&r2r>rNr[rHr-rgrf DEFAULT_UNKrr`rrrrrrrrrWr"rrUrrrrr staticmethodrr__static_attributes__r6rrr r s&iVO+36 ! %&5:,MN,b D.)(-*!2F!,)( ,%NG(K( ,8(6 7 7A200d)>!67(((rr c^\rSrSrSrU4SjrSU4SjjrSU4SjjrSU4Sjjr\ \ 4Sjr \ \ S S4S jr S r S rS rSrSrSrU4SjrU4SjrSrU=r$) TextEncoderi6a2CategoricalEncoder subclass which offers specific methods for encoding text and handle special tokens for training of sequence to sequence models. In detail, aside special token already present in CategoricalEncoder for handling out-of-vocab tokens here special methods to handle beginning of sequence and tokens are defined. Note: update_from_iterable and update_from_didataset here have as default sequence_input=True because it is assumed that this encoder is used on iterables of strings: e.g. >>> from speechbrain.dataio.encoder import TextEncoder >>> dataset = [["encode", "this", "textencoder"], ["foo", "bar"]] >>> encoder = TextEncoder() >>> encoder.update_from_iterable(dataset) >>> encoder.expect_len(5) >>> encoder.encode_label("this") 1 >>> encoder.add_unk() 5 >>> encoder.expect_len(6) >>> encoder.encode_sequence(["this", "out-of-vocab"]) [1, 5] >>> Two methods can be used to add and to the internal dicts: insert_bos_eos, add_bos_eos. >>> encoder.add_bos_eos() >>> encoder.expect_len(8) >>> encoder.lab2ind[encoder.eos_label] 7 >>> add_bos_eos adds the special tokens at the end of the dict indexes >>> encoder = TextEncoder() >>> encoder.update_from_iterable(dataset) >>> encoder.insert_bos_eos(bos_index=0, eos_index=1) >>> encoder.expect_len(7) >>> encoder.lab2ind[encoder.eos_label] 1 >>> insert_bos_eos allows to specify whose index will correspond to each of them. Note that you can also specify the same integer encoding for both. Four methods can be used to prepend and append . prepend_bos_label and append_eos_label add respectively the and string tokens to the input sequence >>> words = ["foo", "bar"] >>> encoder.prepend_bos_label(words) ['', 'foo', 'bar'] >>> encoder.append_eos_label(words) ['foo', 'bar', ''] prepend_bos_index and append_eos_index add respectively the and indexes to the input encoded sequence. >>> words = ["foo", "bar"] >>> encoded = encoder.encode_sequence(words) >>> encoder.prepend_bos_index(encoded) [0, 3, 4] >>> encoder.append_eos_index(encoded) [3, 4, 1] c>[TU]U5 SU;aSU;aURSSUSUSS9 gSU;dSU;a [S5eg)z+Handles special labels such as bos and eos. bos_label eos_labelrr)rr bos_index eos_indexz.Only BOS or EOS specified. Need both for init.N)superrinsert_bos_eosrrrrAs rr!TextEncoder.handle_special_labelsxsm %n5 . ([N-J   !!(5(5   N *k^.KLM M/Lrc">[TU]X5$*Change default for sequence_input to True.)rr2)rr.r/rAs rr2 TextEncoder.update_from_iterablesw+HEErc$>[TU]XU5$r)rr>)rr=r9r/rAs rr>!TextEncoder.update_from_didatasetsw, >  rNc$>[TU]USSSS9$)rTNrs)r/rIrJ)rrN)rr.r/rIrJrAs rrN*TextEncoder.limited_labelset_from_iterables%w5 T6  rcX:Xa'[RS5 URU5 O"URU5 URU5 XlX lg)aAdd sentence boundary markers in the label set. If the beginning-of-sentence and end-of-sentence markers are the same, will just use one sentence-boundary label. This method adds to the end of the index, rather than at the beginning, like insert_bos_eos. Arguments --------- bos_label : hashable Beginning-of-sentence label, any label. eos_label : hashable End-of-sentence label, any label. If set to the same label as bos_label, will just use one sentence-boundary label. IBOS and EOS labels are the same so using just one sentence boundary labelN)rCrrHrr)rrrs r add_bos_eosTextEncoder.add_bos_eossK*  ! LL!  NN9 % NN9 % NN9 %""rrcX:Xa'[RS5 URX5 OOURX5 Uc*[RS5 URX#S-5 OURX$5 XlX lg)a1Insert sentence boundary markers in the label set. If the beginning-of-sentence and end-of-sentence markers are the same, will just use one sentence-boundary label. Arguments --------- bos_label : hashable Beginning-of-sentence label, any label eos_label : hashable End-of-sentence label, any label. If set to the same label as bos_label, will just use one sentence-boundary label. bos_index : int Where to insert bos_label. eos_index = bos_index + 1 eos_index : optional, int Where to insert eos_label. Default: eos_index = bos_index + 1 rNz,EOS label not specified, using BOS label + 1rs)rCrrgrr)rrrrrs rrTextEncoder.insert_bos_eossv0  ! LL!    i 3   i 3  KL!!)];!!)7""rcp[US5(d [S5eURUR5$)(Returns the index to which blank encodesrzBOS label is not set!)rrrrrs r get_bos_indexTextEncoder.get_bos_index0t[))67 7  00rcp[US5(d [S5eURUR5$)rrzEOS label is not set!)rrrrrs r get_eos_indexTextEncoder.get_eos_indexr rcl[US5(d [S5eUR/[U5-$)z/Returns a list version of x, with BOS prependedr*BOS label has not been added to label set!)rr_rlistrrs rprepend_bos_labelTextEncoder.prepend_bos_labels1t[))GH H$q'))rcF[US5(d [S5e[R"U5(aE[R"UR UR /5n[R"X!/5$UR UR /[U5-$)zfReturns a list version of x, with BOS index prepended. If the input is a tensor, a tensor is returned.rr) rr_r is_tensorTensorrrcatr)rrbos_inds rprepend_bos_indexTextEncoder.prepend_bos_indexszt[))GH H ??1  llDLL$@#ABG99g\* * T^^,-Q77rcl[US5(d [S5e[U5UR/-$)z/Returns a list version of x, with EOS appended.r*EOS label has not been added to label set!)rr_rrrs rappend_eos_labelTextEncoder.append_eos_labels1t[))GH HAw$..)))rcF[US5(d [S5e[R"U5(aE[R"UR UR /5n[R"X/5$[U5UR UR /-$)zeReturns a list version of x, with EOS index appended. If the input is a tensor, a tensor is returned.rr) rr_rrrrrrr)rreos_inds rappend_eos_indexTextEncoder.append_eos_indexszt[))GH H ??1  llDLL$@#ABG99a\* *Aw$,,t~~6777rc>[TU]5n[US5(aURUS'[US5(aURUS'U$Nrr)rrrrrrrrAs rrTextEncoder._get_extrassK$& 4 % %"&..F;  4 % %"&..F;  rcf>[TU]U5 SU;a USUlSU;a USUlggr#)rrrrr$s rrTextEncoder._set_extrass= F# & #K0DN & #K0DN !r)rrr)TNrs)rBrrrrrr2r>rN DEFAULT_BOS DEFAULT_EOSrrrr rrrr rrr __classcell__rAs@rrr6su?BN$F LM #F &#P1 1 * 8* 811rrcx^\rSrSrSrU4Sjr\4Sjr\S4SjrSr S Sjr S S jr U4S jr U4S jr S rU=r$)CTCTextEncoderi!aSubclass of TextEncoder which also provides methods to handle CTC blank token. add_blank and insert_blank can be used to add special token to the encoder state. >>> from speechbrain.dataio.encoder import CTCTextEncoder >>> chars = ["a", "b", "c", "d"] >>> encoder = CTCTextEncoder() >>> encoder.update_from_iterable(chars) >>> encoder.add_blank() >>> encoder.expect_len(5) >>> encoder.encode_sequence(chars) [0, 1, 2, 3] >>> encoder.get_blank_index() 4 >>> encoder.decode_ndim([0, 1, 2, 3, 4]) ['a', 'b', 'c', 'd', ''] collapse_labels and collapse_indices_ndim can be used to apply CTC collapsing rules: >>> encoder.collapse_labels(["a", "a", "b", "c", "d"]) ['a', 'b', 'c', 'd'] >>> encoder.collapse_indices_ndim([4, 4, 0, 1, 2, 3, 4, 4]) # 4 is [0, 1, 2, 3] cT>SU;aURUSS9 [TU] U5 g)z&Handles special labels such as blanks. blank_label)raN) insert_blankrrrs rr$CTCTextEncoder.handle_special_labels<s0 N *   N=$A  B %n5rc2URU5 Xlg)zAdd blank symbol to labelset.N)rHr/)rr/s r add_blankCTCTextEncoder.add_blankIs {#&rrc2URX5 Xlg)z(Insert blank symbol at a given labelset.N)rgr/)rr/ras rr0CTCTextEncoder.insert_blankNs +-&rcp[US5(d [S5eURUR5$)z)Returns the index to which blank encodes.r/zBlank label is not set!)rrrr/rs rget_blank_indexCTCTextEncoder.get_blank_indexSs2t]++89 9  !1!122rc.[US5(d [S5eU(aD[U5VVs/sH+up4US:Xd XAUS- :wdMX@R:wdM)UPM- snn$UVs/sHoDUR:wdMUPM sn$s snnfs snf)aApplies the CTC collapsing rules on one label sequence. Arguments --------- x : iterable Label sequence on which to operate. merge_repeats : bool Whether to merge repeated labels before removing blanks. In the basic CTC label topology, repeated labels are merged. However, in RNN-T, they are not. Returns ------- list List of labels with collapsing rules applied. r/Blank label has not been addedrrs)rr_ enumerater/)rr merge_repeatsrwr1s rcollapse_labelsCTCTextEncoder.collapse_labelsYs&t]++;< < !*!  ,HAFeQx/5:>N>N5N ,  ()FqeT5E5E,EEqF F  GsB  B B ,BBc[US5(d [S5e/nUH$nURURXB55 M& U$![a  Of=fUR UR nU(a@[U5VVs/sH!upgUS:Xd XqUS- :wdMXu:wdMUPM# snn$s snnfUVs/sH owU:wdM UPM sn$s snf)aApplies the CTC collapsing rules on arbitrarily label sequence. Arguments --------- x : iterable Label sequence on which to operate. merge_repeats : bool Whether to merge repeated labels before removing blanks. In the basic CTC label topology, repeated labels are merged. However, in RNN-T, they are not. Returns ------- list List of labels with collapsing rules applied. r/r;rrs)rr_rcollapse_indices_ndimrrr/r<)rrr= collapsedr blank_indexrwras rrA$CTCTextEncoder.collapse_indices_ndimws"t]++;< < I   ..yH     ll4#3#34 !*!  ,HAFeQx/5:5I ,  ()Aqe[,@EqA AAs/ A  AA B6%B6,B6 C Ccd>[TU]5n[US5(aURUS'U$Nr/)rrrr/r$s rrCTCTextEncoder._get_extrass2$& 4 ' '$($4$4F= ! rcF>[TU]U5 SU;a USUlggrF)rrr/r$s rrCTCTextEncoder._set_extrass* F# F "%m4D  #r)r/r)rBrrrrr DEFAULT_BLANKr3r0r8r>rArrrr*r+s@rr-r-!sD4 6%2' (5A' 3 G<(BT 55rr-c~[5nURU5 [URR 55$)agLoads the encoder tokens from a pretrained model. This method is useful when you used with a pretrained HF model. It will load the tokens in the yaml and then you will be able to instantiate any CTCBaseSearcher directly in the YAML file. Arguments --------- model_path : str, Path Path to the pretrained model. Returns ------- list List of tokens. )rr"rrr|) model_path label_encoders rload_text_encoder_tokensrNs3" MMz"  %%**, --r)rrrEr)r speechbrainrQspeechbrain.utils.checkpointsrrrspeechbrain.utils.loggerrrBrCrr(r)rJr rr-rNr6rrrRs   0 H     R (R (R (jh1$h1VI5[I5X.r