j= fSSKrSSKrSSKJrJr SSKJr SSKJrJ r SSK r SSK r SSK J r JrJr SSKJr SSKJrJr SSKJrJrJrJr SS KJr \"S S 9"S S 55r\"SS9S5rSrSrSr S\!S\S4Sjr"S-S\!S\#S\!4Sjjr$S.S\!S\#S\#S\#S\#S\!4 Sjjr%S-S\!S\!S\S S!\#S\#4 S"jjr&S\!S\SS\S S\!4S#jr'S-S\!S\SS\S S$\#S\!4 S%jjr(S-S\!S\SS\S S!\#S\!4 S&jjr)S/S\!S'\ \S(\*4S)jjr+S/S\!S'\ \S(\*4S*jjr,S0S\!S'\ \S(\*S+\#4S,jjr-g)1N) dataclassreplace) lru_cache)LiteralOptional)NormalizationCandidateNormalizationMappingNormalizationResult)NormalizerConfig)FSTS FST_PATHS)UniqueOutputPathStream compose_inputtrace_input_spanstransduce_with_spans) TokenParserT)frozencl\rSrSr%\\S'\\S'\\S'\\S'\\S'\\S'\\S'\S 5r S r g ) _NormalizationCandidate$taggedoutput tagger_weightverbalizer_weightbest_verbalizer_weight tagger_rankverbalizer_rankcNURUR-UR- $N)rrr)selfs ښC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\wetext/utils.pyweight_NormalizationCandidate.weight.s$!!D$:$::T=X=XXXN) __name__ __module__ __qualname____firstlineno__str__annotations__floatintpropertyr"__static_attributes__r%r$r!rr$s> K K!! YYr$r)maxsizec@[RRU5$r)kaldifst StdVectorFstread)paths r! _read_fstr63s  % %d ++r$cSnUR(aUS:waURS:XaSnSnUR(aUS:XaURS:XaSn[[UURU5R 5n[[UURU5R 5nUR S 4URS 4URS 4URS 4URS 44nUHRunupU(dM[R"US5 [R"U[[UU 55nMT XE4$)Ntaggerenitntagger_enable_0_to_9 verbalizerzhtnverbalizer_remove_erhua) preprocesstraditional_to_simple) postprocess full_to_half)rBremove_interjections)rB remove_puncts)rBtag_oovolabel) enable_0_to_9operator remove_erhuar6r copyrArCrDrErFr2arcsortcompose) configlang tagger_nameverbalizer_namer8r<postprocessorsenabledgroupnames r!_graphsrV8s9K  E1I, "O tt|40G3 yv7 D E J J LF9T?6??;OLMRRTJ  % %'NO   =>  $ $&MN   ?@ 34 N#1% 7   Z 2!))*i %@PQU@V6WXJ#1  r$cz[U[5(d[U[5(aUS:a [S5eg)Nz nbest must be a positive integer) isinstanceboolr- ValueError)nbests r!_validate_nbestr]Ps4%j&<&< ;<<AJr$c |^^^^^^[[X55m/mSm/n[5nUUUUUU4Sjn[U5U:GadTR 5n T(dU"5(dGO?M7U b,U R U R S4TSSS:a U"5 Mf[R"T5u ppU R5nUb[U RURU RUR U RU RUR 5n[R "TUR URUR"U UU 45 U R$U;a,UR'U R$5 UR)U 5 [U5U:aGMdU(d [+S5eU$)Nrc >T R5nUcg[TT 5RUR5n[ [ UT 55nUR5nUR nTS- mUcg[URURURURURUUR 5n[R"TURURURTUU45 g)NFrXT) poprreordertextrrrankrr"heapqheappushrr) tagged_path reorderedverbalizer_streamverbalized_pathr candidate activatedfrontierrOrI tagger_streamr<s r!activate_next_tag4_normalization_candidates..activate_next_tag\s#'')  h/77 8H8HI 2=J3WX+//1!&& Q  "+         " "  " "        %%))!   r$z-no normalization path for the requested input)rrsetlenpeekr"rcrdheappopr`rrrbrrrrerraddappend RuntimeError)rbrOrIr8r<r\ candidates seen_outputsrnnext_tag_serialrjrhnext_verbalizednext_candidaterkrlrms `` ` @@@r!_normalization_candidatesrUs*=+FGMHIJ5L  D j/E ! %%'$&&   X__hmmQ$G(ST+VXWX/$Y   8= h8O51a+//1  &4  $$''&&00%%$$N NN"))".."22"%     < /   Y-- .   i (E j/E !F JKK r$rbreturn)r9r=jacSnUHnSUs=::aS::dMO MSnM U(dUR5(aS$S$)zx Get the language of the text. Args: text: The text to get the language of. Returns: The language of the text. Fu一u鿿Tr=r9)isdigit)rbcontains_chinesechs r!get_langrsC r %X % %# $t||~~4?4?r$rAcTU(a[SS"U5nUR5$)z Preprocess the text before normalization. Args: text: The text to preprocess. traditional_to_simple: Whether to convert traditional Chinese to simplified Chinese. Returns: The preprocessed text. r@rAr strip)rbrAs r!r@r@s(L!"9:4@ ::<r$rCrDrErFcU(a[SS"U5nU(a[SS"U5nU(a[SS"U5nU(a[SS"U5nUR5$)ag Postprocess the text after normalization. Args: text: The text to postprocess. full_to_half: Whether to convert full-width characters to half-width. remove_interjections: Whether to remove interjections. remove_puncts: Whether to remove punctuations. tag_oov: Whether to tag out-of-vocabulary words. Returns: The postprocessed text. rBrCrDrErFr)rbrCrDrErFs r!rBrBsm&M">248M"#9:4@M"?3D9M"9-d3 ::<r$rOrI)r>r:rJcUS:XaQUS:waK[[R"SU55(agU(a[R"SU5(agg[U5S:$)a  Check if the text should be normalized. Args: text: The text to check. lang: The language of the text. operator: The operator to use. remove_erhua: Whether to remove erhua for TN. Returns: True if the text should be normalized, False otherwise. r>r9z\dTu儿|兒Fr)rZresearchrr)rbrOrIrJs r!should_normalizersR4DDL  %& ' ' BIIj$77 t9q=r$c6[X5RU5$)z~ Reorder the text. Args: text: The text to reorder. lang: The language of the text. Returns: The reordered text. )rra)rbrOrIs r!raras t & . .t 44r$rHc[UUSnU(aUS:waUS:Xa[USSnU"U5R5$)z Tag the text. Args: text: The text to tag. lang: The language of the text. operator: The operator to use. enable_0_to_9: Whether to enable 0-to-9 conversion for ITN. Returns: The tagged text. r8r9r:r;r)rbrOrIrHr8s r!tagrsN$Z !( +F(e*;dE"#9: $<   r$c[UUSnU(aUS:XaUS:Xa[SSSnU"U5R5$)z Verbalize the text. Args: text: The text to verbalize. lang: The language of the text. operator: The operator to use. remove_erhua: Whether to remove erhua for TN. Returns: The verbalized text. r<r=r>r?r)rbrOrIrJr<s r! verbalizersOdH%l3J T)9$Z%&?@ d  ! ! ##r$rNr\c 4[U5 [U=(d [540UD6nUR(aSU;a[R "U5nUR 5nURnUS:Xa [U5nUS:Xa[XAR5n[XeURUR5(aX[XeURUR5n[!XeUR5n[#XeURUR5n[%UUR&UR(UR*UR,5$[XEURUR5(a@[/X5upx[1XEURXxU5V s/sHn U R2PM n n OO[XAR5n[%UUR&UR(UR*UR,5nU/n US:XaU S$U $s sn f)z Normalize the text. Args: text: The text to normalize. config: Optional normalization config object. Returns: The normalized text. 'autorXr)r]rr fix_contractions contractionsfixrrOrr@rArrIrJrrHrarrBrCrDrErFrVrr) rbrNr\kwargspreparedrOrr8r<rjoutputss r! normalizer+sE V1/1 Q>QRF      ' '  NN   9L9LMM$V27xvX^lqr r    r   H&B&BC      ' '  NN  (!71:00 s Hc [U5 [U=(d [540UD6nUR(aSU;a[R "U5nUR 5nURnUS:Xa [U5n[XEURUR5(d][XAR5n[UURUR UR"UR$5n['USSSSS5/$[)X5upx[+XEURXxU5n U V s/sHMn ['U R,U R.U R0U R2U R4U R6S9PMO sn $s sn f)zReturn distinct outputs together with their WFST ranking costs. Costs are tropical weights: lower values rank first. They are not probabilities or calibrated confidence scores. rrgr)rbcost tagger_costverbalizer_costrr)r]rr rrrrrOrrrIrJr@rArBrCrDrErFrrVrrr"rrrr) rbrNr\rrrOrr8r<rxrjs r!normalize_candidatesr^sbE V1/1 )normalize_with_mapping..s A=%kk599 %=s$&delete)r]rr rr[rrrlstriprOrrrIrJrr r rVrrreorder_with_spansrrrrtokensziprvrUtuple)rbrNr\rrrleading_offsetrOrmappingsresultr8r<rxresultsrjparserrg token_spans output_spans input_spansr input_start input_end output_start output_endsourcespokenkinds r!normalize_with_mappingrsyE V1/1 rsM *$^^*, , $ Y Y Y 4,,0= Nb @3 @7#34 @ S  # "!&       <3cW[5IY]jn* 5# 5W%56 5'+BV 5[^ 5 c !12 gk>R cg tw &fk$ $-.$:A+:N$^b$$(01C01*:!;01301f&s&H5E,F&VY&TdiG1 G1 01G1ADG1\`G1r$