,jP @SrSSKrSSKJr SSKJrJrJr SSKr SSK r SSK r SSK J s Jr SSKrSSKJr S\4SjrS!S\S \S \4S jjrS!S\S \S \4S jjrS"S \S\4Sjjr\"SS9S\S\ R64Sj5rS#S\\\ R:\ R64S\S\S\\\\ R<44SjjrS$S\ R6S\S\ R64Sjjr S\ R6S\ RBS\ R64Sjr"S\\ R64Sjr#S r$g)%zModified from https://github.com/openai/whisper/blob/main/whisper/audio.py Add rename_weights() & onnx2torch() & make_non_pad_mask() & mask_to_bias() Copy merge_tokenized_segments() from https://github.com/Mddct/s3tokenizer-long/blob/main/example.py N) lru_cache)ListOptionalUnion) pad_sequence weights_dictc0nUR5HnSU;aUS:Xa XUS'MSU;aXX'M&M(SU;aXX'M6SU;aXX'MDSU;deUSS RS S 5RS S 5RSS5RSS 5RSS5RSS5RSS5nXUSU3'M U$)z Rename onnx weights to pytorch format. Parameters ---------- weight_dict: dict The dict containing weights in onnx format Returns ------- A new weight dict containing the weights in pytorch format. quantizerz,/quantizer/rq/model/layers.0/_codebook/Pow_1zquantizer._codebook.embed project_downpositional_embeddingconvblocksN/.MatMulweightAdd_1biasMulAddzmlp.mlpmlpzfsmn_block.Convfsmn_block.weightencoder.)keysreplace)rnew_weight_dictknew_ks ڟC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\s3tokenizer/utils.py_rename_weightsr! s O     ! BB?K ;<1$%1_"% #q (!-O  q[!-O q= =qrU]]3,44($$+GGV$%%,WUF%;GG!5=*+273D3F,H  3?/Ohug. /%!&  onnx_path torch_pathverbosec [R"U5n0nURRVs0sHnURU_M nnURR GHnUR GHnX;dM SupXhnUS;aSn OyUS;aSn OpUS;aSn OgUS;aS n O^US :XaS n OUUS :XaS n OLUS:XaSn OCURS:Xa'URRSS5n U S-n U S-n O URn U bU bUR n U SnU SnX;a/[RRXn5R5OSnX;a/[RRXo5R5OSnSURl SURl [R"U5n[R"U5nUXI'UXJ'GMv[RRU5R5nSURl [R"U5n[!UR"5S:dW S;aUUW 'GMUR%5XK'GM GM ['U5nU(aOUR)5H-unn[+USUR"SUR,35 M/ [+SU35 AAU(a[R."UU5 gU$s snf)a Open an onnx file and convert to pytorch format. Parameters ---------- onnx_path: str The onnx file to open, typically `speech_tokenizer_v1.onnx` torch_path: str The path to save the torch-formated checkpoint. verbose: bool Logging info or not. Returns ------- A checkpoint dict containing the weights and their names, if torch_path is None. Otherwise save checkpoint dict to the desired path. )NN)zonnx::Conv_1519zencoders.conv1.weightzonnx::Conv_2216encoder.conv1.weight)zonnx::Conv_1520zencoders.conv1.biaszonnx::Conv_2217encoder.conv1.bias)zonnx::Conv_1521zencoders.conv2.weightzonnx::Conv_2218encoder.conv2.weight)zonnx::Conv_1522zencoders.conv2.biaszonnx::Conv_2219encoder.conv2.biaszencoders.positional_embeddingencoder.positional_embeddingquantizer.project_in.bias%quantizer._codebook.project_down.biaszonnx::MatMul_2536'quantizer._codebook.project_down.weightLayerNormalizationz/LayerNormalizationz.weightz.biasNrT)r+ :  PyTorch weights saved to )onnxloadgraph initializernamenodeinputop_typer numpy_helperto_arraycopyflags writeabletorch from_numpylenshapetr!itemsprintdtypesave)r#r$r% onnx_modelrr8initializer_mapr: input_name ln_bias_nameln_weight_name weight_nameln_name ln_inputs scale_name bias_namescaler weight_tensor bias_tensor weight_arraynew_weights_dictrvs r onnx2torchr[Ds(9%JL&++777K +%7  %%**J,/9, -9 " #9K$ #7K$ #9K$ #7K#BB"@K#>>"IK#66"KK||';;"&))"3"34I2"N)09)<'.'8 &*ii !-,2J $ I!*1J )! I(:!--66'3559T6@D '9 ,,55'2448D5?C-1EKK)+/DJJ($)$4$4U$;M"'"2"24"8K1nS'M@ UR#5VVs0sHunnUcM UU_M nnnU(aTUR#5H2unnUcM [%US?UR&S@UR(35 M4 [%SAU35 AU(a[*R,"XA5 gU$s snfs snnf)Cz$ Convert V3 ONNX to PyTorch format. Constantvaluerc&>UT;a1[RRTU5R5nOUT;aTUR5nOg[R "U5nU(a UR S:XaUR5nU$)Nr1)r5r=r>r?rBrCndimrF)r9 transposearrrF constant_maprLs r get_tensor!onnx2torch_v3..get_tensorsz ? "##,,_T-BCHHJC \ !t$))+C   S ! 1Ar"cJ>URHnT"U5nUcMUs $ g)z_Helper to find bias tensor for an Add node. Checks both inputs to see which one is a parameter.N)r;)r:inprFrds r get_bias_tensor&onnx2torch_v3..get_bias_tensors+::C3A}r"z /conv1/Convrr'r1r(z /conv2/Convr)r*z/blocks.rrzencoder.blocks.z attn_ln/Mulrz.attn_ln.weightz attn_ln/AddrNz .attn_ln.biasz mlp_ln/Mulz.mlp_ln.weightz mlp_ln/Addz .mlp_ln.biaszattn/query/MatMulT)raz.attn.query.weightzattn/query/Addz.attn.query.biaszattn/key/MatMulz.attn.key.weightz attn/key/Addz.attn.key.biaszattn/value/MatMulz.attn.value.weightzattn/value/Addz.attn.value.biaszattn/out/MatMulz.attn.out.weightz attn/out/Addz.attn.out.biaszmlp/mlp.0/MatMulz .mlp.0.weightz mlp/mlp.0/Addz .mlp.0.biaszmlp/mlp.2/MatMulz .mlp.2.weightz mlp/mlp.2/Addz .mlp.2.biaszfsmn_block/Convrrr,r-r rr.r2r3r4)F)r5r6r7r8r9r:r< attributer=r>rFoutputr;rD startswithsplitnumelrGrHrErIrBrJ)r#r$r%rKrr8r:attrrhr9opinputsparts block_part block_idxprefixrF init_namerrZrcrdrLs @@@r onnx2torch_v3rwsk 9%JL&++777K +%7O L  %% <<: %99'373D3D3M3M4 LQ0'&    %%yy \\ = 3=fQi3HL/ 06{Q5?q 5J 12 ] "3=fQi3HL/ 06{Q5?q 5J 12__Z ( (JJsOEqJ"((-a0I&yk2F$u;E1I< x78$&2;#D)=QWWY]=>L8=!9:%"+:DVAY:O x~67%"+#D)=QWWY]<=L8H1I?/ x'9:;!T)H1I?/ x'9:;!T)HJJqMT>3 9 ;&&2%7%7%9K%9TQQDAqD%9LK &&(DAq}3qwwiq 23) )*67 <,u\LsR1 R6+R6filesrc[R"U5up#X1:wa%[RRX15"U5nUSnU$)a Open an audio file and read as mono waveform, resampling as necessary Parameters ---------- file: str The audio file to open sr: int The sample rate to resample the audio if necessary Returns ------- A torch.Tensor containing the audio waveform, in float32 dtype. r) torchaudior6 transformsResample)rxryaudio sample_rates r load_audiorVsD $.E%%..{?F !HE Lr")maxsizen_melsreturnc\US;d SU35e[RR[RR[5SS5n[ R "USS9n[R"USU35RU5sSSS5 $!,(df  g=f) aD load the mel filterbank matrix for projecting STFT into a Mel spectrogram. Allows decoupling librosa dependency; saved using: np.savez_compressed( "mel_filters.npz", mel_80=librosa.filters.mel(sr=16000, n_fft=400, n_mels=80), mel_128=librosa.filters.mel(sr=16000, n_fft=400, n_mels=128), ) >PzUnsupported n_mels: assetszmel_filters.npzF) allow_picklemel_N) ospathjoindirname__file__npr6rBrCto)devicer filters_pathfs r _mel_filtersrms Y ?"6vh ?? 77<< 98 13L E 2aD/ 2366v> 3 2 2s (+B B+r~paddingrc|[R"U5(d [U[5(a [ U5nUbUR U5nUS:a[ R"USU45n[R"S5R UR5n[R"USSUSS9nUSSS24R5S -n[URU5nXv-n[R"US S 9R5n [R"XR!5S - 5n U S -S - n U $)a5 Compute the log-Mel spectrogram of Parameters ---------- audio: Union[str, np.ndarray, torch.Tensor], shape = (*) The path to audio or either a NumPy array or Tensor containing the audio waveform in 16 kHz n_mels: int The number of Mel-frequency filters, only 80 is supported padding: int Number of zero samples to pad to the right device: Optional[Union[str, torch.device]] If given, the audio tensor is moved to this device before STFT Returns ------- torch.Tensor, shape = (128, n_frames) A Tensor that contains the Mel spectrogram NriT)windowreturn_complex.r1g|=)ming @g@)rB is_tensor isinstancestrrrFpad hann_windowrstftabsrclamplog10maximummax) r~rrrrr magnitudesfiltersmel_speclog_specs r log_mel_spectrogramrs : ??5 ! ! eS ! !u%E  {ea\*   s # & &u|| 4F ::eS#fT JDc3B3h##%q(J5<<0G#H{{8/557H}}X||~';URS5nUS:aUOUR5R5n[R"SU[R UR S9nURS5RX!5nURS5nXE:nU)$)aMake mask tensor containing indices of non-padded part. The sequences in a batch may have different lengths. To enable batch computing, padding is need to make all sequence in same size. To avoid the padding part pass value to context dependent block such as attention or convolution , this padding part is masked. 1 for non-padded part and 0 for padded part. Parameters ---------- lengths (torch.Tensor): Batch of lengths (B,). Returns: ------- torch.Tensor: Mask tensor containing indices of padded part (B, max_T). Examples: >>> import torch >>> import s3tokenizer >>> lengths = torch.tensor([5, 3, 2]) >>> masks = s3tokenizer.make_non_pad_mask(lengths) masks = [[1, 1, 1, 1, 1], [1, 1, 1, 0, 0], [1, 1, 0, 0, 0]] r)rIrr) sizeritemrBarangeint64r unsqueezeexpand)rr batch_size seq_rangeseq_range_expandseq_length_expandmasks r make_non_pad_maskrs8aJ 1g'++-*<*<*>G Q$#(;;$+NN4I!**1-44ZI))"-  0D 5Lr"rrIcUR[R:XdeU[R[R[R 4;deUR U5nSU- S-nU$)a Convert bool-tensor to float-tensor for flash attention. Parameters ---------- lengths (torch.Tensor): Batch of lengths (B, ?). Returns: ------- torch.Tensor: Mask tensor containing indices of padded part (B, ?). Examples: >>> import torch >>> import s3tokenizer >>> lengths = torch.tensor([5, 3, 2]) >>> masks = s3tokenizer.make_non_pad_mask(lengths) masks = [[1, 1, 1, 1, 1], [1, 1, 1, 0, 0], [1, 1, 0, 0, 0]] >>> new_masks = s3tokenizer.mask_to_bias(masks, torch.float32) new_masks = [[-0.0000e+00, -0.0000e+00, -0.0000e+00, -0.0000e+00, -0.0000e+00], [-0.0000e+00, -0.0000e+00, -0.0000e+00, -1.0000e+10, -1.0000e+10], [-0.0000e+00, -0.0000e+00, -1.0000e+10, -1.0000e+10, -1.0000e+10]] g?g _)rIrBboolfloat32bfloat16float16r)rrIs r mask_to_biasrs]2 :: ## # U]]ENNEMMB BB B 775>D $J( "D Kr"datacHUn[U[5(de[R"UVs/sHo"R S5PM sn[R S9nUVs/sHo"R 5PM nn[USSS9nURSS5U4$s snfs snf)zPadding the data into batch data Parameters ---------- data: List[Tensor], shape of Tensor (128, T) Returns: ------- feats [B, 128, T_max], feats lengths [B] r)rITr) batch_first padding_valuer1) rlistrBtensorrint32rFrra)rsamples feats_lengthsfeats padded_featss r rrsF fd # ## #LLV!)rrN)r)%__doc__r functoolsrtypingrrrnumpyrr5rBtorch.nn.functionalnn functionalrr{torch.nn.utils.rnnrdictr!rrr[rwintrTensorrndarrayrrrrIrrrr"r rs  (( +!$!Hj #j 3j j \%)"'bSb!bbJSc. 4????*15 / bjj%,,. // // U3 ,- . /d%u||%c%%,,%P!u||!EKK!ELL!H7$u||$7*r"