jKISrSSKrSSKJr SSKJr SSKrSSKJr SSKJ r SSK J r SS K Jr SS KJr SS KJr SS KJrJr SS KJr SSKJrJrJr SSKJrJrJrJ r SSK!J"r" \ RF"\$5r%SoS\RLS\RNS\RNS\RNS\RNS-S\(S-S\(S\\4Sjjr)"SS\RL5r*"SS\RL5r+SpS\RNS \(S!\,S-S"\-S#\.4 S$jjr/SqS\RNS%\,\.-S!\,S-S#\.4S&jjr0"S'S(\RL5r1"S)S*\RL5r2"S+S,\RL5r3\"S-S.\55r4"S/S0\RL5r5"S1S2\RL5r6"S3S4\45r7\"S5S69\"S7S8\555r8\"S9S69\"S:S;\555r9\"S\555r:\"S?S69\"S@SA\555r;\"SBS69\"SCSD\555r<\"SES69\"SFSG\555r=SH\R|R~SI\RNSJ\RN4SKjr@SrSL\RNSM\RNS-SJ\RN4SNjjrA"SOSP\RL5rB"SQSR\RL5rC"SSST\RL5rD"SUSV\RL5rE\"SWSX\455rF"SYSZ\RL5rG\"S[S69"S\S]\455rH"S^S_\RL5rI\"S`S69"SaSb\455rJ\"ScS69"SdSe\RL55rK\"SfS69"SgSh\455rL"SiSj\RL5rM\"SkS69"SlSm\455rN/SnQrOg)szPyTorch PatchTST model.N)Callable) dataclass)nn)initialization)ACT2CLS)is_deepspeed_zero3_enabled)FlashAttentionKwargs)BaseModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)NegativeBinomialOutput NormalOutputStudentTOutput) ModelOutputTransformersKwargsauto_docstringlogging)PatchTSTConfigmodulequerykeyvalueattention_maskscalingdropoutkwargsc UcURS5S-n[R"XRSS55U-nUbX-n[R R USS9n[R RXURS9n[R"X5n U RSS5R5n X4$)Nrdim)ptrainingr) sizetorchmatmul transposer functionalsoftmaxrr' contiguous) rrrrrrrr attn_weights attn_outputs ڼC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\transformers/models/patchtst/modeling_patchtst.pyeager_attention_forwardr2's**R.D(<<}}Q':;gEL!#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$c^\rSrSrSrSS\S\S\S\S\S \S \S-4U4S jjjr SS \ RS \ RS-S\ RS-S\S-S\ \ S\\ R\ RS-44 SjjrSrU=r$)PatchTSTAttentionDz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsr is_decoderbias is_causalconfigc>[TU]5 XlX lX0lX-UlXplUR U-UR:wa[SURSUS35eUR S-UlX@l X`l [R"XUS9Ul [R"XUS9Ul[R"XUS9Ul[R"XUS9Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: ).r"r:)super__init__r7r8rhead_dimr< ValueErrorrr9r;rLineark_projv_projq_projout_proj) selfr7r8rr9r:r;r< __class__s r1rAPatchTSTAttention.__init__Gs "" !.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 4@ ii 4@ ii 4@  )TB r3 hidden_stateskey_value_statesroutput_attentionsrreturnc USLnURSSn/UQSPURP7nURU5RU5R SS5n U(aUOUn /U RSSQSPURP7n UR U 5RU 5R SS5n UR U 5RU 5R SS5n [R"URR[5nU"UU U U U4UR(dSO URURUS.UD6unnUR"/UQSP76R!5nUR#U5nUU4$)z#Input shape: Batch x Time x ChannelNr!rr#)rrrN)shaperBrGviewr+rErFr get_interfacer<_attn_implementationr2r'rrreshaper.rH)rIrLrMrrNris_cross_attention input_shape hidden_shape query_statescurrent_stateskv_shape key_states value_statesattention_interfacer0r/s r1forwardPatchTSTAttention.forwardfs.T9$))#2. 88b8$--8 {{=166|DNNqRST -?)]B^))#2.BBDMMB[[055h?II!QO {{>277AKKAqQ (?(M(M KK , ,.E) %8      % $}}C$,,LL/ %  % ! \"));;;;FFH mmK0 L((r3) r<rr7rBr;r9rEr8rHrGrrF)rQFTFN)NNF)__name__ __module__ __qualname____firstlineno____doc__intfloatboolrrAr)Tensorrr tupler`__static_attributes__ __classcell__rJs@r1r5r5DsG  (,CCC C  C  CC%CCD15.2). 0)||0) ,,-0) t+ 0)  $; 0)-.0) u||U\\D00 10)0)r3r5cV^\rSrSrSrS\4U4SjjrS\R4Sjr Sr U=r $)PatchTSTBatchNormzH Compute batch normalization over the sequence length (time) dimension. r<c~>[TU]5 [R"URUR S9Ulg)Neps)r@rAr BatchNorm1dd_modelnorm_eps batchnormrIr<rJs r1rAPatchTSTBatchNorm.__init__s( FOOLr3inputsclURSS5nURU5nURSS5$)z Parameters: inputs (`torch.Tensor` of shape `(batch_size, sequence_length, d_model)`): input for Batch norm calculation Returns: `torch.Tensor` of shape `(batch_size, sequence_length, d_model)` rr#)r+rx)rIr{outputs r1r`PatchTSTBatchNorm.forwards7!!!Q''1%%r3)rx rbrcrdrerfrrAr)rjr`rlrmrns@r1rprps+M~M &ell & &r3rpr{ mask_ratiounmasked_channel_indiceschannel_consistent_masking mask_valuecUS:dUS:a[SUS35eURupVpxURn [USU- -5n U(a*[R "USXyS9n U R SUS5n O[R "XVXyS9n [R"XVXyS9n SU SS2SS2SU 24'[R"U SS9n [R"U SS9n[R"U SUS 9n U RS5R SSSU5n UbSU SS2USS2SS24'URU R5U5nXS 4$) arandom_masking: Mask the input considering the control variables. Args: inputs (`torch.Tensor` of shape `(batch_size, num_channels, sequence_length, num_features)`): The input tensor to mask. mask_ratio (`float`): Masking ratio applied to mask the input data during random pretraining. It is the number between 0 and 1. unmasked_channel_indices (list, *optional*): Indices of channels that will not be masked. channel_consistent_masking (bool, *optional*, defaults to `False`): When true, masking will be same across all channels of a timeseries. Otherwise, masking positions will vary across channels. mask_value (int, *optional*, defaults to 0): Define the value of masked patches for pretraining. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as input Tensor and mask tensor of shape [bs x c x n] rrz Mask ratio z has to be between 0 and 1.deviceNr!r$)r%index.r) rCrRrrgr)randrepeatonesargsortgather unsqueeze masked_fillri)r{rrrr batch_size num_channelssequence_length num_featuresrlen_keepnoisemask ids_shuffle ids_restore inputs_masks r1random_maskingrsA4A~q;zl2MNOO>Dll;Jo ]]F?a*n56H! :q/I Q a0 :_T ::j ODDAyy--2.K-- 4K <<"K 8D >>"  $ $Q1l ;D+23Q(!Q ./$$TYY[*=K V $$r3num_forecast_mask_patchescP[U[5(aU/nUVs/sHnSPM nnURupgp[R"XgXR S9n /n Sn [ U5n [X5HGupUS::dX:a[SUS35e[Xo-U - 5nU RXU/5 U U- n MI [U SS9n X:aU SSXl- -U SS'OX:aU S SX- -U S S'SnU HunnnUU-nSU UU2S S 2U*S 24'UnM [R"U RS5nU Un U RS 5RSSSU 5n UbSU S S 2US S 2S S 24'URU R5U5nUU S 4$s snf) aiForecast masking that masks the last K patches where K is from the num_forecast_mask_patches. If num_forecast_mask_patches is a list, samples in the batch will be randomly masked by numbers defined in the list. Parameters: inputs (`torch.Tensor`): Input of shape `(bs, num_channels, num_patch, patch_length)` num_forecast_mask_patches (`list`): Number of patches to be masked at the end of each batch sample. e.g. 4 or [3, 5]. unmasked_channel_indices (`list`, *optional*): Indices of channels that are not masked. mask_value (`int`, *optional*, defaults to 0): Values in the masked patches will be filled by `mask_value`. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as inputs Tensor and Mask tensor of shape `(bs, num_channels , num_patch)` or `(bs, tsg1, tsg2, num_channels, num_patch)` rrrznum_forecast_mask_patches z6 should be greater than 0 and less than total patches.c US$)Nr#)xs r1"forecast_masking..s!A$r3)rr#r!Nr) isinstancergrRr)zerosrsumziprCappendsortedrandpermrrrri)r{rrr_forecast_mask_ratiosrrrrrt_list total_length total_ratio patch_lengthratiotemp_lenbatch1 patch_lenbatch2permrs r1forecast_maskingrs0+S11%>$?!'@A'@!A'@A>Dll;Jo ;;z WD FL*+K"#<S 1  ?,\N:pq z)K78 |H56  TF /F ay|z'@Aq !  "r 1 )BCr 1 F"( 1h("./VF]A z{ *+#) >>$**Q- (D :D >>"  $ $Q1l ;D+23Q(!Q ./$$TYY[*=K V $$OBs F#cV^\rSrSrSrS\4U4SjjrS\R4Sjr Sr U=r $)PatchTSTPatchifyi-z A class to patchify the time series sequence into different patches Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` r<c>[TU]5 URUlURUlUR UlURUR::a&[ SURSURS35e[URUR5UR- UR -S-UlURUR URS- --nURU- Ul g)NzSequence length (z+) has to be greater than the patch length ()r) r@rAcontext_lengthrr patch_striderCmax num_patchessequence_start)rIr<new_sequence_lengthrJs r1rAPatchTSTPatchify.__init__5s %44"//"//   4#4#4 4#D$8$8#99deievevdwwxy    4 4d6G6GH4K\K\\aeararruvv"//$2C2CtGWGWZ[G[2\\"225HHr3 past_valuesc4URSnX R:wa[SUSURS35eUSS2URS2SS24nUR SUR UR S9nURSS5R5nU$)z Parameters: past_values (`torch.Tensor` of shape `(batch_size, sequence_length, num_channels)`, *required*): Input for patchification Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` zInput sequence length (z%) doesn't match model configuration (r>N) dimensionr(step) rRrrCrunfoldrrr+r.)rIrrr}s r1r`PatchTSTPatchify.forwardFs&++B/ 22 2)/)::_`d`t`t_uuwx Q 3 3 5q89$2C2C$J[J[\!!"b)446 r3)rrrrrrrns@r1rr-s+I~I"5<<r3rcV^\rSrSrSrS\4U4SjjrS\R4Sjr Sr U=r $)PatchTSTMaskingi]al Class to perform random or forecast masking. Parameters: config (`PatchTSTConfig`): model config Returns: x_mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`) Masked patched input mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches)`) Bool tensor indicating True on masked points r<c>>[TU]5 URUlURUlURUlUR UlUR UlURUlUR b[UR 5UlggN) r@rArandom_mask_ratior mask_typerrrrrys r1rAPatchTSTMasking.__init__js !'!9!9*0*K*K')))/)I)I&(.(G(G% ++  ( ( 4,243P3P,QD ) 5r3 patch_inputcdURS:Xa8[UURURURUR S9up#OVURS:Xa-[ UURURUR S9up#O[SURS35eUR5nX#4$)a Parameters: patch_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`, *required*): Patch input Return: masked_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`) Masked patched input mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches)`) Bool tensor indicating True on masked points random)r{rrrrforecast)r{rrrzInvalid mask type .) rrrrrrrrrCri)rIr masked_inputrs r1r`PatchTSTMasking.forwardus >>X %!/"11)-)F)F+/+J+J?? " L$^^z )!1"*.*H*H)-)F)F?? " L$1$..1ACD Dyy{!!r3)rrrrrrrrns@r1rr]s+  R~ R!"5<<!"!"r3rcd^\rSrSrSrS\4U4SjjrS S\RS\ S-4Sjjr S r U=r $) PatchTSTEncoderLayeriz PatchTST encoder layer r<c (>[TU]5 URUl[URUR UR US9UlURS:a [R"UR5O[R"5Ul URS:Xa[U5UlOWURS:Xa/[R "URUR"S9UlO[%URS35eUR(aURS:a [R"UR5O[R"5UlURS:Xa[U5UlOWURS:Xa/[R "URUR"S9UlO[%URS35e[R*"[R,"URUR.UR0S9[2UR4"5UR6S:a [R"UR65O[R"5[R,"UR.URUR0S95UlURS:a [R"UR5O[R"5UlURS:Xa[U5UlOWURS:Xa/[R "URUR"S9UlO[%URS35eUR>Ulg)N)r7r8rr<rrx layernormrsz$ is not a supported norm layer type.r?) r@rAchannel_attentionr5rvnum_attention_headsattention_dropout self_attn path_dropoutrDropoutIdentity dropout_path1 norm_typerpnorm_sublayer1 LayerNormrwrC dropout_path2norm_sublayer2 SequentialrDffn_dimr:ractivation_function ff_dropoutff dropout_path3norm_sublayer3pre_normrys r1rAPatchTSTEncoderLayer.__init__s !'!9!9*nn00,,  AG@S@SVW@WRZZ(;(;<]_]h]h]j   { *"3F";D     ,"$,,v~~6??"SD  0 011UVW W  ! !DJDWDWZ[D[F,?,?!@acalalanD ;.&7&?#!![0&(ll6>>v&W# F$4$4#55Y!Z[[-- IIfnnfnn6;; G F.. / 1-3->->-BBJJv(( ) IIfnnfnn6;; G  AG@S@SVW@WRZZ(;(;<]_]h]h]j   { *"3F";D     ,"$,,v~~6??"SD  0 011UVW W r3N hidden_staterNcURup4pVURX4-XV5nUR(a5URUR U5US9upxXR U5-nO3URXS9upxUR XR U5-5nUR X4XV5nUR(aURSS5R5nURX5-XF5nUR(a5URURU5US9upyXRU5-nO3URXS9upyURXRU5-5nUR X5XF5nURSS5R5nURX4-XV5nUR(a2XRURURU555-nO1URXRURU55-5nUR X4XV5nU4n U(aXR(aUW 4OU4- n U $)ao Parameters: hidden_state (`torch.Tensor` of shape `(batch_size, num_channels, sequence_length, d_model)`, *required*): Past values of the time series output_attentions (`bool`, *optional*): Whether or not to return the output attention of all layers Return: `torch.Tensor` of shape `(batch_size, num_channels, sequence_length, d_model)` )rLrNr#r)rRrSrrrrrVrr+r.rrrrr) rIrrNrnum_input_channelsrrvr0r/channel_attn_weightsoutputss r1r`PatchTSTEncoderLayer.forwardsYDPCUCU@ $(()H/c ==(,"11,?Sd)7) %K(*<*<[*IIL)-\(w %K..|>P>PQ\>]/]^L$++JOe   ! !'11!Q7BBDL',,Z-IK]gL}}48NN"&"5"5l"CWh5C51  ,.@.@.MM 59NN".5C51  $22P>PQUQXQXYeQf>g/ghL$++JOe /  ?U?U &:;\h[j jGr3) rrrrrrrrrrr) rbrcrdrerfrrAr)rjrir`rlrmrns@r1rrs90(~0(dOELLOTD[OOr3rc^\rSrSr%\\S'SrSrSrSr Sr Sr Sr \ R"5S\R 4U4S jj5rS S jrS rU=r$) PatchTSTPreTrainedModeli"r<modelr)timeFTrc6>[TU]U5 [U[5(Gab[ UR R UR R5UR R- UR R-S-nUR R(a$[R"URSS9 US- nURUR U5n[5(aqSSKnUR R#UR$SS9 UR$R'5S:a![R("UR$U5 SSS5 g[R("UR$U5 gg!,(df  g=f)z Initialize weights rg{Gz?)stdrN) modifier_rank)r@ _init_weightsrPatchTSTPositionalEncodingrr<rrr use_cls_tokeninitnormal_ cls_token_init_per deepspeedzeroGatheredParameters position_encnumelcopy_)rIrrrrrJs r1r%PatchTSTPreTrainedModel._init_weights-s/ f% f8 9 9DKK.. 0H0HIDKKLdLdd))*,-.K{{(( V--48q !??4;; DL)++ ^^66v7J7JZ^6_**002Q6 6#6#6 E`_ 6.. =% :`_s AF  Fc<[U[5(aX!lggr)rPatchTSTEncodergradient_checkpointing)rIrrs r1_set_gradient_checkpointing3PatchTSTPreTrainedModel._set_gradient_checkpointingGs f 0 0,1 ) 1r3r)F)rbrcrdrer__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnr)no_gradrModulerrrlrmrns@r1rr"s^ #O &+#N ]]_>BII>>222r3rcR^\rSrSrS\4U4SjjrS\R4SjrSr U=r $)PatchTSTEmbeddingiLr<c>[TU]5 URUlURUlUR(a1[R "UR UR5Ulg[R"5Ul[UR5HGnURR[R "UR UR55 MI gr) r@rArshare_embeddingrrDrrvinput_embedding ModuleListranger)rIr<rrJs r1rAPatchTSTEmbedding.__init__Ms "(";";%55   #%99V-@-@&..#QD #%==?D 6445$$++BIIf6I6I6>>,Z[6r3rc jURSnX R:wa[SURSUS35eUR(aUR U5nU$[ U5Vs/sH$o@RU"USS2USS2SS245PM& nn[ R"USS9nU$s snf)z Parameters: patch_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`, *required*): Patch input for embedding return: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, d_model)` rz&The defined number of input channels (zQ) in the config has to be the same as the number of channels in the batch input (rNr$)rRrrCrrr!r)stack)rIrr embeddingsis r1r`PatchTSTEmbedding.forwardYs)..q1 !8!8 889P9P8QRTTfSgghj    --k:JUZZlTmnTmq..q1+aAqj2IJTmJnZQ7Jos,+B0)rrr rbrcrdrerrAr)rjr`rlrmrns@r1rrLs& \~ \5<<r3rc^\rSrSrSrS\S\4U4Sjjr\S\S\S\ R4Sj5r S\ R4S jrS rU=r$) ripz Class for positional encoding r<rc>[TU]5 URUlURUlUR(aA[R "[ R"SSSUR55Ul US- nURX5Ul URS:a&[R"UR5Ul g[R"5Ul g)Nrr)r@rArrr Parameterr)rrvrrrpositional_dropoutrrrIr<rrJs r1rA#PatchTSTPositionalEncoding.__init__us #11"(";";   \\%++aAv~~*NODN 1 K MM&>6<5N5NQR5RBJJv00 1 XZXcXcXe r3rOc$URS:Xa5[R"[R"XR 5SS9nU$URS:XGa#[R "XR 5n[R"SU5RS5n[R"[R"SUR S5[R"S5UR - *-5n[R"X4-5USS2SSS24'[R"X4-5USS2SSS24'X"R5- nX"R5S -- n[R"US S9nU$[!URS 35e) NrT requires_gradsincosrrr#g@ FzN is not a valid positional encoder. Available types are 'random' and 'sincos'.)positional_encoding_typerr+r)randnrvrarangerexpmathlogsincosmeanrrC)r<rrpositiondiv_terms r1r#PatchTSTPositionalEncoding._init_pesX  * *h 6<< K(P`deL , , 8 ;;{NNCL||A{3==a@Hyya!CQXHY\b\j\jHjFk!klH$)IIh.A$BLADqD !$)IIh.A$BLADqD !'*;*;*==L'+;+;+=+BCL<< EJL 2234BC r3rcxUR(aURXRSS2SS24-5nURURSS2SS24-nUR UR SUR SS5n[R"X14SS9nU$URXR-5nU$)Nrrr!r#r$) rr,rrexpandrRrr)cat)rIrr cls_tokensrs r1r`"PatchTSTPositionalEncoding.forwards   11+@Q@QRSRTVWRW@X2XYK):):2A2q5)AAI"))+*;*;A*>@W@WY[]_`J 99j%>AFL 22;ARAR3RSLr3)rrrr,r)rbrcrdrerfrrgrA staticmethodrr+rr)rjr`rlrmrns@r1rrps]  ~  C  cbll& 5<<  r3rc z^\rSrSrSrS\S\4U4SjjrS S\RS\ S-S \ S-S \ 4S jjr S r U=r$)r iz PatchTST Encoder r<rc,>[TU]U5 SUl[U5Ul[ X5Ul[R"[UR5Vs/sHn[U5PM sn5Ul UR5 gs snf)NF)r@rArrembedderrpositional_encoderrr r!num_hidden_layersrlayers post_init)rIr<rr&rJs r1rAPatchTSTEncoder.__init__sx  &+#*&1 "