oMjr dSSKJr SSKJr SSKJr SSKrSSKJr SSK Js J r SSK J r Jr SSKJrJr SSKJr SS KJr SS KJr \"5(aSS KJrJrJrJrJrJr S S KJ r S SK!J"r" S SK#J$r$ S SK%J&r& S SK'J(r( SSK)J*r*J+r+J,r, \RZ"\.5r/S\R`S\RbS\Rb4Sjr2S\R`S\RbS\Rb4Sjr3S\R`S\RbS\4S\Rb4Sjr5S-S\6S\4S\4S\4S\4S\R`4 Sjjr7\2\3\5S .r8\"S!S"\55r9"S#S$\R`5r:"S%S&\R`5r;"S'S(\R`5r<"S)S*\R`5r="S+S,\(\"\*\ 5r>g).) dataclass)sqrt)AnyN) ConfigMixinregister_to_config) BaseOutputlogging)apply_forward_hook)is_transformers_available) randn_tensor)Dinov2WithRegistersConfigDinov2WithRegistersModelSiglipVisionConfigSiglipVisionModel ViTMAEConfig ViTMAEModel)get_activation)AttentionMixin) Attention)get_2d_sincos_pos_embed) ModelMixin)AutoencoderMixin DecoderOutput EncoderOutputmodelimagesreturnc>U"USS9nSnURSS2US24$)NT)output_hidden_stateslast_hidden_state)rroutputsunused_token_nums ڻC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\diffusers/models/autoencoders/autoencoder_rae.py_dinov2_encoder_forwardr):s.F6G  $ $Q(8(9%9 ::c*U"USSS9nUR$)NT)r"interpolate_pos_encodingr$)rrr&s r(_siglip2_encoder_forwardr-@sFPTUG  $ $$r* patch_sizecURSURSpC[X4-US--5nXRS--X4-:wa [S5e[R"U5R S5R URSS5RUR5RUR5nU"XSS9nURSS2SS24$) Nrrz-Image size should be divisible by patch size.rT)r,r) shapeint ValueErrortorcharange unsqueezeexpandtodevicedtyper%)rrr.hw patch_numnoiser&s r(_mae_encoder_forwardr?Es <<?FLLOqAEZ]*+Iq= AE)HII LL # - -a 0 7 7 Q L O OPVP]P] ^ a abhbnbn oEFDAG  $ $QU ++r* encoder_type hidden_sizenum_hidden_layershead_dimc X-nUS:Xa;[UUSUUS9n[U5nSURlSURlOUS:XaO[ UUSUUS9n[ U5nSURRlSURRlOQUS:Xa<[UUSUUS S 9n[U5nSURlSURlO[S US 35eURS 5 U$)z[TU]5 [R"X5Ul[ U5UlgN)super__init__nnLineardenserintermediate_act_fn)selfrArgrh __class__s r(rlViTMAEIntermediate.__init__s+ YY{> #1*#= r* hidden_statesr cJURU5nURU5nU$rjrorp)rqrts r(forwardViTMAEIntermediate.forwards& =1 00? r*rv)gelu) r\r]r^r_r2strrlr4rarwrc __classcell__rrs@r(reresB>C>C>S>> U\\ellr*rec^\rSrSrS S\S\S\4U4SjjjrS\RS\RS\R4S jr S r U=r $) ViTMAEOutputrArghidden_dropout_probc>[TU]5 [R"X!5Ul[R "U5Ulgrj)rkrlrmrnroDropoutdropout)rqrArgrrrs r(rlViTMAEOutput.__init__s0 YY0> zz"56 r*rt input_tensorr cRURU5nURU5nX-nU$rjror)rqrtrs r(rwViTMAEOutput.forwards, =1  ]3 %4 r*r)rL) r\r]r^r_r2floatrlr4rarwrcr{r|s@r(r~r~sQ7C7C7V[77 U\\RWR^R^r*r~c^\rSrSrSrSSSSSS.S\S \S \S \S \S \S\S\4U4Sjjjr S\ RS\ R4Sjr Sr U=r$) ViTMAELayerzV This matches the naming/parameter structure used in RAE-main (ViTMAE decoder block). T-q=rLry)qkv_biaslayer_norm_epsrattention_probs_dropout_probrhrArGrgrrrrrhc>[T U]5 X-S:wa[SUSU35e[UUX-UUS9Ul[ XUS9Ul[XUS9Ul[R"XS9Ul [R"XS9Ul g)Nrz hidden_size=z* must be divisible by num_attention_heads=) query_dimheadsdim_headrrQ)rArgrh)rArgreps) rkrlr3r attentionre intermediater~outputrm LayerNormlayernorm_beforelayernorm_after) rqrArGrgrrrrrhrrs r(rlViTMAELayer.__init__s   , 1{m+UViUjk #!% 70  /#U_ ##^q  !# [ M!||KLr*rtr cURURU55nX!-nURU5nURU5nUR X15nU$rj)rrrrr)rqrtattention_output layer_outputs r(rwViTMAELayer.forwardsX>>$*?*? *NO(8 ++M: ((6 {{<? r*)rrrrr)r\r]r^r_r`r2boolrrzrlr4rarwrcr{r|s@r(rrs %%(.1 MM! M  M  MM#M',MMMBU\\ellr*rc^\rSrSrSrS$S\S\S\S\S\S \S \S \S \S \S\S\S\S\4U4Sjjjr S\ RS\ R4Sjr S\ RS\ R4Sjr S%S\ RS\\\4S-4SjjrSSSS.S\ RS\S \S!\S\\\ R-4 S"jjrS#rU=r$)& RAEDecoderz Decoder implementation ported from RAE-main to keep checkpoint compatibility. Key attributes (must match checkpoint keys): - decoder_embed - decoder_pos_embed - decoder_layers - decoder_norm - decoder_pred - trainable_cls_token TrAdecoder_hidden_sizedecoder_num_hidden_layersdecoder_num_attention_headsdecoder_intermediate_size num_patchesr. num_channelsrFrrrrrhc>[TU]5 X lXplXlXlX`l[R"XSS9Ul [US-5n[X/SSSS9nURSURS5R5S S 9 [R"[!U5Vs/sHn[#UUUU U U U US 9PM sn5Ul[R&"X+S 9Ul[R"X'S -U-SS9UlS Ul[R."[0R2"SSU55Ulgs snf)NT)rQ?rpt) cls_token extra_tokens output_typedecoder_pos_embedrF persistent)rArGrgrrrrrhrr)rkrlrr.rrFrrmrn decoder_embedr2rregister_bufferr6r ModuleListrangerdecoder_layersr decoder_norm decoder_predgradient_checkpointing Parameterr4zerostrainable_cls_token)rqrArrrrrr.rrFrrrrrh grid_size pos_embed_rrs r(rlRAEDecoder.__init__sB" #6 $($&YY{dS S() + dX\  0)2E2Ea2H2N2N2P]bc mm89 :A 3(C&?%#1(;1M) :   LL)<QII&9q=<;W^bc&+##%<< AqBU0V#W ' s-E embeddingsr cURSS- nURRSS- nURSS2SSS24nURSS2SS2SS24nURRSnURSSSU5RSSSS5n[R "USX#- 4SSS9nURSSSS5R SSU5n[R"URS5U4SS 9$) Nrrr0rrbicubicF) scale_factormode align_cornersdim) r1rreshapepermuteF interpolateviewr4catr6)rqrembeddings_positions num_positionsclass_pos_embedpatch_pos_embedrs r(r,#RAEDecoder.interpolate_pos_encoding&s )//2Q6..44Q7!; 00Aq900AB:$$**2.)11!QC@HHAqRST-- 1AB   *11!Q1=BB1b#Nyy/33A6HaPPr*xcURup#nX0R:XaU$[US-5=pVURX%Xd5R SSSS5n[URS-5[URS-54n[ R "XSSS9nUR SSSS5R5RX RU5nU$) NrrrrrbilinearFsizerr) r1rr2rrrr contiguousr)rqrblcr;r< target_sizes r(interpolate_latentRAEDecoder.interpolate_latent8s''a   HAsF  IIaA ! ) )!Q1 54++S013t7G7G7L3MN MM!Je T IIaAq ! , , . 3 3A7G7G Kr*Npatchified_pixel_valuesoriginal_image_sizecURURpCUbUOURUR4nUupVXS-nXc-nXx-URS:wa![ SURSSUSU35eURSn UR U UUUUU5n[ R"SU5nUR U UXs-X-5n U $)Nrz5The number of patches in the patchified pixel values z9, does not match the number of patches on original image *rznhwpqc->nchpwq)r.rrFr1r3rr4einsum) rqrrr.roriginal_heightoriginal_width num_patches_h num_patches_w batch_size pixel_valuess r( unpatchifyRAEDecoder.unpatchifyCs*#'??D4E4EL#6#B Z^ZiZiHj +>''5 &4  (,C,I,I!,L LGH_HeHefgHhGijcdqcrrstAsBC -2215 "9"A"A       # #(,,/?AX"Y.66    &  &  r*F)r,drop_cls_token return_dictrtr,rrcURU5nU(a USS2SS2SS24nURU5nOURU5nURRURSSS5n[ R "Xv/SS9nU(a$U(d [S5eURU5nO URnXXRURURS9-nURH n U "U5nM URU5nURU5n U SS2SS2SS24n U(dU 4$[!U S9$)Nrrr0rz:interpolate_pos_encoding only supports drop_cls_token=Truer9r:)rZ)rrrr7r1r4rr3r,rr8r9r:rrrrX) rqrtr,rrrx_rr layer_modulerZs r(rwRAEDecoder.forwardbs/   } - 1ab!8B((,B((+B,,33BHHQKRH IIyo1 - #! !]^^ $ = =a @  $ 6 6 000PP  //L(7M0))-8 ""=112q!9 v..r*) rrrrrrrFrrr.r)rIrrrITrrLrLryrj)r\r]r^r_r`r2rrrzrlr4rar,rtuplerrXrwrcr{r|s@r(rrs #&)*+-)- %%(.1 3X3X!3X$' 3X &) 3X $' 3X3X3X3X3X3X3X#3X',3X3X3XjQ5<<QELLQ$ ELL U\\ %,,UZ[^`c[cUdgkUkF*/$ $/||$/#' $/  $/  $/ E%,,/ /$/$/r*rc*<^\rSrSrSrSrS/rS/r\S-S\ S\ S \ S \ S \ S \ S \ S\ S\ S\ S\ S-S\ S\ S-S\ S-S\ \ -\ R-S-S\ \ -\ R-S-S\S\S\S\4(U4Sjjj5rS.S\ RS\ R$S-S\ R4SjjrS\ RS\ R4S jrS\ RS\ R4S!jrS"\ RS\ R4S#jrS"\ RS\ R4S$jrS.S\ RS\ R$S-S\ R4S%jjr\S/S\ RS&\S\ R$S-S\\ \ R-4S'jj5rS"\ RS\ R4S(jr\S0S"\ RS&\S\\ \ R-4S)jj5rS/S*\ RS&\S\ R$S-S\\ \ R-4S+jjrS,r U=r!$)1AutoencoderRAEia" Representation Autoencoder (RAE) model for encoding images to latents and decoding latents to images. This model uses a frozen pretrained encoder (DINOv2, SigLIP2, or MAE) with a trainable ViT decoder to reconstruct images from learned representations. This model inherits from [`ModelMixin`]. Check the superclass documentation for its generic methods implemented for all models (such as downloading or saving). Args: encoder_type (`str`, *optional*, defaults to `"dinov2"`): Type of frozen encoder to use. One of `"dinov2"`, `"siglip2"`, or `"mae"`. encoder_hidden_size (`int`, *optional*, defaults to `768`): Hidden size of the encoder model. encoder_patch_size (`int`, *optional*, defaults to `14`): Patch size of the encoder model. encoder_num_hidden_layers (`int`, *optional*, defaults to `12`): Number of hidden layers in the encoder model. patch_size (`int`, *optional*, defaults to `16`): Decoder patch size (used for unpatchify and decoder head). encoder_input_size (`int`, *optional*, defaults to `224`): Input size expected by the encoder. image_size (`int`, *optional*): Decoder output image size. If `None`, it is derived from encoder token count and `patch_size` like RAE-main: `image_size = patch_size * sqrt(num_patches)`, where `num_patches = (encoder_input_size // encoder_patch_size) ** 2`. num_channels (`int`, *optional*, defaults to `3`): Number of input/output channels. encoder_norm_mean (`list`, *optional*, defaults to `[0.485, 0.456, 0.406]`): Channel-wise mean for encoder input normalization (ImageNet defaults). encoder_norm_std (`list`, *optional*, defaults to `[0.229, 0.224, 0.225]`): Channel-wise std for encoder input normalization (ImageNet defaults). latents_mean (`list` or `tuple`, *optional*): Optional mean for latent normalization. Tensor inputs are accepted and converted to config-serializable lists. latents_std (`list` or `tuple`, *optional*): Optional standard deviation for latent normalization. Tensor inputs are accepted and converted to config-serializable lists. noise_tau (`float`, *optional*, defaults to `0.0`): Noise level for training (adds noise to latents during training). reshape_to_2d (`bool`, *optional*, defaults to `True`): Whether to reshape latents to 2D (B, C, H, W) format. use_encoder_loss (`bool`, *optional*, defaults to `False`): Whether to use encoder hidden states in the loss (for advanced training). Frzdecoder.decoder_pos_embedNr@encoder_hidden_sizeencoder_patch_sizeencoder_num_hidden_layersrrrrr.encoder_input_sizerFrencoder_norm_meanencoder_norm_std latents_mean latents_std noise_tau reshape_to_2duse_encoder_lossscaling_factorc >^[TU]5 U[;a,[SUS[ [R 5535eS[ S[ 4U4SjjmS[R[-[-S-S[RS-4SjnU"U5nURT"U5T"U5S9 Xl [U5Ul[U5Ul[U5Ul[%U5nURU-S:wa[S URS US 35e[%U 5nUS::a [S 5e['UUUUS 9Ul[UUlURU-S-n[%[-U55nUU-U:wa[SUS35eUU-nU cUn O)[%U 5n U U:wa[SU SUSUSUS 3 5eU c/SQn Uc/SQn[R."U [R0S9R3SSSS5n[R."U[R0S9R3SSSS5nUR5SUSS9 UR5SUSS9 U"U5nUc[R6"S5nUR5SUSS9 Uc[R8"S5nUR5SUSS9 [;[%U5[%U5[%U5[%U5[%U5[%U5[%U5[%U 5[%U 5S 9 Ul[%U5Ul[%U5Ul [%U 5Ul!S!Ul"g)"NrNz'. Available: valuer cj>[U[R5(a,UR5R 5R 5$[U[ 5(aUVs/sH nT"U5PM sn$[U[5(aUVs/sH nT"U5PM sn$U$s snfs snfrj) isinstancer4radetachcputolistrlist)r v_to_config_compatibles r(r6AutoencoderRAE.__init__.._to_config_compatibles%..||~))+2244%'':?@%Q-a0%@@%&&:?@%Q-a0%@@LA@s &B+B0cUcg[U[R5(aUR5R 5$[R "U[R S9$)Nr:)rr4rarclonetensorfloat32)r s r(_as_optional_tensor4AutoencoderRAE.__init__.._as_optional_tensorsE}%..||~++--<<U]]; ;r*)rrrzencoder_input_size=z) must be divisible by encoder_patch_size=.zCpatch_size must be a positive integer (this is decoder_patch_size).)r@rAr.rBrzComputed num_patches=z must be a perfect square.z image_size=z1 must equal decoder_patch_size*sqrt(num_patches)=z for patch_size=z and computed num_patches=)g ףp= ?gv/?gCl?)gZd;O?gy&1?g?rrr encoder_meanTr encoder_std _latents_mean _latents_std) rArrrrrr.rrFF)#rkrl_ENCODER_FORWARD_FNSr3sortedkeysrr4rarrrrrrrrr r2rVencoder_encoder_forward_fnrrrrrronesrdecoderrdecoder_patch_sizedecoder_image_size use_slicing) rqr@rrrrrrrr.rrFrrrrrrrr r rlatents_std_tensorr)rgridderived_image_sizeencoder_mean_tensorencoder_std_tensorlatents_mean_tensorrrrs @r(rlAutoencoderRAE.__init__s0  3 3(nVL`LeLeLgEhDij     >ghzg{{|} !_  "bc c#1%+)7 #  $8 #E ..2DDJ 4 $% $;+ %4[MA[\] ]/$6  +JZJ// !*-^_q^rs&&8%99ST_S``ac  $ 5   #4 #ll+4P2,?  &"'++a.  _.AdS  %!&A  ^-?DQ"/0 #$7 8&)*C&D(+,G(H&)*C&DK(-.\*:   {+"%&8"9"%j/!r*r generatorr c UR[R"URS54SURS- --UR UR US9-nX[URX!R UR S9--$)Nr)rr)r9r:r3)r3r9r:) rr4randrndimr9r:r r1)rqrr3 noise_sigmas r(_noisingAutoencoderRAE._noisingDssnnuzz VVAYL4166A:. .qxxqwwZc(  aggS[S[cdcjcj!kkkkr*c|URu p#nX0R:wdX@R:wa,[R"XRUR4SSS9nURR UR URS9nURR UR URS9nX- U- $)NrFrr) r1rrrrr8r9r:r)rqrrr;r<meanstds r(_resize_and_normalize$AutoencoderRAE._resize_and_normalizeKsWW 1 '' '10G0G+G 00$2I2IJQZjoA  ##188177#C!!!ACr*cURRURURS9nURRURURS9nX-U-$)Nr)rr8r9r:r)rqrr;r<s r(_denormalize_image!AutoencoderRAE._denormalize_imageUsT  ##188177#C!!!Aw~r*zcURRURURS9nURRURURS9nX- US-- $Nrgh㈵>r r8r9r:r!rqrBrrs r(_normalize_latents!AutoencoderRAE._normalize_latentsZs[)),,AHHAGG,L ''**!((!''*J  [4%788r*cURRURURS9nURRURURS9nXS--U-$rDrErFs r(_denormalize_latents#AutoencoderRAE._denormalize_latents_s[)),,AHHAGG,L ''**!((!''*J $&',66r*cURU5nURRS:Xa1URURXRR 5nOURURU5nUR (aURS:aURX2S9nUR(akURupEn[[U55nXw-U:wa[SUS35eURSS5R5R!XFXw5nOUnUR#U5nURR$S:waXRR$-nU$) NrJrr3zToken length n=z/ is not a perfect square; cannot reshape to 2D.rr?)r=rUr@r&r%rtrainingrr8rr1r2rr3 transposerrrGr ) rqrr3tokensrnrsiderBs r(_encodeAutoencoderRAE._encodeds&  & &q ) ;; # #u ,--dllA{{?]?]^F--dllA>F ==T^^a/]]6]?F   llGA!tAw> ''/''.xxqxxx((r*c$UR(aWURSS:aD[R"UR S5Vs/sHo0R U5PM snSS9nOUR U5nU(dU4$[ US9$s snf)Nrrrsample)r+r1r4rrXr`r)rqrBrz_slicedecodeds r(decodeAutoencoderRAE.decodesq    Qiiaggaj Qj7g!6j QWXYGll1oG: G,, !RsB rdcvURUSUS9SnURUSS9SnU(dU4$[US9$)a Args: sample (`torch.Tensor`): Input sample. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`DecoderOutput`] instead of a plain tuple. generator (`torch.Generator`, *optional*): A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make sampling deterministic. Returns: [`DecoderOutput`] or `tuple`: If `return_dict` is True, a [`DecoderOutput`] is returned, otherwise a plain `tuple` is returned. F)rr3rr^rc)r[rgr)rqrdrr3rZrfs r(rwAutoencoderRAE.forwardsJ ++f%9+MaP++g5+9!<: G,,r*) r&r(r*r)r%rrrrr r+)rEr rrrrrrKNrNNNNrLTFrNrj)TN)T)"r\r]r^r_r` _supports_gradient_checkpointing_no_split_modules"_keys_to_ignore_on_load_unexpectedrrzr2rrr4rarrrl Generatorr8r=r@rGrJrTr rr[r`rrgrwrcr{r|s@r(rrsE,^(-$&*E)F&%#&"$)+#&)*+-)-"%!%)-(,;?:>"!& #+D!D!!D! D! $' D! ! D!$'D!&)D!$'D!D! D!$JD!D! $;D!+D! UlU\\1D8!D!"E\ELL047#D!$%D!&'D!()D!*+D!D!Ll%,,l5??T3IlUZUaUal u||   ELLU\\ 9ELL9U\\9 7ell7u||7 %//D2HTYT`T`8]a - -,0 -DIOOVZDZ - u||, , - -))%,,)$- -4-=SXY^YeYeSfCf--cg-ll-15-IN[_I_- u||, ,--r*r)@)? dataclassesrmathrtypingrr4torch.nnrmtorch.nn.functional functionalrconfiguration_utilsrrutilsr r utils.accelerate_utilsr utils.import_utilsr utils.torch_utilsr transformersrrrrrr activationsrrrattention_processorrrrmodeling_utilsrvaerrr get_loggerr\loggerModulerar)r-r2r?rzrVr"rXrer~rrrr[r*r(rs" B(8;-)&+0'??   H %;299;ell;u||; %BII%u||% % , ,5<<,S,UZUaUa, ce00$'0580MP0\_0YY0h&'   z      299 -"))-`b/b/Ju-Z1A;u-r*