oMj* SSKrSSKrSSKrSSKJs Jr SSKJr SSKJ r SSK J r SSK J r Jr SSKJr SS \R"S \S \S \S\S\S\R"4SjjrSS\S\\\\4-S\S\S\S\R.S-S\S\R"4SjjrSS\S\\\\4-S\S\S\S\R44 SjjrSS\R.S-S\4SjjrSSjrSSjrSSjrS r S!r!"S"S#\RD5r#"S$S%\RD5r$"S&S'\RD5r%"S(S)\RD5r&SS+\S,\S-\S.\\\4S-S\R.S-S\R"\\R"\R"4-4 S/jjr'SS0\\\\4S+\S\R.S-S\R"\\R"\R"4-4S1jjr(SS\R.S-S\4S2jjr)SS3jr*SS4jr+SS5jr,S6S SSS*\RZ4S7\S8\R4\-S+\4S9jjr.SS:\R"S;\R"\\R"-S,\S<\S=\S\\R"\R"44 S>jjr/S:\R"4S?jr0"S@SA\RD5r1"SBSC\RD5r2"SDSE\RD5r3"SFSG\RD5r4"SHSI\RD5r5"SJSK\RD5r6"SLSM\RD5r7"SNSO\RD5r8"SPSQ\RD5r9"SRSS\RD5r:"STSU\RD5r;"SVSW\RD5r<"SXSY\RD5r="SZS[\RD5r>"S\S]\RD5r?"S^S_\RD5r@"S`Sa\RD5rA"SbSc\RD5rB"SdSe\RD5rC"SfSg\RD5rD"ShSi\RD5rE"SjSk\RD5rF"SlSm\RD5rG"SnSo\RD5rHSprI"SqSr\RD5rJ"SsSt\RD5rK"SuSv\RD5rL"SwSx\RD5rM"SySz\RD5rN"S{S|\RD5rO"S}S~\RD5rP"SS\RD5rQ"SS\RD5rR"SS\RD5rSg)N)nn) deprecate)maybe_adjust_dtype_for_device)FP32SiLUget_activation) AttentionF timesteps embedding_dimflip_sin_to_cosdownscale_freq_shiftscale max_periodreturnc[UR5S:XdS5eUS-n[R"U5*[R "SU[R URS9-nXvU- - n[R"U5nUSS2S4R5USSS24-nXH-n[R"[R"U5[R"U5/SS9nU(a)[R"USS2US24USS2SU24/SS9nUS-S:Xa*[RRRUS 5nU$) a This matches the implementation in Denoising Diffusion Probabilistic Models: Create sinusoidal timestep embeddings. Args timesteps (torch.Tensor): a 1-D Tensor of N indices, one per batch element. These may be fractional. embedding_dim (int): the dimension of the output. flip_sin_to_cos (bool): Whether the embedding order should be `cos, sin` (if True) or `sin, cos` (if False) downscale_freq_shift (float): Controls the delta between frequencies between dimensions scale (float): Scaling factor applied to the embeddings. max_period (int): Controls the maximum frequency of the embeddings Returns torch.Tensor: an [N x dim] Tensor of positional embeddings. rzTimesteps should be a 1d-arrayrr)startenddtypedeviceNdim)rrrr)lenshapemathlogtorcharangefloat32rexpfloatcatsincosr functionalpad) r r r rrrhalf_dimexponentembs کC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\diffusers/models/embeddings.pyget_timestep_embeddingr,s36 y 1 $F&FF $!H$$u||XU]]9;K;K(H&::;H ))H C AtG  " " $s47| 3C +C ))UYYs^UYYs^4" =CiiQ \*C9H9 ,=>BGqAhh!!%%c<8 J? embed_dim spatial_size temporal_sizespatial_interpolation_scaletemporal_interpolation_scaler output_typecUS:Xa [UUUUUS9$US-S:wa [S5e[U[5(aX4nSU-S-nUS-n[R "USU[R S9U- n [R "USU[R S9U- n [R"XS S 9n [R"U SS 9n U RS SUSUS/5n [X{S S9n [R "X%[R S9U- n [XS S9nU SSS2SS24n U RUSU RSU-S9n USS2SSS24nURUSUS-SS 9n[R"X/SS 9nU$)aP Creates 3D sinusoidal positional embeddings. Args: embed_dim (`int`): The embedding dimension of inputs. It must be divisible by 16. spatial_size (`int` or `tuple[int, int]`): The spatial dimension of positional embeddings. If an integer is provided, the same size is applied to both spatial dimensions (height and width). temporal_size (`int`): The temporal dimension of positional embeddings (number of frames). spatial_interpolation_scale (`float`, defaults to 1.0): Scale factor for spatial grid interpolation. temporal_interpolation_scale (`float`, defaults to 1.0): Scale factor for temporal grid interpolation. Returns: `torch.Tensor`: The 3D sinusoidal positional embeddings of shape `[temporal_size, spatial_size[0] * spatial_size[1], embed_dim]`. np)r/r0r1r2r3r"`embed_dim` must be divisible by 4rrrxyindexingrrptr4Nr output_sizer)_get_3d_sincos_pos_embed_np ValueError isinstanceintrrr meshgridstackreshape!get_2d_sincos_pos_embed_from_grid!get_1d_sincos_pos_embed_from_gridrepeat_interleaverconcat)r/r0r1r2r3rr4embed_dim_spatialembed_dim_temporalgrid_hgrid_wgridpos_embed_spatialgrid_tpos_embed_temporal pos_embeds r+get_3d_sincos_pos_embedrVQs<d*%'(C)E   1}=>>,$$$3 I *"a\\,q/& NQl lF \\,q/& NQl lF >>&4 8D ;;t #D <<A|A Q@ AD9:K_cd\\-emm LOk kF:;Mcgh*$1*5);;1*;*A*A!*D}*T<,AtQJ7+==Q,q/)q> 0D"MI r-c6Sn[SSUSS9 US-S:wa [S5e[U[5(aX4nS U-S-nUS-n[R "US [R S 9U- n[R "US[R S 9U- n [R"X5n [R"U SS 9n U RS S US US/5n [Xj5n [R "U[R S 9U- n [X|5n U [RSS2SS24n [R"XSS 9n U SS2[RSS24n [R"XSUS -S S 9n [R"X/SS 9nU$)aN Creates 3D sinusoidal positional embeddings. Args: embed_dim (`int`): The embedding dimension of inputs. It must be divisible by 16. spatial_size (`int` or `tuple[int, int]`): The spatial dimension of positional embeddings. If an integer is provided, the same size is applied to both spatial dimensions (height and width). temporal_size (`int`): The temporal dimension of positional embeddings (number of frames). spatial_interpolation_scale (`float`, defaults to 1.0): Scale factor for spatial grid interpolation. temporal_interpolation_scale (`float`, defaults to 1.0): Scale factor for temporal grid interpolation. Returns: `np.ndarray`: The 3D sinusoidal positional embeddings of shape `[temporal_size, spatial_size[0] * spatial_size[1], embed_dim]`. z`get_3d_sincos_pos_embed` uses `torch` and supports `device`. `from_numpy` is no longer required. Pass `output_type='pt' to use the new version now.output_type=='np'0.33.0F standard_warnr7rr8r9rraxisrNr)rrCrDrEr6rr rFrGrHrIrJnewaxisrepeat concatenate)r/r0r1r2r3deprecation_messagerMrNrOrPrQrRrSrTrUs r+rBrBs: ? !8-@PUV1}=>>,$$$3 I *"aYY|Abjj 9>&4 8D ;;t #D <<Ay|Yq\: ;D1){[I\A%LL%++|.G"H)!TZ[\ r-cUS:XaSn[SSUSS9 [UUS9$US-S :wa [S 5e[US-US US 9n[US-US US 9n[R "XE/S S 9nU$)a+ This function generates 2D sinusoidal positional embeddings from a grid. Args: embed_dim (`int`): The embedding dimension. grid (`torch.Tensor`): Grid of positions with shape `(H * W,)`. Returns: `torch.Tensor`: The 2D sinusoidal positional embeddings with shape `(H * W, embed_dim)` r6z`get_2d_sincos_pos_embed_from_grid` uses `torch` and supports `device`. `from_numpy` is no longer required. Pass `output_type='pt' to use the new version now.rXrYFrZ)r/rQrr embed_dim must be divisible by 2r?rr)r$get_2d_sincos_pos_embed_from_grid_nprCrJrrL)r/rQr4rbemb_hemb_wr*s r+rIrI!sd C  %x1DTYZ3  1};<< .i1nd1gS^ _E -i1nd1gS^ _E ,,~1 -C Jr-cXUS:XaSn[SSUSS9 [XS9$US-S :wa [S 5eUc$[[R UR 5n[R"US-UR US 9nX`S - -nSSU-- nURS5n[R"X5n[R"U5n[R"U5n [R"X/SS9n U(a/[R"U S S 2US-S 24U S S 2S US-24/SS9n U $)a This function generates 1D positional embeddings from a grid. Args: embed_dim (`int`): The embedding dimension `D` pos (`torch.Tensor`): 1D tensor of positions with shape `(M,)` output_type (`str`, *optional*, defaults to `"np"`): Output type. Use `"pt"` for PyTorch tensors. flip_sin_to_cos (`bool`, *optional*, defaults to `False`): Whether to flip sine and cosine embeddings. dtype (`torch.dtype`, *optional*): Data type for frequency calculations. If `None`, defaults to `torch.float32` on MPS devices (which don't support `torch.float64`) and `torch.float64` on other devices. Returns: `torch.Tensor`: Sinusoidal positional embeddings of shape `(M, D)`. r6z`get_1d_sincos_pos_embed_from_grid` uses `torch` and supports `device`. `from_numpy` is no longer required. Pass `output_type='pt' to use the new version now.rXz0.34.0FrZ)r/posrrrnNr:@r.'rrr)r$get_1d_sincos_pos_embed_from_grid_nprCrrfloat64rrrHouterr$r%rLr#) r/rsr4r rrbomegaoutemb_sinemb_cosr*s r+rJrJBs)d C  %x1DTYZ3iQQ1};<< }-emmSZZH LLa % HE _E %, E ++b/C ++c !CiinGiinG ,,)q 1CiiQ Q 0013q:JIN:J7J3KLRST Jr-c[U[5(aX4n[R"US[RS9USU- - U- n[R"US[RS9USU- - U- n[R "Xv5n[R "USS9nURSSUSUS/5n[X5n U(a2US:a,[R"[R"X0/5U /SS9n U $)a Creates 2D sinusoidal positional embeddings. Args: embed_dim (`int`): The embedding dimension. grid_size (`int`): The size of the grid height and width. cls_token (`bool`, defaults to `False`): Whether or not to add a classification token. extra_tokens (`int`, defaults to `0`): The number of extra tokens to add. interpolation_scale (`float`, defaults to `1.0`): The scale of the interpolation. Returns: pos_embed (`np.ndarray`): Shape is either `[grid_size * grid_size, embed_dim]` if not using cls_token, or `[1 + grid_size*grid_size, embed_dim]` if using cls_token rr\rr]r) rDrEr6rr rFrGrHrorark) r/rerfrgrhrirOrPrQrUs r+rjrjss.)S!!* YYy|2:: 6)A,:R SVi iF YYy|2:: 6)A,:R SVi iF ;;v &D 88Dq !D <<Ay|Yq\: ;D4YEI\A%NNBHHl-F$G#SZ[\ r-cUS-S:wa [S5e[US-US5n[US-US5n[R"X#/SS9nU$)a' This function generates 2D sinusoidal positional embeddings from a grid. Args: embed_dim (`int`): The embedding dimension. grid (`np.ndarray`): Grid of positions with shape `(H * W,)`. Returns: `np.ndarray`: The 2D sinusoidal positional embeddings with shape `(H * W, embed_dim)` rrrnrr])rCrvr6ra)r/rQrprqr*s r+roros]1};<< 1aa IE 0aa IE ..%a 0C Jr-clUS-S:wa [S5e[R"US-[RS9nX S- -nSSU-- nUR S5n[R "S X5n[R "U5n[R"U5n[R"XE/S S 9nU$) a This function generates 1D positional embeddings from a grid. Args: embed_dim (`int`): The embedding dimension `D` pos (`numpy.ndarray`): 1D tensor of positions with shape `(M,)` Returns: `numpy.ndarray`: Sinusoidal positional embeddings of shape `(M, D)`. rrrnr\rtr.rurzm,d->mdrr]) rCr6rrwrHeinsumr$r%ra)r/rsryrzr{r|r*s r+rvrvs1};<< IIi1nBJJ 7E _E %, E ++b/C ))Is *CffSkGffSkG ..'+! 4C Jr-cR^\rSrSrSrSU4SjjrSrSrSrU=r $) PatchEmbedia 2D Image to Patch Embedding with support for SD3 cropping. Args: height (`int`, defaults to `224`): The height of the image. width (`int`, defaults to `224`): The width of the image. patch_size (`int`, defaults to `16`): The size of the patches. in_channels (`int`, defaults to `3`): The number of input channels. embed_dim (`int`, defaults to `768`): The output dimension of the embedding. layer_norm (`bool`, defaults to `False`): Whether or not to use layer normalization. flatten (`bool`, defaults to `True`): Whether or not to flatten the output. bias (`bool`, defaults to `True`): Whether or not to use bias. interpolation_scale (`float`, defaults to `1`): The scale of the interpolation. pos_embed_type (`str`, defaults to `"sincos"`): The type of positional embedding. pos_embed_max_size (`int`, defaults to `None`): The maximum size of the positional embedding. c V>[TU]5 X-X#--n XplX`lXl[ R "XEX34X8S9UlU(a[ R"USSS9Ul OSUl X0l X-X#-sUl Ul X-Ul XlU (aU n O[U S-5n U cSUlgU S:Xa[[#UU URURSS9nU (aS OSnUR%S UR'5R)S 5US 9 g[+S U 35e)N kernel_sizestridebiasFgư>)elementwise_affineeps?sincosr>)rirhr4TrUr persistentzUnsupported pos_embed_type: )super__init__flatten layer_normpos_embed_max_sizerConv2dproj LayerNormnorm patch_sizeheightwidthrirhrErUrlregister_bufferr" unsqueezerC)selfrrr in_channelsr/rrrrhpos_embed_typer num_patchesrerUr __class__s r+rPatchEmbed.__init__s. +0CD  $"4II 0HQ[    Y5dSDIDI$"("68K TZ-#6  *IK,-I  !!DN x '/..$($<$< I"4J  ioo.?.I.I!.LYc d;N;KLM Mr-c0URc [S5eXR-nX R-nXR:a[SUSURS35eX R:a[SUSURS35eURU- S-nURU- S-nURR SURURS 5nUSS2X3U-2XDU-2SS24nUR SS UR S 5nU$) z2Crops positional embeddings for SD3 compatibility.Nz.`pos_embed_max_size` must be set for cropping.zHeight (z/) cannot be greater than `pos_embed_max_size`: .zWidth (rrr)rrCrrUrHr)rrrtopleftspatial_pos_embeds r+cropped_pos_embedPatchEmbed.cropped_pos_embeds>  " " *MN N??*( ++ +6("QRVRiRiQjjkl  ** *% OPTPgPgOhhij &&/A5''%/A5 NN221d6M6MtOfOfhjk-aV|1CTSXLEXZ[.[\-55a=N=T=TUW=XY  r-c hURbURSSup#O7URSUR-URSUR-p2URU5nUR(a!UR S5R SS5nUR (aURU5nURcURUR5$UR(aURX#5nOURU:wdURU:wad[URRSX#4URUR UR"SS9nUR%5R'S5nO URnX-RUR5$)Nrrrr>)r/rerirhrr4r)rrrrr transposerrrUtorrrrrlrirhrr"r)rlatentrrrUs r+forwardPatchEmbed.forward+sW  " " ."LL-MFE"LL,?bAQUYUdUdAdE6" <<^^A&00A6F ??YYv&F >> !99V\\* *  " "..v=I{{f$ e(;3"nn2226%o"nn(,(@(@!== $  &OO-77:  NN "&&v||44r-) rirrrhrrrrUrrr) rr9FTTrrN) __name__ __module__ __qualname____firstlineno____doc__rrr__static_attributes__ __classcell__rs@r+rrsA& 5Nn!.55r-rc6^\rSrSrSrSU4SjjrSrSrU=r$)LuminaPatchEmbediKa^ 2D Image to Patch Embedding with support for Lumina-T2X Args: patch_size (`int`, defaults to `2`): The size of the patches. in_channels (`int`, defaults to `4`): The number of input channels. embed_dim (`int`, defaults to `768`): The output dimension of the embedding. bias (`bool`, defaults to `True`): Whether or not to use bias. cn>[TU]5 Xl[R"X-U-UUS9Ulg)N in_features out_featuresr)rrrrLinearr)rrrr/rrs r+rLuminaPatchEmbed.__init__Vs4 $II"/+="  r-c:URUSR5nUR=p4UR5upVpxXs-X-pUR XVXX5R SSSSSS5nUR S5nURU5nUR SS5n[R"URSURS[RURS9n UU Xx4/U-USU 2SU 24R SS5RS54$) a Patchifies and embeds the input tensor(s). Args: x (list[torch.Tensor] | torch.Tensor): The input tensor(s) to be patchified and embedded. Returns: tuple[torch.Tensor, torch.Tensor, list[tuple[int, int]], torch.Tensor]: A tuple containing the patchified and embedded tensor(s), the mask indicating the valid patches, the original image size(s), and the frequency tensor(s). rrr7rr9rrN) rrrsizeviewpermuterrronesrint32r) rx freqs_cis patch_height patch_width batch_sizechannelrr height_tokens width_tokensmasks r+rLuminaPatchEmbed.forward_sLL1- %)__4 -.VVX* V&,&R| FF: \ _ g g q!Q1  IIaL IIaL IIaOzz!''!*aggaj AHHU  _  * n}nm|m3 4 < [TU]5 XlX lX@lXlXplXlXlXl Xl Xl Xl Xl Uc[R"X4X4XS9UlO$[R""X1-U-U-U5Ul[R""XT5UlU(dU(a&UnUR'XU 5nUR)SUUS9 gg)Nr pos_embeddingr)rrrrr/rrrrrr2r3rrrrrr text_proj_get_positional_embeddingsr)rrrrr/rrrrrrrr2r3rrrrrs r+rCogVideoXPatchEmbed.__init__s$ $("*(**D'#6 +F(,H))B&1R.   Z4LU_DI  +":Z"G,"VXabDI>= $(I:J ;;MYfgM  -J W)Jr-rc XR-nX R-nUS- UR-S-nXV-U-n[URXe4UURUR USS9n U R SS5n U RSURU-URSS9n U RSS2URS24RU 5 U $)Nrr>)rr4rF requires_grad) rrrVr/r2r3r new_zerosrdatacopy_) rrrrrpost_patch_heightpost_patch_widthpost_time_compression_framesrrjoint_pos_embeddings r+r.CogVideoXPatchEmbed._get_positional_embeddingss*__<'??:(5(9d>]>]']`a'a$':=YY / NN  1 (  , ,  - - &--a3 +55 t''+5t~~UZ6    D$<$<$>!>?EEmT""r- text_embeds image_embedsc URU5nURup4pVnURcxURSXVU5nUR U5nUR "X4/URSSQ76nUR S5RSS5nUR SS5nOURnURn URSSSSS5nURX4U -XU-XU-X5nURSSSSS SSS 5R SS 5R SS5nUR U5n[R"X/SS 9R5n UR(dUR(aUR(a+URU:wdUR U:wa [#S 5eUS- UR$-S-n UR U:wd URU:wdUR&U :waUR)XgXR*S 9n O UR,n U R/U R0S9n X-n U $)a Args: text_embeds (`torch.Tensor`): Input text embeddings. Expected shape: (batch_size, seq_length, embedding_dim). image_embeds (`torch.Tensor`): Input image embeddings. Expected shape: (batch_size, num_frames, channels, height, width). Nrrr9rrr7rraIt is currently not possible to generate videos at a different resolution that the defaults. This should only be the case with 'THUDM/CogVideoX-5b-I2V'.If you think this is incorrect, please open an issue at https://github.com/huggingface/diffusers/issues.rr\)rrrrHrrrrrrrr# contiguousrrrrrCrrrrrrr) rrrr num_frameschannelsrrpp_tembedspre_time_compression_framesrs r+rCogVideoXPatchEmbed.forwardsKnn[1 :F:L:L7 %    $'//HeLL99\2L',,Z]lFXFXYZY[F\]L'//2<L'//#-saKQJPQL(//1aAq!QGOOPQSTU]]^_abcL99\2L  'Q *,   ) )T-S-S554;L;LPU;UY]YkYkouYu  ,6>T=\=\*\_`*` '""f,$$-%%)DD $ ? ?#>}}!@! !% 2 2 ),,6<<,@M+F r-)r/rrrrrrrr2rr3rrr)rNriTZ<1r7g?r.TTN)rrrrrEboolr"rrrTensorrrrrrs@r+rrsU#'"*+#&-2.1*.26!/X/XDj/X /X  /X  /X/X/X/X/X%(/X!/X&+/X',/X$(/X ,0!/X" #/X/Xdhl# #03#DG#QVQ]Q]`dQd# #285<<8u||88r-rc ^\rSrSrS S\S\S\S\S\4 U4SjjjrS\RS \RS \R4S jrS r U=r $)CogView3PlusPatchEmbedir hidden_sizertext_hidden_sizercN>[TU]5 XlX lX0lX@lXPl[R"XS--U5Ul [R"XB5Ul [X%USS9nURXUU5nURSUR5SS9 g)Nrr>)rir4rUFr)rrrr rr rrrrrrlrHrr")rrr rr rrUrs r+rCogView3PlusPatchEmbed.__init__s &&$ 0"4IIkM9;G #3A+ 7IW[ %%&8kZ  [)//*;Nr- hidden_statesencoder_hidden_statesrcZURup4pVXPR-S:wdX`R-S:wa [S5eXPR-nX`R-nURX4XPRX`R5nUR SSSSSS5R 5nURX5U-X@R-UR-5nUR U5nURU5n[R"X!/SS9nURSnURSU2SU24RXV-S 5n[R"XpR4URURS 9n [R"X/SS9S n X-R!UR5$) Nrz0Height and width must be divisible by patch sizerr7rr9rrrr)N.)rrrCrrrrrrr#rUrHrkr rrr) rr rrrrr text_lengthimage_pos_embedtext_pos_embedrUs r+rCogView3PlusPatchEmbed.forward"s-:-@-@* V OO #q (EOO,Cq,HOP P??*(%**:Y^`o`op %--aAq!Q?JJL %**:~wQ`Q`G`cgcrcrGrs  -0 $/D E #8"HaP ,11!4 ..&&5&9AA&.RTU ** +?3H3HQ`QgQg II~?QG R )--m.A.ABBr-)r rrrrr r)ri rr rrrrrErrrrrrrs@r+rrs $"% OOO O  O  OO4CU\\C%,,C[`[g[gCCr-rTthetause_real grid_typemax_sizec ^^^ USLa [S5eUS:XaUupUumm [R"U SU STS- -T- TU[RS9n [R"U SU ST S- -T - T U[RS9n [R"TU[RS9n [R"STTS- -T- TU[RS9n OUS:XatUupUumm [R"X[RS9n [R"X[RS9n [R"TU[RS9n O [S5eUS -nUS -S -nUS -S -n[ UXSS 9n[ UXSS 9n[ UXSS 9nUU U4S jnUunnUunnUunnUS:XaUSTUSTnnUSTUSTnnUST UST nnU"UUU5nU"UUU5nUU4$)a RoPE for video tokens with 3D structure. Args: embed_dim: (`int`): The embedding dimension size, corresponding to hidden_size_head. crops_coords (`tuple[int]`): The top-left and bottom-right coordinates of the crop. grid_size (`tuple[int]`): The grid size of the spatial positional embedding (height, width). temporal_size (`int`): The size of the temporal dimension. theta (`float`): Scaling factor for frequency computation. grid_type (`str`): Whether to use "linspace" or "slice" to compute grids. Returns: `torch.Tensor`: positional embedding with shape `(temporal_size * grid_size[0] * grid_size[1], embed_dim/2)`. TzJ `use_real = False` is not currently supported for get_3d_rotary_pos_embedlinspacerrr:slicez%Invalid value passed for `grid_type`.r7r9)rrc >USS2SSSS24RSTTS5nUSSS2SSS24RTSTS5nUSSSS2SS24RTTSS5n[R"XU/SS9nURTT-T-S5nU$)Nrr)expandrr#r)freqs_tfreqs_hfreqs_wfreqs grid_size_h grid_size_wr1s r+combine_time_height_width:get_3d_rotary_pos_embed..combine_time_height_widths!T4*+22  [" $4*+22 2{B $a*+22 ;B  w 'R  K '+ 5r  r-N)rCrrr rget_1d_rotary_pos_embed)!r/ crops_coordsrer1rrrrrrstoprOrPrSmax_hmax_wdim_tdim_hdim_wr r!r"r&t_cost_sinh_cosh_sinw_cosw_sinr%r$r$r%s! ` @@r+get_3d_rotary_pos_embedr6?s6>teffJ" #, [ !Hd1gq1K?U[chcpcp  !Hd1gq1K?U[chcpcp mF%--P }  12]BMZ`hmhuhu  g  #, [e%--He%--HmF%--P@AA NE NQ E NQ E&eV4PG%eV4PG%eV4PG&LE5LE5LE5G^m,eN].Cu\k*E,;,?u\k*E,;,?u #E5% 8C #E5% 8C 8Or-rhcUupxUupUupn [R"SX3S- -U- X6[RS9n[R"USUSU S- -U - X[RS9n[R"USUSU S- -U - X[RS9nUS-nUS-nUS-n[UX- USSS9n[UX- USSS9n[UUU - USSS9nUUUXU4$)Nrrr:r9TF)rrrepeat_interleave_real)rrr r()r/r)rer1rhrrrr*r$r%interpolation_scale_tinterpolation_scale_hinterpolation_scale_wrSrOrPr-r.r/r r!r"s r+get_3d_rotary_pos_embed_allegror<s:KE(KJ]G2G ^^ =A- . > didqdqF^^ a$q'[1_- ;[_d_l_lF^^ a$q'[1_- ;[_d_l_lF NE NE NE& v-UTbgG& v-UTbgG& v--UTbgG GWff <>&4 8D ;;t #D <<A/ 12/ 0D1)HUI r-cxUupE[R"USUSUSS[RS9n[R"USUSUSS[RS9n[R"Xv5n[R"USS9nUR SS/UR SSQ5n[XUS9n U $) a RoPE for image tokens with 2d structure. Args: embed_dim: (`int`): The embedding dimension size crops_coords (`tuple[int]`) The top-left and bottom-right coordinates of the crop. grid_size (`tuple[int]`): The grid size of the positional embedding. use_real (`bool`): If True, return real part and imaginary part separately. Otherwise, return complex numbers. Returns: `torch.Tensor`: positional embedding with shape `( grid_size * grid_size, embed_dim/2)`. rF)endpointrrr]rNr>)r6rr rFrGrHrr@) r/r)rerrr*rOrPrQrUs r+r?r?s"KE [[q47IaL5PRPZPZ [F [[q47IaL5PRPZPZ [F ;;v &D 88Dq !D <<A/ 12/ 0D1)HUI r-cPUS-S:Xde[US-USRS5US9n[US-USRS5US9nU(a=[R"USUS/SS9n[R"USUS/SS9nXV4$[R"X4/SS9nU$)a Get 2D RoPE from grid. Args: embed_dim: (`int`): The embedding dimension size, corresponding to hidden_size_head. grid (`np.ndarray`): The grid of the positional embedding. use_real (`bool`): If True, return real part and imaginary part separately. Otherwise, return complex numbers. Returns: `torch.Tensor`: positional embedding with shape `( grid_size * grid_size, embed_dim/2)`. r7rrrr>rr)r(rHrr#)r/rQrrprqr%r$r*s r+r@r@s q=A   $QQ+h E $QQ+h Eiiq58,!4iiq58,!4xiiA. r-c@US-S:Xde[US-XUS9n[US-X#US9nURUSUS-S5RSUSS5nURSX S-S5RUSSS5n[R"XV/SS9R S5nU$)a; Get 2D RoPE from grid. Args: embed_dim: (`int`): The embedding dimension size, corresponding to hidden_size_head. grid (`np.ndarray`): The grid of the positional embedding. linear_factor (`float`): The linear factor of the positional embedding, which is used to scale the positional embedding in the linear layer. ntk_factor (`float`): The ntk factor of the positional embedding, which is used to scale the positional embedding in the ntk layer. Returns: `torch.Tensor`: positional embedding with shape `( grid_size * grid_size, embed_dim/2)`. r7rr) linear_factor ntk_factorrrr)r(rr`rr#r)r/len_hlen_wrFrGrprqr*s r+get_2d_rotary_pos_embed_luminarJ?s$ q=A   #Qz E $Qz E JJuaa 3 : :1eQ JE JJq%a 3 : :5!Q JE ))UN + 3 3A 6C Jr-@rrsc NUS-S:Xde[U[5(a[R"U5n[U[R 5(a[R "U5nX%-nSU[R"SUSXqRS9U- -- U- n[R"X5nURRS:Hn U (aUR5nU(aU(a}UR5RSSURSS-S9R5n UR5RSSURSS-S9R5n X4$U(a[R"UR5UR5/SS 9R5n [R"UR5UR5/SS 9R5n X4$[R "[R""U5U5n U $) a Precompute the frequency tensor for complex exponentials (cis) with given dimensions. This function calculates a frequency tensor with complex exponentials using the given dimension 'dim' and the end index 'end'. The 'theta' parameter scales the frequencies. The returned tensor contains complex values in complex64 data type. Args: dim (`int`): Dimension of the frequency tensor. pos (`np.ndarray` or `int`): Position indices for the frequency tensor. [S] or scalar theta (`float`, *optional*, defaults to 10000.0): Scaling factor for frequency computation. Defaults to 10000.0. use_real (`bool`, *optional*): If True, return real part and imaginary part separately. Otherwise, return complex numbers. linear_factor (`float`, *optional*, defaults to 1.0): Scaling factor for the context extrapolation. Defaults to 1.0. ntk_factor (`float`, *optional*, defaults to 1.0): Scaling factor for the NTK-Aware RoPE. Defaults to 1.0. repeat_interleave_real (`bool`, *optional*, defaults to `True`): If `True` and `use_real`, real part and imaginary part are each interleaved with themselves to reach `dim`. Otherwise, they are concateanted with themselves. freqs_dtype (`torch.float32` or `torch.float64`, *optional*, defaults to `torch.float32`): the dtype of the frequency tensor. Returns: `torch.Tensor`: Precomputed frequency tensor with complex exponentials. [S, D/2] rrr.rnpurr@rr)rDrErrr6ndarray from_numpyrrxtyper"r%rKrr$r#polar ones_like) rrsrrrFrGr8 freqs_dtyper#is_npu freqs_cos freqs_sinrs r+r(r(`sH 7a<<#sll3#rzz""s#  E uaa{::VY\\]^ann  KK #E \\  % 'F  *IIK11! TUYZHZ1[aac IIK11! TUYZHZ1[aac ## IIuyy{EIIK8bAGGI IIuyy{EIIK8bAGGI ##KK 6> r-rruse_real_unbind_dim sequence_dimcNU(GaUupVUS:XaUSSSS2SS24nUSSSS2SS24nO0US:XaUSSS2SSS24nUSSS2SSS24nO[SUS35eURUR5URUR5peUS:Xa\UR"/URSSQSPSP76R S5upx[ R"U*U/SS9RS5n ObUS :XaMUR"/URSSQSPSP76R S 5upx[ R"U*U/SS9n O[S US 35eUR5U-U R5U--RUR5n U $[ R"UR5R"/URSSQSPSP765n URS5n[ R"X-5RS5n U RU5$) a Apply rotary embeddings to input tensors using the given frequency tensor. This function applies rotary embeddings to the given query or key 'x' tensors using the provided frequency tensor 'freqs_cis'. The input tensors are reshaped as complex numbers, and the frequency tensor is reshaped for broadcasting compatibility. The resulting tensors contain rotary embeddings and are returned as real tensors. Args: x (`torch.Tensor`): Query or key tensor to apply rotary embeddings. [B, H, S, D] xk (torch.Tensor): Key tensor to apply freqs_cis (`tuple[torch.Tensor]`): Precomputed frequency tensor for complex exponentials. ([S, D], [S, D],) Returns: tuple[torch.Tensor, torch.Tensor]: tuple of modified query tensor and key tensor with rotary embeddings. rNrz`sequence_dim=z` but should be 1 or 2.rrr9rz`use_real_unbind_dim=z` but should be -1 or -2.)rCrrrHrunbindrrGrr#r"rview_as_complexr view_as_realtype_as) rrrrWrXr%r$x_realx_imag x_rotatedrzx_outs r+apply_rotary_embrbs* 1 dD!Q&'CdD!Q&'C Q dAtQ&'CdAtQ&'C~l^;RST T66!((#SVVAHH%5S " $YY< FFqII B &YY< .apply_1d_ropeskk##AtQM2kk##AtQM24v||B/14445vc6<Z7[BRC9"5 $~';';'='CCGG UUr-r9rrrrr)chunkrr#) rr positionsrkr0r1r2r3r4r5thws r+apply_rotary_emb_allegrorrs{V6?2NUNUNUggaRg GA!a1u4Aa1u4Aa1u4A 1)$A Hr-c ^^\rSrSrS S\S\S\S\S\S-4 U4SjjjrS S jrS rU=r $) TimestepEmbeddingiNrtime_embed_dimact_fnout_dim post_act_fncB>[T U]5 [R"XU5UlUb[R"XaSS9UlOSUl[ U5UlUbUnOUn[R"X(U5UlUcSUl g[ U5Ul g)NFr) rrrrlinear_1 cond_projr actlinear_2post_act) rrrurvrwrx cond_proj_dimsample_proj_biastime_embed_dim_outrs r+rTimestepEmbedding.__init__s  +?OP  $YY}NDN!DN!&)  !( !/  .FVW   DM*;7DMr-cUbXRU5-nURU5nURbURU5nURU5nURbUR U5nU$r)r|r{r}r~r)rsample conditions r+rTimestepEmbedding.forwardsj  nnY77Fv& 88 XXf%Fv& == $]]6*F r-)r}r|r{r~r)siluNNNTr) rrrrrEstrrrrrrs@r+rtrts] "&888 8  8 4Z 88@  r-rtc z^\rSrSrS S\S\S\S\4U4SjjjrS\RS\R4S jr S r U=r $) Timestepsi num_channelsr rrcR>[TU]5 XlX lX0lX@lgr)rrrr rr)rrr rrrs r+rTimesteps.__init__s% (.$8! r-r rcp[UURURURURS9nU$)N)r rr)r,rr rr)rr t_embs r+rTimesteps.forward&s8&     00!%!:!:**   r-)rr rr)r) rrrrrErr"rrrrrrrs@r+rrsKS4W\eh%,,r-rcD^\rSrSrSrSS\S\4U4SjjjrSrSr U=r $) GaussianFourierProjectioni1z-Gaussian Fourier embeddings for noise levels.embedding_sizerc8>[TU]5 [R"[R "U5U-SS9UlX@lXPlU(aGU?[R"[R "U5U-SS9Ul URUlU? gg)NFr) rrr Parameterrrandnweightrr W)rrrset_W_to_weightrr rs r+r"GaussianFourierProjection.__init__4sz ll5;;~#>#FV[\ .  \\%++n"="EUZ[DF&&DK r-cUR(a[R"U5nUSS2S4URSSS24-S-[R-nUR (aA[R "[R"U5[R"U5/SS9nU$[R "[R"U5[R"U5/SS9nU$)Nrrr) rrrr6pir r#r%r$)rrx_projrzs r+r!GaussianFourierProjection.forwardCs 88 ! A1d7dkk$'22Q6>   ))UYYv. &0ABKC ))UYYv. &0ABKC r-)rr rr)r.TTF) rrrrrrEr"rrrrrs@r+rr1s17ns ! 05    r-rcB^\rSrSrSrSS\S\4U4SjjjrSrSrU=r $) SinusoidalPositionalEmbeddingiPaCApply positional information to a sequence of embeddings. Takes in a sequence of embeddings with shape (batch_size, seq_length, embed_dim) and adds positional embeddings to them Args: embed_dim: (int): Dimension of the positional embedding. max_seq_length: Maximum sequence length to apply positional embeddings r/max_seq_lengthc>[TU]5 [R"U5R S5n[R "[R"SUS5[ R"S5*U- -5n[R"SX!5n[R"X4-5USSS2SSS24'[R"X4-5USSS2SSS24'URSU5 g)NrrrrKpe) rrrrrr!rrrkr$r%r)rr/rpositiondiv_termrrs r+r&SinusoidalPositionalEmbedding.__init__\s <</99!<99U\\!Y:txx?P>PS\>\]^ [[N 68#671aA:8#671aA: T2&r-cRURup#nXRSS2SU24-nU$r)rr)rr_ seq_lengths r+r%SinusoidalPositionalEmbedding.forwardes.77q ;J;' 'r-) rrrrrrErrrrrs@r+rrPs* '#'s''r-rcF^\rSrSrSrS\S\S\S\4U4SjjrSrS rU=r $) ImagePositionalEmbeddingsikaU Converts latent image classes into vector embeddings. Sums the vector embeddings with positional embeddings for the height and width of the latent space. For more details, see figure 10 of the dall-e paper: https://huggingface.co/papers/2102.12092 For VQ-diffusion: Output vector embeddings are used as input for the transformer. Note that the vector embeddings for the transformer are different than the vector embeddings from the VQVAE. Args: num_embed (`int`): Number of embeddings for the latent pixels embeddings. height (`int`): Height of the latent image i.e. the number of height embeddings. width (`int`): Width of the latent image i.e. the number of width embeddings. embed_dim (`int`): Dimension of the produced vector embeddings. Used for the latent pixel, height, and width embeddings. num_embedrrr/c6>[TU]5 X lX0lXlX@l[ R"URU5Ul[ R"URU5Ul [ R"URU5Ul gr) rrrrrr/r Embeddingr* height_emb width_emb)rrrrr/rs r+r"ImagePositionalEmbeddings.__init__sf   ""<< :,,t{{I>djj)<r-cJURU5nUR[R"URUR S9R SUR55nURS5nUR[R"URUR S9R SUR55nURS5nX4-nUR SURUR-S5nX%SS2SURS2SS24-nU$)Nrrrr) r*rrrrrrrrrr)rindexr*rrpos_embs r+r!ImagePositionalEmbeddings.forwardshhuo__U\\$++ell%S%X%XYZ\`\g\g%hi  ))!, NN5<< 5<<#P#U#UVWY]YcYc#de ''* (,,q$++ ":B?A~1~q011 r-)r*r/rrrrrrrs@r+rrks=.=== =  =$r-rc\^\rSrSrSrU4SjrSSjrSS\R4Sjjr Sr U=r $) LabelEmbeddingia Embeds class labels into vector representations. Also handles label dropout for classifier-free guidance. Args: num_classes (`int`): The number of classes. hidden_size (`int`): The size of the vector embeddings. dropout_prob (`float`): The probability of dropping a label. c>[TU]5 US:n[R"X-U5UlXlX0lgNr)rrrrembedding_table num_classes dropout_prob)rrr ruse_cfg_embeddingrs r+rLabelEmbedding.__init__s; (1,!||K,K[Y&(r-cUc:[R"URSURS9UR:nO[R "US:H5n[R "X0RU5nU$)z2 Drops labels to enable classifier-free guidance. rrr)rrandrrrtensorwherer)rlabelsforce_drop_idsdrop_idss r+ token_dropLabelEmbedding.token_drops_  !zz&,,q/&--H4K\K\\H||Na$78HX'7'7@ r-rcURS:nUR(aU(dUbURX5nURU5nU$r)rtrainingrr)rrr use_dropout embeddingss r+rLabelEmbedding.forwardsD''!+ MMk~/I__V[TU]5 X@l[R"X RU-5Ul[R"X5Ulgr)rrrrrrr)rrrrrrs r+rTextImageProjection.__init__sC %:"IIo7Q7QTg7gh>Gr-rrcURSnURU5nURX0RS5nUR U5n[ R "XA/SS9$)Nrrrr)rrrHrrrr#)rrrrimage_text_embedss r+rTextImageProjection.forwardsd &&q) !--l;-55jB\B\^`ann[1 yy+9qAAr-)rrr)rr rrs@r+rrsk#"#&%' H H H! H # H H B5<< Bu|| B Br-rcd^\rSrSrS S\S\S\4U4SjjjrS\R4SjrSr U=r $) ImageProjectionirrrc>[TU]5 X0l[R"XRU-5Ul[R "U5Ulgr)rrrrrrrr)rrrrrs r+rImageProjection.__init__sD %:"IIo7Q7QTg7ghLL!45 r-rcURSnURURURRR55nUR X R S5nURU5nU$)Nrr)rrrrrrHrr)rrrs r+rImageProjection.forwardsl!''* ((9J9J9Q9Q9W9W)XY #++J8R8RTVW yy. r-)rrr)rrrrrs@r+rrsI ##&%' 6 6! 6 # 6 6ELLr-rcN^\rSrSrSU4SjjrS\R 4SjrSrU=r $)IPAdapterFullImageProjectionic~>[TU]5 SSKJn U"XSSS9Ul[ R "U5UlgNr FeedForwardgelu)mult activation_fn)rr attentionrffrrr)rrrrrs r+r%IPAdapterFullImageProjection.__init__s3 *oZ`aLL!45 r-rcBURURU55$r)rr)rrs r+r$IPAdapterFullImageProjection.forward syy.//r-)rr)rr rrrrrrrrrrrs@r+rrs60ELL00r-rcN^\rSrSrSU4SjjrS\R 4SjrSrU=r $)IPAdapterFaceIDImageProjectionic>[TU]5 SSKJn X@lX lU"XU-USS9Ul[R"U5Ul gr) rrrr num_tokensrrrrr)rrrrrrrs r+r'IPAdapterFaceIDImageProjection.__init__sD *$#6 oZ/OVZjpqLL!45 r-rcURU5nURSURUR5nUR U5$)Nr)rrHrrr)rrrs r+r&IPAdapterFaceIDImageProjection.forwards: GGL ! IIb$//4+C+C Dyy|r-)rrrr)rrrrrrs@r+rrs6ELLr-rc6^\rSrSrSU4SjjrSSjrSrU=r$)CombinedTimestepLabelEmbeddingsic>[TU]5 [SSSS9Ul[ SUS9Ul[ XU5Ulg)NrTrrr rrru)rrr time_projrttimestep_embedderrclass_embedder)rrr class_dropout_probrs r+r(CombinedTimestepLabelEmbeddings.__init__s? "T`ab!2sS`!a,[I[\r-cURU5nURURUS95nURU5nXR-nU$Nr\)rrrr)rtimestep class_labels hidden_dtypetimesteps_proj timesteps_emb conditionings r+r'CombinedTimestepLabelEmbeddings.forward&sL1..~/@/@|/@/TU **<8 $3 r-)rrr)g?rrrrrrrrrrs@r+rrs]r-rc.^\rSrSrU4SjrSrSrU=r$)"CombinedTimestepTextProjEmbeddingsi1c~>[TU]5 [SSSS9Ul[ SUS9Ul[ X!SS9UlgNrTrrrrrv)rrrrrtrPixArtAlphaTextProjection text_embedderrr pooled_projection_dimrs r+r+CombinedTimestepTextProjEmbeddings.__init__2s@ "T`ab!2sS`!a67Ldjkr-cURU5nURURURS95nUR U5nXE-nU$r)rrrrr)rrpooled_projectionr r pooled_projectionsr s r+r*CombinedTimestepTextProjEmbeddings.forward9sU1..~/@/@GXG^G^/@/_` !//0AB$9 r-)rrrrrs@r+rr1slr-rc.^\rSrSrU4SjrSrSrU=r$)*CombinedTimestepGuidanceTextProjEmbeddingsiDc>[TU]5 [SSSS9Ul[ SUS9Ul[ SUS9Ul[X!SS9Ulgr) rrrrrtrguidance_embedderrrrs r+r3CombinedTimestepGuidanceTextProjEmbeddings.__init__EsP "T`ab!2sS`!a!2sS`!a67Ldjkr-cURU5nURURURS95nURU5nUR URURS95nXW-nUR U5n X-n U $r)rrrrr!r) rrguidancerr r  guidance_proj guidance_embtime_guidance_embrr s r+r2CombinedTimestepGuidanceTextProjEmbeddings.forwardMs1..~/@/@GXG^G^/@/_` x0 --m.>.>EVE\E\.>.]^ )8!//0AB(= r-)r!rrrrrs@r+rrDsl  r-rc ^\rSrSrSS\S\S\S\4U4SjjjrS\RS\RS \RS \RS \RS \R4 S jr Sr U=r $)&CogView3CombinedTimestepSizeEmbeddingsi\r condition_dimr timesteps_dimc>[TU]5 [USSS9Ul[USSS9Ul[ XAS9Ul[X1SS9Ulg)NTrrrrr) rrrrcondition_projrtrrcondition_embedder)rr r+rr,rs r+r/CogView3CombinedTimestepSizeEmbeddings.__init__]sP " tjkl']TXopq!2}!k";[TU]5 [R"[R "US-U5US-- 5Ul[R"X"5Ul[R"X"5Ul [R"X"5Ul [R"X$=(d U5Ul X0l gNrr) rrrrrrpositional_embeddingrk_projq_projv_projc_projr=)rr<r/r=r>rs r+r HunyuanDiTAttentionPool.__init__s $&LL[1_i1X[dfi[i1i$j!ii 5 ii 5 ii 5 ii +BC "r-c JURSSS5n[R"URSSS9U/SS9nXRSS2SSS24R UR 5-n[R"S0SUSS_SU_S U_S URS _S UR_S URR_SURR_SURR_SS_S[R"URRURRURR/5_SS_SS_SS_SS_SUR R_SUR R_SS_SUR"_SS_6upUR%S5$)NrrrTrkeepdimrquerykeyvalueembed_dim_to_checkrr= q_proj_weight k_proj_weight v_proj_weightin_proj_weight in_proj_biasbias_kbias_v add_zero_attnF dropout_pout_proj_weight out_proj_biasuse_separate_proj_weightr need_weightsr)rrr#meanrArrremulti_head_attention_forwardrr=rCrrBrDrrErsqueeze)rrrs r+rHunyuanDiTAttentionPool.forwards IIaA  IIqvv!Tv2A6A > ))!T1*588A A-- BQ%    !wwr{  nn  ++,,  ++,, ++,,   DKK$4$4dkk6F6F HXHX#YZ      !KK.. ++**! "&*# $]]% &' *yy|r-)rErBr=rArCrDr rrrrrErrrrrs@r+r;r;}s4#C#C#C#UX##r-r;c>^\rSrSrSU4SjjrSSjrSrU=r$)-HunyuanCombinedTimestepTextSizeStyleEmbeddingic0>[TU]5 [SSSS9Ul[ SUS9Ul[SSSS9Ul[X4SUS9UlXPl U(a%[R"SU5Ul S U-U-nOUn[UUS -US S 9Ulg) NrTrrrr)r=r>rr7 silu_fp32)rr rrv)rrrrrtr size_projr;pooler"use_style_cond_and_image_meta_sizerrstyle_embedderrextra_embedder)rr rseq_lenrrg extra_in_dimrs r+r6HunyuanCombinedTimestepTextSizeStyleEmbedding.__init__s "T`ab!2sS`!a"T`ab- ABW  3U/ -"$,,q-"@D "]25JJL0L7$%)&  r-cURU5nURURUS95nURU5nUR(ajUR UR S55nURUS9nUR SS5nURU5n [R"XU /SS9n O[R"U/SS9n XpRU 5-n U $)Nr\rrcrr) rrrrfrgrerrhrr#ri) rrrimage_meta_sizestyler r r rstyle_embedding extra_condr s r+r5HunyuanCombinedTimestepTextSizeStyleEmbedding.forwards1..~/@/@|/@/TU "[[)>?  2 2"nn_-A-A"-EFO-00|0DO-222w?O#11%8O$6#Y_`aJ$6#7Q?J$':'::'FF r-)rirfrerhrrrg)rrTrrrs@r+raras!# +/  Dr-rac2^\rSrSrSU4SjjrSrSrU=r$)&LuminaCombinedTimestepCaptionEmbeddingic >[TU]5 [USSS9Ul[ X1S9Ul[ R"[ R"U5[ R"UUSS95Ul g)NTrrrz) rrrrrtrr Sequentialrrcaption_embedder)rr rfrequency_embedding_sizers r+r/LuminaCombinedTimestepCaptionEmbedding.__init__se "14^a "3?W!t " LL, - II# ! r-cDURU5nURURURS95nUR 5R S5nX&-R SS9UR SS9- nURU5nURU5nXX-n U $Nr\rrr)rrrrr"rsumry) rr caption_feat caption_mask time_freq time_embedcaption_mask_floatcaption_feats_pool caption_embedr s r+r.LuminaCombinedTimestepCaptionEmbedding.forwardsNN8, ++ILL|?Q?QL,RS *//1;;B?*?DDDKN`NdNdijNdNkk/22<@--.@A !1 r-)ryrr)rrsrrrs@r+rurus "  r-ruc ^\rSrSrSS\S\S\S\S\SS4 U4S jjjrSS \RS \RS \RS \RS-4Sjjr Sr U=r $)%MochiCombinedTimestepCaptionEmbeddingir rrrunum_attention_headsrNc>[TU]5 [USSS9Ul[ XAS9Ul[ XSUS9Ul[R"X25Ul g)NTrwrr)rr/r>) rrrrrtrMochiAttentionPoolrfrr caption_proj)rr rrrurrs r+r.MochiCombinedTimestepCaptionEmbedding.__init__sT "PTkno!2~!l( 3Zg  IInLr-rrencoder_attention_maskr cURU5nURURUS95nURX#5nUR U5nXg-n X4$r)rrrrfr) rrrrr rtime_embrrr s r+r-MochiCombinedTimestepCaptionEmbedding.forwards_NN8, ))),,\,*JK![[)>W(()>? 4 ))r-)rrfrr)rrr) rrrrrErrrrrrrrrs@r+rrs "#$ MM #M M  M ! M MM,,0 *""* %||*!& * kkD( **r-rcB^\rSrSrSS\S\S\4U4SjjjrSrSrU=r$) TextTimeEmbeddingi' encoder_dimrur=c>[TU]5 [R"U5Ul[ X15Ul[R"X5Ul[R"U5Ul gr) rrrrnorm1AttentionPoolingpoolrrnorm2)rrrur=rs r+rTextTimeEmbedding.__init__(sJ \\+. $Y< IIk: \\.1 r-cURU5nURU5nURU5nURU5nU$r)rrrr)rr s r+rTextTimeEmbedding.forward/s@ =1  -0  -0  =1 r-)rrrr)@r_rs@r+rr's,2C2222r-rcv^\rSrSrS S\S\S\4U4SjjjrS\RS\R4SjrS r U=r $) TextImageTimeEmbeddingi7rrruc>[TU]5 [R"X5Ul[R "U5Ul[R"X#5Ulgr)rrrrrr text_norm image_proj)rrrrurs r+rTextImageTimeEmbedding.__init__8s> >Bn5))ODr-rrcpURU5nURU5nURU5nXC-$r)rrr)rrrtime_text_embedstime_image_embedss r+rTextImageTimeEmbedding.forward>s;>>+6>>*:;!OOL9 33r-)rrr)rrrcrrs@r+rr7sHEsE3E^aEE 45<<4u||44r-rcZ^\rSrSrSS\S\4U4SjjjrS\R4SjrSr U=r $) ImageTimeEmbeddingiIrruc>[TU]5 [R"X5Ul[R "U5Ulgr)rrrrrr image_normrrrurs r+rImageTimeEmbedding.__init__Js. ))OD,,~6r-rcJURU5nURU5nU$r)rr)rrrs r+rImageTimeEmbedding.forwardOs' OOL9 OO,=>  r-)rrrrcrrs@r+rrIs077377 !ELL!!r-rcr^\rSrSrS S\S\4U4SjjjrS\RS\R4SjrSr U=r $) ImageHintTimeEmbeddingiVrruc@>[TU]5 [R"X5Ul[R "U5Ul[R"[R"SSSSS9[R"5[R"SSSSS9[R"5[R"SSSSSS9[R"5[R"SSSSS9[R"5[R"SSSSSS9[R"5[R"SSSSS9[R"5[R"SS SSSS9[R"5[R"S S SSS95Ul g) Nr9rr)paddingrr)rr`rr7) rrrrrrrrxrSiLUinput_hint_blockrs r+rImageHintTimeEmbedding.__init__Ws ))OD,,~6 " IIaQ * GGI IIb"a + GGI IIb"a1 5 GGI IIb"a + GGI IIb"a1 5 GGI IIb"a + GGI IIb#q!A 6 GGI IIc1a +! r-rhintcnURU5nURU5nURU5nX24$r)rrr)rrrrs r+rImageHintTimeEmbedding.forwardms: OOL9 OO,=>$$T* &&r-)rrrrrrs@r+rrVs;  3  ,'ELL' ''r-rc2^\rSrSrSU4SjjrSrSrU=r$)riuc>[TU]5 X0l[R"[ R "SU5US-- 5Ul[R"X"URS9Ul [R"X"URS9Ul [R"X"URS9Ul Xl X R-Ul g)Nrrr\)rrrrrrrrArrBrCrDr= dim_per_head)rr=r/rrs r+rAttentionPooling.__init__xs  $&LLQ 1JYX[^1[$\!ii DJJG ii DJJG ii DJJG "%7r-c^^ UR5um p#U U4SjnURSSS9TRRUR5-n[ R "XQ/SS9nU"TRU55nU"TRU55nU"TRU55nS[R"[R"TR55- n [ R"SXi-Xy-5n [ R"U R5SS9R!U R5n [ R"SX5n U R#T SS5R%SS 5n U SS2S SS24$) Nc>URTSTRTR5nURSS5nUR TTR-STR5nURSS5nU$)Nrrr)rr=rrrH)rbsrs r+r'AttentionPooling.forward..shapesgr2t~~t/@/@AA Aq!A "t~~-r43D3DEA Aq!AHr-rTrHrz bct,bcs->btsrz bts,bcs->bctrr)rr[rArrrr#rCrBrDrsqrtrrsoftmaxr"rPrHr) rrlengthrr class_tokenqkvrrars ` @r+rAttentionPooling.forwards7FFHF ffDf1D4M4M4P4PQRQXQX4YY II{&A . $++k* + $++a. ! $++a. !DIIdii(9(9:;;naiCv||~26;;FLLI LL 3 IIb"a * *1a 0Aqzr-)rrrBr=rArCrDrrrs@r+rrus8""r-rc ^\rSrSrSS\S\S\S-SS4U4Sjjjr\SS .S \RS \RS\R4S jj5r S \RS \RS\R4S jr Sr U=r $)riNrr/r>rc >[TU]5 U=(d UUlXl[R "USU-5Ul[R "X"5Ul[R "X R5Ulg)Nr) rrr>rrrto_kvto_qto_out)rrr/r>rs r+rMochiAttentionPool.__init__s\ $1 #6 YYy!i-8 IIi3 ii ??; r-FrIrrcFURS5URS5:XdeURS5URS5:XdeUSS2SS2S4RURS9nXRSSS9R SS9- nX-RSUS9nU$)z Pool tokens in x using mask. NOTE: We assume x does not require gradients. Args: x: (B, L, D) tensor of tokens. mask: (B, L) boolean tensor indicating which tokens are not padding. Returns: pooled: (B, D) tensor of pooled tokens. rrNr\TrH)min)rrrr~clamp)rrrIpooleds r+ pool_tokensMochiAttentionPool.pool_tokenssvvayDIIaL(((vvayDIIaL(((Aq$J"""1hh1dh399a9@@(Aw7 r-cURS5nUSS2SSSS24R5n[R"USSS9nUR XSS9n[ R "XQ/SS9nURU5nURUSS2S 45nX0R-nURSSURU45nURSS 5nURS5upURSURU45nURS5n[R"XyXS S 9nURS5R!SS5nUR#U5nU$) a Args: x (`torch.Tensor`): Tensor of shape `(B, S, D)` of input tokens. mask (`torch.Tensor`): Boolean ensor of shape `(B, S)` indicating which tokens are not padding. Returns: `torch.Tensor`: `(B, D)` tensor of pooled tokens. rN)rrT)rLrrrrr9rw) attn_maskrV)rrrer'rrr#rrr unflattenrrZrscaled_dot_product_attentionr]rr) rrrDrx_poolkvrhead_dimrrs r+rMochiAttentionPool.forwardsJ FF1ID$)*//1 EE)V48 !!!4!8 IIvkq )ZZ] IIa1g 000 \\!a!9!98D E \\!Q yy| KKD44h? @ KKN  * *1SV W IIaL A & KKNr-)rr>rrrr)rrrrrEr staticmethodrrr BoolTensorrrrrs@r+rrs "& <  < <$J <  < <DIu||5<<ell(**U-=-=*%,,**r-rcURSSup#S[R"U5U- -nUSRURUR S9nXAR S5-n[R"UR5UR54SS9nURSS S S S5RX#US-S -5nU$) z Args: embed_dim: int box: a 3-D tensor [B x N x 4] representing the bounding boxes for GLIGEN pipeline Returns: [B x N x embed_dim] tensor of positional embeddings Nrd)NNNr:rrrrr9r7) rrrrrrrrGr$r%rrH)r/boxr num_boxesr*s r+#get_fourier_embeds_from_boundingboxrs IIbqMJ %,,y)I5 6C   " "#**CII " FC b! !C ++swwy#'')," 5C ++aAq! $ , ,ZIPQMTUDU VC Jr-c@^\rSrSrSU4SjjrSSjrSrU=r$)GLIGENTextBoundingboxProjectionic l>[TU]5 XlX lX@lUS-S-Ul[ U[5(aUSnUS:Xa[R"[R"URUR -S5[R"5[R"SS5[R"5[R"SU55Ul [RR[R"UR/55UlGOUS:XGa[R"[R"URUR -S5[R"5[R"SS5[R"5[R"SU55Ul[R"[R"URUR -S5[R"5[R"SS5[R"5[R"SU55Ul[RR[R"UR/55Ul[RR[R"UR/55Ul[RR[R"UR /55Ulg)Nrr7r text-onlyz text-image)rr positive_lenrwfourier_embedder_dim position_dimrDtuplerrxrrlinearsrrrknull_positive_feature linears_text linears_imagenull_text_featurenull_image_featurenull_position_feature)rrrw feature_type fourier_freqsrs r+r(GLIGENTextBoundingboxProjection.__init__s ( $1!)A-1 gu % %ajG ; &== $++d.?.??E  #s#  #w' DL*/););EKKIZIZH[<\)]D & \ ) "  $++d.?.??E  #s#  #w' !D "$ $++d.?.??E  #s#  #w' "D &+XX%7%7 TEVEVDW8X%YD "&+hh&8&8dFWFWEX9Y&ZD #%*XX%7%7 TEVEVDW8X%Y"r-cURS5n[URU5nURR SSS5n X-SU- U --nUbQUR R SSS5n X2-SU- U --nUR [R"X8/SS95n U $URS5nURS5nURR SSS5n URR SSS5n Xd-SU- U --nXu-SU- U --nUR[R"Xh/SS95nUR[R"Xx/SS95n[R"X/SS9n U $)Nrrr) rrrrrrrrr#rrrr)rboxesmaskspositive_embeddings phrases_masks image_masksphrases_embeddingsimage_embeddingsxyxy_embedding xyxy_null positive_nullobjs text_null image_null objs_text objs_images r+r'GLIGENTextBoundingboxProjection.forward8s#=T=V=VX]^..33Aq"= (/1u9 2II  * 66;;Aq"EM#6"=Um@["[ << +>*OUW XYD& *33B7M%//3K..33Aq"=I0055aB?J"4!Cq=GX\eFe!e /=[T^@^^ ))%))5G4X^`*abI++EII7G6X^`,abJ99i4!^\rSrSrSrSS\4U4SjjjrSrSrU=r $))PixArtAlphaCombinedTimestepSizeEmbeddingsikz For PixArt-Alpha. Reference: https://github.com/PixArt-alpha/PixArt-alpha/blob/0f55e922376d8b797edd44d25d0e7464b260dcab/diffusion/model/nets/PixArtMS.py#L164C9-L168C29 use_additional_conditionsc>[TU]5 X l[SSSS9Ul[ SUS9UlX0lU(a/[SSSS9Ul[ SUS9Ul [ SUS9Ul gg)NrTrrr) rroutdimrrrtrradditional_condition_projresolution_embedderaspect_ratio_embedder)rr size_emb_dimrrs r+r2PixArtAlphaCombinedTimestepSizeEmbeddings.__init__ssu " "T`ab!2sS`!a)B& $-6CY]tu-vD *'8SYe'fD $):s[g)hD & %r-cURU5nURURUS95nUR(aUR UR 55RU5nUR U5RUS5nUR UR 55RU5n URU 5RUS5n U[R"X/SS9-n U $Un U $r}) rrrrrrrrHrrr#) rr resolution aspect_ratiorr r r resolution_embaspect_ratio_embr s r+r1PixArtAlphaCombinedTimestepSizeEmbeddings.forwards1..~/@/@|/@/TU  ) )!;;JR>R>TUXXYef #99:JKSST^`bc (599n5W]^+__L)Lr-)rrrrrrrF) rrrrrrrrrrrs@r+rrks& it i i  r-rc6^\rSrSrSrSU4SjjrSrSrU=r$)riz Projects caption embeddings. Also handles dropout for classifier-free guidance. Adapted from https://github.com/PixArt-alpha/PixArt-alpha/blob/master/diffusion/model/nets/PixArt_blocks.py c^>[TU]5 UcUn[R"XSS9UlUS:Xa[R "SS9UlOEUS:Xa[R"5UlO$US:Xa[5UlO[SU35e[R"X#SS9Ul g) NTr gelu_tanhtanh) approximaterrdzUnknown activation function: ) rrrrr{GELUact_1rrrCr~)rrr rrvrs r+r"PixArtAlphaTextProjection.__init__s   &L kZ^_ [ V4DJ v DJ { "!DJ[TU]5 SSKJn [R "U5Ul[R "U5Ul[UUUSS9Ul [R"[R "U5U"XSUSS95Ul g)NrrF) query_dimr-r.out_biasrrrr) rrrrrrln0ln1r attnrxr)rr,r-r.r/rrs r+r*IPAdapterPlusImageProjectionBlock.__init__sx *<< +<< +    -- LL $  f9[` a r-cURU5nURU5n[R"XB/SS9nUR X$5U-nUR U5U-nU$)Nrr)r4r5rr#r6r)rrlatentsresidualrs r+r)IPAdapterPlusImageProjectionBlock.forwards_ $ ((7# % +@*JPR S))G;hF'''"W,r-)r6rr4r5)rrrr7) rrrrrEr"rrrrrs@r+r+r+sU           .r-r+c^\rSrSrSrSS\S\S\S\S\S\S \S \S S 4U4S jjjrS\RS \R4Sjr Sr U=r $)IPAdapterPlusImageProjectioniaResampler of IP-Adapter Plus. Args: embed_dims (int): The feature dimension. Defaults to 768. output_dims (int): The number of output channels, that is the same number of the channels in the `unet.config.cross_attention_dim`. Defaults to 1024. hidden_dims (int): The number of hidden channels. Defaults to 1280. depth (int): The number of blocks. Defaults to 8. dim_head (int): The number of head channels. Defaults to 64. heads (int): Parallel attention heads. Defaults to 16. num_queries (int): The number of queries. Defaults to 8. ffn_ratio (float): The expansion ratio of feedforward network hidden layer channels. Defaults to 4. r, output_dims hidden_dimsdepthr-r. num_queriesr/rNc >[T U]5 [R"[R "SXs5US-- 5Ul[R"X5Ul[R"X25Ul [R"U5Ul [R"[U5V s/sHn [X5Xh5PM sn 5Ulgs sn fr@)rrrrrrr9rproj_inproj_outrnorm_out ModuleListranger+layers) rr,r>r?r@r-r.rAr/rrs r+r%IPAdapterPlusImageProjection.__init__s ||EKK;$L{\_O_$_` yy9  +;  [1 mmafglam nam\] .{e Wam n  ns4CrcURRURS5SS5nURU5nURHnUnU"XU5nM UR U5nUR U5$)zcForward pass. Args: x (torch.Tensor): Input Tensor. Returns: torch.Tensor: Output Tensor. rr)r9r`rrCrHrDrE)rrr9blockr:s r+r$IPAdapterPlusImageProjection.forwardso,,%%affQiA6 LLO[[EHA1G!--(}}W%%r-)r9rHrErCrD)rrr7rrrr7 rrrrrrEr"rrrrrrrs@r+r=r=s "               .&&%,,&&r-r=c^\rSrSrSrSS\S\S\S\S\S\S \S \S \S \S \SS4U4SjjjrS\RS\R4Sjr Sr U=r $)"IPAdapterFaceIDPlusImageProjectioni acFacePerceiverResampler of IP-Adapter Plus. Args: embed_dims (int): The feature dimension. Defaults to 768. output_dims (int): The number of output channels, that is the same number of the channels in the `unet.config.cross_attention_dim`. Defaults to 1024. hidden_dims (int): The number of hidden channels. Defaults to 1280. depth (int): The number of blocks. Defaults to 8. dim_head (int): The number of head channels. Defaults to 64. heads (int): Parallel attention heads. Defaults to 16. num_tokens (int): Number of tokens num_queries (int): The number of queries. Defaults to 8. ffn_ratio (float): The expansion ratio of feedforward network hidden layer channels. Defaults to 4. ffproj_ratio (float): The expansion ratio of feedforward network hidden layer channels (for ID embeddings). Defaults to 4. r,r>r?id_embeddings_dimr@r-r.rrAr/ ffproj_ratiorNc >[TU]5 SSKJn XlXlSUlSUlSUlU "XAU-SU S9Ul [R"U5Ul [R"X15Ul[R"X5Ul[R"U5Ul[R""[%U5V s/sHn ['XXz5PM sn 5Ulgs sn f)NrrFr.r)rr)rrrrrr/ clip_embedsshortcutshortcut_scalerrrrrrCrDrErFrGr+rH)rr,r>r?rQr@r-r.rrAr/rRrrrs r+r+IPAdapterFaceIDPlusImageProjection.__init__ s *$# ! 1 3JZ`gst LL, yy9  *:  [1 mm`efk`l m`l[\ .zU V`l m  msC3 id_embedsc:URURR5nURU5nUR SUR UR 5nURU5nUnURUR5nUR SURSURS5nURHnUnU"XBU5nM URU5nURU5nUR(aXRU--nU$)zwForward pass. Args: id_embeds (torch.Tensor): Input Tensor (ID embeds). Returns: torch.Tensor: Output Tensor. rrr9)rrTrrrHrr/rrCrrHrDrErUrV)rrXr9rTrrKr:rzs r+r*IPAdapterFaceIDPlusImageProjection.forward= sLL!1!1!7!78 IIi( %%b$//4>>J IIi( ll4#3#34   K$5$5a$8+:K:KA:N O[[EHA1G!--(mmG$ ==11C77C r-) rTr/rHrrErrrCrDrUrV) rrrMrr7rrr7rr7rrNrs@r+rPrP s$!$! ! !  !  !  ! ! ! ! ! ! !  ! ! F%,,r-rPc ^\rSrSrSrSS\S\S\S\SS4 U4S jjjrS \RS \RS \RS\R4S jr Sr U=r $)!IPAdapterTimeImageProjectionBlockiY aBlock for IPAdapterTimeImageProjection. Args: hidden_dim (`int`, defaults to 1280): The number of hidden channels. dim_head (`int`, defaults to 64): The number of head channels. heads (`int`, defaults to 20): Parallel attention heads. ffn_ratio (`int`, defaults to 4): The expansion ratio of feedforward network hidden layer channels. hidden_dimr-r.r/rNc t>[TU]5 SSKJn [R "U5Ul[R "U5Ul[UUUUSSS9Ul U"XSUSS9Ul [R"5Ul [R"USU-5Ul[R "U5UlS[ R""[ R""U55- URlURR'5 SURlSURlg)NrrF)r1rr-r.rr2rr3r7)rrrrrrr4r5r r6rr adaln_silur adaln_proj adaln_normrrrfuse_projectionsto_kto_v)rr]r-r.r/rrs r+r*IPAdapterTimeImageProjectionBlock.__init__g s *<< +<< + *   jFQZafg'')))JJ?,,z2dii ((;<<  ""$  r-rr9 timestep_embcURURU55nURSSS9upVpxUn URU5nUR U5SUSS2S4--USS2S4-nUR Sn UR RU5n [R"X4SS9n UR RU 5RSSS9upU R SnXR R-nU RU SUR RU5RSS5n U RU SUR RU5RSS5n URU SUR RU5RSS5nXR R-XR R-RSS5-n[R"UR!5SS9R#UR$5nUU-nURSS5R'U SUR RU-5nUR R(S"U5nUR R(S"U5nX)-nUn UR+U5SUSS2S4--USS2S4-nUR-U5U -$) zForward pass. Args: x (`torch.Tensor`): Image features. latents (`torch.Tensor`): Latent features. timestep_emb (`torch.Tensor`): Timestep embedding. Returns: `torch.Tensor`: Output latent features. r7rrNrrrr)r`r_rmr4r5rr6rrr#rr.rrrrr"rPrrHrrar)rrr9rfr* shift_msa scale_msa shift_mlp scale_mlpr:rrJkv_inputrKrL inner_dimrrs r+r)IPAdapterTimeImageProjectionBlock.forward sy oodool;<58YYqaY5H2 i HHQK((7#q9QW+='=>1d7ASS]]1%  w'99a\r2YY__X.44QB4? IIbM  / :r499??HEOOPQSTUhhz2tyyAKKAqQ :r499??HEOOPQSTU))//)cIIOO.C-N-NrSU-VVv||~26;;FLLI5.##Aq)11*b$))//T\B\]))""1%g.))""1%g.$//'*a)AtG2D.DE RSUYRYHZZwww(**r-)rar`r_r6rr4r5)rMrr7) rrrrrrErrrrrrrs@r+r\r\Y s       B1+1+ 1+ELL1+]b]i]i1+1+r-r\c^\rSrSrSrSS\S\S\S\S\S\S \S \S \S \S \SS4U4SjjjrS\RS\RS\ \R\R44Sjr Sr U=r $)IPAdapterTimeImageProjectioni a*Resampler of SD3 IP-Adapter with timestep embedding. Args: embed_dim (`int`, defaults to 1152): The feature dimension. output_dim (`int`, defaults to 2432): The number of output channels. hidden_dim (`int`, defaults to 1280): The number of hidden channels. depth (`int`, defaults to 4): The number of blocks. dim_head (`int`, defaults to 64): The number of head channels. heads (`int`, defaults to 20): Parallel attention heads. num_queries (`int`, defaults to 64): The number of queries. ffn_ratio (`int`, defaults to 4): The expansion ratio of feedforward network hidden layer channels. timestep_in_dim (`int`, defaults to 320): The number of input channels for timestep embedding. timestep_flip_sin_to_cos (`bool`, defaults to True): Flip the timestep embedding order to `cos, sin` (if True) or `sin, cos` (if False). timestep_freq_shift (`int`, defaults to 0): Controls the timestep delta between frequencies between dimensions. r/r>r]r@r-r.rAr/timestep_in_dimtimestep_flip_sin_to_costimestep_freq_shiftrNc >[T U]5 [R"[R "SXs5US-- 5Ul[R"X5Ul[R"X25Ul [R"U5Ul [R"[U5V s/sHn [X5Xh5PM sn 5Ul[!XU 5Ul[%XSS9Ulgs sn f)Nrrrr)rrrrrrr9rrCrDrrErFrGr\rHrrrttime_embedding)rr/r>r]r@r-r.rAr/rrrsrtrrs r+r%IPAdapterTimeImageProjection.__init__ s ||EKK;$KjZ]o$]^ yy7  *9  Z0 mm`efk`l m`l[\ .zU V`l m  #?Nab/TZ[ ns4C7rrcURU5RURS9nURU5nURR UR S5SS5nURU5nXSS2S4-nURH nU"XU5nM URU5nURU5nXC4$)zForward pass. Args: x (`torch.Tensor`): Image features. timestep (`torch.Tensor`): Timestep in denoising process. Returns: `tuple`[`torch.Tensor`, `torch.Tensor`]: The pair (latents, timestep_emb). r\rrN) rrrrvr9r`rrCrHrDrE)rrrrfr9rKs r+r$IPAdapterTimeImageProjection.forward s~~h/222A **<8 ,,%%affQiA6 LLO QW% %[[EA 5G!--(--($$r-)r9rHrErCrDrvr) ii rMr7rrorr7i@Tr)rrrrrrErrrrrrrrrs@r+rqrq s:")-#$\\\ \  \  \\\\\#'\!\ \\2%%%% V[VbVbHbBc%%r-rqc^\rSrSrS\\R \\R -4U4Sjjr\ S\ 4Sj5r S\\ R4SjrSrU=r$) MultiIPAdapterImageProjectioni IPAdapterImageProjectionLayerscX>[TU]5 [R"U5Ulgr)rrrrFimage_projection_layers)rr|rs r+r&MultiIPAdapterImageProjection.__init__ s  ')}}5S'T$r-rc,[UR5$)zNumber of IP-Adapters loaded.)rr~)rs r+num_ip_adapters-MultiIPAdapterImageProjection.num_ip_adapters s4//00r-rcF/n[U[5(d Sn[SSUSS9 URS5/n[ U5[ UR 5:wa-[ S[ U5S[ UR 535e[XR 5HupEURS URSpvURXg-4URS S-5nU"U5nURXg4URSS-5nURU5 M U$) NzYou have passed a tensor as `image_embeds`.This is deprecated and will be removed in a future release. Please make sure to update your script to pass `image_embeds` as a list of tensors to suppress this warning.zimage_embeds not a list1.0.0FrZrzGimage_embeds must have the same length as image_projection_layers, got z and rr) rDlistrrrr~rCziprrHappend)rrprojected_image_embedsrb image_embedimage_projection_layerr num_imagess r+r%MultiIPAdapterImageProjection.forward sP!# ,--@  /:M]b c(22156L | D$@$@ A AYZ]^jZkYllqruvzwSwSsTrUV 47|EaEa3b /K%0%6%6q%9;;L;LQ;O %--z/F.H;K\K\]^]_K`.`aK0=K%--z.FIZIZ[\[]I^.^_K " ) )+ 6 4c&%r-)r~)rrrrrrModulerrpropertyrErrrrrrrs@r+r{r{ s\UtBIIryyIY7YU111&D$6&&r-r{c\rSrSrSrSrg) FluxPosEmbedi6 c<Sn[SSU5 SSKJn U"U0UD6$)NzImporting and using `FluxPosEmbed` from `diffusers.models.embeddings` is deprecated. Please import it from `diffusers.models.transformers.transformer_flux`.rrr)r)rtransformers.transformer_fluxr)clsargskwargsrbrs r+__new__FluxPosEmbed.__new__7 s+}.'+>??T,V,,r-rN)rrrrrrrr-r+rr6 s-r-r)Frrru)r.r.Nr6)r.r.)Frr.rNr6)r6)r6FN)Frr.r)ruTrNN))r.r.r.ruN)TNr6)Tr)Trr)Trnumpyr6rtorch.nn.functionalrr&reutilsrutils.torch_utilsr activationsrr attention_processorr rrErr"r,rrrrVrNrBrlrIrJrjrorvrrrrrr6r<rAr?r@rJr r(rbrrrtrrrrrrrrrrrrr*r;rarurrrrrrrrrrrr+r=rPr\rqr{rrr-r+rs9  =1* ""# 3||333 3  3  3 \\3t*-*-"&GGc3h'GG"' G #( G LL4  GG \\G\*-*- ??c3h'??"' ? #( ? ZZ ?J"&A LL4 AAHB.d_a#L,8}5}5@3 ryy3 lC"))CL5CRYY5Cz'+"&b  b  bbCHo$b LL4 b \\E%,, 455bT7F"&(= ueU23 (=  (= LL4 (= \\E%,, 455(=Xos/?D||d?R/hk/d8DH  @ @ c @ @L! 6  ||6 ||eELL116 6  6  6  5<< %& 6 r   $- -` & >BII6> >B!RYY!HB"))B6bii. 0299 0 RYY  bii&&0RYYB%bii%P:BII:zRYYD *BII *F   4RYY4$ ! !'RYY'>/ryy/dOOd*ZbiiZz" "J : B8&2998&vMM``+ `+HN%299N%b%&BII%&P-299-r-