oMjZSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKJr SSKJ r SSK J r J r SSK rSSKrSSKJs Jr \R(R+5(aSSKJrJr SSKJr SSKJr OSrSrSrSrSS KJr SS KJ r SS K!J"r" SS K#J$r$J%r%J&r&J'r'J(r(J)r)J*r*J+r+ \+"5(a1SSK,r,\,RZR\R_5(aSSK.r.\'"5(aSS K0J1r1 \("5(aSSK2J3r3 \*"5(aSSK4J5r5 \)"5(aSSK6r6S\74Sjr8Sr9SSSSS.S\RtS\RtS\;S\;S\RtS-S\7S\<\Rt\Rt\Rt44Sjjr=S\>4S jr?SOS!\S"\"S#\RtS$\RtS%\RtS&\RtS'\RtS(\;S\<\Rt\Rt44S)jjr@S!\S\A\>\Rt44S*jrB\R4S+\R"R\E\R"R-4S,jjrFS-\A\>\Rt4S.\>S/\R"R4S0jrGS1\A\>\R"R4S\A\>\ 44S2jrHSPS3\>S4\7S5\;S6\;S7\;S8\R\>-S9\RS-4S:jjrKSQS3\>4S;jjrLS<rM\S=S>.S?\R"R\ -S8\>\R-S@\N4SAjj5rOSBrPSCrQ/SDQrRSRSEjrSS\A4SFjrTS\A4SGjrUSSS+\R"RS8\>\R-S@\NSH\NSI\NSJ\A\>\ 4S-SK\V\W\R"RS-S\4SLjjrX"SMSN5rYg)TN)contextmanager)partial)AnyIterable) CPUOffloadShardingStrategy)FullyShardedDataParallel)transformer_auto_wrap_policy)UNet2DConditionModel)DiffusionPipeline)SchedulerMixin)convert_state_dict_to_diffusersconvert_state_dict_to_peft deprecateis_accelerate_availableis_peft_availableis_torch_npu_availableis_torchvision_availableis_transformers_available) get_logger)set_peft_model_state_dict) transformsseedc6[R"U5 [RRU5 [R"U5 [ 5(a [R RU5 g[RRU5 g)z Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch`. Args: seed (`int`): The seed to set. Returns: `None` N) randomrnptorch manual_seedrnpumanual_seed_allcuda)rs ڦC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\diffusers/training_utils.pyset_seedr$=s[ KKIINN4 d !!$' ""4(cURnUS-nSU- S-nURURS9UR5n[ UR 5[ UR 5:a3USn[ UR 5[ UR 5:aM3UR UR 5nURURS9UR5n[ UR 5[ UR 5:a3USn[ UR 5[ UR 5:aM3UR UR 5nXV- S-nU$)a_ Computes SNR as per https://github.com/TiankaiHang/Min-SNR-Diffusion-Training/blob/521b624bd70c67cee4bdf49225915f5945a872e3/guided_diffusion/gaussian_diffusion.py#L847-L849 for the given timesteps using the provided noise scheduler. Args: noise_scheduler (`NoiseScheduler`): An object containing the noise schedule parameters, specifically `alphas_cumprod`, which is used to compute the SNR values. timesteps (`torch.Tensor`): A tensor of timesteps for which the SNR is computed. Returns: `torch.Tensor`: A tensor containing the computed SNR values for each timestep. ??device).N)alphas_cumprodtor*floatlenshapeexpand)noise_scheduler timestepsr,sqrt_alphas_cumprodsqrt_one_minus_alphas_cumprodalphasigmasnrs r# compute_snrr9QsF %33N(#-%(>%9c$A!.00 8H8H0I)TZZ\ !'' (3y+? ?1)< !'' (3y+? ?  & &y 7E$A$D$DIL\L\$D$]^g$h$n$n$p! +11 2S5I I(Ei(P% +11 2S5I I ) 0 0 AE =Q C Jr%r(i) lambda_conf temperatureper_token_weights ignore_indexlogitslabelsr;r<r=r>returnchURS:a"[S[UR5S35eURURSS:wa8[S[UR5S[UR5S35eUS::a[S US35eUR U5nUc[ R "XRS 9nOkURUR:wa8[S [UR5S [UR5S35eURURS 9nURSn[R"URSU5URS5S US9RU5n XvRUR5-R5RS5n X-URU R5-R5U - n US:Xa.[ R"SUR U RS9n XU 4$[ R""5 UR%SS9n XmR'U5-nSSS5 UR)5nUS:waU[)U5- n[ R*"USS9n[ R,"UR5R.n[ R0"URU55nUU-RSS9RURS 9*nUWRUR5-R5RS5nUU-URUR5-R5U- n U [)U5U --nUX4$!,(df  GNC=f)a Computes a confidence-aware training loss for token classification-style heads. This loss combines: - `loss_sft`: standard supervised cross-entropy on all non-ignored labels. - `loss_conf`: an entropy penalty applied only on tokens that are already predicted correctly. Args: logits (`torch.Tensor`): Logits of shape `(..., vocab_size)`. labels (`torch.Tensor`): Labels of shape `(...)`, matching `logits.shape[:-1]`. Values set to `ignore_index` are excluded from both losses. lambda_conf (`float`, *optional*, defaults to `0.0`): Weight for the confidence term. temperature (`float`, *optional*, defaults to `1.0`): Temperature used for the entropy term only. Lower values sharpen the distribution and change the strength of the confidence gradients. per_token_weights (`torch.Tensor`, *optional*): Optional weights of shape `(...)` to reweight both losses per token (e.g. schedule-aware weights). Tokens with weight `0` contribute nothing. ignore_index (`int`, *optional*, defaults to `-100`): Ignore index for labels. Returns: `Tuple[torch.Tensor, torch.Tensor, torch.Tensor]`: `(loss, loss_sft, loss_conf)`. r+z.`logits` must have at least 2 dims, got shape .Nz:`labels` shape must match `logits.shape[:-1]`, got labels=z logits=rz`temperature` must be > 0, got )dtypez9`per_token_weights` shape must match `labels` shape, got z != none) reductionr>r r:)r*rE)dimr()ndim ValueErrortupler0ner ones_likerEr-F cross_entropyreshape reshape_assum clamp_minzerosr*no_gradargmaxeqr.softmaxfinfotinylog)r?r@r;r<r=r>validweights vocab_size per_token_nll denom_sftloss_sft loss_confpredcorrect scaled_logitsprobseps log_probsentropy denom_conflosss r#compute_confidence_aware_lossrmvsQ<{{QI%PVP\P\J]I^^_`aa ||v||CR((Hv||I\H]]efklrlxlxfyezz{ |  a:;-qIJJ IIl #E //& =  " "fll 2KERcRiRiLjKkkopuv|wCwCqDpEEFG $&&V\\&:b!JOOr:&r!  j 88GMM22779CCAFI'%((=3F3F*GGLLNQZZHcKK6==O 9,, }}}$''&/) LLNMc%k(:: MM-R 0E ++ekk " ' 'C %//#./I !&&2&.11 1EEGGJJw}}55::<FFqIJ7"WZZ %>>CCE RI eK(94 4D  $$# s #N"" N1interpolation_typec0[5(d [S5eUS:Xa[RRnU$US:Xa[RR nU$US:Xa[RR nU$US:Xa[RRnU$US:Xa[RRnU$US:Xa[RRnU$US:Xa[RRnU$[S US 35e) a Maps a string describing an interpolation function to the corresponding torchvision `InterpolationMode` enum. The full list of supported enums is documented at https://pytorch.org/vision/0.9/transforms.html#torchvision.transforms.functional.InterpolationMode. Args: interpolation_type (`str`): A string describing an interpolation method. Currently, `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos` are supported, corresponding to the supported interpolation modes in torchvision. Returns: `torchvision.transforms.InterpolationMode`: an `InterpolationMode` enum used by torchvision's `resize` transform. zhPlease make sure to install `torchvision` to be able to use the `resolve_interpolation_mode()` function.bilinearbicubicboxnearest nearest_exacthamminglanczoszThe given interpolation mode z is not supported. Currently supported interpolation modes are `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos`.) r ImportErrorrInterpolationModeBILINEARBICUBICBOXNEAREST NEAREST_EXACTHAMMINGLANCZOSrK)rninterpolation_modes r#resolve_interpolation_moders> $ % % v  Z''99BB& % y ('99AA" ! u $'99==  y ('99AA   .'99GG  y ('99AA  y ('99AA  +,>+?@m n  r%unetr2r3noise noisy_latentstargetencoder_hidden_statesdream_detail_preservationc\URRUR5USSS4nSU- S-n X-n Sn [R"5 U"XBU5R n SSS5 SupUR RS:XaLU nX>- R5nURU 5 URX-5n URU5n X4$UR RS:Xa [S5e[SUR R35e!,(df  N=f) a Implements "DREAM (Diffusion Rectification and Estimation-Adaptive Models)" from https://huggingface.co/papers/2312.00210. DREAM helps align training with sampling to help training be more efficient and accurate at the cost of an extra forward step without gradients. Args: `unet`: The state unet to use to make a prediction. `noise_scheduler`: The noise scheduler used to add noise for the given timestep. `timesteps`: The timesteps for the noise_scheduler to user. `noise`: A tensor of noise in the shape of noisy_latents. `noisy_latents`: Previously noise latents from the training loop. `target`: The ground-truth tensor to predict after eps is removed. `encoder_hidden_states`: Text embeddings from the text model. `dream_detail_preservation`: A float value that indicates detail preservation level. See reference. Returns: `tuple[torch.Tensor, torch.Tensor]`: Adjusted noisy_latents and target. Nr(r')NNepsilon v_predictionz/DREAM has not been implemented for v-predictionzUnknown prediction type ) r,r-r*rrVsampleconfigprediction_typedetachmul_addNotImplementedErrorrK)rr2r3rrrrrr,r5 dream_lambdard_noisy_latents_targetpredicted_noise delta_noises r# compute_dream_and_update_latentsrs*:%3366y7G7GHTXZ^`dIdeN%(>%9c$A!1KL D M.CDKK +N--:.668 &&**+H+VW**[)  ""    / /> A!"STT3O4J4J4Z4Z3[\]] s D D+c0nUR5H[up#[US5(dM[US5nUcM)UR5nUR 5HupgXqUSU3'M M] U$)z@ Returns: A state dict containing just the LoRA parameters. set_lora_layer lora_layerz.lora.) named_moduleshasattrgetattr state_dictitems)rlora_state_dictnamemodulercurrent_lora_layer_sdlora_layer_matrix_name lora_params r#unet_lora_state_dictr.s O**,  6+ , , 6J%(2(=(=(?%:O:U:U:W6*OYtfF3I2J$KL;X - r%modelc[U[5(dU/nUHCnUR5H,nUR(dMUR U5UlM. ME g)z Casts the training parameters of the model to the specified data type. Args: model: The PyTorch model whose parameters will be cast. dtype: The data type to which the model parameters will be cast. N) isinstancelist parameters requires_gradr-data)rrEmparams r#cast_training_paramsrAsM eT " " \\^E""""XXe_ $r%rprefix text_encodercUR5VVs0sH1up4URU5(dMURUS5U_M3 nnn[[ U55n[ X%SS9 gs snnf)a, Sets the `lora_state_dict` into `text_encoder` coming from `transformers`. Args: lora_state_dict: The state dictionary to be set. prefix: String identifier to retrieve the portion of the state dict that belongs to `text_encoder`. text_encoder: Where the `lora_state_dict` is to be set. default) adapter_nameN)r startswithreplacerrr)rrrkvtext_encoder_state_dicts r#!_set_state_dict_into_text_encoderrRss3B2G2G2I2I$!Q\\Z`Ma%199VR !A%2I99XYp9qrlR[\ s A.A.modules_to_savec0nUR5H,up#UcM URSR5XS3'M. U$)Nr_lora_adapter_metadata)r peft_configto_dict)r metadatas module_namers r#_collate_lora_metadataresPI.446  @F@R@RS\@]@e@e@gI %;< = 7 r%weighting_scheme batch_size logit_mean logit_std mode_scaler* generatorcvUS:XaB[R"X#U4XVS9n[RRR U5nU$US:XaT[R "U4XVS9nSU- U[R "[RU-S- 5S-S- U--- nU$[R "U4XVS9nU$)z Compute the density for sampling the timesteps when doing SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://huggingface.co/papers/2403.03206v1. logit_normal)meanstdsizer*rmode)rr*rr r+) rnormalnn functionalsigmoidrandcosmathpi)rrrrrr*rus r#%compute_density_for_timestep_samplingrms >) LLjzmTZ p HH   ' ' * H V # JJZM& N EJ%))DGGaK!O"<"AA"E"IJ J H JJZM& N Hr%cUS:XaUS-R5nU$US:Xa)SSU-- SUS---nS[RU-- nU$[R"U5nU$)z Computes loss weighting scheme for SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://huggingface.co/papers/2403.03206v1. sigma_sqrtgcosmapr r+)r.rrrrN)rsigmas weightingbots r#compute_loss_weighting_for_sd3rsz<'T\((*  X %!f*nq619},3'  OOF+ r%cV[R"5 [RR 5(a[RR 5 g[R RR 5(a[RR 5 g[5(a[RR 5 g[[S5(aC[RR 5(a[RR 5 ggg)zN Runs garbage collection. Then clears the cache of the available accelerator. xpuN) gccollectrr" is_available empty_cachebackendsmpsr torch_npur rrrHr%r# free_memoryrsJJL zz       ( ( * *   ! ! !!#   599#9#9#;#; $< r%T)offloadmodulesrc' # U(a[SU55(+nU(a4UVs/sH&n[UR55RPM( nnO![ U5S:XdeUSR/nUHnUR U5 M Sv U(a'[ UW5HupFUR U5 M ggs snf!U(a'[ UW5HupFUR U5 M ff=f7f)a Context manager that, if offload=True, moves each module to `device` on enter, then moves it back to its original device on exit. Args: device (`str` or `torch.Device`): Device to move the `modules` to. offload (`bool`): Flag to enable offloading. c3B# UHn[U[5v M g7fN)rr ).0rs r# !offload_models..sMW:a):;;Wsr rN)anynextrr*r/r-zip)r*rris_modelroriginal_devicesorig_devs r#offload_modelsrsMWMMM ELMWQ\\^ 4 ; ;W M w<1$ $$ ' 1 12 A DDL "7,<= X >  N "7,<= X > s(*D-C >DC4D0DDc^U(d [S5eUR5RS5n/nUHn[R"SU5nU(d[SUS35e[ UR S55n[ UR S55nUS::dUS::a [S 5eUS -S:wd US -S:wa[R"S US US 35 URXV45 M U(d [S5eU$![an[SUSU35UeSnAff=f)zGParses a string defining buckets into a list of (height, width) tuples.zBucket string cannot be empty.;z^\s*(\d+)\s*,\s*(\d+)\s*$zInvalid bucket format: 'z'. Expected 'height,width'.r r+rz,Bucket dimensions must be positive integers.zBucket dimension (,z.) not divisible by 8. This might cause issues.z Invalid integer in bucket pair 'z': Nz.No valid buckets found in the provided string.) rKstripsplitrematchintgroupwarningswarnappend) buckets_str bucket_pairsparsed_bucketspair_strrheightwidthes r#parse_buckets_stringrs/ 9::$$&,,S1LN 5x@7zA\]^ ^ YQ(F A'E{eqj !OPPzQ%!)q. 26(!E7Bpqr  ! !6/ 2! IJJ   Y?zQCPQWX X Ys'B D  D,D''D,c[S5nSn[U5H%unupg[X-X-- 5nX::dM!UnUnM' U$)z6Finds the closes bucket to the given height and width.infN)r. enumerateabs) hwbucket_options min_metricbest_bucket_idx bucket_idxbucket_hbucket_wmetrics r#find_nearest_bucketrsSuJO,5n,E( (XQ\AL01  J(O -F r%) )r r ))rr)r+)r r+)r r)rr ) )r"r!)r r+)r+r cjX-nUbUVVs/sHupEXT4PM nnnO[n/n[5nUHuupX- n X;- S-nXK-n[U[XA- 5U-5n[U[XQ- 5U-5nXE4U;dMQUR XE45 UR XE45 Mw U$s snnf)aGenerate a discrete list of ``(height, width)`` buckets for a target pixel budget. Buckets are sized so that ``height * width`` is close to ``resolution ** 2`` while spanning a range of aspect ratios, with each dimension rounded to a multiple of ``divisibility``. When ``base_resolutions`` (a list of ``(height, width)`` pairs, e.g. a model's preferred resolutions) is provided, its aspect ratios are used instead of the default set and rescaled to the target pixel budget. Args: resolution (`int`): Target resolution; the pixel budget is ``resolution ** 2``. divisibility (`int`, *optional*, defaults to 16): Each bucket dimension is rounded to a multiple of this value (16 satisfies the patch-size constraints of the Flux/Z-Image VAEs). base_resolutions (`list[tuple[int, int]]`, *optional*): ``(height, width)`` pairs whose aspect ratios seed the ladder instead of the default aspect-ratio set. Returns: `list[tuple[int, int]]`: The unique ``(height, width)`` buckets. r')_DEFAULT_BUCKET_ASPECT_RATIOSsetmaxroundrr) resolution divisibilitybase_resolutions target_pixelsrr aspect_ratiosbucketsseenrwrhaspects r#generate_aspect_ratio_bucketsr2s$+M#.>?.>FQ!.> ? 5 G 5D  # + J  eA$45 D E  eA$45 D E 6  HHaV  NNA6 "  N!@sB/c UR5VVs0sHSupU[U[R5(a,UR 5R 5R 5OU_MU snn$s snnfr)rrrTensorrcpu contiguous) state_dictsrrs r#_to_cpu_contiguousr8)sW_j_p_p_r s_rW[WXA 1ell0K0Kqxxz~~**,QR R_r ss ssAA2c0n[URSS5nUc [S5eURRnUc[R US'U$UR =(d [R US'U$)zL Extract and convert FSDP config from Accelerator into PyTorch FSDP kwargs. fsdp_pluginNzLAccelerate isn't configured to handle FSDP. Please update your installation.sharding_strategy)rstaterKr:r FULL_SHARDr;) acceleratorkwargs fsdp_stater:s r# get_fsdp_kwargs_from_acceleratorrA-s F**M4@Jghh##//K&6&A&A"# M'2&C&C&bGWGbGb"# Mr%use_orig_paramslimit_all_gathers fsdp_kwargstransformer_layer_clscT[[5nUcJ[URRR S5nUR SUR35 [[U1S9nUU(a [US9OSUUUS.n U(aU RU5 [U40U D6n U $)u) Wrap a model with FSDP using common defaults and optional transformer auto-wrapping. Args: model: Model to wrap device: Target device (e.g., accelerator.device) offload: Whether to enable CPU parameter offloading use_orig_params: Whether to use original parameters limit_all_gathers: Whether to limit all gathers fsdp_kwargs: FSDP arguments (sharding_strategy, etc.) — usually from Accelerate config transformer_layer_cls: Classes for auto-wrapping (if not using policy from fsdp_kwargs) Returns: FSDP-wrapped model Nrz8transformer_layer_cls is not provided, auto-inferred as )rE)offload_params) device_id cpu_offloadrBrCauto_wrap_policy) r__name__typerlanguage_modellayersinforr rupdateFSDP) rr*rrBrCrDrEloggerrJr fsdp_models r#wrap_with_fsdprTDs2 !F$ $U[[%?%?%F%Fq%I J NOdOmOmNnop;TiSjk=Dz9$*., F k"e&v&J r%c<\rSrSrSrSS\\RRS\ S\ S\ S\ S \ \ -S \ \ -S \ S \ S-S \ \\ 4S-4Sjjr\SS Sjj5rSrS\ S\ 4Sjr\R("5S\\RR4Sj5rS\\RRSS4SjrS!SjrS"S!SjjrS\ 4SjrS\\RRSS4SjrS\\RRSS4SjrS\ SS4SjrSrg)#EMAModeliwz. Exponential Moving Average of models weights Nrdecay min_decayupdate_after_stepuse_ema_warmup inv_gammapowerforeach model_cls model_configc  [U[RR5(a Sn [ SSU SS9 UR 5nSnU R SS5bS n [ SSU SS9 U SnU R S S5bS n [ S SU SS9 U S n[U5nUV s/sH oR5R5PM" sn Ul U R S S5b S n [ S SU SS9 URU S S9 SUl X l X0lX@lXPlX`lXplSUlSUlXlXlXlgs sn f)a Args: parameters (Iterable[torch.nn.Parameter]): The parameters to track. decay (float): The decay factor for the exponential moving average. min_decay (float): The minimum decay factor for the exponential moving average. update_after_step (int): The number of steps to wait before starting to update the EMA weights. use_ema_warmup (bool): Whether to use EMA warmup. inv_gamma (float): Inverse multiplicative factor of EMA warmup. Default: 1. Only used if `use_ema_warmup` is True. power (float): Exponential factor of EMA warmup. Default: 2/3. Only used if `use_ema_warmup` is True. foreach (bool): Use torch._foreach functions for updating shadow parameters. Should be faster. device (str | torch.device | None): The device to store the EMA weights on. If None, the EMA weights will be stored on CPU. @crowsonkb's notes on EMA Warmup: If gamma=1 and power=1, implements a simple average. gamma=1, power=2/3 are good values for models you plan to train for a million or more steps (reaches decay factor 0.999 at 31.6K steps, 0.9999 at 1M steps), gamma=1, power=3/4 for models you plan to train for less (reaches decay factor 0.999 at 10K steps, 0.9999 at 215.4k steps). zzPassing a `torch.nn.Module` to `ExponentialMovingAverage` is deprecated. Please pass the parameters of the module instead.z9passing a `torch.nn.Module` to `ExponentialMovingAverage`1.0.0F standard_warnT max_valueNzCThe `max_value` argument is deprecated. Please use `decay` instead. min_valuezGThe `min_value` argument is deprecated. Please use `min_decay` instead.r*z=The `device` argument is deprecated. Please use `to` instead.r)r)rrrModulerrgetrcloner shadow_paramsr-temp_stored_paramsrWrXrYrZr[r\optimization_stepcur_decay_valuer]r^r_)selfrrWrXrYrZr[r\r]r^r_r?deprecation_messageps r#__init__EMAModel.__init__|sgF j%((// 2 2D  K##   $..0J"N ::k4 ( 4"g  k7,?u U;'E ::k4 ( 4"k  k7,?u U{+I*% :DE*Qggi..0*E ::h % 1"a  h)> !ij j    $lm m**4+<+<=__& -   .:. U%%'( d#r%rkc[SXR- S- 5nUS::agUR(a#SSX R- -UR*-- nO SU-SU-- n[ X0R 5n[X0R5nU$)z> Compute the decay factor for the exponential moving average. rr r: )r&rYrZr[r\minrWrX)rmrksteprls r# get_decayEMAModel.get_decays1'*@*@@1DE 19   1tnn'<#<$**"LLO 4xBI6Oozz:o~~>r%c T[U[RR5(aSn[ SSUSS9 UR 5n[ U5nU=RS- slURUR5nX0l SU- n[R"5nUR(Ga~[5(aK[RR R#5(a[ R$R'USS9nU UVs/sHofR((dMUPM nn[+UR,U5VVs/sHupUR((dMUPM n nn[/U5[/U5:au[R0"[+UR,U5VVs/sHupUR((aMUPM snnUVs/sHofR((aMUPM snSS 9 [R2"U [R4"X5US 9 SSS5 g[+UR,U5Hup[5(aK[RR R#5(a[ R$R'USS9nU UR((aUR7XHU- -5 OUR9U5 SSS5 M gs snfs snnfs snnfs snf!,(df  g=f!,(df  M=f) NzPassing a `torch.nn.Module` to `ExponentialMovingAverage.step` is deprecated. Please pass the parameters of the module instead.z>passing a `torch.nn.Module` to `ExponentialMovingAverage.step`raFrbr ) modifier_rankT) non_blocking)r6)rrrrfrrrrkrrl contextlib nullcontextr]r transformers integrations deepspeedis_deepspeed_zero3_enabledzeroGatheredParametersrrrir/_foreach_copy_ _foreach_sub_ _foreach_subsub_copy_) rmrrnrWone_minus_decaycontext_managerr params_grads_param s_params_grads r#r EMAModel.stepso j%((// 2 2D  P##   $..0J*%  !#t556$e)$002 <<<(**|/H/H/R/R/m/m/o/o"+.."C"CJ^b"C"c 2<T*@S@Su* T25d6H6H*2U!2UY^YlYlG2U!{#c*o5((7:4;M;Mz7Zv7Z^Wbgbubu7Zv,6RJ5>Q>QJR%) ##!5#5#5m#QYh!$#&d&8&8*"E,..<3L3L3V3V3q3q3s3s&/nn&G&G]a&G&bO$** _%%HI e, %_ #F#U! wR!,%_sgLK1.K14LK6*K60AL4K< K< LL 4L :0L*:L1L L L' c [U5nUR(a~[R"UVs/sHo"RPM sn[ UR U5VVs/sH*up2URUR5RPM, snn5 g[ UR U5HCup2URRURUR5R5 ME gs snfs snnf)a1 Copy current averaged parameters into given collection of parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored moving averages. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. N) rr]rrrrrir-r*r)rmrrrs r#rEMAModel.copy_to6s*% <<  )344EHI[I[]gEhiEh>7ELL)..Ehi  #&d&8&8*"E   ELL!9!>!>?#F 5is C9!1C> clURVs/sHoR5PM snUlgs snf)z Move internal buffers of the ExponentialMovingAverage to pinned memory. Useful for non-blocking transfers for offloading EMA params to the host. N)ri pin_memory)rmros r#rEMAModel.pin_memoryIs* 7;6H6HI6Hlln6HIIs1c URVs/sH7nUR5(aURXUS9OURXS9PM9 snUlgs snf)z Move internal buffers of the ExponentialMovingAverage to `device`. Args: device: like `device` argument to `torch.Tensor.to` )r*rEr)r*rN)riis_floating_pointr-)rmr*rErros r#r- EMAModel.toQse''  (""$$ DD,D GV? @(   s>Ac URURURURURUR UR URS.$)z Returns the state of the ExponentialMovingAverage as a dict. This method is used by accelerate during checkpointing to save the ema state dict. rWrXrkrYrZr[r\rir)rms r#rEMAModel.state_dict`sNZZ!%!7!7!%!7!7"11ZZ!//  r%cUVs/sH.o"R5R5R5PM0 snUlgs snf)z Saves the current parameters for restoring later. Args: parameters: Iterable of `torch.nn.Parameter`. The parameters to be temporarily stored. N)rr5rhrj)rmrrs r#storeEMAModel.storess6NX"XZE<<>#5#5#7#=#=#?Z"X"Xs5AcURc [S5eUR(aV[R"UVs/sHo"R PM snURVs/sHo3R PM sn5 OD[ URU5H*up2UR RUR 5 M, SUlgs snfs snf)a Restore the parameters stored with the `store` method. Useful to validate the model with EMA parameters without: affecting the original optimization process. Store the parameters before the `copy_to()` method. After validation (or model saving), use this to restore the former parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored parameters. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. NzGThis ExponentialMovingAverage has no `store()`ed weights to `restore()`)rj RuntimeErrorr]rrrrr)rmrrc_params r#restoreEMAModel.restore|s  " " *hi i <<  )344SWSjSj6kSj||Sj6k #&d&=&=z"J   .#K#'56ks C #C rcH[R"U5nURSUR5UlURS:dURS:a [ S5eURSUR 5Ul[ UR [5(d [ S5eURSUR5Ul[ UR[5(d [ S5eURS UR5Ul [ UR[5(d [ S 5eURS UR5Ul [ UR[5(d [ S 5eURS UR5Ul [ UR[[45(d [ S5eURSUR5Ul[ UR[[45(d [ S5eURSS5nUb]X l[ UR[ 5(d [ S5e[#SUR55(d [ S5egg)z Loads the ExponentialMovingAverage state. This method is used by accelerate during checkpointing to save the ema state dict. Args: state_dict (dict): EMA state. Should be an object returned from a call to :meth:`state_dict`. rWr:r(zDecay must be between 0 and 1rXzInvalid min_decayrkzInvalid optimization_steprYzInvalid update_after_steprZzInvalid use_ema_warmupr[zInvalid inv_gammar\z Invalid powerriNzshadow_params must be a listc3V# UHn[U[R5v M! g7fr)rrr4)rros r#r+EMAModel.load_state_dict..s O@4@&J   D &Y);); <YY'(588+=+=">'4'2.F$.F4.Fr%rV)r()NNNr5Nr)r"N)TTTNN)Zrrrrrrrr functoolsrtypingrrnumpyrrtorch.nn.functionalrrrO distributedrtorch.distributed.fsdprrr rQtorch.distributed.fsdp.wrapr modelsr pipelinesr schedulersrutilsrrrrrrrrrrrraccelerate.loggingrpeftr torchvisionrrrr$r9r4r.rLrmrrrrrfloat32rfrrrrr* Generatorrrrrrrrr$r2r8rAr%rLrTrVrHr%r#rsZ %   !!##CGHJ D#' ((&     **EEGG-.&)3)("R-1S% LLS% LLS% S%  S% ||d* S%S% 5<<u|| 34S%l)3)h(+3# 3##3#||3# << 3# << 3# LL 3#!<<3# %3# 5<< %&3#l3S%,,=N8O&PU}}-$uxx2G G-"]#u||+,]69]IN]&Dehhoo1E,F4PSUXPX>!&(,        LL3   % 6S$  nrUXX__/@@#PUP\P\J\gk>8  !$NtttT4 ")-?C/ 88??/ %,, // /  / c3h$& /tEHHOO45</ /fLFLFr%