MjddlZddlmZmZmZmZddlZddlZddl m Z m Z ddl m Z mZmZddlmZmZerddlZGddee ZdS)N)ListOptionalTupleUnion) ConfigMixinregister_to_config)KarrasDiffusionSchedulersSchedulerMixinSchedulerOutput) deprecateis_scipy_availablec!eZdZdZdeDZdZeddddddd dd d d gd d ddfdedede de e de de de de de de de edede dede e fdZed Zed!ZdGd"efd#Z dHd$eed fd%ee ejfd&e e e d'e ee d fde ee d ff d(Zd)ejd*ejfd+Zd,Zd-Zd'e d.e d/ejfd0Zd d1d2ejd)ejd*ejfd3Zd d d4d2ejd)ejd5ed*ejfd6Zd d d d7d8ejd9ejd:ejd5ed*ejf d;ZdId<Z d=Z! dJd2ejd>eeejfd)ejd?e d*ee"e#ff d@Z$d)ejd*ejfdAZ%dBejdCejdDej&d*ejfdEZ'dFZ(d S)KFlowUniPCMultistepScheduleru `UniPCMultistepScheduler` is a training-free framework designed for the fast sampling of diffusion models. This model inherits from [`SchedulerMixin`] and [`ConfigMixin`]. Check the superclass documentation for the generic methods the library implements for all schedulers such as loading and saving. Args: num_train_timesteps (`int`, defaults to 1000): The number of diffusion steps to train the model. solver_order (`int`, default `2`): The UniPC order which can be any positive integer. The effective order of accuracy is `solver_order + 1` due to the UniC. It is recommended to use `solver_order=2` for guided sampling, and `solver_order=3` for unconditional sampling. prediction_type (`str`, defaults to "flow_prediction"): Prediction type of the scheduler function; must be `flow_prediction` for this scheduler, which predicts the flow of the diffusion process. thresholding (`bool`, defaults to `False`): Whether to use the "dynamic thresholding" method. This is unsuitable for latent-space diffusion models such as Stable Diffusion. dynamic_thresholding_ratio (`float`, defaults to 0.995): The ratio for the dynamic thresholding method. Valid only when `thresholding=True`. sample_max_value (`float`, defaults to 1.0): The threshold value for dynamic thresholding. Valid only when `thresholding=True` and `predict_x0=True`. predict_x0 (`bool`, defaults to `True`): Whether to use the updating algorithm on the predicted x0. solver_type (`str`, default `bh2`): Solver type for UniPC. It is recommended to use `bh1` for unconditional sampling when steps < 10, and `bh2` otherwise. lower_order_final (`bool`, default `True`): Whether to use lower-order solvers in the final steps. Only valid for < 15 inference steps. This can stabilize the sampling of DPMSolver for steps < 15, especially for steps <= 10. disable_corrector (`list`, default `[]`): Decides which step to disable the corrector to mitigate the misalignment between `epsilon_theta(x_t, c)` and `epsilon_theta(x_t^c, c)` which can influence convergence for a large guidance scale. Corrector is usually disabled during the first few steps. solver_p (`SchedulerMixin`, default `None`): Any other scheduler that if specified, the algorithm becomes `solver_p + UniC`. use_karras_sigmas (`bool`, *optional*, defaults to `False`): Whether to use Karras sigmas for step sizes in the noise schedule during the sampling process. If `True`, the sigmas are determined according to a sequence of noise levels {σi}. use_exponential_sigmas (`bool`, *optional*, defaults to `False`): Whether to use exponential sigmas for step sizes in the noise schedule during the sampling process. timestep_spacing (`str`, defaults to `"linspace"`): The way the timesteps should be scaled. Refer to Table 2 of the [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://huggingface.co/papers/2305.08891) for more information. steps_offset (`int`, defaults to 0): An offset added to the inference steps, as required by some model families. final_sigmas_type (`str`, defaults to `"zero"`): The final `sigma` value for the noise schedule during the sampling process. If `"sigma_min"`, the final sigma is the same as the last sigma in the training schedule. If `zero`, the final sigma is set to 0. cg|] }|j S)name).0es EC:\Users\johnn\OneDrive\Documents\AI\shared\utils\fm_solvers_unipc.py z&FlowUniPCMultistepScheduler.Is>>>qAF>>>iflow_prediction?Fgףp= ?Tbh2Nlinspacerzeronum_train_timesteps solver_orderprediction_typeshift thresholdingdynamic_thresholding_ratiosample_max_value predict_x0 solver_typelower_order_finaldisable_correctorsolver_ptimestep_spacing steps_offsetfinal_sigmas_typec| dvr4| dvr|dnt| d|j| |_d|_t jdd|z |ddd}d|z }tj | tj }|s||zd|dz |zzz }||_ ||z|_ dg|z|_dg|z|_d |_| |_| |_d|_d|_d|_|j d |_ |j d|_|j d |_dS) N)bh1r)midpointheunlogrhor)r'z is not implemented for rrdtypercpu)rNotImplementedError __class__r&num_inference_stepsnprcopytorch from_numpytofloat32sigmas timesteps model_outputs timestep_listlower_order_numsr)r* last_sample _step_index _begin_indexitem sigma_min sigma_max)selfrr r!r"use_dynamic_shiftingr#r$r%r&r'r(r)r*r+r,r-alphasr@s r__init__z$FlowUniPCMultistepScheduler.__init__Ls* n , ,<<<''E'::::)"LLDNLLNNN%#' Q$7 702226$B$88< v!&)),,5=,AA# =V^q(- V';(<=F "55"Vl2"Vl2 !!2   knn  R--//Q,,..rc|jS)zg The index counter for current timestep. It will increase 1 after each scheduler step. )rFrKs r step_indexz&FlowUniPCMultistepScheduler.step_indexs rc|jS)zq The index for the first timestep. It should be set from pipeline with `set_begin_index` method. rGrPs r begin_indexz'FlowUniPCMultistepScheduler.begin_indexs   rrTc||_dS)z Sets the begin index for the scheduler. This function should be run from pipeline before the inference. Args: begin_index (`int`): The begin index for the scheduler. NrS)rKrTs rset_begin_indexz+FlowUniPCMultistepScheduler.set_begin_indexs(rr9devicer@mucF|jjr|td|=tj|j|j|dzdd}|jjr||d|}n| |jj }||zd|dz |zzz }|jj dkr"d|j dz |j dz dz}n/|jj d krd}ntd |jj ||jj z}tj ||ggtj}t!j||_t!j||t j |_t-||_dg|jjz|_d|_d|_|jr!|j|j| d|_d|_|jd |_dS)a Sets the discrete timesteps used for the diffusion chain (to be run before inference). Args: num_inference_steps (`int`): Total number of the spacing of the time steps. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. NzR you have to pass a value for `mu` when `use_dynamic_shifting` is set to be `True`rr3rrIr?rzC`final_sigmas_type` must be one of 'zero', or 'sigma_min', but got rWr5rWr6) configrL ValueErrorr:rrJrIr; time_shiftr"r-alphas_cumprodr concatenateastyper?r<r=r@r>int64rAlenr9r rBrDrEr* set_timestepsrFrG)rKr9rWr@rXr" sigma_lastrAs rrez)FlowUniPCMultistepScheduler.set_timestepssP" ; +  d  >[!4!""#$$$(DFF3B30F ; + =__Rf55FF} )V^q(- V';(<=F ; (K 7 7t2155-a01367JJ [ *f 4 4JJuVZVaVsuu T[<< *!"###)6"*#5#5 &v.. ))44778..$'y>>   K $%!" = Q M ' '(@ ' P P P  knn  rsamplereturnc,|j}|j^}}}|tjtjfvr|}|||tj|z}| }tj ||j j d}tj |d|j j}|d}tj || ||z }|j||g|R}||}|S)as "Dynamic thresholding: At each sampling step we set s to a certain percentile absolute pixel value in xt0 (the prediction of x_0 at timestep t), and if s > 1, then we threshold xt0 to the range [-s, s] and then divide by s. Dynamic thresholding pushes saturated pixels (those near -1 and 1) inwards, thereby actively preventing pixels from saturation at each step. We find that dynamic thresholding results in significantly better photorealism as well as better image-text alignment, especially when using very large guidance weights." https://arxiv.org/abs/2205.11487 rdim)minmax)r5shaper<r?float64floatreshaper:prodabsquantiler]r$clampr% unsqueezer>)rKrgr5 batch_sizechannelsremaining_dims abs_sampless r_threshold_samplez-FlowUniPCMultistepScheduler._threshold_samples% 06 - H~  6 6 6\\F Hrw~7N7N,NOOZZ\\ N  >A G G G K 1$+6    KK    QB    HF~FFF5!! rc ||jjzSNr]rrKsigmas r _sigma_to_tz'FlowUniPCMultistepScheduler._sigma_to_t st{666rcd|z |fSNrrrs r_sigma_to_alpha_sigma_tz3FlowUniPCMultistepScheduler._sigma_to_alpha_sigma_ts5y%rrtcltj|tj|d|z dz |zzz Sr)mathexp)rKrXrrs rr_z&FlowUniPCMultistepScheduler.time_shifts/x||tx||q1uqy5.@@AArrg model_outputc6t|dkr|dn|dd}|+t|dkr |d}ntd|tddd|j|j}||\}}|jrk|jj d kr|j|j}|||zz } ntd |jj d |jj r| | } | S|jj d kr|j|j}|d|z |zz } ntd |jj d |jj r4|j|j}|||zz } | | } || z} | S) a Convert the model output to the corresponding type the UniPC algorithm needs. Args: model_output (`torch.Tensor`): The direct output from the learned diffusion model. timestep (`int`): The current discrete timestep in the diffusion chain. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. Returns: `torch.Tensor`: The converted model output. rtimestepNrz/missing `sample` as a required keyward argumentrA1.0.0zPassing `timesteps` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`rzprediction_type given as zi must be one of `epsilon`, `sample`, `v_prediction` or `flow_prediction` for the UniPCMultistepScheduler.) rdpopr^r r@rQrr&r]r!r#r|) rKrrgargskwargsrralpha_tsigma_tx0_predepsilons rconvert_model_outputz0FlowUniPCMultistepScheduler.convert_model_outputs,"$ii!mm477J1M1M >4yy1}}a EGGG   Z     DO,77>> ? {*.???+do6 7\#99 \ 0K\\\ {' :0099N{*.???+do6 AK<#?? \ 0K\\\ {' 1+do6 7\#990099&0Nr)rgorderrc t|dkr|dn|dd}|+t|dkr |d}ntd|+t|dkr |d}ntd|tddd |j}|jd }|d } |} |jr#|j||| j} | S|j |j dz|j |j } } | | \}} | | \}} tj |tj | z }tj |tj | z }||z }|j}g}g}td|D]}|j |z }||dz }| |j |\}}tj |tj |z }||z |z }||||| z |z |d tj|| }g}g}|jr| n|}tj|} | |z dz }!d}"|jjd kr|}#n3|jjdkrtj|}#nt-td|dzD][}|tj||dz ||!|"z|#z |"|dzz}"|!|z d|"z z }!\tj|}tj|| }t|dkrtj|d}|dkrtjdg| j|}$nbtj|dd dd f|dd || j}$nd}|jr8| | z | z|| z| zz }%|tjd|$|}&nd}&|%||#z|&zz } n7||z | z| | z| zz }%|tjd|$|}&nd}&|%| |#z|&zz } | | j} | S)a One step for the UniP (B(h) version). Alternatively, `self.solver_p` is used if is specified. Args: model_output (`torch.Tensor`): The direct output from the learned diffusion model at the current timestep. prev_timestep (`int`): The previous discrete timestep in the diffusion chain. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. order (`int`): The order of UniP at this timestep (corresponds to the *p* in UniPC-p). Returns: `torch.Tensor`: The sample tensor at the previous timestep. r prev_timestepNrz0 missing `sample` as a required keyward argumentrz/ missing `order` as a required keyward argumentrzPassing `prev_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`r3rr\r/rrjrZr5rWk,bkc...->bc...)rdrr^r rBrCr*step prev_sampler@rQrr<logrWrangeappendtensorr&expm1r]r'r7powstackr5linalgsolver>einsum)'rKrrgrrrrmodel_output_lists0m0xx_trsigma_s0ralpha_s0lambda_t lambda_s0hrWrksD1sisimialpha_sisigma_si lambda_sirkRbhhh_phi_1h_phi_k factorial_iB_hrhos_px_t_pred_ress' rmultistep_uni_p_bh_updatez5FlowUniPCMultistepScheduler.multistep_uni_p_bh_update^s2$'t99q==Qfjj T7#7# >4yy1}}a FHHH =4yy1}}Q EGGG  $ ^    !.   # r "  = -$$\2q99ECJ K!(;r))rKrrErrrrrrrrrmodel_trrrrrrrrWrrrrrrrrrrrrrrrrrhos_crcorr_resD1_ts) rmultistep_uni_c_bh_updatez5FlowUniPCMultistepScheduler.multistep_uni_c_bh_updatesB8$'t99q==Qfjj T7#7#  4yy1}}"1g  JLLL  4yy1}}"1g  JLLL =4yy1}}Q DFFF  $ ^    !. r " # K8$+ Oa ;!77@@!99(CC(9W%% '(:(::Ih''%)H*=*== y #q% ' 'AAE*B"QU8,B!%!=!=dk"o!N!N Hh (++ei.A.AAIi'1,B JJrNNN JJR2~ & & & & 3l3v...  ? )aRR+b//B," ; "e + +CC [ $ - -+b//CC%'' 'q%!)$$ 5 5A HHUYsAE** + + + HHW{*S0 1 1 1 1q5 KlQ_4GG KNN L6 * * * s88a<<+cq)))CCC A::\3%qwvFFFFF\''1--0088;;AGDDF ? HX%)Gg,=,BBD <(96#2#;LLRrArWrrFrG)rKrs r_init_step_indexz,FlowUniPCMultistepScheduler._init_step_indexsd   #(EL11 >#;;t~'<==#66x@@D   #0D   rr return_dictc|jtd|j|||jdko|jdz |jvo|jdu}|||}|r#|||j||j}t|j j dz D]2}|j |dz|j |<|j |dz|j |<3||j d<||j d<|j jr5t|j j t!|j|jz } n |j j } t| |jdz|_|jdksJ||_||||j} |j|j j kr|xjdz c_|xjdz c_|s| fSt+| S) a Predict the sample from the previous timestep by reversing the SDE. This function propagates the sample with the multistep UniPC. Args: model_output (`torch.Tensor`): The direct output from learned diffusion model. timestep (`int`): The current discrete timestep in the diffusion chain. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. return_dict (`bool`): Whether or not to return a [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`. Returns: [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`: If return_dict is `True`, [`~schedulers.scheduling_utils.SchedulerOutput`] is returned, otherwise a tuple is returned where the first element is the sample tensor. NzaNumber of inference steps is 'None', you need to run 'set_timesteps' after creating the schedulerrrr)rrErrr3)rrgr)r)r9r^rQrr)rErr this_orderrr]r rBrCr(rlrdrArDrrFr ) rKrrrgr generator use_correctormodel_output_convertrrrs rrz FlowUniPCMultistepScheduler.stepsE4  # +s  ? "  ! !( + + + Oa  ) Oa t'= = )  D (  $88  9 ) )  33"6 ,"o 4Ft{/!344 > >A$($6q1u$=D q !$($6q1u$=D q ! !!52!)2 ; ( 2T[5 00!_-..JJ1Jj"3a799""""!44%/5   4;#; ; ;  ! !Q & ! ! A "> !;7777rc|S)a? Ensures interchangeability with schedulers that need to scale the denoising model input depending on the current timestep. Args: sample (`torch.Tensor`): The input sample. Returns: `torch.Tensor`: A scaled input sample. r)rKrgrrs rscale_model_inputz-FlowUniPCMultistepScheduler.scale_model_inputs  roriginal_samplesnoiserAc j|j|j}|jjdkrft j|rRj|jt j ||jt j}n9j|j ||j}j  fd|D}n4j j g|j dz}nj g|j dz}|| }t|j t|j kr?|d}t|j t|j k?|\}}||z||zz} | S)Nr[mpsr4c<g|]}|Sr)r)rrrrKs rrz9FlowUniPCMultistepScheduler.add_noise.. s:''+=>>rrr3)r@r>rWr5typer<is_floating_pointrAr?rTrQrnflattenrdrvr) rKrrrAr@ step_indicesrrr noisy_samplesrs ` @r add_noisez%FlowUniPCMultistepScheduler.add_noises#*2B2H JJ  " '5 0 0U5L66 0"&!2!2 'u}"3">"> !  'u}%>>II"&!2!23C3J!K!K ! %5%<==I   #"LL_ ( O,yq/AALL!,- 0BBL|$,,..%+%5%;!>"22Wu_D rc|jjSr~rrPs r__len__z#FlowUniPCMultistepScheduler.__len__s {..r)r)NNNNNr~)TN))__name__ __module__ __qualname____doc__r _compatiblesrrintstrrrpboolrr rNpropertyrQrTrVrr<rWrerr|rrr_rrrrrr rrr IntTensorrrrrrrrs22h?>$=>>>L E(, !#4%(!&!&05&)#$&*+-'+$. !/5#7/7/!$7/7/! 7/ E? 7/7/).7/$7/7/7/ $7/ $Cy7/%7/"7/ !7/" (}#7/7/7/7/r  X !!X!((3((((15+/(,+/.2 CC"39-Cc5<'(Ce% C U5$;' ( C eTk*+ CCCCL$ $$$$$N777   BUB5BU\BBBB $ EEElE E  EEEEV $ FFFlF F  F FFFFX%)$( LLL <L\ L \ L  L LLLL\ # # # # 1 1 1 "& T8T8 <T8S%,./T8\T8 T8 !&ou&< = T8T8T8T8l ',|"','|'? '  ''''R/////rr)rtypingrrrrnumpyr:r<diffusers.configuration_utilsrr%diffusers.schedulers.scheduling_utilsr r r diffusers.utilsr r scipy.statsscipyrrrrrs  //////////// IIIIIIIIDDDDDDDDDD:9999999L /L /L /L /L /.+L /L /L /L /L /r