Mj~ddlZddlZddlmZmZmZmZddlZddl Z ddl m Z m Z ddl mZmZmZddlmZmZddlmZer dZ d dZGd d ee ZdS) N)ListOptionalTupleUnion) ConfigMixinregister_to_config)KarrasDiffusionSchedulersSchedulerMixinSchedulerOutput) deprecateis_scipy_available) randn_tensorcjtjdd|dzd|}||zd|dz |zzz }|SNr)nplinspace)sampling_stepsshiftsigmas ?C:\Users\johnn\OneDrive\Documents\AI\shared\utils\fm_solvers.pyget_sampling_sigmasrsD K1nq0 1 1/>/ BE U]a519"55 6E Lc f||td|dttj|jjv}|std|jd|jd ||d||j}t|}n|dttj|jjv}|std|jd|jd ||d||j}t|}n|j|fd |i||j}||fS) NzYOnly one of `timesteps` or `sigmas` can be passed. Please choose one to set custom values timestepszThe current scheduler class zx's `set_timesteps` does not support custom timestep schedules. Please check whether you are using the correct scheduler.)rdevicesigmaszv's `set_timesteps` does not support custom sigmas schedules. Please check whether you are using the correct scheduler.)rrr) ValueErrorsetinspect signature set_timesteps parameterskeys __class__rlen) schedulernum_inference_stepsrrrkwargsaccepts_timesteps accept_sigmass rretrieve_timestepsr-s!3 g   '3  i5 6 6 A F F H H,J,JJ  by/Bbbb  M)FMMfMMM' !)nn   C  i5 6 6 A F F H H%J%JJ  `y/B```  GvfGGGGG' !)nn  3MMFMfMMM' ) ))rc *eZdZdZdeDZdZeddddddd dd d d dd ed ddfde de de de ede dedede de de de de e dede e de fdZ ed Zed!ZdFd#e fd$Z dGd%ee dfd&ee ejfd'e eed(e eedfde eedff d)Zd*ejd+ejfd,Zd-Zd.Zd(ed/ed0ejfd1Zdd2d3ejd*ejd+ejfd4Zddd5d3ejd*ejd6e ejd+ejfd7Zddd5d8eejd*ejd6e ejd+ejfd9Zdd2d8eejd*ejd+ejfd:ZdHd;Z d<Z! dId3ejd=ee ejfd*ejd>e ejd?e d+ee"e#ff d@Z$d*ejd+ejfdAZ%dBejd6ejdCej&d+ejfdDZ'dEZ(dS)JFlowDPMSolverMultistepSchedulera `FlowDPMSolverMultistepScheduler` is a fast dedicated high-order solver for diffusion ODEs. This model inherits from [`SchedulerMixin`] and [`ConfigMixin`]. Check the superclass documentation for the generic methods the library implements for all schedulers such as loading and saving. Args: num_train_timesteps (`int`, defaults to 1000): The number of diffusion steps to train the model. This determines the resolution of the diffusion process. solver_order (`int`, defaults to 2): The DPMSolver order which can be `1`, `2`, or `3`. It is recommended to use `solver_order=2` for guided sampling, and `solver_order=3` for unconditional sampling. This affects the number of model outputs stored and used in multistep updates. prediction_type (`str`, defaults to "flow_prediction"): Prediction type of the scheduler function; must be `flow_prediction` for this scheduler, which predicts the flow of the diffusion process. shift (`float`, *optional*, defaults to 1.0): A factor used to adjust the sigmas in the noise schedule. It modifies the step sizes during the sampling process. use_dynamic_shifting (`bool`, defaults to `False`): Whether to apply dynamic shifting to the timesteps based on image resolution. If `True`, the shifting is applied on the fly. thresholding (`bool`, defaults to `False`): Whether to use the "dynamic thresholding" method. This method adjusts the predicted sample to prevent saturation and improve photorealism. dynamic_thresholding_ratio (`float`, defaults to 0.995): The ratio for the dynamic thresholding method. Valid only when `thresholding=True`. sample_max_value (`float`, defaults to 1.0): The threshold value for dynamic thresholding. Valid only when `thresholding=True` and `algorithm_type="dpmsolver++"`. algorithm_type (`str`, defaults to `dpmsolver++`): Algorithm type for the solver; can be `dpmsolver`, `dpmsolver++`, `sde-dpmsolver` or `sde-dpmsolver++`. The `dpmsolver` type implements the algorithms in the [DPMSolver](https://huggingface.co/papers/2206.00927) paper, and the `dpmsolver++` type implements the algorithms in the [DPMSolver++](https://huggingface.co/papers/2211.01095) paper. It is recommended to use `dpmsolver++` or `sde-dpmsolver++` with `solver_order=2` for guided sampling like in Stable Diffusion. solver_type (`str`, defaults to `midpoint`): Solver type for the second-order solver; can be `midpoint` or `heun`. The solver type slightly affects the sample quality, especially for a small number of steps. It is recommended to use `midpoint` solvers. lower_order_final (`bool`, defaults to `True`): Whether to use lower-order solvers in the final steps. Only valid for < 15 inference steps. This can stabilize the sampling of DPMSolver for steps < 15, especially for steps <= 10. euler_at_final (`bool`, defaults to `False`): Whether to use Euler's method in the final step. It is a trade-off between numerical stability and detail richness. This can stabilize the sampling of the SDE variant of DPMSolver for small number of inference steps, but sometimes may result in blurring. final_sigmas_type (`str`, *optional*, defaults to "zero"): The final `sigma` value for the noise schedule during the sampling process. If `"sigma_min"`, the final sigma is the same as the last sigma in the training schedule. If `zero`, the final sigma is set to 0. lambda_min_clipped (`float`, defaults to `-inf`): Clipping threshold for the minimum value of `lambda(t)` for numerical stability. This is critical for the cosine (`squaredcos_cap_v2`) noise schedule. variance_type (`str`, *optional*): Set to "learned" or "learned_range" for diffusion models that predict variance. If set, the model's output contains the predicted Gaussian variance. cg|] }|j Sr)name).0es r z*FlowDPMSolverMultistepScheduler.}s>>>qAF>>>rriflow_prediction?Fgףp= ? dpmsolver++midpointTzeroinfNnum_train_timesteps solver_orderprediction_typer thresholdingdynamic_thresholding_ratiosample_max_valuealgorithm_type solver_typelower_order_finaleuler_at_finalfinal_sigmas_typelambda_min_clipped variance_type invert_sigmasc\| dvrd| d}tdd|| dvr6| dkr|d nt| d |j| d vr4| d vr|d nt| d |j| dvr| dkrt d| d| dd|_t jdd|z |ddd}d|z }tj | tj }|s||zd|dz |zzz }||_ ||z|_dg|z|_d|_d|_d|_|j d|_|j d|_dS)N dpmsolver sde-dpmsolverzalgorithm_type zn is deprecated and will be removed in a future version. Choose from `dpmsolver++` or `sde-dpmsolver++` insteadz+algorithm_types dpmsolver and sde-dpmsolver1.0.0)rLr8rMsde-dpmsolver++deisr8)rBz is not implemented for )r9heun)logrhobh1bh2r9)rCr8rOr:z`final_sigmas_type` z' is not supported for `algorithm_type` z$. Please choose `sigma_min` instead.rr7dtyper)r rNotImplementedErrorr&rr)rrcopytorch from_numpytofloat32rr model_outputslower_order_nums _step_index _begin_indexitem sigma_min sigma_max)selfr<r=r>ruse_dynamic_shiftingr?r@rArBrCrDrErFrGrHrIdeprecation_messagealphasrs r__init__z(FlowDPMSolverMultistepScheduler.__init__sm( ; ; ;#sN#s#s#s  CW) + + + "   ''''}'EEEE)%OOt~OOQQQ 2 2 2666''J'????)"LLDNLLNNN "# # #'8F'B'BV'8VVaoVVV  $( Q$7 702226$B$88< v!&)),,5=,AA# =V^q(- V';(<=F "55"Vl2 ! R--//Q,,..rc|jS)zg The index counter for current timestep. It will increase 1 after each scheduler step. )rarfs r step_indexz*FlowDPMSolverMultistepScheduler.step_indexs rc|jS)zq The index for the first timestep. It should be set from pipeline with `set_begin_index` method. rbrls r begin_indexz+FlowDPMSolverMultistepScheduler.begin_indexs   rrrpc||_dS)z Sets the begin index for the scheduler. This function should be run from pipeline before the inference. Args: begin_index (`int`): The begin index for the scheduler. Nro)rfrps rset_begin_indexz/FlowDPMSolverMultistepScheduler.set_begin_indexs(rr)rrmuc|jjr|td|=tj|j|j|dzdd}|jjr||d|}n| |jj }||zd|dz |zzz }|jj dkr"d|j dz |j dz dz}n/|jj d krd}ntd |jj ||jj z}tj ||ggtj}t!j||_t!j||t j |_t-||_dg|jjz|_d|_d|_d|_dS) a Sets the discrete timesteps used for the diffusion chain (to be run before inference). Args: num_inference_steps (`int`): Total number of the spacing of the time steps. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. NzR you have to pass a value for `mu` when `use_dynamic_shifting` is set to be `True`rrVr7rdr?r:zC`final_sigmas_type` must be one of 'zero', or 'sigma_min', but got rrX)configrgrrrrerdrZ time_shiftrrFalphas_cumprodr< concatenateastyper^r[r\rr]int64rr'r)r=r_r`rarb)rfr)rrrsr sigma_lastrs rr#z-FlowDPMSolverMultistepScheduler.set_timestepss" ; +  d  >[!4!""#$$$(DFF3B30F ; + =__Rf55FF} )V^q(- V';(<=F ; (K 7 7t2155-a01367JJ [ *f 4 4JJuVZVaVsuu T[<< *!"###)6"*#5#5 &v.. ))44778..$'y>>   K $%!" rsamplereturnc,|j}|j^}}}|tjtjfvr|}|||tj|z}| }tj ||j j d}tj |d|j j}|d}tj || ||z }|j||g|R}||}|S)ar "Dynamic thresholding: At each sampling step we set s to a certain percentile absolute pixel value in xt0 (the prediction of x_0 at timestep t), and if s > 1, then we threshold xt0 to the range [-s, s] and then divide by s. Dynamic thresholding pushes saturated pixels (those near -1 and 1) inwards, thereby actively preventing pixels from saturation at each step. We find that dynamic thresholding results in significantly better photorealism as well as better image-text alignment, especially when using very large guidance weights." https://arxiv.org/abs/2205.11487 r)dim)minmax)rXshaper[r^float64floatreshaperprodabsquantilerwr@clamprA unsqueezer])rfr~rX batch_sizechannelsremaining_dims abs_sampless r_threshold_samplez1FlowDPMSolverMultistepScheduler._threshold_sample$s% 06 - H~  6 6 6\\F Hrw~7N7N,NOOZZ\\ N  >A G G G K 1$+6    KK    QB    HF~FFF5!! rc ||jjzSNrwr<rfrs r _sigma_to_tz+FlowDPMSolverMultistepScheduler._sigma_to_tJst{666rcd|z |fSNrrrs r_sigma_to_alpha_sigma_tz7FlowDPMSolverMultistepScheduler._sigma_to_alpha_sigma_tMs5y%rrtcltj|tj|d|z dz |zzz Sr)mathexp)rfrsrrs rrxz*FlowDPMSolverMultistepScheduler.time_shiftQs/x||tx||q1uqy5.@@AArr~ model_outputct|dkr|dn|dd}|+t|dkr |d}ntd|tddd|jjd vrk|jjd kr|j|j}|||zz }ntd |jjd |jj r| |}|S|jjd vr|jjd kr|j|j}|d|z |zz }ntd |jjd|jj r4|j|j}|||zz }| |}||z}|SdS)a+ Convert the model output to the corresponding type the DPMSolver/DPMSolver++ algorithm needs. DPM-Solver is designed to discretize an integral of the noise prediction model, and DPM-Solver++ is designed to discretize an integral of the data prediction model. The algorithm and model type are decoupled. You can use either DPMSolver or DPMSolver++ for both noise prediction and data prediction models. Args: model_output (`torch.Tensor`): The direct output from the learned diffusion model. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. Returns: `torch.Tensor`: The converted model output. rtimestepNrz/missing `sample` as a required keyward argumentrrNPassing `timesteps` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`rUr6zprediction_type given as zr must be one of `epsilon`, `sample`, `v_prediction`, or `flow_prediction` for the FlowDPMSolverMultistepScheduler.rKzq must be one of `epsilon`, `sample`, `v_prediction` or `flow_prediction` for the FlowDPMSolverMultistepScheduler.) r'poprr rwrBr>rrmr?r) rfrr~argsr*rsigma_tx0_predepsilons rconvert_model_outputz4FlowDPMSolverMultistepScheduler.convert_model_outputUs0"$ii!mm477J1M1M >4yy1}}a EGGG   Z    ; %)K K K{*.???+do6 7\#99 e 0Keee {' :0099N[ '+I I I{*.???+do6 AK<#?? d 0Kddd {' 1+do6 7\#990099&0N!J Irr~noiserc t|dkr|dn|dd}t|dkr|dn|dd}|+t|dkr |d}ntd|tdd d |tdd d |j|jdz|j|j} }||\} }|| \} } tj| tj|z } tj| tj| z } | | z }|j j d kr)|| z |z| tj | d z z|zz }n.|j j dkr'| | z |z|tj |d z z|zz }n|j j dkrw|J|| z tj | z|z| dtj d|zz z|zz|tj d tj d|zz z|zz}np|j j dkr`|J| | z |zd|tj |d z zz|zz |tj tj d|zd z z|zz}|S)a One step for the first-order DPMSolver (equivalent to DDIM). Args: model_output (`torch.Tensor`): The direct output from the learned diffusion model. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. Returns: `torch.Tensor`: The sample tensor at the previous timestep. rrNr prev_timestepr50 missing `sample` as a required keyward argumentrrNrPassing `prev_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`r8r7rLrOrM@) r'rrr rrmrr[logrwrBrsqrt)rfrr~rrr*rrrsigma_salpha_talpha_slambda_tlambda_shx_ts rdpm_solver_first_order_updatez=FlowDPMSolverMultistepScheduler.dpm_solver_first_order_updates;&"$ii!mm477J1M1M#&t99q==Qfjj T7#7# >4yy1}}a FHHH   Z     $ ^     ;t':;T[ O>77@@77@@9W%% '(:(::9W%% '(:(:: x  ; % 6 6$%(/).A2)<)>AM(NNCC[ '; 6 6$%(/).1);)=@L(MMCC[ '+< < <$$$g% 1" 5?q59TAX#6#667<GHUZeiQ.?.?(?@@@5HICC[ '? : :$$$g%/#uy||c12346B3CCUZ !a%(8(83(>???%GHC rmodel_output_listc  t|dkr|dn|dd}t|dkr|dn|dd}|+t|dkr |d}ntd|tddd |tddd |j|jdz|j|j|j|jdz } } }||\} }|| \} } || \} } tj| tj|z }tj| tj| z }tj| tj| z }|d |d }}||z ||z }}||z }|d |z ||z z}}|j j dkr|j j dkrK|| z |z| tj | d z z|zz d| tj | d z zz|zz }n|j j dkrL|| z |z| tj | d z z|zz | tj | d z |z d zz|zz}nc|j j dkr|j j dkrI| | z |z|tj |d z z|zz d|tj |d z zz|zz }n|j j dkrJ| | z |z|tj |d z z|zz |tj |d z |z d z z|zz }n|j j dkrZ|J|j j dkr|| z tj | z|z| dtj d|zz z|zzd| dtj d|zz zz|zz|tj d tj d |zz z|zz}n|j j dkr|| z tj | z|z| dtj d|zz z|zz| d tj d|zz d|zz d zz|zz|tj d tj d |zz z|zz}n3|j j dkr"|J|j j dkr{| | z |zd|tj |d z zz|zz |tj |d z z|zz |tj tj d|zd z z|zz}n|j j dkr| | z |zd|tj |d z zz|zz d|tj |d z |z d z zz|zz |tj tj d|zd z z|zz}|S)a One step for the second-order multistep DPMSolver. Args: model_output_list (`List[torch.Tensor]`): The direct outputs from learned diffusion model at current and latter timesteps. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. Returns: `torch.Tensor`: The sample tensor at the previous timestep. r timestep_listNrrr5rrNPassing `timestep_list` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`rrVrr7r8r9rurQrLrOrrMr)r'rrr rrmrr[rrwrBrCrr)rfrr~rrr*rrrsigma_s0sigma_s1ralpha_s0alpha_s1r lambda_s0 lambda_s1m0m1rh_0r0D0D1rs r(multistep_dpm_solver_second_order_updatezHFlowDPMSolverMultistepScheduler.multistep_dpm_solver_second_order_updates&$'t99q==Qfjj T7#7# #&t99q==Qfjj T7#7# >4yy1}}a FHHH  $ ^     $ ^    K!+ , K ( K!+ ,$  77@@!99(CC(!99(CC(9W%% '(:(::Ih''%)H*=*== Ih''%)H*=*== "2&(9"(=BI%y9'<3 1WcBh27+B ; % 6 6{&*44(*f459aR==3#672=>@C59aR==3#67A9;=A>>(F22(*f459aR==3#672=>EIqbMMC$71#4yy1}}a EGGG  $ ^     $ ^    K!+ , K ( K!+ , K!+ , 1 -8X 77@@!99(CC(!99(CC(!99(CC(9W%% '(:(::Ih''%)H*=*== Ih''%)H*=*== Ih''%)H*=*== &r*,= -"2&B*I ,A9yCX3q#'B Bh27+cBh27-Cd R27^t 4 4R"Wo$+ . ; % 6 6h&&0uy!}}s23r9: 1"  3q83>?2EF 1"  3a 71a4?#EF"LMCC[ '; 6 6h&&0G5:Yq\\C5G5ILN4OO ! s 2a7#=>"DE ! s 2Q 6!Q$>DEKLC rc||j}||k}t|dkrdnd}||Sr)rnonzeror'rc)rfrschedule_timestepsindicesposs rindex_for_timestepz2FlowDPMSolverMultistepScheduler.index_for_timestepsW  %!% %1::<< w<#;;t~'<==#66x@@D   #0D   rrvariance_noise return_dictc\|jtd|j|||jt |jdz ko?|jjp3|jjot |jdkp|jj dk}|jt |jdz ko#|jjot |jdk}| ||}t|jj dz D]} |j | dz|j | <||j d<|tj}|jjd vr*|(t%|j||jtj } n7|jjd vr'||jtj } nd} |jj dks |jdks|r|||| } nW|jj dks |jdks|r||j || } n||j |} |j|jj kr|xjdz c_| |j} |xjdz c_|s| fSt7| S)a Predict the sample from the previous timestep by reversing the SDE. This function propagates the sample with the multistep DPMSolver. Args: model_output (`torch.Tensor`): The direct output from learned diffusion model. timestep (`int`): The current discrete timestep in the diffusion chain. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. generator (`torch.Generator`, *optional*): A random number generator. variance_noise (`torch.Tensor`): Alternative to generating noise with `generator` by directly providing the noise for the variance itself. Useful for methods such as [`LEdits++`]. return_dict (`bool`): Whether or not to return a [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`. Returns: [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`: If return_dict is `True`, [`~schedulers.scheduling_utils.SchedulerOutput`] is returned, otherwise a tuple is returned where the first element is the sample tensor. NzaNumber of inference steps is 'None', you need to run 'set_timesteps' after creating the schedulerrr:r5rrV)rMrO) generatorrrXrvr) prev_sample)r)rrmrr'rrwrErDrFrranger=r_r]r[r^rBrrrr`rrrrXrar ) rfrrr~rrrrDlower_order_secondirrs rstepz$FlowDPMSolverMultistepScheduler.steps>  # +s  ? "  ! !( + + +"_DN0C0Ca0GG5 K & 4 [ * Gs4>/B/BR/G 4 K )V 3  $#dn2E2E2II7"k;7!$.11B6 00f0MM t{/!344 > >A$($6q1u$=D q ! !!-25=)) ; %*+ + +/=/E "##*m %%%EE [ '+O O O"%%#*m&%%EEE ; #q ( (D,AA,E,EIZ,E<<V5=::KK [ % * *d.Ca.G.GK].GGG"6H@@KKFF"6G33K  4;#; ; ;  ! !Q & ! !"nn\%788  A "> !;7777rc|S)a= Ensures interchangeability with schedulers that need to scale the denoising model input depending on the current timestep. Args: sample (`torch.Tensor`): The input sample. Returns: `torch.Tensor`: A scaled input sample. r)rfr~rr*s rscale_model_inputz1FlowDPMSolverMultistepScheduler.scale_model_input s  roriginal_samplesrc j|j|j}|jjdkrft j|rRj|jt j ||jt j}n9j|j ||j}j  fd|D}n4j j g|j dz}nj g|j dz}|| }t|j t|j kr?|d}t|j t|j k?|\}}||z||zz} | S)NrvmpsrWc<g|]}|Sr)r)r2rrrfs rr4z=FlowDPMSolverMultistepScheduler.add_noise..Es:''+=>>rrrV)rr]rrXtyper[is_floating_pointrr^rprmrflattenr'rr) rfrrrr step_indicesrrr noisy_samplesrs ` @r add_noisez)FlowDPMSolverMultistepScheduler.add_noise/s#*2B2H JJ  " '5 0 0U5L66 0"&!2!2 'u}"3">"> !  'u}%>>II"&!2!23C3J!K!K ! %5%<==I   #"LL_ ( O,yq/AALL!,- 0BBL|$,,..%+%5%;!>"22Wu_D rc|jjSrrrls r__len__z'FlowDPMSolverMultistepScheduler.__len__Xs {..r)r)NNNNNr)NNT))__name__ __module__ __qualname____doc__r _compatiblesorderrrintstrrboolrjpropertyrmrprrrr[rrr#rrrrrxrrrrrrr rrr IntTensorrrrrrr/r/EsH55n?>$=>>>L E$(0!$"",1"%+%"&$+1%*U5\\M'+##F/F/ F/F/ F/  F/F/%*F/ F/F/F/ F/F/$C=F/"F/  }!F/"#F/F/F/F/P  X !!X!((3((((15+/(,+/.2 =!=!"39-=!c5<'(=!e% =! U5$;' ( =! eTk*+ =!=!=!=!D# #####L777   BUB5BU\BBBB $ GGGlG G  GGGG\ $(, DDDlD D  % D DDDDV $(, kkk -k k  % k kkkkd $ QQQ -Q Q  QQQQf # # # # 1 1 1$15 [8[8l[8U\)*[8 [8 !. [8[8 % &[8[8[8[8|  ',|    ','|'? '  ''''R/////rr/)NNNN)r!rtypingrrrrnumpyrr[diffusers.configuration_utilsrr%diffusers.schedulers.scheduling_utilsr r r diffusers.utilsr r diffusers.utils.torch_utilsrrr-r/rrrr sX  //////////// IIIIIIIIDDDDDDDDDD:9999999444444    %*%*%*%*PT /T /T /T /T /nkT /T /T /T /T /r