IhjUSSKrSSKJr SSKJr SSKJr SSKJr SSK J r SSK r SSK J r SSK rSSKJr S S KJrJr S S KJrJrJr S S KJr S \SS4SjrS \SS4SjrS$S \S\SS4SjjrS\S\4Sjr \S5r!\S%S \ RDS\4Sjj5r#"SS5r$"SS5r%"SS5r&S&S \ RDS\'S\(S \ \)\S!4S"\S\*\\ \+\44 S#jjr,g!\a SrNf=f)'N)contextmanager)deepcopy)update_wrapper) MethodType)Optional)nn)tqdm) PeftConfig PeftModel) LoraLayer LoraModeldora)BaseTunerLayermodelreturncr[R"UR5n[UR5S:Xa~SUR;amSUR;a\[ URR 5n[U[UR55RSS9 [X 5Ulgggg)ap Updates the forward signature of the PeftModel to include parents class signature model (`PeftModel`): Peft model to update the forward signature Example: ```python >>> from transformers import WhisperForConditionalGeneration >>> from peft import get_peft_model, LoraConfig >>> from peft.helpers import update_forward_signature >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en") >>> peft_config = LoraConfig(r=8, lora_alpha=32, lora_dropout=0.1, target_modules=["q_proj", "v_proj"]) >>> peft_model = get_peft_model(model, peft_config) >>> update_forward_signature(peft_model) ``` argskwargs__doc____name____annotations__assignedN) inspect signatureforwardlen parametersr__func__rtypeget_base_modelr)rcurrent_signaturers ښC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\peft/helpers.pyupdate_forward_signaturer'&s* ))%--8  ( ()Q. '22 2 )44 45==112 T%..0199Dn #72 5 3 /c[US5(dg[R"UR5n[ UR 5S:Xa SUR ;aSUR ;d)[ UR 5S:XamSUR ;a\[ URR5n[U[UR55RSS9 [X 5Ulggg) a: Updates the generate signature of a PeftModel with overriding generate to include parents class signature model (`PeftModel`): Peft model to update the generate signature Example: ```python >>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer >>> from peft import get_peft_model, LoraConfig, TaskType >>> from peft.helpers import update_generate_signature >>> model_name_or_path = "bigscience/mt0-large" >>> tokenizer = AutoTokenizer.from_pretrained(model_name_or_path) >>> model = AutoModelForSeq2SeqLM.from_pretrained(model_name_or_path) >>> peft_config = LoraConfig( ... task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1 ... ) >>> peft_model = get_peft_model(model, peft_config) >>> update_generate_signature(peft_model) >>> help(peft_model.generate) ``` generateNrrrr rr) hasattrrrr*r r!rr"rr#r$r)rr%r*s r&update_generate_signaturer,Hs0 5* % %))%..9  ( ()Q. '22 2 )44 4 ** +q 0XARA]A]5]ENN334  %%' ( 1 1? $H46^ 0r(methodcUS:Xa [U5 gUS:Xa [U5 gUS:Xa[U5 [U5 g[SUS35e)a Updates the signature of a PeftModel include parents class signature for forward or generate method model (`PeftModel`): Peft model to update generate or forward signature method (`str`): method to update signature choose one of "forward", "generate", "all" Example: ```python >>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer >>> from peft import get_peft_model, LoraConfig, TaskType >>> from peft.helpers import update_signature >>> model_name_or_path = "bigscience/mt0-large" >>> tokenizer = AutoTokenizer.from_pretrained(model_name_or_path) >>> model = AutoModelForSeq2SeqLM.from_pretrained(model_name_or_path) >>> peft_config = LoraConfig( ... task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1 ... ) >>> peft_model = get_peft_model(model, peft_config) >>> update_signature(peft_model) >>> help(peft_model.generate) ``` rr*allzmethod zE is not supported please choose one of ['forward', 'generate', 'all']N)r'r, ValueError)rr-s r&update_signaturer1qsP0 ' : !%( 5 '!%(76(*opqqr(model_name_or_pathc^Sn[R"U5 U$![a SnU$f=f)z Check if the model is a PEFT model. Args: model_name_or_path (`str`): Model id to check, can be local or on the Hugging Face Hub. Returns: `bool`: True if the model is a PEFT model, False otherwise. TF)r from_pretrained Exception)r2 is_peft_models r&check_if_peft_modelr7sAM""#56    s  ,,c#6# [U[[45(d[S[ U535e0nUR 5Hmn[U[ 5(dMURR5X#'URR5VVs0sH upEXEU-_M snnUlMo U(d [S5eSv UR5H up6XclM gs snnf!UR5H up6XclM f=f7f)a Context manager to temporarily rescale the scaling of the LoRA adapter in a model. The original scaling values are restored when the context manager exits. This context manager works with the transformers and diffusers models that have directly loaded LoRA adapters. For LoRA, applying this context manager with multiplier in [0, 1] is strictly equivalent to applying [wise-ft](https://huggingface.co/papers/2109.01903) (see [#1940](https://github.com/huggingface/peft/issues/1940) for details). It can improve the performances of the model if there is a distribution shiftbetween the training data used for fine-tuning, and the test data used during inference. Warning: It has been reported that when using Apple's MPS backend for PyTorch, it is necessary to add a short sleep time after exiting the context before the scales are fully restored. Args: model: The model containing `LoraLayer` modules whose scaling is to be adjusted. multiplier (float or int): The multiplier that rescales the `scaling` attribute. Must be of type float or int. Raises: ValueError: If the model does not contain any `LoraLayer` instances, indicating that the model does not support scaling. Example: ```python >>> model = ModelWithLoraLayer() >>> multiplier = 0.5 >>> with rescale_adapter_scale(model, multiplier): ... outputs = model(**inputs) # Perform operations with the scaled model >>> outputs = model(**inputs) # The original scaling values are restored here ``` z1Argument multiplier should be of type float, got z6scaling is only supported for models with `LoraLayer`sN) isinstancefloatint TypeErrorr#modulesr scalingcopyitemsr0)r multiplieroriginal_scalingmodulekvr>s r&rescale_adapter_scalerFsH j5#, / /KDQ[L\K]^__--/ fi ( ('-~~':':'<  $U(a[T U]"X4SS0UD6upVO[T U]"X4SS0UD6nSnSnSnUR5H1n [U [5(dMXyR 5-nUS- nM3 US:aXx- nXW-n U(aX4$U $)a Compute loss with Monteclora variational regularization. This method extends the standard compute_loss by adding the variational loss (KL divergence + entropy) from Monteclora samplers to the task loss. Args: model: The model being trained inputs: Input batch return_outputs: Whether to return model outputs along with loss **kwargs: Additional arguments Returns: loss or (loss, outputs) depending on return_outputs return_outputsTFNgrr )super compute_lossr=r9r_get_monteclora_loss) selfrinputsrQr task_lossoutputsmonteclora_lossnum_lora_modelsrC total_loss __class__s r&rS#MontecloraTrainerMixin.compute_loss2s !&!5e!cTX!c\b!c Iw,U[5[TZ[IGmmoF&),,"14O4O4Q"Q1$& Q -?O0 (6 $FJFr()F)r __module__ __qualname____firstlineno__rrS__static_attributes__ __classcell__)r\s@r&rOrO s$L'G'Gr(rOcL\rSrSrSrS S\SS4SjjrSrSrS S\4S jjr S r g) DoraCachingi\aLContext manager to enable DoRA caching, which improves speed of DoRA inference at the expense of memory. With active caching, the materialized LoRA weight (B @ A) and the weight norm (base weight + LoRA weight) are cached. Even within the caching context, if the model is in training mode, caching is disabled. When the model switches to training mode, the cache will be cleared. Example: ```py >>> from peft.helpers import DoraCaching >>> model.eval() # put in eval model for caching to work >>> with DoraCaching(): # use as a context manager ... output = model(inputs) >>> dora_caching = DoraCaching() >>> dora_caching(enabled=True) # permanently enable caching >>> output = model(inputs) >>> dora_caching(enabled=False) # permanently disable caching >>> output = model(inputs) ``` enabledrNcXlSUlgNrf prev_valuerUrfs r&__init__DoraCaching.__init__xs r(cX[RUlUR[lgrh)rENABLE_DORA_CACHINGrjrf)rUs r& __enter__DoraCaching.__enter__|s22#'<< r(c<UR[lSUlgrh)rjrro)rUr#value tracebacks r&__exit__DoraCaching.__exit__s#'?? r(cU[lgrh)rrorks r&__call__DoraCaching.__call__s #* r(riT) rr_r`rarboolrlrprurxrbr^r(r&rere\s560+++r(rec \rSrSrSrSS\R S\S\\ \ S4S\ 4S jjr SS jr SS \\S \\S\\S \\ 4SjjrSS \\S \\S\\S \\ 4SjjrSrg)KappaTuneSelectoria@ Lightweight utility to compute per-module / per-parameter condition numbers and return the best LoRA targets. Supports: - Classic nn.Linear modules (target_modules in LoraConfig) - Modern fused MoE weights stored as 3D nn.Parameter (gate_up_proj / down_proj, gate_proj / up_proj, etc.) used in Llama-4, Qwen2_MoE, Qwen3_MoE, Mixtral, OLMoE and similar models. These are returned via target_parameters. Notes: - Condition-number computation requires running SVD and can take several minutes on very large models. A progress bar can be shown/disabled via `show_progress`. Nrmax_dim_size_to_analyzemoe_param_suffixes. show_progresscdXlX lX@lU=(d SUlSUlSUlg)N)z .gate_up_projz .down_projz .gate_projz.up_proj)rr~rr_condition_numbers_parameter_condition_numbers)rUrr~rrs r&rlKappaTuneSelector.__init__s: '>$*"4# 9  ?CHL)r(rc^^ TRbg0nTRR5VVs/sH)up#[U[R 5(dM&X#4PM+ nnnTR (a [USSS9OUnUGHup#URn[Gb[US5(aC[RRURUR5R5nO[US5(ar[UR S5(aW[RR#UR R$UR R&5R5nOQURR)5R5nO(URR)5R5n[+U4SjUR,55(aGMi[.R0R3UR5UR7S5S 55nUSUS S -- R95n XU'GM UTl0n TRR;5V ^ V s/sHFum n [+U 4S jTR<55(dM,U R?5S :XdMBT U 4PMH n n n TR (a [U S SS9OU nUHum n U RR)5R5nUR,tnn[+U4SjU55(aMX/n[AU5HTnUUn[.R0R3U5nUSUS S -- R95n URCU 5 MV [EU5[GU5- n XT 'M U Tl$gs snnfs sn n f)NzComputing SVD (linear layers)layer)descunit quant_statestateCBc3@># UHoTR:v M g7frhr~.0dimrUs r& 4KappaTuneSelector._compute_kappas..sI#555rg:0yE>c3F># UHnTRU5v M g7frh)endswith)rs param_names r&rrs!K3Ja:&&q))3Js!zComputing SVD (MoE parameters)paramc3@># UHoTR:v M g7frhrrs r&rrsN#555r)%rrrIr9rLinearrr weightbnbr+ functionaldequantize_4bitdatarr:rint8_vectorwise_dequantrSCBdetachanyshapetorchlinalgsvdvalsviewsizeitemnamed_parametersrrrangeappendsumr r)rUcondition_numbers module_namerClinear_modules linear_iterrwSkappaparameter_condition_numbersrr moe_paramsmoe_iter num_experts expert_shapekappas expert_idxexpert_ws` ` r&_compute_kappas!KappaTuneSelector._compute_kappass7  " " . /1(,zz'?'?'A 'A# &")), "[ !'A  !! &EG T  $/ K]]F6=1166v{{FDVDVW]]_AVW--'&,,2M2M>>v||PVP\P\P`P`aggiA **,224AKK&&(..0IIII $$QVVAFF1Ir%:;AqTQrUT\*002E-2k *#$/&#49;#&*ZZ%@%@%B %B! EK43J3JKK PUPYPYP[_`P` Z %B  VZUgUgD"B Qmw "* J !!#))+A)* &K,NNNNF#K0 Z=LL((212.446 e$ 1 K#f+-E6; 3"* -H)s B s%O O 6+O%O;Otop_p num_modules thresholdcUR5 UR(d/$[URR5SS9nUb/[ U[ U55nUSUVVs/sHupgUPM snn$Ub;[ S[[ U5U-55nUSUVVs/sHupgUPM snn$UbUVVs/sHuphX::dM UPM snn$UVVs/sHupgUPM snn$s snnfs snnfs snnfs snnf)Nc US$Nr r^xs r&4KappaTuneSelector.get_best_targets..s qQRtr(keyr )rrsortedr@minr maxr;) rUrrrsorted_modulesrDrL_rs r&get_best_targets"KappaTuneSelector.get_best_targetss &&I 7 7 = = ?^T  "K^!45A(6r(:;(:WTD(:; ;  As3~.678A(6r(:;(:WTD(:; ;  ,:QN[Te>PDNQ Q$23NN33<<R3'C/%C5? C;C;DcUR5 UR(d/$[URR5SS9nUb/[ U[ U55nUSUVVs/sHupgUPM snn$Ub;[ S[[ U5U-55nUSUVVs/sHupgUPM snn$UbUVVs/sHuphX::dM UPM snn$UVVs/sHupgUPM snn$s snnfs snnfs snnfs snnf)Nc US$rr^rs r&r>KappaTuneSelector.get_best_target_parameters..s XYZ[X\r(rr )rrrr@rr rr;) rUrrr sorted_paramsrDrLrrs r&get_best_target_parameters,KappaTuneSelector.get_best_target_parameterss 00It@@FFHn]  "K]!34A(5bq(9:(9WTD(9: :  As3}-567A(5bq(9:(9WTD(9: :  ,9PM[TU=ODMP P$12MM22;;Q2r)rrr~rrr)@NT)rN)NNN)rr_r`rarrModuler;rtuplestrr{rlrr:listrrrbr^r(r&r}r}s "(-8<" MyyM"%M%U38_5 M  M&?HDnr4e_4:B3-4[cdi[j4 c4.nr3e_3:B3-3[cdi[j3 c33r(r}rr~r.rcr[UUUUS9nURUS9nURUS9nU(dSnUUS.$)a One-liner convenience function for KappaTune target selection. Returns both target_modules and target_parameters. Args: model (nn.Module): Base model whose weights will be analyzed for condition numbers. top_p (float, optional): Select the top fraction of candidate modules/parameters with the lowest condition numbers. max_dim_size_to_analyze (int, optional): Upper bound on the maximum matrix dimension analyzed via SVD. Defaults to 16384. moe_param_suffixes (Optional[tuple[str, ...]], optional): Parameter-name suffixes used to identify fused MoE tensors that should be returned via `target_parameters`. If None, sensible defaults are used. show_progress (bool, optional): Whether to display a progress bar while computing condition numbers (SVD-based) across candidate tensors/modules. Disable in CI or other non-interactive environments. Defaults to True. )r~rr)rN)target_modulestarget_parameters)r}rr)rrr~rrselectorrrs r&find_kappa_target_modulesrs`0! 7-# H..U.;N ;;%;H  ). r()r/rz)g?rNT)-r contextlibrr?r functoolsrtypesrtypingrrr bitsandbytesr ImportError tqdm.autor peft_modelr r tuners.lorar rrtuners.tuners_utilsrr'r,rr1r{r7rFrrMrOrer}r:r;rdictrrr^r(r&rs%$ -33/3I3$3D&5Y&54&5R rI rs rt rFCD*9%9%x!Hryy!H$!H!HHNGNGb)+)+XM3M3d#(48 ) 99) )!)!sCx1 )  )  #xS " "# )y CsC--C87C8