j92RSSKJr SSKrSSKJr SSKJrJr SSKJ r J r J r J r \ R"\5r\(a SSKJr SS KJr \ "5(aSSKrSS jrSS jrSS jrSS jrSSjrSSSjjrSrSSSjjrSSjrSSjrS SjrS!Sjr S!Sjr!S!Sjr"g)") annotationsN) timedelta) TYPE_CHECKING TypeGuard)is_torch_availableis_torch_distributed_availableis_torch_greater_or_equalloggingDTensor)DistributedConfigc^[5(dg[RR5$)NF)r torch distributedis_initializedڬC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\transformers/distributed/utils.py!_is_torch_distributed_initializedr$s# ) + +    + + --rcD[5(dgSSKJn [X5$)NFrr )r torch.distributed.tensorr isinstance)objr s r is_dtensorr*s ) + +0 c ##rc^[5(dg[RR5$)Nr)rrrget_rankrrr_get_torch_distributed_rankr2s# , . .    % % ''rc^[5(dg[RR5$)Nr)rrrget_world_sizerrr!_get_torch_distributed_world_sizer"8s# , . .    + + --rc|[5=(a, [[RR SS55S:H$)N LOCAL_RANKz-1r)rintosenvirongetrrris_local_dist_rank_0r)>s+ , . _3rzz~~lTX7Y3Z^_3__rc [RR5(GdUc([RR 5R nUS:Xa[ RS5 Sn[[RS5n[[RS5n[[RS5nSS S S S S S.nURU5nSnUS:wa5[[U5RU5 [R"X5n[RRUUUU[!SS9S9 gg!["an[%S5UeSnAff=f)zInitialize torch.distributed if not already initialized. If `device_type` is not given, it is detected from the current accelerator. Nmps^PyTorch's built-in DeviceMesh/DTensor stack does not support an MPS mesh. Falling back to CPU.cpuRANKr$ WORLD_SIZEncclglooxcclhcclneurontpu_dist)cudar-xpuhpur4tpur)hours)backendrank world_size device_idtimeoutzWe tried to initialize torch.distributed for you, but it failed. Make sure you init torch distributed in your script to use distributed training.)rrr_C_get_acceleratortypelogger warning_oncer%r&r'r(getattr set_devicedeviceinit_process_groupr ExceptionOSError) device_typer< local_rankr= backend_mapr;r>es r_ensure_torch_distributedrOBsO    + + - -  ((335::K %    p  K" rzz&)*DRZZ 56JRZZ 56J"! K"ook2G Ie#{+66zB!LLA    0 0%#!* 1 = .L ^  s.CE E EEc2[5(dg[RR5RnUS:wa;[R R [[U5R5/S9 g[R R 5 g)u#Barrier bound to the current accelerator device. Passing `device_ids` is required when the process group was initialized without a `device_id`; with it, the call is a no-op compared to plain `barrier()`. Safe to call when torch.distributed has not been initialized — returns immediately. Nr-) device_ids) rrr@rArBrbarrierrEcurrent_device)rKs r_distributed_barrierrTtsl - . .((++-22Ke !!ge[.I.X.X.Z-[!\ !!#rcdUbUc [S5eUbUb [S5eUGc[S5(d [S5e[RR 5R nUS:Xa[RS5 Sn[[U5nUS:waTUR[[RS55 UR5n[R"XF5nUnO![R"U5nU=(d 0n[R R#UR U45nX24$UR$S :a S UR&;a [S 5eUS n[R"UR(S [[RS535nX24$) z Sets up the device mesh and initialized the backend for tensor parallelism. This function is called when the model is loaded and the TP plan is set to 'auto'. z-tp_plan has to be set when tp_size is passed.zY`tp_plan` and `device_map` are mutually exclusive. Choose either one for parallelization.2.5z3Tensor parallel is only supported for `torch>=2.5`.r+r,r-r$rtpzsWhen using `tp_plan` and n-d `device_mesh`, it must contain a 'tp' dimension. Please provide a valid `device_mesh`.:) ValueErrorr rJrr@rArBrCrDrErFr%r&r'rSrGrinit_device_meshndimmesh_dim_namesrK)tp_plantp_size device_mesh device_maprKrSindex tp_devices rinitialize_tensor_parallelismrcswHIIz5tuu(//OP Phh//166 %    p  K  4 %   % %c"**\*B&C D"113E [8I"J [1I$*J''88'T   ""   a ;555 <&d+K\\[%<%<$=Qs2::lC[?\>]"^_  ""rcURS:a[S5(d [S5e[RR 5R nUS:wa_[[RRSS55n[[U5RU5 [R"X5nO[R"U5nURn//peUS:a"URU5 URS5 [RR!U[#U5[#U5S9n[%U5S:a UR'S R)U55 X74$) Nr2.7z?FSDP2 requires `torch>=2.7` (distributed checkpoint save/load).r-r$rfsdpr\_) fsdp_sizer rJrr@rArBr%r&r'r(rErFrGappendrrZtuplelen_flattenjoin)distributed_configrKrLr`ridimsnamesmeshs r)initialize_fully_sharded_data_parallelismrss##a'0I%0P0PWXX((++-22Ke a89 {#..z:\\+: \\+. ",,Ib%1} I V    - -k5;W\]bWc - dD 4y1} chhuo&  rc[S5(d [S5e[RR 5R n[ U5 [RR5nURnX#:wa[SUSUS35eUS:wa_[[RRSS55n[[U5R!U5 [R""X5nO[R""U5nX#:XdSUSUS35e[RR%X4S S 9nXV4$) NrVzBPipeline parallelism with DistributedConfig requires `torch>=2.5`.z world_size (z) must be equal to pp_size ()r-r$r)pprg)r rJrr@rArBrOrr!pp_size RuntimeErrorr%r&r'r(rErFrGrZ)rorKr=rwrLr`rrs rinitialize_pipeline_parallelismrys %U + +Z[[((++-22Kk*""113J ((G\*5QRYQZZ[\]]e a89 {#..z:\\+: \\+.  cL =2.7`.r)StateDictOptionsget_model_state_dictT)full_state_dict cpu_offload)options)r rJ'torch.distributed.checkpoint.state_dictr|r}r)modelr|r}rr~s rgather_full_state_dictrsM %U + +HII_tFG*5BO"$) Irc [S5(d [S5eSSKJs Jn SSKJn SSKJn U"U5nURUU"USSS9S 9 [5 g) uSave model parameters as standard HF-format sharded safetensors using DCP + HuggingFaceStorageWriter with consolidation enabled. Every rank first writes its own shard in parallel under `/sharded/`, then a consolidation pass reads those shards and emits HF-compatible `model-*-of-N.safetensors` (+ index) at `/`. The result is a directory `from_pretrained` reads through its normal path — no special flag needed at load time. rer{rN)HuggingFaceStorageWriter)r}T)pathsave_distributedenable_consolidation)storage_writer) r rJtorch.distributed.checkpointr checkpoint'torch.distributed.checkpoint.hf_storagerrr}saverT)rcheckpoint_dirdcprr} state_dicts r!save_model_checkpoint_distributedrs` %U + +HII/.PL%e,JHH/!!%  rc[S5(d [S5eSSKJs Jn SSKJn U"X5nURSU0US9 g)zSave optimizer state via DCP.rer{rN)get_optimizer_state_dict optimizer checkpoint_id)r rJrrrrrr)rrrrroptimizer_state_dicts rsave_optimizer_distributedr"sE $U + +HII/.P3EEHHk/ 0HOrc[S5(d [S5eSSKJs Jn SSKJnJn U"X5nURSU0US9 U"XU5 g)zLoad optimizer state via DCP.rer{rN)rset_optimizer_state_dictrr) r rJrrrrrrload)rrrrrrrs rload_optimizer_distributedr0sP $U + +HII/.j3EEHHk/ 0HOU/CDr)returnbool)robjectrzTypeGuard[DTensor])rr%)N)rKz str | None)NNN)r]zstr | dict[str, str] | Noner^z int | None)ror)rzdict[str, torch.Tensor])rstrrNone)# __future__rr&datetimertypingrrutilsrr r r get_logger__name__rCrr configuration_utilsrrrrrr"r)rOrTrcrsryrrrrrrrrs# +jj   H %06. $( . `/d $$dh,# (,#3=,#^>)6&B P Er