ˍjz0SSKrSSKJrJr "SS5rg)N) AcceleratorDistributedTypec v\rSrSrSrSrSrSS\S\RRS\ S\ 4S jjr S rS rS rg )LocalSGDu A helper class to support local SGD on top of Accelerator. It simply runs a given number of updates independently on each device, and averages model weights every K synchronization step. It should be used only in the multi-GPU (or multi-CPU) setup without extensions such as DeepSpeed. In particular, this is a simple implementation that cannot support scenarios such as model parallelism. Although we are not aware of the true origins of this simple approach, the idea of local SGD is quite old and goes back to at least: Zhang, J., De Sa, C., Mitliagkas, I., & Ré, C. (2016). [Parallel SGD: When does averaging help?. arXiv preprint arXiv:1606.07365.](https://huggingface.co/papers/1606.07365) We credit the term Local SGD to the following paper (but there might be earlier references we are not aware of). Stich, Sebastian Urban. ["Local SGD Converges Fast and Communicates Little." ICLR 2019-International Conference on Learning Representations. No. CONF. 2019.](https://huggingface.co/papers/1805.09767) cUR(a9URR5UlURR 5 U$N)enabledmodelno_syncmodel_sync_obj __enter__selfs ڢC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\accelerate/local_sgd.pyrLocalSGD.__enter__)s5 <<"&**"4"4"6D     ) ) + cUR(a-UR5 URRXU5 ggr )r _sync_and_avg_model_paramsr __exit__)rtypevaluetbs rrLocalSGD.__exit__0s0 <<  + + -    ( (b 9 r acceleratorr local_sgd_stepsr c UR[R[R[R[R [R [R[R[R[R[R4 ;a [S5eU=(a UR[R:gUl SUlUR(aXlX lX0lgg)ai Constructor. Args: model (`torch.nn.Module): The model whose parameters we need to average. accelerator (`Accelerator`): Accelerator object. local_sgd_steps (`int`): A number of local SGD steps (before model parameters are synchronized). enabled (`bool): Local SGD is disabled if this parameter set to `False`. zILocalSGD is supported only for CPUs and GPUs (no DeepSpeed or MegatronLM)rN)distributed_typerNO MULTI_CPU MULTI_GPU MULTI_XPU MULTI_MLU MULTI_HPU MULTI_SDAA MULTI_MUSA MULTI_NPU MULTI_NEURONNotImplementedErrorr num_stepsrr r)rrr rr s r__init__LocalSGD.__init__6s  ' '     % %  % %  % %  % %  % %  & &  & &  % %  ( ( 0 &&qr rU;#?#??CUCU#U  <<* J#2  rcU=RS- slUR(dgURUR-S:XaUR5 gg)zN This function makes a "step" and synchronizes model parameters if necessary. Nr)r*r rrrs rstep LocalSGD.stepXsD !||  >>D00 0A 5  + + - 6rc4URR5 URR5 URR 5H,nURR UR SS9UlM. SSS5 g!,(df  g=f)z8 Synchronize + Average model parameters across all GPUs mean) reductionN)rwait_for_everyoneautocastr parametersreducedata)rparams rr#LocalSGD._sync_and_avg_model_paramscsl **,    & & (..0!--44UZZ64R 1) ( (s A B  B)rr rr r r*N)T)__name__ __module__ __qualname____firstlineno____doc__rrrtorchnnModuleintboolr+r/r__static_attributes__rrrrsH*: 3K 3 3Z] 3hl 3D .Srr)r@ acceleraterrrrFrrrHs 3XSXSr