ICj: SSKrSSKrSSKrSSKrSSKrSSKrSSKJrJr SSK J r SSKJ r SSK J r SSKJrJr SSKJr SSKJrJrJr SS KJr SSKrSSKJr SSKJr SSKJs J r! SS K"J#r# SS K$J%r% SS K&J'r'J(r(J)r* SS K+J,r, SSK-J.r.J/r/ SSK0J1r1 SSK2J3r3J4r4J5r5 SSK6J7r7 SSK8J9r9J:r:J;r; SSKr>J?r? SSK@JArAJBrBJCrCJDrD SSKJErEJFrF SSKGJHrI SSKJJKrKJLrLJMrMJNrNJOrO SSKPJQrQJRrRJSrSJTrTJUrUJVrVJWrW SSKXJYrY \V(a)\Z"S\["S\RR555r^OSr^\T(aSr_Sr`\RR5r^O8\U(aSr_Sr`O,\W(aS r_S!r`\RR5r^OS"r_S#r`S$r^"S%S&\5rb"S'S(\5rc"S)S*\R\5reS+\RS,\RS-\ 4S.jrgS/rhS~S+\RS0\i4S1jjrjSS2jrkS3rlS4rmSS+\RS5\i4S6jjrnS+\RS7\i4S8jroS+\RS9\i4S:jrp"S;S<5rq"S=S>\e5rr"S?S@\e5rs"SASB\s5rt"SCSD\s5ru"SESF\e5rv"SGSH\v5rw"SISJ\R5rx"SKSL\s5ry"SMSN\R5rz"SOSP\R5r|"SQSR\R5r}\RSS\ 4STj5r\RSU\ 4SVj5r\RSW\ 4SXj5r\RSY\ 4SZj5r\RS[\ 4S\j5r\\RS]\ 4S^j55r\\RS_\ 4S`j55r\\RSa\ 4Sbj55r\\RSc\ 4Sdj55rSe\ S-\ Sf\Sg\4ShjrSSi\RSj\RSk\\Sl44Smjjr"SnSo\M5r"SpSq5r"SrSs\\L5r"StSu\\K5rSSv\S-4Swjjr"SxSy\R5r"SzS{\R5r"S|S}\R5rg)N)ABCabstractmethod)Callable) nullcontext)deepcopy)autoEnumwraps)Anycast no_type_check)mock) checkpoint) DeviceMesh) CPUOffload fully_shardFullyShardedDataParallel) TrainingState)FSDPParamGroupRegisterPostBackwardFunction)#NO_RESHARD_AFTER_FORWARD_STRATEGIES)BackwardPrefetchMixedPrecisionShardingStrategy)ShardedGradScaler)always_wrap_policyModuleWrapPolicywrap)distribute_tensorDTensorShard)ColwiseParallelparallelize_moduleRowwiseParallelSequenceParallel)TransformerDecoderLayerTransformerEncoderLayer)DistributedDataParallel)MultiProcContinuousTestMultiProcessTestCaseMultiThreadedTestCase run_subtests TEST_SKIPS) FILE_SCHEMAget_cycles_per_ms set_rng_seed TEST_CUDATEST_HPUTEST_WITH_ROCMTEST_XPU) has_tritoncudancclzhpu:0hcclxpuxcclcpuglooc0\rSrSr\"5r\"5rSrg) FSDPInitMode]N)__name__ __module__ __qualname____firstlineno__rNO_FSDP RECURSIVE__static_attributes__rDڱC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\torch/testing/_internal/common_fsdp.pyrBrB]sfGIrLrBc>\rSrSr\"5r\"5r\"5rSrg)DEVICEInitModefrDN) rErFrGrHr DEVICE_BEFORE DEVICE_AFTER DEVICE_NEVERrKrDrLrMrOrOfsFM6L6LrLrOc\rSrSrSr\S\\RS44Sj5r \S\R4Sj5r \S Sj5r \ \S \ S \ S\R4S j55rS rg) FSDPTestModelozVThis defines the interface expected from all models used commonly for FSDP unit tests.return.cg)z*Returns an input for the model as a tuple.NrDselfdevices rM get_inputFSDPTestModel.get_inputs rLcg)z,Returns the loss given the input and output.NrD)rZinputoutputs rMget_lossFSDPTestModel.get_lossxr^rLNcg)z named_buffersdistget_world_sizerangeall_gather_objectAssertionErrorzip)rvrwrx param_nameparamnamed_module_states buffer_namebuffer world_size_olist rank0_statesstatep1p2s rM_assert_module_statesrs3"'!7!7!9!9 J \\^'')*!9#(#6#6#8#8 K mmo))+,#8$$]3J , -,aT,E -5]K8LCDDqr = !@A A #L E GQWa b !F .s+D+D!* D'c6[R"[5$N)rqr[ DEVICE_TYPErDrLrM get_devtypers << $$rL zero_bufferscU(a[R"U5O [5nU UR5H1n[R "5 UR 5 SSS5 M3 U(aEUR5H1n[R "5 UR 5 SSS5 M3 SSS5 g!,(df  M=f!,(df  M_=f!,(df  g=f)zBZeros the parameters and optionally buffers of ``model`` in place.N)FSDPsummon_full_paramsr parametersrqno_gradzero_buffers)rvr summon_fullctxrrs rM _zero_modelrs -8$ ! !% ([]C %%'E !( --/]]_LLN%_*  %_ s;)C)C$;C)C0 C) C C) C& !C)) C7cU(dUR[5nU(aUR5 UR5$r)torhalf state_dict)rv cpu_offloadrs rM_get_state_dictrs. %     rLc pSRUVs/sHo"bU[U5OSPM sn5$s snf)Nrnone)joinstr)test_name_mappingriss rM subtest_namers9 88IMNAm 3q6 " ?N Ns3cUR5H=up#UR[R"S5:wdM+UR5X'M? US:XaUOS/n[R "U5 [ [[[R4US5nUHnXR[5X'M U$)Nr>r) itemsr[rqr>rbroadcast_object_listr dictrrrrr)rankrrrrs rM_broadcast_state_dictrs(--/ <<5<<. .%*YY[J "0 19Z$ /Eu%d3 ,-uQx8J !+!7!:!:;!G ! rLrecursec[R"XS9 [[UR 555sSSS5 $!,(df  g=f)a? Returns the full unsharded parameters of ``model``. Any FSDP-managed parameters offloaded to CPU are moved to GPU in the returned list. Args: recurse (bool): If ``False``, only unshards the parameters immediate to ``model``; if ``True``, recurses through the module hierarchy rooted at ``model``. )rN)rrrlistr)rvrs rMget_full_paramsrs4   8U--/01 9 8 8s "A Amove_to_devicec>U(aUR[5$U$r)rr)rvrs rM_move_to_devicers$2588K ==rL wrap_fsdpc2U(dU$[U/UQ70UD6$rr)rvrrirjs rM_maybe_wrap_fsdprs!5CtE'CD'CF'CCrLcH\rSrSrS\S\4SjrS\4SjrS\4SjrSrS r g ) DummyProcessGrouprsizecXlX lgr_rank_size)rZrrs rM__init__DummyProcessGroup.__init__s   rLrWcUR$r)rrZs rMrDummyProcessGroup.rank zzrLcUR$r)rrs rMrDummyProcessGroup.sizerrLcB[R"5nSnXClU$)Ncd[RR5nURS5 U$)Nr@)rqfuturesFuture set_result)futures rM get_future/DummyProcessGroup.allreduce..get_futures'+0==+?+?+AF   a MrL)rMockr)rZrirj dist_waitrs rM allreduceDummyProcessGroup.allreduces IIK   *rLrN) rErFrGrHintrrrrrKrDrLrMrrs2Scc rLrc^\rSrSrS\R S\S\S\4U4SjjrSr Sr S r S r \ SS\R S \S\S \\\4S -S\S\S\R&\-4Sjj5rSrSrU=r$)TransformerWithSharedParamsi r{device_init_modeadd_bn deterministicc>[TU]5 UR5UlUR5UlU(a[ R "S5 SnSn[R"XV5Ul [R"USSSSS9Ul [R"Xe5Ul URRURlURSURRR!U455 URS [ R""UR$[ R&S 95 SUlU(a)[ RR+UR(5O[ RR-5UlU[0R2:XaUR5[65nU(aUR95 gg) Nrr8g?)d_modelnum_encoder_layersnum_decoder_layersdim_feedforwarddropout vocab_bias long_buffer)dtype)superrrrrrq manual_seedrt Embedding embed_tokens Transformer transformerLinear output_projweightregister_buffernew_ones zeros_likerlongbs BatchNorm1dIdentitybnrOrQrreval)rZr{rrrd_vocabr __class__s rMr$TransformerWithSharedParams.__init__ s] JJL **,    a LL:>>    99W6#'"3"3":":  $++22;;WJG      T__EJJ ? 39%((&&tww/uxx?P?P?R ~;; ;77;'D  IIK rLc$[R"SUR-5 [R"SUS9R SUR 5n[R"UR S-US9R SUR 5nX#4$)Nr@ r[r7)rqrrarangeviewr)rZr[srctgts rMr\%TransformerWithSharedParams.get_input7sj !dii-(ll2f-221dgg>ll477Q;v6;;AtwwGzrLcURU5nX0R-URRU5-nURU5nUR U5nUR X45nUR U5$r)rrrtype_asrrr)rZsrc_idstgt_idsr r xs rMforward#TransformerWithSharedParams.forward=sr(OO#d&6&6&>&>s&CC(ggcl   S &""rLcUup4[RRURSUR S55URS5SS9$)Nsum) reduction)rt functional cross_entropyr r)rZr`rarr s rMrb$TransformerWithSharedParams.get_lossEsG}}** KKFKKO ,chhrle+  rLc$UR5 grbackwardres rMrg(TransformerWithSharedParams.run_backwardK  rLNfsdp_init_mode fsdp_kwargsrWcxUc0nU[R:Xa)[U[5(aUSnOUn[ XbXT5$U[R :XaSU;a[ [[15nOURS5nSU;a?US[R[R1;a[U[5(dSnOUn[U[5(aUSn OUn [ XXT5n [U U4SU0UD6n U[R:XaU R!["5n U $[%SU35e)a Initializes a :class:`TransformerWithSharedParams` instance. Args: fsdp_init_mode (FSDPInitMode): If ``NO_FSDP``, then does not wrap any modules with FSDP. If ``RECURSIVE``, then wraps with top-level FSDP. By default, the top-level FSDP uses the ``ModuleWrapPolicy`` for encoder and decoder layers, but a different auto wrap policy may be specified via ``fsdp_kwargs``. device_init_mode (DEVICEInitMode): Determines model movement to DEVICE. fsdp_kwargs (Optional[Dict[str, Any]]): Optional keyword arguments forwarded to the FSDP constructor. deterministic (bool): Whether to make the model deterministic across constructions. add_bn (bool): Whether to include batch norm in the model. Nrauto_wrap_policysharding_strategyUnsupported FSDP init mode: )rBrI isinstancerprrJrr(r'popr HYBRID_SHARD_HYBRID_SHARD_ZERO2rrOrRrr ValueError) r{r rr!rrpgr#fsdp_pg tformer_pgm fsdp_models rMrl TransformerWithSharedParams.initNsW6  K \11 1%''1X.f |55 5!4#3//$ $/??3E#F ${2 34$113C3W3WXY"5%00%''"1X " +fA"2 J  >#>#>>']];7  77GHIIrLcUR/$r)rrs rMget_ignored_modules/TransformerWithSharedParams.get_ignored_moduless  !!rL)rrrrrrr)NFT)rErFrGrHr ProcessGrouprOboolrr\rrbrgrsrBrrr rtrurrlr2rK __classcell__rs@rMrr s(  ()( (  (T #  .2# KJ  KJ$KJ)KJ#s(^d* KJ  KJ  KJ T KJKJZ""rLrc^\rSrSrS\R S\S\S\4U4SjjrSr Sr S r S r \ SS\R S \S\S \\\4S -S\S\R&4 Sjj5rSrU=r$)NestedWrappedModuleir{rrrc >^^^[TU]5 TR5UlTR5UlU[ R :HnUUU4SjnU(a[R"S5 [R"[[R"SS5U5U"[R"U"[[R"SS5U55[[R"SS5U555U"[[R"SS5U55[[R"SS5U55Ul g)Nc0>T(a [UT40TD6$U$rrlayerr!r{rs rM _maybe_wrap1NestedWrappedModule.__init__.._maybe_wrapE58K88LrLrrr7r)rrrrrrOrQrqrrt Sequentialrrmodule rZr{rrrr!rr>rs `` ` rMrNestedWrappedModule.__init__s JJL **,)^-I-II     a mm BIIaO^ <   !R0@. QR#BIIb"$5~F   "a(8.I J BIIaO^ <   rLcv[R"SUR-5 [R"SSUS94$)Nr@r7rr)rqrrrandrYs rMr\NestedWrappedModule.get_inputs. !dii-( 1a/11rLc$URU5$rrBrZrs rMrNestedWrappedModule.forward{{1~rLc&UR5nU$r)rrZr`rarfs rMrbNestedWrappedModule.get_lossszz| rLc$UR5 grrres rMrg NestedWrappedModule.run_backwardrrLNr r!rWcUc0nU[R:Xa [USUUS9$U[R:Xa;[U4SUUS.UD6nU[R :XaUR [5nU$[SU35e)a Initializes a :class:`NestedWrappedModule` instance. Args: fsdp_init_mode (FSDPInitMode): If ``NO_FSDP``, then does not wrap any modules with FSDP. If ``RECURSIVE``, then wraps some nested modules with FSDP but not the top-level module. The model may later be wrapped with a top-level FSDP external to this method if desired. device_init_mode (DEVICEInitMode): Determines model movement to DEVICE. fsdp_kwargs (Optional[Dict[str, Any]]): Optional keyword arguments forwarded to the FSDP constructor. deterministic (bool): Whether to make the model deterministic across constructions. FrrrTr%) rBrIr9rJrOrRrrr*)r{r rr!rr/s rMrlNestedWrappedModule.inits.  K \11 1&!1+   |55 5,!1+   J >#>#>>']];7  77GHIIrLrBrrNF)rErFrGrHrr4r5rOrr\rrbrgrsrBrrr rtrurlrKr6r7s@rMr9r9s     )    @2 .2# +J  +J$+J)+J#s(^d* +J  +J  +J+JrLr9c v^\rSrSr\S S\R S\S\S\ \ \ 4S-S\ 4 U4Sjjj5r S rU=r$) AlwaysWrapNestedWrappedModuleiNr{r rr!rc:>[[[] U[RUUUS9nU[R:XaU$U[R :XaHU=(d 0n[ U4S[0UD6nU[R:XaUR[5nU$g)z Initializes a :class:`NestedWrappedModule` instance, but unlike :meth:`NestedWrappedModule.init`, for the ``RECURSIVE`` init mode, this wraps with top-level FSDP and the ``always_wrap_policy()`` auto wrap policy. )r{r rr!rr#N) rrXrlrBrIrJrrrOrRrr)r{r rr!rrvr/rs rMrl"AlwaysWrapNestedWrappedModule.inits )+H '//-#'    \11 1L |55 5%+KeX6HXKXJ>#>#>>']];7   6rLrDrV)rErFrGrHrsrr4rBrOrrr r5rlrKr6r7s@rMrXrXs^ .2#   $)#s(^d*   rLrXc ^\rSrSrS\R S\S\S\4U4Sjjr\ S Sj5r \ SS\R S \ S\S \ \ \4S-S\4 S jj5rS rU=r$)NonUniformReqGradNWMi r{rrrc >^^^[[U] 5 TR5UlTR 5UlU[ R:HnUUU4SjnU(a[R"S5 [R"[[R"SS5U5U"[R"U"[[R"SS5U55[[R"SS5U555U"[R"[[R"SS5U5[[R"SS5U5555Ulg)Nc0>T(a [UT40TD6$U$rrr<s rMr>2NonUniformReqGradNWM.__init__.._maybe_wrap4r@rLrrr7r)rr9rrrrrOrQrqrrtrArrrBrCs `` ` rMrNonUniformReqGradNWM.__init__!s !413JJL **,)^-I-II     a mm BIIaO^ <   !R0@. QR#BIIb"$5~F   #BIIb!$4nE#BIIaO^D   rLNcUR5H3up#[R"X5(aM"URS5 M5 grV)r~rematchrequires_grad_)rv req_grad_masknps rM_set_nonuniform_req_grad-NonUniformReqGradNWM._set_nonuniform_req_gradKs4**,DA88M--  '-rLr r!c[R"S5nU[R:Xa#[ USUUS9n[R Xe5 U$U[R :XaUUc0n[ U4SUUS.UD6nU[R:XaUR[5n[R Xu5 U$[SU35e)a Initializes a :class:`NestedWrappedModule` instance, but unlike :meth:`NestedWrappedModule.init`, it wraps a second :class:`torch.nn.Sequential` container to enable the desired non-uniform ``requires_grad`` ``use_orig_params=True`` tests. For both ``RECURSIVE`` and ``NO_FSDP`` init modes, freezes all parameters except the last two to validate ``ShardedGradScaler`` support for ranks with no (non-zero sized) local shards in FSDP ``use_orig_params=True`` mode. zmodule\.2.*\.1.*FrSTr%) rbcompilerBrIr\rhrJrOrRrrr*)r{r rr!rreq_grad_pattern ddp_modelr/s rMrlNonUniformReqGradNWM.initQs(::&9: \11 1,!1+ I ! 9 9) V  |55 5" -!1+   J >#>#>>']];7 9 9* W 77GHIIrLrUrnrV)rErFrGrHrr4r5rOrrsrhrBrrr rlrKr6r7s@rMr\r\ s(   ( ( ) (  ( T((  .2# +J  +J$+J)+J#s(^d* +J  +J+JrLr\c ^\rSrSrSrS\R S\S\4U4SjjrSr Sr S r S r \ S \\S \S\S\S \4 Sj5rSrU=r$)ModuleWithDelayizThis class wraps a :class:`FSDPTestModel` to optionally add a delay after computing the loss and/or before the gradient reduction.rBdelay_after_loss_msdelay_before_reduction_mscF>[TU]5 X lX0lXlgr)rrrqrrrB)rZrBrqrrrs rMrModuleWithDelay.__init__s! #6 )B& rLc8URRU5$r)rBr\rYs rMr\ModuleWithDelay.get_inputs{{$$V,,rLc$URU5$rrIrJs rMrModuleWithDelay.forwardrLrLcbURRX5nURS:a[(d [(a%[ R "URS- 5 U$[(a=[RR[UR[5-55 U$Nr) rBrbrqr3r5timesleepr2rqr9_sleeprr0rNs rMrbModuleWithDelay.get_losss}{{##E2  # #a 'x88 433d:;  !!#d&>&>ARAT&T"UV rLc^^[RRmUU4Sjn[R"SU5 TR R U5 SSS5 g!,(df  g=f)Nc8>TRS:a[(a>[RR [ TR[ 5-55 O9[(d [(a#[R"TRS- 5 T"U0UD6$rz) rrr2rqr9r~rr0r3r5r|r})rirjorig_reduce_scatterrZs rM_delayed_reduce_scatter=ModuleWithDelay.run_backward.._delayed_reduce_scattersq--19JJ%%D::=N=PPQXJJt==DE&77 7rLz'torch.distributed.reduce_scatter_single)rq distributedreduce_scatter_singlerpatchrBrg)rZrfrrs` @rMrgModuleWithDelay.run_backwardsR#//EE 8ZZ 57N  KK $ $T *   s A  A. module_class model_args model_kwargsc<[UR"U0UD6UU5$)a Args: module_class (Type[FSDPTestModel]): Wrapped module class to which to add delays. model_args: Positional arguments forwarded to the ``module_class`` ``init()``. delay_after_loss_ms (int): Delay after computing the loss/before the optimizer step (in ms). delay_before_reduction_ms (int): Delay before reduce-scattering gradients (in ms). model_kwargs: Keyword arguments forwarded to the ``module_class`` ``init()``. )rprl)rrqrrrrs rMrlModuleWithDelay.inits**   z :\ :  %  rL)rqrrrB)rErFrGrHrortrurrr\rrbrgrstyperUr rlrKr6r7s@rMrprpsF   ! $' -+$ =)  ! $'     rLrpc\rSrSr\\R SSSS4S\RS\ S\S\ \ \ 4S-S \ S \S \4S jj5rS rg)NestedWrappedModuleWithDelayiNFrr{r rr!rrqrrc >[R[UUUUUUUS9$)Nr{r rr!rrqrr)rprlr9rs rMrl!NestedWrappedModuleWithDelay.inits4## )-#' 3&?$  rLrD)rErFrGrHrsrOrRrr4rBrrr r5rrlrKrDrLrMrrs,:+F+F-1##$)*    $ ) #s(^d*    !  $'  rLrc.^\rSrSrU4SjrSrSrU=r$)DummyDDPic.>[TU]5 Xlgr)rrrB)rZrBrs rMrDummyDDP.__init__s  rLc&UR"U0UD6$rrIrZrirjs rMrDummyDDP.forwards{{D+F++rLrIrErFrGrHrrrKr6r7s@rMrrs,,rLrc^\rSrSrS\R S\S\S\S\4 U4Sjjr Sr S r \ SS\R S \ S\S \\\4S -S\S\4 S jj5rSrU=r$)MixtureOfExpertsir{rrdelay_before_free_msrc >[TU]UUUUS9 XlX@lX lU[ R :HUlU(a#[R"SUR-5 SnSnSn [[R"Xx5UR5n [SU R555UlU R5H n SU lM U(a[R"S5 [[R"X5UR5n U(aF[R$R'UR5/5n [)X40UD6n [)X40UD6n [R*"[[R"X5UR5U U [[R"X5UR55Ulg) N)r{rrr*rrrc3@# UHoR5v M g7fr)numel).0rgs rM ,MixtureOfExperts.__init__.. s$L8K1WWYY8KTr)rrr{rrrOrQrrqrrrrtrrrnum_expert_paramsexpertr new_grouprrArB)rZr{rrrrr!d_expertd_sharedd_inputrrgshared expert_grouprs rMrMixtureOfExperts.__init__s{ -'   $8!"..2N2NN    b499n - 8!>@S@ST!$$L8I8I8K$L!L""$AAH%    a  8!>@S@ST  ,,66L&>+>F&7;7Fmm BIIg8$:M:M N   BIIh8$:M:M N   rLc^^TRS:aTRSn[U[5(ag[R R RRmUU4Sjn[R"SU5 TRU5sSSS5 $TRU5$!,(df  N=f)Nrr8c>[(a>[RR[ TR [ 5-55 O9[(d [(a#[R"TR S- 5 T"U0UD6$)Nr{) r2rqr9r~rrr0r3r5r|r})rirj orig_reshardrZs rM_delayed_reshard2MixtureOfExperts.forward.._delayed_reshard(sc y )) 9 9 {{1~ s B// B=cUR5 UR(d[R"5 UR 5H|n[ US5(aMUR cM%UR RUR5 [RRUR URS9 M~ SSS5 gg!,(df  g=f)Nrrz) rrrqrrhasattrgraddiv_rr all_reducer{)rZrfrgs rMrgMixtureOfExperts.run_backward:s ~~*Aq(++ vv) DOO4))44QVV4::4N +! s2C-AC C Nr r!c Uc0nU[R:Xa [USUUUS9$U[R:Xa<[U4SUUUS.UD6nU[R :XaUR [5nU$[SU35e)a Initializes a :class:`MixtureOfExperts` instance. Args: fsdp_init_mode (FSDPInitMode): If ``NO_FSDP``, then does not wrap any modules with FSDP. If ``RECURSIVE``, then wraps some nested modules with FSDP, including the expert and shared layers, but not the top-level module. The model may later be wrapped with a top-level FSDP external to this method if desired. device_init_mode (DEVICEInitMode): Determines model movement to DEVICE. fsdp_kwargs (Optional[Dict[str, Any]]): Optional keyword arguments forwarded to the FSDP constructor. deterministic (bool): Whether to make the model deterministic across constructions. delay_before_free_ms (int): Delay before resharding expert parameters in the forward pass (in ms). F)rrrrTr%) rBrIrrJrOrRrrr*)r{r rr!rrr/s rMrlMixtureOfExperts.initFs4  K \11 1#!1%9+  |55 5)!1%9+   J >#>#>>']];7  77GHIIrL)rr{rBrrr)NFr)rErFrGrHrr4r5rOrrrrgrsrBrrr rlrKr6r7s@rMrrs2   2 2 ) 2 " 2  2 h0 O .2#$% 0J  0J$0J)0J#s(^d* 0J  0J " 0J0JrLrc ^\rSrSrSSSSS.S\S\R S-S \S \S \4 U4S jjjjrS \RS\R4Sjr Sr Sr U=r $)MLPizNTFr7)bias with_bufferdim_multiplierdimr[rrrc>[TU]5 [R"XU-X#S9Ul[R"XQ-XUS9UlU(a'UR S[R"U4US95 gSUl g)N)r[rrr) rrrtrin_projout_projrrqrandnr)rZrr[rrrrs rMr MLP.__init__{sf yys&:6U  ."6QUV   5;;vf+M NDKrLrrWcURU5n[R"U5nURU5n[R"U5nURbX R-nU$r)rFrelurr)rZrzs rMr MLP.forwardsQ LLO FF1I MM!  FF1I ;; "KKArLcURb4[RRR UR5 ggr)rrqrtrlnormal_rs rMreset_parametersMLP.reset_parameterss+ ;; " HHMM ! !$++ . #rL)rrrr)rErFrGrHrrqr[r5rrrrrrKr6r7s@rMrrzs'+ !  t#    "%,,//rLrcV^\rSrSrSS.S\S\4U4SjjjrS\S\S \S S4S jrSr U=r $) MLPStackiF)with_seq_parallelmlp_dimrc>[USS9[U5[USS9/nU(a$UR[R"USS95 [TU]"U6 X lg)N)rFr)rappendrt LayerNormrrr)rZrrmodulesrs rMrMLPStack.__init__sV  * L  * $   NN2<<e< = '"!2rLtp_meshdp_meshuse_activation_checkpointingrWc [SS9[SS9[SS9[SS9[SS9UR(a[[S5S9O [5S.nUR(a [ SS9US'[ XUS9 UHDn[ U[R5(aM$U(a [U5 [U4S U0UD6 MF [U4S U0UD6 U$) NF)use_local_outputr@)output_layouts)z 0.in_projz 0.out_projz 1.in_projz 1.out_projz 2.in_projz 2.out_proj) sequence_dim3) device_meshparallelize_planmesh) r#r%rr"r&r$r&rtrrr)rZrrrr!rrBs rM parallelizeMLPStack.parallelizes)%@)5A(%@)5A(%@%%*qB "    ! !$4!$D S !4GWXF&",,//+6"  [TU]5 [R"X5Ul[R"X5Ul[R "5UlX lgr) rrrtrlin1lin2ReLUrr)rZrrrs rMrDoubleLinear.__init__sA IIc' IIc' GGI !2rLrrWcUR(a@URURU55URURU554$URURU55$r)rrrrrJs rMrDoubleLinear.forwardsQ  ! !99TYYq\*DIIdiil,CC Cyy1&&rL)rrrrT)rErFrGrHrorr5rrqrrrprrKr6r7s@rMrrsY 3C3D33'' u||U\\) *U\\ 9''rLrnew_all_gather_into_tensorc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)rall_gather_singlebarrier)rorig_all_gathers rMpatch_all_gatherrsN,,OLLN7D1  !0 !01A>A!A>"A;;A>new_foreach_all_gatherc#@# [RRRRR n[ R"5 U[RRRRlSv [ R"5 U[RRRRlg![ R"5 U[RRRRlf=f7fr)rqrr _fully_shard_fsdp_param_groupforeach_all_gatherrr)rorig_foreach_all_gathers rMpatch_foreach_all_gatherrs ++==PP LLN ''99L    # ++==P  # ++==P BDCA DA DDnew_foreach_reducec#@# [RRRRR n[ R"5 U[RRRRlSv [ R"5 U[RRRRlg![ R"5 U[RRRRlf=f7fr)rqrrrrforeach_reducerr)rorig_foreach_foreach_reduces rMpatch_foreach_reducer s ++==LL  LLN ''99H    ' ++==L  ' ++==Lrnew_reduce_scatter_tensorc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)rrr)r rs rMpatch_reduce_scatterrsO44LLN!:D9  %8" %8"rnew_all_reducec## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)rrr)rorig_all_reduces rMpatch_all_reducersIooOLLN$DO*  ) )r new_unshardc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)runshardrr)r orig_unshards rM patch_unshardr*P"))LLLN(N.  !- !-r new_reshardc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)rreshardrr)rrs rM patch_reshardr7rrnew_post_backwardc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)r post_backwardrr)rorig_post_backwards rMpatch_post_backwardr!DsQ(55LLN#4N :  '9$ '9$r new_backwardc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)rrrr)r" orig_backwards rM*patch_register_post_backward_hook_backwardr%QsS199MLLN,8 )>  0=$- 0=$-rrrirjc[U5S:aUSnOSU;aUSnO[SUSU35eU"U5 U"U0UD6$)Nrraz,Cannot get reduce-scatter output from args: z kwargs: )lenr)clsrrxrirjras rMreduce_scatter_with_assertr)^sa 4y1}a V !;D6F8 T  f  / //rLreplicated_modulesharded_moduleprefixes_to_ignore.c[UR5UR5SS9GHuupEupgUnUHn URU S5nM URXH5 UR U[ 5 [ U[ 5(d [S5eURURp[U 5[S5[S54:Xa [S5e[XZU 5n URUR5U R55 URcURUR5 GMUR!UR5 [URX5n UR UR[ 5 [ UR[ 5(d [S5eURURR5U R55 GM g)NTr|z&Expected sharded_param to be a DTensorrzmFSDP's (Shard(0), Shard(0)) layout differs from distribute_tensor(), so we cannot check for equality using itz+Expected sharded_param.grad to be a DTensor)rr~replace assertEqualassertIsInstancer!r&rr placementsrpr"r to_localr assertIsNoneassertIsNotNone)r(r*r+r,replicated_namereplicated_param sharded_name sharded_paramclean_sharded_nameprefixrr2sharded_ref_paramsharded_ref_grads rMcheck_sharded_parityr>qs OR**,'')OJ+-Jl *(F!3!;!;FB!G ) < ]G4-11 !IJ J(44m6N6Nj  q58 4 4 ; ..>jQ  ..02C2L2L2NO  (   ]// 0  M../,-=-B-BDU ]//9-,,g66 !NO O  **3357G7P7P7RS9OrLcJ^\rSrSr\S5rU4SjrSrSrSr Sr U=r $)FSDPTestMultiThreadic[$r DEVICE_COUNTrs rMrFSDPTestMultiThread.world_sizerLcB>[TU]5 UR5 gr)rsetUp_spawn_threadsrZrs rMrGFSDPTestMultiThread.setUps   rLc [U/UQ70UD6$rr-rs rMr- FSDPTestMultiThread.run_subtestsD242622rLc@[RR5 grrq_dynamoresetrs rMperThreadSetUp"FSDPTestMultiThread.perThreadSetUp rLc@[RR5 grrPrs rMperThreadTearDown%FSDPTestMultiThread.perThreadTearDownrUrLrD) rErFrGrHpropertyrrGr-rSrWrKr6r7s@rMr@r@s. 3rLr@c#d\rSrSrSrSrSrSrSr\ S5r S$S \ RS \ S \S \S\S-S\S\S-S\S\S\\\4S-4SjjrSSS\"5SSSS S S S SS4 S\\S\S\S\S-S\ S\S\S\S-S\S-S\S-S\S \S\S\S!\\\4S-S\\\4S-4 S"jjrS#rg)% FSDPTestMixiniz Mixin class containing shared test utilities for FSDP tests. Provides common helper methods for both FSDPTest and FSDPTestContinuous. c:URX!R5 gr)r0r)rZr/rs rM_check_cpu_offload FSDPTestMixin._check_cpu_offloads &<&<=rLc:URX!R5 gr)r0backward_prefetch)rZr/r`s rM_check_backward_prefetch&FSDPTestMixin._check_backward_prefetchs *,H,HIrLc:URX!R5 gr)r0forward_prefetch)rZr/rds rM_check_forward_prefetch%FSDPTestMixin._check_forward_prefetchs )+F+FGrLc [U/UQ70UD6$rrLrs rMr-FSDPTestMixin.run_subtestsrNrLc U"U5nXlX6lURSS5n[SURSUR35 [ S:wa`[ RR5UR:a4[R"[SUR3R5 U(a^[ RRRR R#5n[$R&"SURUUS9 OC[$R&"UR([*[-UR5URS 9 Sn UR[2-n [4(d [6(a[ RR9U 5 U /n [$R:"U S 9 [ R<R?5 [A5 URCX$5 [ R<R?5 [$R:"U S 9 [$RD"5 g![.a@n S U R0S ;a'[R"[S R5 eSn A ff=f)Nfake_pgF dist init r=, world=r> multi-gpu-fakebackendrrstore init_methodrprr recompilerbackend_unavailable device_ids)#r file_namegetprintrrrq accelerator device_countsysexitr. exit_codetesting _internalrrj FakeStorerinit_process_grouprsDISTRIBUTED_BACKENDr RuntimeErrorrirCr2r5set_device_indexrrQrRr1run_testdestroy_process_group r(r test_namerxpiperjrZrjrqerw device_ids rM_runFSDPTestMixin._runs9~ "**Y. TYYKx/@AB % E$5$5$B$B$Dt$V HHZ*T__,= >?II J  //;;CCMMO''"# '' $ 0 0/"4??3  II , 9    . .y 9[ +  i&  + ""$1 affQi'$9:DDE   s 7A$IAI J  ;JJ NFrv num_stepsautocastlrfsdp_cpu_offload save_modelmixed_precisionenable_sharded_grad_scaler use_pure_fp16sharded_grad_scaler_kwargsc U=(a URn [UR55Rn U c0n [ S SU0U D6n [ R RUR5USS9n[U5GHtnUR5 [ RR[US9 URR[ R"[55nU (dU(aW[U[ 5(dB[U[ R"5(aUR%5nO['SU55nU"U6nU (ap[U[ 5(a[UR([*;aGUR5H3nUR-UR[ R"S55 M5 URR/UU5R1U 5nSSS5 U R3W5nU(d1U (d*UR4[ R6:wa [9S5eOU (a+UR-UR4[ R:5 Ot[U[ 5(a5Uc [9S5eUR-UR4UR<5 O*UR-UR4[ R65 URR?U5 U (a\[U[ 5(aGUR5H3nUR-UR[ R"S55 M5 U RAU5 U RC5 U(dGMURE5RG5VVs0sHunnUURI5_M nnn[KU5 URMU5 GMw [U[ 5(aURO[PRR5 WRU5$!,(df  GNN=fs snnf) Nenabledg?)rmomentum)rc3@# UHoR5v M g7fr)r)rrs rMr9FSDPTestMixin._train_for_several_steps..s%>1ffhhrr>zQloss data type should be float32, as the original parameter data type is float32.z'Expected mixed_precision to not be NonerD)+offload_paramsnextrr[rrqoptimSGDr zero_gradamprrrBr\r&rrrrrpr$rr0rbrscalerfloat32rfloat16 param_dtypergstepupdaterrclonerload_state_dict _assert_staterIDLEr)rZrvrrrrrrrrrcpu_offload_params model_devicesharded_grad_scalerrrr`rargrfkvrs rM_train_for_several_steps&FSDPTestMixin._train_for_several_stepss@.Q2B2Q2QE,,./66 % -)+ &/ . 2L    0 0 2rCHy!A OO ##K#B ..u||K/HI _Zt=T=T!%66 %  %%>%> >'"5$////>?#--/((5<<3FG0||,,UF;>>|L-C.',,T2D"=::.(:/ !$$TZZ?t,,&.,E$$TZZ1L1LM$$TZZ? LL % %d +!j&=&=))+A$$QXXu||E/BC, $ $U +  & & (z7<7G7G7I7O7O7QR7Qtq!al7Q RE"%%j1{"~ eT " "    2 2 3{{}CBnSs9EQ:Q Q r8T model_classr r ref_init_fn num_itersrr`r$rduse_orig_params init_kwargsc @U[R:Xa [S5eUc0nSnURR 5nUR "UR[R[ R4SS0UD6nUcC[(a[U[/[S9nO+[S:Xa [U5nO[UU/US9nOU"U5nU(aUR5nURUUU SLUUU U UUS9 n[UR55nURUUU U U U S .5 UR "URUUU4SS0UD6n['U[(5(d[)UUR40UD6nU(aUR5nU[ R*:XaUR-[5nUSL=(a UR.nU=(a U[ R*:HnU=(a U[ R*:gnU(aI[0R2"S5nUR5HnUR5UR2U5 M! U(aUR7[8S [35O [;5nU URUUS UUUU U UUS9 n SSS5 U(agU(a^[0R2"S5nUR5HnUR5UR2U5 M! W R-[5n [=U5n![0R>RAUW S S9 U cU(dUR5UU!SSS9 ggg![ a n[#S US [%U535UeSnAff=f!,(df  N=f)a Tests FSDP training against a reference, which defaults to DDP but may be customized with ``ref_init_fn``. Args: model_class (Type[FSDPTestModel]): A model class that inherits from ``FSDPTestModel``, which defines the expected interface. fsdp_init_mode (FSDPInitMode): The mode to initialize the FSDP-wrapped model. This should not be ``NO_FSDP``. ref_init_fn (Optional[Callable]): A callable to invoke that wraps a non-wrapped model to construct the reference model, where this wrapper should provide data parallel semantics. If ``None``, then the callable defaults to the DDP constructor. z.Expects an FSDP init mode that wraps with FSDPN{Gz?rT)rw output_devicer>)rrrrrrr)rr`r$rrdrz Initializing z raised error zOAn FSDP-managed module with parameter CPU offloading enabled has parameters on F)rrrrrrrr) check_dtypezFSDP did not match DDP) exact_devicemsg)!rBrIrrwrrlrOrQr3DDPrrrrrr Exceptionr*rr&rrRrrrqr[r0assertRaisesRegexrrrr assert_close)"rZrr rrrrrr`r$rrdrrrrrr!rrrv ref_modelref_loss ddp_paramsr/rrexpects_device_errorexpects_cpu_device cpu_devicercontext fsdp_lossfsdp_unsharded_paramss" rM_test_fsdp_parityFSDPTestMixin._test_fsdp_parityMsD \11 1 !QR R  K !!&&(       ( (       x{m; %J 4&M #E*I !(I00  $D0(+'A''A1  )..01 *%6%6#2$4#2    Y$))""   #   J*d++j$*<*<L LJ #*J ~:: :#{3J$D0O[5O5O  N/>3N3NN   N/>3N3NN  e,J#..0  z:1$  " "%%0M3   55!,% /+E++E6 I  e,J#..0  z:1! [1I / ;  ""8YE"J  "=   %!,  ,9 "K Y}[MAxPQWX X YFWs$."M" N" N ,NN  Nrxr)rNFNFFN)rErFrGrHror]rarer- classmethodrrtrurr5floatrrrrr rrrUrBrOrrrrrKrDrLrMr[r[s >JH34%4%v.2 15+0#<@YyyYY Y  Y %t+ YY($.Y%)YY%)cNT$9Y@(,",,595915!& %+0#-1<@#h-(h%h) h _ h  hh h,d2h,d2h($.hhh%)hh #s(^d*!h"%)cNT$9#hhrLr[c^\rSrSrU4Sjr\S5r\S5r\S\4Sj5r \S5r \ S5r S r U=r$) FSDPTestich>[TU]5 S[RS'UR 5 g)N0TORCH_NCCL_DESYNC_DEBUG)rrGosenviron_spawn_processesrIs rMrGFSDPTest.setUps)  14 ,- rLc[$rrBrs rMrFSDPTest.world_sizerErLc>[RR5$r)rdistributed_c10d_get_default_grouprs rMrwFSDPTest.process_groups$$7799rLrWcgrVrDrs rMdestroy_pg_upon_exitFSDPTest.destroy_pg_upon_exit srLc*[UR3$r)r/rxrs rMrsFSDPTest.init_methodst~~.//rLc U"U5nXlX6lURSS5n[SURSUR35 [ R R5UR:a4[R"[SUR3R5 U(a^[ RRRRR!5n["R$"SURUUS9 OC["R$"UR&[([+UR5URS9 Sn UR[0-n [2(d [4(a[ R R7U 5 U /n ["R8"U S 9 [ R:R=5 [?5 URAX$5 [ R:R=5 ["R8"U S 9 ["RB"5 g![,a@n S U R.S ;a'[R"[S R5 eSn A ff=f) NrjFrkrlrmrnrorrrtrrurv)"rrxryrzrrqr{r|r}r~r.rrrrrjrrrrsrrrrirCr2r5rrrQrRr1rrrs rMr FSDPTest._runs9~ "**Y. TYYKx/@AB    ) ) +doo = HHZ*T__,= >?II J  //;;CCMMO''"# '' $ 0 0/"4??3  II , 9    . .y 9[ +  i&  + ""$1 affQi'$9:DDE   s -A$H7AH77 J;I<<Jr)rErFrGrHrGrYrrwr5rrsrrrKr6r7s@rMrrss ::d004%4%rLrc^\rSrSr%Sr\r\\S'\ S\ 4Sj5r \ S\ 4Sj5r \ U4Sj5r U4SjrU4S jr\S 5rS rU=r$) FSDPTestContinuousiJz FSDP test base class using MultiProcContinuousTest for faster test execution. This class reuses worker processes across tests, reducing process spawn overhead. Use this for tests that don't require fresh process state between tests. rrWc[$r)rr(s rM backend_strFSDPTestContinuous.backend_strSs""rLc[$r)rrs rM device_typeFSDPTestContinuous.device_typeWsrLc`>S[RS'[RR 5U:a*[ R "[SU3R5 U[-n[(d [(a[RRU5 [TU]9XU5 g)Nrrrm)rrrqr{r|r}r~r.rrCr2r5rr_init_pg)r(rr rdvz_filerrs rMrFSDPTestContinuous._init_pg[s 14 ,-    ) ) +j 8 HHZ*ZL 9:DD E<' 9    . .y 9 95rLc>[TU]5 URUR:wa[R "5 [ RR5 [5 gr) rrGrMAIN_PROCESS_RANKrrrqrQrRr1rIs rMrGFSDPTestContinuous.setUpks>   99.. . LLN rLc>URUR:wa[R"5 [TU]5 [ RR5 gr) rrrrrtearDownrqrQrRrIs rMrFSDPTestContinuous.tearDownus9 99.. . LLN  rLc.URR$r)rr+rs rMrw FSDPTestContinuous.process_group|s~~   rLrD)rErFrGrHrorCrr__annotations__rrrrrrGrrYrwrKr6r7s@rMrrJsz #J"#C##C 6 6!!rLrcompile_compute_on_modulecD^^^U4Sjm"SS[5mUU4SjnU$)Nc>[RRR"U0UD6 Tb[ UST5(aUSR 5 gg)Nr)rqrrrr&rk)rirjrs rM!fully_shard_with_compiled_compute=compiled_fsdp_test..fully_shard_with_compiled_computesS **D;F; $ , G.1 1  GOO 1 rLc0\rSrSr\"5r\"5rSrg)*compiled_fsdp_test..FullyShardModeirDN)rErFrGrHrEAGERCOMPILED_COMPUTErKrDrLrMFullyShardModers6rLrc6>^[T5UUU4Sj5nU$)Nc>[RRRnTGHFnUTR:wa&[ 5(d[ R"SSS9 M:[RRRn[RR5 UTR:XaUnO@UTR:Xa"S[RRl TnO[SU35eUTRUR'T"U0UD6 [RR5 UTRUR'U[RRl GMI g)Nz0Inductor on GPU needs Triton and recent GPU archr8) stacklevelr@z!Need to implement FullyShardMode=)rqrrrrr6warningswarn _inductorconfigcompile_threadsrrNotImplementedError __globals__rE) rirjoriginal_fully_shardmodeoriginal_compile_threadsfully_shard_patchrrfuncs rMwrapper6compiled_fsdp_test..decorator..wrappers,(-(9(9(>(>(J(J &>/// MMJWX+0??+A+A+Q+Q(!!))+>///(<%^<<<=>EOO**:(I%-;D6B CT  !5!>!>?d%f%!!))+BV  !5!>!>?9Q&&67'rLr )rrrrs` rM decorator%compiled_fsdp_test..decorators" t R  R>rL)r )rrrrs` @@rMcompiled_fsdp_testrs"""!F rLc2^\rSrSrSU4SjjrSrSrU=r$) SkipModuleicX>[TU]5 [R"SSSS9UlgN Fr)rrrtrlinrIs rMrSkipModule.__init__s" 99R%0rLc$URU5$rr!rJs rMrSkipModule.forwardsxx{rLr$rnrr7s@rMrrs1rLrc.^\rSrSrU4SjrSrSrU=r$) NestedLinearic>[TU]5 U(a8[[R"SSSS9R [ 55Ulg[R"SSSS9R [ 5Ulgr)rrrrtrrr nested_linear)rZ fsdp_wraprs rMrNestedLinear.__init__sV  !%biiBU&C&F&F{&S!TD !#2r!>!A!A+!ND rLc$URU5$rr)rJs rMrNestedLinear.forwards!!!$$rLr-rr7s@rMr'r'sO%%rLr'c.^\rSrSrU4SjrSrSrU=r$) SkipModelic>[TU]5 [R"SSSS9R [ 5Ul[5R [ 5Ul[[US9[ S9Ul g)Nr Fr)r*)r) rrrtrrrlinearr linear_skiprr'r))rZ double_nestrs rMrSkipModel.__init__sW iiBU366{C %r@r[rrrrrr'r0rDrLrMrVs #$"++ 44 ?S IRRFF FH+q#a!8!8!:;299>d>DBIID$D.Q"-Q"h[J-[J|$7D]J.]J@J mJ Z ? .,ryy,JJ*JJZ/"))/@*r}}*Z'299'6 111  X  "  X  " 9H99 *X** .x.. .x.. :8:: >X>>0!00  0  0.+- "Tyy"TII"Tc3h "TJ/&LL^ O%}2O%d4!(?4!n/$+/d %299 %  rL