oMj|CSSKJr SSKJrJr SSKrSSKJr SSK J r \(a\ "\ 5r \"SS55r \"SS 55r\"S S 55r\"S S 9"SS55r\"S S 9"SS55r\\\-\\\-\\S4-4r\\\-\\S4-r\\\\-4rS\S\R4S\\4Sjrg)) dataclass) TYPE_CHECKINGLiteralN) get_loggerc\rSrSr%SrSr\S-\S'Sr\S-\S'Sr \ \S'Sr \ S \S 'Sr \RR R"S-\S 'S r\ \S 'S r\ \S'Sr\\S'Sr\\S'Sr\R.\S'Sr\RR R"\S'Sr\RR R"\S'Sr\RR R"\S'Sr\RR R"\S'Sr\\S'Sr\\S'Sr\S\ \\44Sj5r!\S\ \"\"44Sj5r#S\S\S\R.S \RR R"4Sjr$S r%g)!ContextParallelConfig(a3 Configuration for context parallelism. Args: ring_degree (`int`, *optional*, defaults to `1`): Number of devices to use for Ring Attention. Sequence is split across devices. Each device computes attention between its local Q and KV chunks passed sequentially around ring. Lower memory (only holds 1/N of KV at a time), overlaps compute with communication, but requires N iterations to see all tokens. Best for long sequences with limited memory/bandwidth. Number of devices to use for ring attention within a context parallel region. Must be a divisor of the total number of devices in the context parallel mesh. ulysses_degree (`int`, *optional*, defaults to `1`): Number of devices to use for Ulysses Attention. Sequence split is across devices. Each device computes local QKV, then all-gathers all KV chunks to compute full attention in one pass. Higher memory (stores all KV), requires high-bandwidth all-to-all communication, but lower latency. Best for moderate sequences with good interconnect bandwidth. convert_to_fp32 (`bool`, *optional*, defaults to `True`): Whether to convert output and LSE to float32 for ring attention numerical stability. rotate_method (`str`, *optional*, defaults to `"allgather"`): Method to use for rotating key/value states across devices in ring attention. Currently, only `"allgather"` is supported. ulysses_anything (`bool`, *optional*, defaults to `False`): Whether to enable "Ulysses Anything" mode, which supports arbitrary sequence lengths and head counts that are not evenly divisible by `ulysses_degree`. When enabled, `ulysses_degree` must be greater than 1 and `ring_degree` must be 1. ring_anything (`bool`, *optional*, defaults to `False`): Whether to enable "Ring Anything" mode, which supports arbitrary sequence lengths. When enabled, `ring_degree` must be greater than 1 and `ulysses_degree` must be 1. mesh (`torch.distributed.device_mesh.DeviceMesh`, *optional*): A custom device mesh to use for context parallelism. If provided, this mesh will be used instead of creating a new one. This is useful when combining context parallelism with other parallelism strategies (e.g., FSDP, tensor parallelism) that share the same device mesh. The mesh must have both "ring" and "ulysses" dimensions. Use size 1 for dimensions not being used (e.g., `mesh_shape=(2, 1, 4)` with `mesh_dim_names=("ring", "ulysses", "fsdp")` for ring attention only with FSDP). N ring_degreeulysses_degreeTconvert_to_fp32 allgather)ralltoall rotate_methodmeshFulysses_anything ring_anything_rank _world_size_device_mesh_flattened_mesh _ring_mesh _ulysses_mesh_ring_local_rank_ulysses_local_rankcURcSUlURcSUlURS:XaURS:Xa [S5eURS:dURS:a [S5eURS:wa[ SURS35eUR (a6URS:Xa [S5eURS:a [S5eUR (a6URS:Xa [S 5eURS:a [S 5eUR (aUR (a [S 5egg) NzfEither ring_degree or ulysses_degree must be greater than 1 in order to use context parallel inferencezF`ring_degree` and `ulysses_degree` must be greater than or equal to 1.rz=Only rotate_method='allgather' is supported for now, but got .zIulysses_degree must be greater than 1 for ulysses_anything to be enabled.z8ulysses_anything cannot be enabled when ring_degree > 1.zCring_degree must be greater than 1 for ring_anything to be enabled.z8ring_anything cannot be enabled when ulysses_degree > 1.z:ulysses_anything and ring_anything cannot both be enabled.)r r ValueErrorrNotImplementedErrorrrselfs ڱC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\diffusers/models/_modeling_parallel.py __post_init__#ContextParallelConfig.__post_init__dsO    # D     &"#D    q T%8%8A%=x    a 4#6#6#:ef f    ,%OPTPbPbOccde   ""a' !lmm!# ![\\   1$ !fgg""Q& ![\\  T%7%7YZ Z&8 returnc2URUR4$N)r r r"s r$ mesh_shape ContextParallelConfig.mesh_shapes  $"5"566r'cg)z$Dimension names for the device mesh.ringulyssesr"s r$mesh_dim_names$ContextParallelConfig.mesh_dim_namess#r'rank world_sizedevicec XlX lX0lX@lURUR -U:a)[ SUR SURSUS35eURSR5UlURSUl URSUl URR5Ul URR5Ul g)NzThe product of `ring_degree` (z) and `ulysses_degree` (z") must not exceed the world size (z).r.r/r0)rrrrr r r _flattenrrrget_local_rankrrr#r4r5r6rs r$setupContextParallelConfig.setups %    !1!1 1J >01A1A0BBZ[_[n[nZopRS]R^^`a  $zz*;<EEG**V,!ZZ 2 $ > > @#'#5#5#D#D#F r') rrrrrrrrrr r )&__name__ __module__ __qualname____firstlineno____doc__r int__annotations__r r boolrrrtorch distributed device_mesh DeviceMeshrrrrrr6rrrrrrr%propertytupler+strr2r;__static_attributes__r1r'r$r r (s"H#Kt"!%NC$J% OT 6AM723A<@D%   ' ' 2 2T 9@#d"M4E3K GU\\ 6:E5   ( ( 3 3:@DOU&&22==D;?J!!--88?>BM5$$00;;B c ##[:7E#s(O77#c3h##G#G3G GEL]L]LiLiLtLtGr'r c~\rSrSr%SrSr\\S'Sr\ RRRS-\S'Sr \\S'Sr\\S'Sr\ R \S 'Sr\ RRR\S 'Sr\\S 'S rS \S\S\ R S\ RRR4SjrSrg)TensorParallelConfiga Configuration for tensor parallelism. Tensor parallelism shards weight matrices (column-wise and row-wise) across devices. Each device computes a partial result; an AllReduce/AllGather at layer boundaries reconstructs the full output. Uses `torch.distributed.tensor.parallelize_module` with `ColwiseParallel` / `RowwiseParallel` sharding styles. Supported device types are `"cuda"` and `"neuron"`. Args: tp_degree (`int`, defaults to `1`): Number of devices to shard across. Must be a divisor of the number of attention heads (and FFN hidden dimensions) of the model being parallelised. mesh (`torch.distributed.device_mesh.DeviceMesh`, *optional*): A custom device mesh to use. If provided, `tp_degree` is inferred from `mesh.size()` and the argument is ignored. Useful when combining TP with other parallelism strategies (e.g. CP) that share the same mesh. r tp_degreeNrrrrr _tp_degreec:URS:a [S5eg)Nrz`tp_degree` must be >= 1.)rPr r"s r$r%"TensorParallelConfig.__post_init__s >>A 89 9 r'r4r5r6c^XlX lX0lX@lUR 5Ulgr*)rrrrsizerQr:s r$r;TensorParallelConfig.setups$ %  ))+r')rrrrQr)r=r>r?r@rArPrBrCrrErFrGrHrrrr6rrQr%r;rLr1r'r$rNrNs"Is<@D%   ' ' 2 2T 9@E3K GU\\ 6:E5   ( ( 3 3:J:&#&3& &EL]L]LiLiLtLt&r'rNc H\rSrSr%SrSr\S-\S'Sr\ S-\S'Sr \ \S'Sr \ \S'Sr \R\S'Sr\R"R$R&\S 'S rSS .S \ S \ S\RS\R"R$R&S-4SjjrSrg)ParallelConfiga Configuration for applying different parallelisms. Args: context_parallel_config (`ContextParallelConfig`, *optional*): Configuration for context parallelism. tensor_parallel_config (`TensorParallelConfig`, *optional*): Configuration for tensor parallelism. Ncontext_parallel_configtensor_parallel_configrrrrcPURbURb [S5egg)NzCombining context parallelism and tensor parallelism in a single `ParallelConfig` is not supported. Please specify only one of `context_parallel_config` or `tensor_parallel_config`.)rZr[r r"s r$r%ParallelConfig.__post_init__s5  ' ' 38S8S8_d 9` 3r')rr4r5r6rcXlX lX0lX@lURbURR XX45 UR bUR R XX45 ggr*)rrrrrZr;r[r:s r$r;ParallelConfig.setupsb %   ' ' 3  ( ( . .t N  & & 2  ' ' - -d M 3r')rrrr)r=r>r?r@rArZr rCr[rNrrBrrrEr6rrFrGrHr%r;rLr1r'r$rXrXs=A2T9@:>047>E3K GU\\ 6:E5   ( ( 3 3:AE NNN N ++66= NNr'rXT)frozencL\rSrSr%Sr\\S'Sr\S-\S'Sr\ \S'Sr S r g) ContextParallelInputa Configuration for splitting an input tensor across context parallel region. Args: split_dim (`int`): The dimension along which to split the tensor. expected_dims (`int`, *optional*): The expected number of dimensions of the tensor. If provided, a check will be performed to ensure that the tensor has the expected number of dimensions before splitting. split_output (`bool`, *optional*, defaults to `False`): Whether to split the output tensor of the layer along the given `split_dim` instead of the input tensor. This is useful for layers whose outputs should be split after it does some preprocessing on the inputs (ex: RoPE). split_dimN expected_dimsF split_outputcVSURSURSURS3$)NzContextParallelInput(split_dim=, expected_dims=z, split_output=))rdrerfr"s r$__repr__ContextParallelInput.__repr__sK00@@PQUQcQcPddstxuFuFtGGHI Ir'r1) r=r>r?r@rArBrCrerfrDrjrLr1r'r$rbrbs, N $M3:$L$Ir'rbc>\rSrSr%Sr\\S'Sr\S-\S'SrSr g)ContextParallelOutputi a Configuration for gathering an output tensor across context parallel region. Args: gather_dim (`int`): The dimension along which to gather the tensor. expected_dims (`int`, *optional*): The expected number of dimensions of the tensor. If provided, a check will be performed to ensure that the tensor has the expected number of dimensions before gathering. gather_dimNrec<SURSURS3$)Nz!ContextParallelOutput(gather_dim=rhri)rnrer"s r$rjContextParallelOutput.__repr__s&24??2CCSTXTfTfSgghiir'r1) r=r>r?r@rArBrCrerjrLr1r'r$rmrm s" O $M3:$jr'rm.rUgroupr(c [R"US9n[[R"US95nSU;aSO[R R 5n[U5Vs/sH'n[R"SU[RS9PM) nn[R"U[R"U/U[RS9US9 UVs/sHowSR5PM nnU$s snfs snf)zkGather the local size from all ranks. size: int, local size return: list[int], list of size from all ranks )rqcpu)r)r6dtyper) distget_world_sizerK get_backendrE acceleratorcurrent_acceleratorrangeemptyint64 all_gathertensoritem)rUrqr5 comm_backends gather_device_gathered_sizesss r$gather_size_by_commrXs"$$51J((u56M"m3E9J9J9^9^9`MZ_`jZklZkUVekk$}EKKPZkNlOO dVME ,::>adiik>N: m;s %.C2C7) dataclassesrtypingrrrEtorch.distributedrFruutilsrr=loggerr rNrXrbrmdictrKrBlistrJContextParallelInputTypeContextParallelOutputTypeContextParallelModelPlan ProcessGrouprr1r'r$rs\$")  H  pGpG pGf $&$& $&N )N)N )NX $III0 $jjj0 #I#d+?&@@5I]_bIbCccc 2D9N4OORWXmorXrRss %=@Y%Y YZV c $*;*; S r'