Kgj SrSSKrSSKJr SSKJrJr SSKJ r J r J r "SS\RR5r\RR S S\R"S\ S \R"S-S \R"S-4S jj5rg) av Context Parallel support for Token Shift. Token shift has a 1-token dependency on previous tokens: y[t] = x[t-1] - x[t] (for t > 0) y[0] = cache - x[0] (cache is the last token from previous rank) In CP mode, non-first ranks need the last token from the previous rank as cache. Backward: non-last ranks need to send the last token's gradient to previous rank. N)token_shift_bwdtoken_shift_fwd) FLACPContextconv_cp_send_recv_bwdconv_cp_send_recv_fwdc\rSrSrSr\S\RS\RS-S\S\ RS-S\ \RS-\ 44 S j5r \SS \RS \RS-S\ RS-S \S \ SS4 Sjj5r\S\RS\RS-S\RS-S\S-4Sj5r\S\R4Sj5rSrg)TokenShiftCPFunctiona$ Context Parallel version of TokenShift. Forward: 1. Get last token from previous rank to construct cache 2. Call token_shift_fwd with cache Backward: 1. Call token_shift_bwd to get dx 2. Sync communication: add next rank's first token gradient to current rank's last token x cu_seqlensNcontextgroupreturnc@UcgURSnSnSnUR(dUR5S:XaURSS:XdSUR35eURS5nUSSR 5n[ X5n Ub[ U5S- OSn [R"XURURS9n[US S5nUS:aU SUS'XV4$URS5nUSSR 5n[ X5n XV4$) amPrepare cache for CP forward pass by communicating with previous rank. Args: x: Input tensor of shape [1, T, D] cu_seqlens: Cumulative sequence lengths context: CP context group: Process group for communication Returns: cache: Cache tensor of shape [N, D] or None pre_num_tokens: Number of tokens from previous rank for the first sequence N)NrrzCP requires [1, T, D], got devicedtypepre_num_conv_tokens) shape is_first_rankdimsqueeze contiguousrlentorchzerosrrgetattr) r r r rDcachepre_num_tokensx_2d last_tokenprev_last_tokenN_s ڨC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\fla/modules/token_shift_cp.py_prepare_cache_for_cp*TokenShiftCPFunction._prepare_cache_for_cp's&& = GGBK$$557aras  DQQn$5>>22n$b'+)-  || t#<<$& r,