j: %SSKJr SSKJr SSKrSSKJs Jr SSK J r SSK J r SSK JrJr SSKJr SS KJrJr SS KJrJrJr \"5(aSS KJr SS KJrJr O \R<r\"S SS9r\"5r \(aSSK!J"r" \RF"\$5r%S\S\4Sjr&S\S\4Sjr'S\(S\(S\(S\(S\)4 Sjr*S\(S\(S\(S\(S\)4 Sjr+S\(S\4Sjr,S\(S\R<S\4Sjr-S \R<S\4S!jr.S\(S\4S"jr/S\(S\4S#jr0S\(S\4S$jr1S\(S\R<S\4S%jr2S&\R<S\4S'jr3S(\R<S\4S)jr4S*\S+\(S,\(S\4S-jr5S.\R<S-S/\(S,\(S\R<S-4S0jr6S \R<S.\R<S-S/\(S,\(S\R<4 S1jr7S2\RpS\Rp4S3jr9ShS&\R<S-S4\(S/\(S+\(S,\(S5\(S-S\)4S6jjr:S&\R<S-S/\(S5\(S-S\)4S7jr;ShS&\R<S-S/\(S5\(S-S\)4S8jjrjr>\"S?S@SASB9SS\*SSSSCSSCSD4 SE\(S4\(S/\(S+\(S,\(S*\S.\R<S-SF\(S-SG\)SH\)S?\)SI\)SJ\R~\@-S\R<S-4SKjj5rASS\*S\RSCSCSD4SE\(S4\(S/\(S+\(S,\(S*\S.\R<S-SL\RSH\)SI\)SJ\R~\@-S\R<4SMjjrDSS\*S4SE\(S4\(S/\(S+\(S,\(S*\S.\R<S-4SNjjrESS\*SSD4SE\(S4\(S/\(S+\(S,\(S*\S.\R<S-SJ\R~\@-S\4SOjjrF"SPSQ\5rG\G"5rH\G\ISR'SS\R<S\R<S-4STjrJShSU\ SV\R<S.\R<\-S-SW\ S-SS\R<S-SX\(S-SY\R<S-S\K\)\R<\-S-\(\(44SZjjrLSiSU\ SV\R<S.\R<S-SW\ S-SS\R<S-S[\S-S\\S-S \R<S-SX\(S-SG\)S\R<\-S-4S]jjrMSjSU\ SV\R<S.\R<S-SY\R<S-SW\ S-S[\S-S\\S-SH\)S\R<\-S-4S^jjrNSiSU\ SV\R<S.\R<S-SW\ S-SS\R<S-S[\S-S\\S-S \R<S-SX\(S-SG\)S\R<\-S-4S_jjrOSjSU\ SV\R<S.\R<S-SY\R<S-SW\ S-S[\S-S\\S-SH\)S\R<\-S-4S`jjrPSjSU\ SV\R<S.\R<S-SW\ S-SS\R<S-S[\S-S\\S-SX\(S-SG\)S\R<\-S-4SajjrQShSU\ SV\R<S.\R<S-SW\ S-S\R<S-4 SbjjrR\M\O\Q\O\O\M\"\MSCSc9\"\MSCSc9\R\R\M\RSd.\M\RSe.Sf. rSSkSU\ SV\R<S.\R<S-SW\ S-SS\R<S-S[\S-S\\S-S \R<S-4SgjjrTg)l)Callable)partialN)Cache)PreTrainedConfig)is_torch_xpu_availablelogging)deprecate_kwarg)GeneralInterfaceis_flash_attention_requested)is_torch_flex_attn_availableis_torch_greater_or_equal is_tracing)_DEFAULT_SPARSE_BLOCK_SIZE) BlockMaskcreate_block_maskz2.6T) accept_dev)TransformGetItemToIndexmask_functionsreturnc^^[ST55(d[ST35eU4SjnU$)zKReturns a mask function that is the intersection of provided mask functionsc38# UHn[U5v M g7fNcallable.0args ڨC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\transformers/masking_utils.py and_masks..37x}}.All inputs should be callable mask_functions: c>URS[RS9nTH'nXE"XX#5RUR5-nM) U$N)dtype)new_onestorchbooltodevice batch_idxhead_idxq_idxkv_idxresultmaskrs rand_maskand_masks..and_mask6sH%**5"Dd9FII&--XXF# all RuntimeError)rr5s` r and_masksr;1s4 77 7 7KNK[\]] Or7c^^[ST55(d[ST35eU4SjnU$)zDReturns a mask function that is the union of provided mask functionsc38# UHn[U5v M g7frrrs rr or_masks..Ar"r#r$c>URS[RS9nTH'nXE"XX#5RUR5-nM) U$r&) new_zerosr*r+r,r-r.s ror_maskor_masks..or_maskDsH5::6"Dd9FII&--XXF# r7r8)rrAs` ror_masksrC?s4 77 7 7KNK[\]] Nr7r/r0r1r2c X2:*$)z2 This creates a basic lower-diagonal causal mask. r'r/r0r1r2s rcausal_mask_functionrFMs  ?r7c US:$)zw This creates a full bidirectional mask. NOTE: It is important to keep an index-based version for non-vmap expansion. rr'rEs rbidirectional_mask_functionrHTs A:r7sliding_windowc T^S[S[S[S[S[4 U4SjjnU$)zy This is an overlay depicting a sliding window pattern. Add it on top of a causal mask for a proper sliding window mask. r/r0r1r2rc>X2T- :$rr'r/r0r1r2rIs r inner_mask*sliding_window_overlay..inner_maskcs...r7intr+rIrMs` rsliding_window_overlayrR]s3 /c/S//c/d/ r7 chunk_size left_paddingc X^^S[S[S[S[S[4 UU4SjjnU$)z This is an overlay depicting a chunked attention pattern. Add it on top of a causal mask for a proper chunked attention mask. r/r0r1r2rc2>UTU- T-UTU- T-:H$rr')r/r0r1r2rSrTs rrM#chunked_overlay..inner_maskos.i00Z?ELYbLcDchrCrrrr7rO)rSrTrMs`` rchunked_overlayrXis? scsSsscsdss r7block_sequence_idsc T^S[S[S[S[S[4 U4SjjnU$)aj This is an overlay depicting a blockwise masking pattern. Instead of a single token, each block consists of arbitrary length tokens. In causal setup, each block can attend to prev block causally and can't attend to future blocks. Within one block the attention is always bidirectional. Mostly used in MLLMs when non-text data attends bidirectionally to itself. r/r0r1r2rc0>TX4nTX4nXE:HUS:-$)Nrr')r/r0r1r2q_groupkv_grouprYs rrM%blockwise_overlay..inner_mask~s/$Y%56%i&78#1 55r7rO)rYrMs` rblockwise_overlayr_us36c6S66c6d6 r7c4[[U5[5$)zI This return the mask_function function to create a sliding window mask. )r;rRrFrIs r#sliding_window_causal_mask_functionrbs +N;=Q RRr7c T^S[S[S[S[S[4 U4SjjnU$)zF This is an overlay depicting a bidirectional sliding window pattern. r/r0r1r2rc$>[X#- 5T:*$)zA token can attend to any other token if their absolute distance is within the (inclusive) sliding window size (distance <= sliding_window).)absrLs rrM8sliding_window_bidirectional_overlay..inner_masks5>"n44r7rOrQs` r$sliding_window_bidirectional_overlayrgs3 5c5S55c5d5 r7c4[[U5[5$)zW This return the mask_function function to create a bidirectional sliding window mask. )r;rgrHras r*sliding_window_bidirectional_mask_functionris 9.IKf ggr7c4[[X5[5$)zL This return the mask_function function to create a chunked attention mask. )r;rXrF)rSrTs rchunked_causal_mask_functionrks _Z>@T UUr7 padding_maskc T^S[S[S[S[S[4 U4SjjnU$)zL This return the mask_function function corresponding to a 2D padding mask. r/r0r1r2rc>TX4$rr')r/r0r1r2rls rrM)padding_mask_function..inner_masksI-..r7rO)rlrMs` rpadding_mask_functionrps3 /c/S//c/d/ r7packed_sequence_maskc T^S[S[S[S[S[4 U4SjjnU$)zT This return the mask_function function corresponding to a 2D packed sequence mask. r/r0r1r2rc>TX4TX4:H$rr')r/r0r1r2rqs rrM1packed_sequence_mask_function..inner_masks#I$459MiN_9```r7rO)rqrMs` rpacked_sequence_mask_functionrus9 acaSaacada r7 mask_functionq_offset kv_offsetc \^^^S[S[S[S[S[4 UUU4SjjnU$)z This function adds the correct offsets to the `q_idx` and `kv_idx` as the torch API can only accept lengths, not start and end indices. r/r0r1r2rc$>T"XUT-UT-5$rr')r/r0r1r2rxrvrws rrM0add_offsets_to_mask_function..inner_masksY%(2BFYDVWWr7rO)rvrwrxrMs``` radd_offsets_to_mask_functionr|s? XcXSXXcXdXX r7attention_mask kv_lengthcUnUbFX-URS- =nS:a,[RRR USU45nU$)z` From the 2D attention mask, prepare the correct padding mask to use by potentially padding it. r)shaper*nn functionalpad)r}r~rxlocal_padding_maskpadding_lengths rprepare_padding_maskrsY(!'3n6J6J26NN NNRS S!&!4!4!8!8!^I\!]  r7cjX#-URS- =nS:a[R"USU4SS9nU$)z Pads the `block_sequence_ids` in case the total length is less than `kv_length`. Usually that happens with `StaticCache` generation or generating without cache. Pads to the right with `-1`. rr)rvalue)rFr)rYr}r~rxrs rmaybe_pad_block_sequence_idsrsF$/2D2J2J22NNNRSSUU#5A~;NVXY r7tensorcDUR5UR5:H$)ziSimilar to `tensor.all()`, but uses an implementation with `tensor.sum()`, which is actually much faster.)sumnumel)rs rfast_allrs ::<6<<> ))r7q_lengthlocal_attention_sizecpUb=URSU:a*[R"X RS9U-nUSS2U4n[ U5(agUbX%:agUS:XdX!:XaUb[ U5(agUS:Xa7Ub3[ USS2SU245(a[ USS2US24)5(agg)a Detects whether the causal mask can be ignored in case PyTorch's SDPA is used, rather relying on SDPA's `is_causal` argument. In case no token is masked in the 2D `padding_mask` argument, if `query_length == 1` or `key_value_length == query_length`, we rather rely on SDPA `is_causal` argument to use causal/non-causal masks, allowing to dispatch to the flash attention kernel (that can otherwise not be used if a custom `attn_mask` is passed). Nrr-FrTr)rr*aranger-rr)rlrr~rwrxr mask_indicess r_ignore_causal_mask_sdpars L$6$6r$:Y$F||I6I6IJYV #A|O4 ,'I,M A .\5IXVbMcMc1},q)8)|*D!E!E(T`abdldmamTnSnJoJo r7c^[U5(agUbX:agUcgUR5$)z XPU-specific logic for determining if we can skip bidirectional mask creation. For XPU devices, we have special handling: - Skip if no padding and no local attention constraint FT)rr9rlr~rs r _can_skip_bidirectional_mask_xpurs<,'I,M    r7c[(a [XU5$[U5(d!UbUR5(a UbX:agg)a Detects whether the bidirectional mask can be ignored in case PyTorch's SDPA is used. In case no token is masked in the 2D `padding_mask` argument and no local attention constraint applies (i.e. `local_attention_size` is None or `kv_length < local_attention_size`), we skip mask creation, allowing to dispatch to the flash attention kernel (that can otherwise not be used if a custom `attn_mask` is passed). TF)_is_torch_xpu_availablerrr9rs r_ignore_bidirectional_mask_sdpar5sL0 I]^^ | $ $  !\%5%5%7%7 ! )Y-M r7cJ/SQnUHn[R"XSS9nM U$)a: Used to vmap our mask_functions over the all 4 dimensions (b_idx, h_idx, q_idx, kv_idx) of the inputs. Using vmap here allows us to keep the performance of vectorized ops, while having a single set of primitive functions between attention interfaces (i.e. between flex and sdpa/eager, FA2 being a bit different). ))NNNr)NNrN)NrNN)rNNNr)in_dimsout_dims)r*vmap)rv dimensionsdimss r_vmap_expansion_sdparTs*nJ =K  r7 batch_indices head_indices q_indices kv_indicescbUSS2SSS4nUSSS2SS4nUSSSS2S4nUSSSSS24nXX#4$)a Used to broadcast our mask_functions over the all 4 dimensions (b_idx, h_idx, q_idx, kv_idx) of the inputs. Allows the usage of any index-based mask function without relying on vmap. NOTE: This is limited to index based functions only and is not guaranteed to work otherwise. Reference: - https://github.com/huggingface/optimum-onnx/blob/c123e8f4fab61b54a8e0e31ce74462bcacca576e/optimum/exporters/onnx/model_patcher.py#L362-L365 Nr')rrrrs r_non_vmap_expansion_sdparas["!T4"56Mat 34L$a-.ID$a/0J  ==r7allow_torch_fixz5.18.0zIt has no effect anymore.)versionadditional_messageFcpu batch_size local_sizeallow_is_causal_skipallow_is_bidirectional_skipuse_vmapr-c  Z[XbU5nU(a[XX#XG5(agU (a[XU5(agUb[U[ U55n[ R "X S9n[ R "SU S9n[ R "XS9U-n[ R "X,S9U-nU (d&U"[UUUU56nURUSX5nU$[(a)[5 [U5"UUUU5nSSS5 U$[S5e!,(df  U$=f)u Create a 4D boolean mask of shape `(batch_size, 1, query_length, kv_length)` where a value of True indicates that the element should take part in the attention computation, and False that it should not. This function can only be used with torch>=2.5, as the context manager is otherwise not available. Args: batch_size (`int`): The batch size of the input sequence. q_length (`int`): The size that the query states will have during the attention computation. kv_length (`int`): The size that the key and value states will have during the attention computation. kv_offset (`int`, optional): An optional offset to indicate at which first position the key and values states will refer to. q_offset (`int`, optional): An optional offset to indicate at which first position the query states will refer to. mask_function (`Callable`): The mask factory function describing the mask pattern. attention_mask (`torch.Tensor`, optional): The 2D attention mask corresponding to padded tokens of shape (batch_size, number_of_seen_tokens+q_length) local_size (`int`, optional): The size of the local attention, if we do not use full attention. This is used only if `allow_is_causal_skip=True` to try to skip mask creation if possible. allow_is_causal_skip (`bool`, optional): Whether to allow to return `None` for the mask under conditions where we can use the `is_causal` argument in `torch.sdpa` instead. Default to `True`. allow_is_bidirectional_skip (`bool`, optional): Whether to allow to return `None` for the mask under conditions where we do not have to add any bias, i.e. full attention without any padding. Default to `False`. allow_torch_fix (`bool`, optional): Deprecated and has no effect. Will be removed in version 5.18.0. use_vmap (`bool`, optional): Whether to use `vmap` during the mask construction or not. Allows powerful custom patterns that may not be index-based (for the cost of speed performance). By default `False`. device (`torch.device` or `str`, optional): An optional device to create the mask on. ## Creating a simple causal mask: To create the following causal mask: 0 ■ ⬚ ⬚ ⬚ ⬚ 1 ■ ■ ⬚ ⬚ ⬚ 2 ■ ■ ■ ⬚ ⬚ 3 ■ ■ ■ ■ ⬚ 4 ■ ■ ■ ■ ■ You can do ```python >>> sdpa_mask(batch_size=1, q_length=5, kv_length=5) >>> tensor([[[[ True, False, False, False, False], [ True, True, False, False, False], [ True, True, True, False, False], [ True, True, True, True, False], [ True, True, True, True, True]]]]) ``` ## Creating a sliding window mask: To create the following sliding window mask (`sliding_window=3`): 0 ■ ⬚ ⬚ ⬚ ⬚ 1 ■ ■ ⬚ ⬚ ⬚ 2 ■ ■ ■ ⬚ ⬚ 3 ⬚ ■ ■ ■ ⬚ 4 ⬚ ⬚ ■ ■ ■ You can do ```python >>> sdpa_mask(batch_size=1, q_length=5, kv_length=5, mask_function=sliding_window_causal_mask_function(3)) >>> tensor([[[[ True, False, False, False, False], [ True, True, False, False, False], [ True, True, True, False, False], [False, True, True, True, False], [False, False, True, True, True]]]]) ``` ## Creating a chunked attention mask To create the following chunked attention mask (`chunk_size=3`): 0 ■ ⬚ ⬚ ⬚ ⬚ 1 ■ ■ ⬚ ⬚ ⬚ 2 ■ ■ ■ ⬚ ⬚ 3 ⬚ ⬚ ⬚ ■ ⬚ 4 ⬚ ⬚ ⬚ ■ ■ You can do ```python >>> sdpa_mask(batch_size=1, q_length=5, kv_length=5, mask_function=chunked_causal_mask_function(3, torch.zeros(1, dtype=int))) >>> tensor([[[[ True, False, False, False, False], [ True, True, False, False, False], [ True, True, True, False, False], [False, False, False, True, False], [False, False, False, True, True]]]]) ``` NrrrzThe vmap functionality for mask creation is only supported from torch>=2.6. Please update your torch version or use `use_vmap=False` with index-based masks.) rrrr;rpr*rrexpand#_is_torch_greater_or_equal_than_2_6rr ValueError)rrr~rwrxrvr}rrrrrr-kwargsrl batch_arange head_arangeq_arange kv_aranges r sdpa_maskrts4p(9ML  8 Y!!"'F|`j'k'k!-1F|1TU << :L,,q0K||H4x?H Y6BI &(@{\dfo(pq'..z2xS"  - ,% &1-@{\dfopN'   _  ' & s 1D D*r(c  U RSS5n [SUUUUUUUSUU U S. U D6n U bV[R"U5Rn[R "U [R "SU RUS9U5n U $)a Create a 4D float mask of shape `(batch_size, 1, query_length, kv_length)` where a value of 0 indicates that the element should take part in the attention computation, and -inf (minimum value for the given `dtype`) that it should not. Args: batch_size (`int`): The batch size of the input sequence. q_length (`int`): The size that the query states will have during the attention computation. kv_length (`int`): The size that the key and value states will have during the attention computation. q_offset (`int`, optional): An optional offset to indicate at which first position the query states will refer to. kv_offset (`int`, optional): An optional offset to indicate at which first position the key and values states will refer to. mask_function (`Callable`): The mask factory function describing the mask pattern. attention_mask (`torch.Tensor`, optional): The 2D attention mask corresponding to padded tokens of shape (batch_size, number_of_seen_tokens+q_length) dtype (`torch.dtype`, optional): The dtype to use for the mask. By default, `torch.float32`. allow_is_bidirectional_skip (`bool`, optional): Whether to allow to return `None` for the mask under conditions where we do not have to add any bias, i.e. full attention without any padding. Default to `False`. use_vmap (`bool`, optional): Whether to use `vmap` during the mask construction or not. Allows powerful custom patterns that may not be index-based (for the cost of speed performance). By default `False`. device (`torch.device` or `str`, optional): An optional device to create the mask on. rNF) rrr~rwrxrvr}rrrr-gr-r(r')poprr*finfominwhererr-)rrr~rwrxrvr}r(rrr-r_r4 min_dtypes r eager_maskrs\  )40A  #%"$?   D KK&** {{4c$++U!SU^_ Kr7c xUb6USS2U*S24nURSU:XaUR5(aSnU$)a8 Create the attention mask necessary to use FA2. Since FA2 is un-padded by definition, here we simply return `None` if the mask is fully causal, or we return the 2D mask which will then be used to extract the seq_lens. We just slice it in case of sliding window. Args: batch_size (`int`): The batch size of the input sequence. q_length (`int`): The size that the query states will have during the attention computation. kv_length (`int`): The size that the key and value states will have during the attention computation. q_offset (`int`, optional): An optional offset to indicate at which first position the query states will refer to. kv_offset (`int`, optional): An optional offset to indicate at which first position the key and values states will refer to. mask_function (`Callable`): The mask factory function describing the mask pattern. attention_mask (`torch.Tensor`, optional): The 2D attention mask corresponding to padded tokens of shape (batch_size, number_of_seen_tokens+q_length) Nr)rr9)rrr~rwrxrvr}rs rflash_attention_maskr`sJ>!'I:;7    "i /N4F4F4H4H!N r7c Ub[U5(dURS[-S-[-n XRS- n [(d1U S:a+[R R RUSSU 4S9n[XbU5n [U[U 55n[XSU5n[UUSUUU[S9n U $)ad Create a 4D block mask which is a compressed representation of the full 4D block causal mask. BlockMask is essential for performant computation of flex attention. See: https://pytorch.org/blog/flexattention/ Args: batch_size (`int`): The batch size of the input sequence. q_length (`int`): The size that the query states will have during the attention computation. kv_length (`int`): The size that the key and value states will have during the attention computation. q_offset (`int`, optional): An optional offset to indicate at which first position the query states will refer to. kv_offset (`int`, optional): An optional offset to indicate at which first position the key and values states will refer to. mask_function (`Callable`): The mask factory function describing the mask pattern. attention_mask (`torch.Tensor`, optional): The 2D attention mask corresponding to padded tokens of shape (batch_size, number_of_seen_tokens+q_length) device (`torch.device` or `str`, optional): An optional device to create the mask on. Nrr)rr)mask_modBHQ_LENKV_LENr-_compile) rrflex_default_block_sizerr*rrrrr;rpr|r) rrr~rwrxrvr}r-rpad_lenrl block_masks rflex_attention_maskrsD!(>*B*B#((+/FF!KOff003322w{"XX0044^1STV]R^4_N+NyQ !-1F|1TU 1)TM#  4J r7c&\rSrSr\\\\\\S.rSr g)AttentionMaskInterfacei)sdpaeagerflash_attention_2flash_attention_3flash_attention_4flex_attentionr'N) __name__ __module__ __qualname____firstlineno__rrrr_global_mapping__static_attributes__r'r7rrrs111- Or7rALL_MASK_ATTENTION_FUNCTIONS position_idscUSS2SS24S- n[R"XSS9nUS:gRS5n[U5(d USS2S4S:HR 5(agU$)a Find the indices of the sequence to which each new query token in the sequence belongs when using packed tensor format (i.e. several sequences packed in the same batch dimension). Args: position_ids (`torch.Tensor`) A 2D tensor of shape (batch_size, query_length) indicating the positions of each token in the sequences. Returns: A 2D tensor where each similar integer indicates that the tokens belong to the same sequence. For example, if we pack 3 sequences of 2, 3 and 1 tokens respectively along a single batch dim, this will return [[0, 0, 1, 1, 1, 2]]. If the there is only one sequence in each batch item (and we don't compile), then we return `None` indicating no packed sequences. This is the same as [[0, 0, 0, 0, 0, 0]] for the example above. Nrr)prependdimr)r*diffcumsumrr9)rfirst_dummy_value position_diffrqs rfind_packed_sequence_indicesrsw*%QU+a/JJ|BOM)Q.66r: * + +1Eae1LPQ1Q0V0V0X0X r7config inputs_embedspast_key_values layer_idxencoder_hidden_statesc [U[R[45(a"[ UR 5S:Xa SUSSSSS4$UR [R;agUb8URS:Xa(URUR[RS9nUR SnUbaURU5n[U[R5(aURUR5OUnURXu5upO,SnUcUbUR SOUn Sn OUR S SpSn UbDUcAUc>UR Sn XR S:waURU S 5n[!U5n S X+XyX4$) a Perform some common pre-processing of the mask arguments we get from the modeling code. Mostly determine the key-value length and offsets, and if we should early exit or not. Args: config (`PreTrainedConfig`): The model config. inputs_embeds (`torch.Tensor`): The input embeddings of shape (batch_size, query_length, hidden_dim). This is used only to infer the batch size, query length and dtype. attention_mask (`torch.Tensor`, optional): The 2D attention mask corresponding to padded tokens of shape (batch_size, number_of_seen_tokens+q_length). It can also be an already prepared 4D mask, in which case it is returned as-is. past_key_values (`Cache`, optional): The past key values, if we use a cache. position_ids (`torch.Tensor`, optional) A 2D tensor of shape (batch_size, query_length) indicating the positions of each token in the sequences. layer_idx (`int`, optional): If `past_key_values` is not None, this is the layer index of the cache from which to get the key-value length and offset. Indeed, for hybrid caches, different layers may return different lengths. encoder_hidden_states (`torch.Tensor`, optional): The input embeddings of shape (batch_size, kv_length, hidden_dim). If provided, it is used instead of `inputs_embeds` to infer the kv length. Returns: early_exit (`bool`): Whether we should early exit mask creation, and return the mask as-is. attention_mask (`torch.Tensor` or `BlockMask` or `None`): The attention mask to either return immediately, or to use in downstream mask creation. packed_sequence_mask (`torch.Tensor`, optional): In case we detected packed sequence format, this is a tensor where each similar integer indicates that the tokens belong to the same sequence. q_length (`int`): The size that the query states will have during the attention computation. kv_length (`int`): The size that the key and value states will have during the attention computation. q_offset (`int`, optional): An optional offset to indicate at which first position the query states will refer to. kv_offset (`int`): An offset to indicate at which first position the key and values states will refer to. TN)TNNNNNNrrrrF) isinstancer*Tensorrlenr_attn_implementationrrndimr,r-r+get_query_offsetget_mask_sizesrr) rrr}rrrrrrwr~rxrqrs r_preprocess_mask_argumentsrsf.5<<";<<^EYEYAZ^_A_^T4tTAA""*F*V*VV7!n&9&9Q&>'**-2F2Fejj*Y""1%H""33I>9C8U\\8Z8Z8;;}334`h.==hR 9  !:O:[-33A6aiII$2#7#7#;Qy N$:?V"((+ ++A. .'..z2>L;LI .X ``r7or_mask_functionand_mask_functionc [USS5(d[UUUUUUU S9$Uc?[US5(a,SUR;aURR S5nOSn[ XX#XH5upppnU (aU$UR SURURnnn[n[URnSnU =(a [USS5=(a U S:H(+n Ub&[(d [S 5e[UU5nSn SnUb&[(d [S 5e[UU5nSn SnU b[U[!U 55nSn Ub#[#XrX5n[U[%U55nSn U"UU U UUUUU UUUUS 9 nU$) aD Create a standard causal mask based on the attention implementation used (stored in the config). If `past_key_values` has a hybrid cache structure, this function will return the mask corresponding to one of the "full_attention" layers (to align to what is needed in the `modeling_xxx.py` files). Args: config (`PreTrainedConfig`): The model config. inputs_embeds (`torch.Tensor`): The input embeddings of shape (batch_size, query_length, hidden_dim). This is used only to infer the batch size, query length and dtype. attention_mask (`torch.Tensor`, optional): The 2D attention mask corresponding to padded tokens of shape (batch_size, number_of_seen_tokens+q_length). It can also be an already prepared 4D mask, in which case it is returned as-is. past_key_values (`Cache`, optional): The past key values, if we use a cache. position_ids (`torch.Tensor`, optional) A 2D tensor of shape (batch_size, query_length) indicating the positions of each token in the sequences. or_mask_function (`Callable`, optional): An optional mask function to combine with the causal mask function (by doing the union of both). This is useful to easily overlay another mask on top of the causal one, for example for image tokens handling. and_mask_function (`Callable`, optional): An optional mask function to combine with the causal mask function (by doing the intersection of both). This is useful to easily overlay another mask on top of the causal one, for example for image tokens handling. block_sequence_ids (`torch.Tensor`, *optional*): A tensor of same shape as input IDs indicating to which block or group each token belongs to. Tokens from the same block will keep a bidirectional mask within the block, attending causally to the past. Index `-1` can be used for blocks that have to keep complete causality within itself. layer_idx (`int`, *optional*): The cache layer to size the mask against. By default, the first "full_attention" layer is used, which is correct whenever all layers of a given mask type have seen the same tokens. Pass it explicitly for caches where layers of the same type hold different lengths (e.g. per-depth MTP streams). allow_is_causal_skip (`bool`, optional): Whether to allow returning `None` (and relying on `sdpa`'s `is_causal` argument) when the mask would be a plain causal mask. Set to `False` to always materialize the mask, e.g. when it is later concatenated with another mask. Defaults to `True`. is_causalTrrrr is_slidingFris_compileablerLUsing `or_mask_function` or `and_mask_function` arguments require torch>=2.6) rrr~rwrxrvr}rr(rrr-)getattrcreate_bidirectional_maskhasattrrindexrrr(r-rFrrrrrCr;rurr_)rrr}rrrrrYrr early_exitrqrr~rwrxrr(r-mask_factory_functionmask_interfacer causal_masks rcreate_causal_maskr asf 6; - -(   +-/(<   ?L 1 1e?Y?Y6Y'2288?II #6.[gs_J 4 U^ - 3 3A 6 8K8K]MaMavJ01&2M2MNN H0!159Kh!m5#22kl l ()>@P Q$$22kl l )*?AR S$' )*?A^_sAt u$%9:L^gs ()>@QRd@e f$!+%1 K r7c [US5(a,SUR;aURRS5n OSn [XX$SX5upppnU (aU$UbUOUnURSUR nnUR n[n[URnSnUb&[(d [S5e[UU5nSnSnUb&[(d [S5e[UU5nSnSnU"UU U UUUUSUUUUUS9 nU$)a Create a standard bidirectional mask based on the attention implementation used (stored in the config). Args: config (`PreTrainedConfig`): The model config. inputs_embeds (`torch.Tensor`): The input embeddings of shape (batch_size, query_length, hidden_dim). This is only used to infer metadata such as the batch size, query length, dtype, and device. attention_mask (`torch.Tensor`, optional): The 2D attention mask corresponding to padded tokens of shape (batch_size, kv_length). It can also be an already prepared 4D mask of shape (batch_size, 1, query_length, kv_length), in which case it is returned as-is. encoder_hidden_states (`torch.Tensor`, optional): The input embeddings of shape (batch_size, kv_length, hidden_dim). If provided, it is used instead of `inputs_embeds` to infer the batch size, kv length and dtype. past_key_values (`Cache`, optional): The past key values, if we use a cache. or_mask_function (`Callable`, optional): An optional mask function to combine with the base mask function (by doing the union of both). This is useful to easily overlay another mask on top, for example for image tokens handling. and_mask_function (`Callable`, optional): An optional mask function to combine with the base mask function (by doing the intersection of both). This is useful to easily overlay another mask on top, for example for image tokens handling. allow_is_bidirectional_skip (`bool`, optional): Whether to allow returning `None` (no bias) when the mask is plain bidirectional with no padding. Set to `False` to always materialize the mask, e.g. when it is later concatenated with another mask. Defaults to `True`. rFrNrT) rrr~rwrxrvr}rrr(rrr-)rrrrrr(r-rHrrrrrCr;)rrr}rrrrrrrrrrr~rwrxembedsrr(r-r r rs rrrsSR --%?;U;U2U#..44U;  Oi~iOKJY)&;&G "]F QJ ! !F71&2M2MNN H #22kl l ()>@P Q&+#$22kl l )*?AR S&+#$+%"$?N r7c \[USS5(d[UUUUUUU S9$Uc?[US5(a,SUR;aURR S5nOSn[ XX#XH5upppnU (aU$[USS5nUc [ S5eURSURURnnn[U5n[URnS nU =(a [US S 5=(a U S :H(+n Ub&[(d [ S 5e[UU5nS n SnUb&[(d [ S 5e[UU5nS n SnU b[U[!U 55nS n Ub#[#XrX5n[U[%U55nS n U"UU U UUUUU UUUUUS 9 nU$)a Create a sliding window causal mask based on the attention implementation used (stored in the config). This type of attention pattern was mostly democratized by Mistral. If `past_key_values` has a hybrid cache structure, this function will return the mask corresponding to one of the "sliding_attention" layers (to align to what is needed in the `modeling_xxx.py` files). Args: config (`PreTrainedConfig`): The model config. inputs_embeds (`torch.Tensor`): The input embeddings of shape (batch_size, query_length, hidden_dim). This is used only to infer the batch size, query length and dtype. attention_mask (`torch.Tensor`, optional): The 2D attention mask corresponding to padded tokens of shape (batch_size, number_of_seen_tokens+q_length). It can also be an already prepared 4D mask, in which case it is returned as-is. past_key_values (`Cache`, optional): The past key values, if we use a cache. position_ids (`torch.Tensor`, optional) A 2D tensor of shape (batch_size, query_length) indicating the positions of each token in the sequences. or_mask_function (`Callable`, optional): An optional mask function to combine with the sliding causal mask function (by doing the union of both). This is useful to easily overlay another mask on top of the sliding causal one, for example for image tokens handling. and_mask_function (`Callable`, optional): An optional mask function to combine with the sliding causal mask function (by doing the intersection of both). This is useful to easily overlay another mask on top of the sliding causal one, for example for image tokens handling. block_sequence_ids (`torch.Tensor`, *optional*): A tensor of same shape as input IDs indicating to which block or group each token belongs to. Tokens from the same block will keep a bidirectional mask within the block, attending causally to the past. Index `-1` can be used for blocks that have to keep complete causality within itself. layer_idx (`int`, *optional*): The cache layer to size the mask against. By default, the first "full_attention" layer is used, which is correct whenever all layers of a given mask type have seen the same tokens. Pass it explicitly for caches where layers of the same type hold different lengths (e.g. per-depth MTP streams). allow_is_causal_skip (`bool`, optional): Whether to allow returning `None` (and relying on `sdpa`'s `is_causal` argument) when the mask would be a plain causal mask. Set to `False` to always materialize the mask, e.g. when it is later concatenated with another mask. Defaults to `True`. rTrNrrrIJCould not find a `sliding_window` argument in the config, or it is not setFrrr rrr~rwrxrvr}rrr(rrr-)r(create_bidirectional_sliding_window_maskrrrrrrr(r-rbrrrrCr;rurr_)rrr}rrrrrYrrrrqrr~rwrxrIrr(r-r r rr s r!create_sliding_window_causal_maskrMs h 6; - -7   +-/(<   ?L 1 1do>X>X6X'2288>II #6.[gs_J 4 U^V%5t@P Q$$22kl l )*?AR S$' )*?A^_sAt u$%9:L^gs ()>@QRd@e f$!+%1!K r7c X[US5(a,SUR;aURRS5n OSn [XX$SX5upppnU (aU$[ USS5nUc [ S5eUbUOUnUR SURnnURn[U5n[URnSnUb&[(d [ S5e[UU5nSnSnUb&[(d [ S5e[UU5nSnSnU"UU U UUUUSUUUUUUS 9nU$) a Create a standard bidirectional sliding window mask based on the attention implementation used (stored in the config). Args: config (`PreTrainedConfig`): The model config. inputs_embeds (`torch.Tensor`): The input embeddings of shape (batch_size, query_length, hidden_dim). This is only used to infer metadata such as the batch size, query length, dtype, and device. attention_mask (`torch.Tensor`, optional): The 2D attention mask corresponding to padded tokens of shape (batch_size, kv_length). It can also be an already prepared 4D mask of shape (batch_size, 1, query_length, kv_length), in which case it is returned as-is. encoder_hidden_states (`torch.Tensor`, optional): The input embeddings of shape (batch_size, kv_length, hidden_dim). If provided, it is used instead of `inputs_embeds` to infer the batch size, kv length and dtype. past_key_values (`Cache`, optional): The past key values, if we use a cache. or_mask_function (`Callable`, optional): An optional mask function to combine with the base mask function (by doing the union of both). This is useful to easily overlay another mask on top, for example for image tokens handling. and_mask_function (`Callable`, optional): An optional mask function to combine with the base mask function (by doing the intersection of both). This is useful to easily overlay another mask on top, for example for image tokens handling. allow_is_bidirectional_skip (`bool`, optional): Whether to allow returning `None` (no bias) when the mask is plain bidirectional with no padding. Set to `False` to always materialize the mask, e.g. when it is later concatenated with another mask. Defaults to `True`. rTrNrIrFr)rrr~rwrxrvr}rrrr(rrr-)rrrrrrrr(r-rirrrrCr;)rrr}rrrrrrrrrrr~rwrxrIrrr(r-r r rs rrrsvR --$/:T:T2T#..44T:  Oi~iOKJY)V%5t@P Q&+#$22kl l )*?AR S&+##+%"$?!N r7c Uc?[US5(a,SUR;aURRS5nOSn[XX#XG5upppnU (aU$[ USS5nUc [ S5e[ U5(aX-U:a [ S5eURSURURnnnUb4URSS 9[R"U5:HRSS 9nO[R"UU[S 9n[!UU5n["UR$nS nU=(a [ US S 5=(a U S :H(+nUb&[&(d [ S5e[)UU5nS nSnUb&[&(d [ S5e[+UU5nS nSnU b[+U[-U 55nS nU"UU U U UUUUUUUUUS9 nU$)aE Create a chunked attention causal mask based on the attention implementation used (stored in the config). This type of attention pattern was mostly democratized by Llama4. If `past_key_values` has a hybrid cache structure, this function will return the mask corresponding to one of the "chunked_attention" layers (to align to what is needed in the `modeling_xxx.py` files). Args: config (`PreTrainedConfig`): The model config. inputs_embeds (`torch.Tensor`): The input embeddings of shape (batch_size, query_length, hidden_dim). This is used only to infer the batch size, query length and dtype. attention_mask (`torch.Tensor`, optional): The 2D attention mask corresponding to padded tokens of shape (batch_size, number_of_seen_tokens+q_length). It can also be an already prepared 4D mask, in which case it is returned as-is. past_key_values (`Cache`, optional): The past key values, if we use a cache. position_ids (`torch.Tensor`, optional) A 2D tensor of shape (batch_size, query_length) indicating the positions of each token in the sequences. or_mask_function (`Callable`, optional): An optional mask function to combine with the chunked causal mask function (by doing the union of both). This is useful to easily overlay another mask on top of the chunked causal one, for example for image tokens handling. and_mask_function (`Callable`, optional): An optional mask function to combine with the chunked causal mask function (by doing the intersection of both). This is useful to easily overlay another mask on top of the chunked causal one, for example for image tokens handling. layer_idx (`int`, *optional*): The cache layer to size the mask against. By default, the first "full_attention" layer is used, which is correct whenever all layers of a given mask type have seen the same tokens. Pass it explicitly for caches where layers of the same type hold different lengths (e.g. per-depth MTP streams). allow_is_causal_skip (`bool`, optional): Whether to allow returning `None` (and relying on `sdpa`'s `is_causal` argument) when the mask would be a plain causal mask. Set to `False` to always materialize the mask, e.g. when it is later concatenated with another mask. Defaults to `True`. NrTrattention_chunk_sizezQCould not find an `attention_chunk_size` argument in the config, or it is not setzFlash attention cannot handle chunked attention, and the key-value length is larger than the chunk size so the chunked pattern cannot be respected. You should use another `attn_implementation` when instantiating the modelr)rrFrrrr)rrrrrrr rr(r-rr* zeros_likerzerosrPrkrrrrCr;ru)rrr}rrrrrrrrqrr~rwrxrSrr(r-left_padding_tokensr r rr s rcreate_chunked_causal_maskr;s;Z ?L 1 1do>X>X6X'2288>II #6.[gs_J 4 U^!7>Jlmm$F++ 0E 0R }  !. 3 3A 6 8K8K]MaMavJ!-444<@P@PQ_@``eejlem#kk*V3O8EXY1&2M2MNN H0!159Kh!m5#22kl l ()>@P Q$$22kl l )*?AR S$' )*?A^_sAt u$!+%1K r7c UbURS:wagURSS:Xag[U5(d[R"US:H5(agUSS2URS*S24R 5$)uReturn the 2D padding mask for mamba / linear-attention layers, sized to the local sequence. Returns ``None`` (so the consumer skips masking entirely) when any of: - the input mask is missing or is already a custom 4D attention mask (no 2D padding signal); - the current forward is a single-token decode step (a generated token is never padding; this also keeps the growing 2D mask out of the compiled decode graph); - the mask is all-ones (un-padded batch — the masking multiply would be a no-op), skipped only outside trace/compile so the graph specialisation stays stable. Otherwise we trim the mask to the trailing ``inputs_embeds.shape[1]`` positions so it aligns with the current forward's local sequence and the consumer can multiply directly without further slicing. Note that this includes multi-token forwards continuing from a cache (chunked prefill, cache continuation): padding inside the new segment must still be zeroed out, otherwise it leaks into the recurrent state. Nrr)rrrr*r9 contiguous)rrr}rrs rcreate_recurrent_attention_maskrs{,!4!4!91" n % %%))Na4G*H*H !m11!4466 7 B B DDr7)r)full_attentionlinear_attention)sliding_attentionr) rr chunked_attentioncompressed_sparse_attentionheavily_compressed_attentionminimax_m3_sparsedeepseek_sparse_attentionqwen_sparse_attentionrconvhybridhybrid_slidingc UR5n U UUUUUUUS.n [U S5(a[U R5n [ SU 55(aU$0n U HXn [ U n[ U[5(a-UR5HunnX;dM U"S0U D6X'M MNU"S0U D6X'MZ U $[U SS5b [S0U D6$[U SS5b [S0U D6$[S0U D6$)aR This function mimics how we create the masks in the `modeling_xxx.py` files, and is used in places like `generate` in order to easily create the masks in advance, when we compile the forwards with Static caches. Args: config (`PreTrainedConfig`): The model config. inputs_embeds (`torch.Tensor`): The input embeddings of shape (batch_size, query_length, hidden_dim). This is used only to infer the batch size, query length and dtype. attention_mask (`torch.Tensor`, optional): The 2D attention mask corresponding to padded tokens of shape (batch_size, number_of_seen_tokens+q_length). It can also be an already prepared 4D mask, in which case it is returned as-is. past_key_values (`Cache`, optional): The past key values, if we use a cache. position_ids (`torch.Tensor`, optional) A 2D tensor of shape (batch_size, query_length) indicating the positions of each token in the sequences. or_mask_function (`Callable`, optional): An optional mask function to combine with the other mask function (by doing the union of both). This is useful to easily overlay another mask on top of the causal one, for example for image tokens handling. and_mask_function (`Callable`, optional): An optional mask function to combine with the other mask function (by doing the intersection of both). This is useful to easily overlay another mask on top of the causal one, for example for image tokens handling. block_sequence_ids (`torch.Tensor`, *optional*): A tensor of same shape as input IDs indicating to which block or group each token belongs to. Tokens from the same block will keep a bidirectional mask within the block, attending causally to the past. Index `-1` can be used for blocks that have to keep complete causality within itself. )rrr}rrrrrY layer_typesc32# UH o[;v M g7fr)&LAYER_PATTERN_TO_MASK_FUNCTION_MAPPING)r layer_types rr ,create_masks_for_generate..)siZhJ!GGZhsrINrr') get_text_configrsetr+anyr-rdictitemsrrrr )rrr}rrrrrYreffective_config mask_kwargslayer_patterns causal_masks layer_patternrvactual_patternactual_functions rcreate_masks_for_generater<s)P--/#&(*$,.0 K//-99: iZhi i i! ! +MB=QM-..7D7J7J7L3NO%97F7U7U 48M /<.Jk.J +, !#3T : F0?;?? !#94 @ L)8K88  , ,,r7r)NNNNNT)NNNNT)NNNN)Ucollections.abcr functoolsrr*torch.nn.functionalrrr cache_utilsrconfiguration_utilsrutilsrr utils.deprecationr utils.genericr r utils.import_utilsr rr!torch.nn.attention.flex_attentionrrrrrrr,torch._dynamo._trace_wrapped_higher_order_opr get_loggerrloggerr;rCrPr+rFrHrRrXr_rbrgrirkrprur|rr BoolTensorrrrrrrr-strrfloat32r(rrrrr__annotations__rtuplerr rrrrrr-r<r'r7rrOs % 12.I !!gNN I&?RV&W#02&T   H % x H  h 8 C3sCTX3#cSV[_ 3 8   5<< H %,,8$SSS   hshxhVSV VQYV      C TW \d  )<  Y\ afamamptat    6;llT6I VY fi  \\ *U%%*%*:*:*(, +,,%+++ +  + * + +\,,%* <(,,,%* >  X ><<>/4||>HM >bgbnbn>&"HIde 2*.!!%(- !&cccc c  c  cLL4'cd cc"&ccc LL3 c \\DcfcT2*.(-!&BBBB B  B  BLL4'B ;;B"&BB LL3 B \\BR2*.(((( (  (  (LL4'(^2*.!&:::: :  :  :LL4': LL3 ::z - 8N7O4O u||  t@S N26ea ea<<eaLL9,t3eaT\ ea ,,% ea Tz ea!<<$.ea 4 )D0#s :;eaZ)-(,)-.2 !%B B<<BLL4'BT\ B ,,% B o B $B t+BTzBB \\I$BR26$((,)-(,d d<<dLL4'd!<<$. d T\ d o d $d"&d \\I$dX)-(,)-.2 !%G G<<GLL4'GT\ G ,,% G o G $G t+GTzGG \\I$G\26$((,)-(,a a<<aLL4'a!<<$. a T\ a o a $a"&a \\I$aR)-(,)- !% <<LL4'T\  ,,%  o  $Tz \\I$L%) E E<<ELL4'ET\ E  \\D ED):3#D$E+!();RW!X$%7eT7 +!3Ihi,>Tst*&.)-(,)-.2N- N-<<N-LL4'N-T\ N- ,,% N- o N- $N- t+N-r7