j SSKrSSKJr SSKJr SSKJrJr SSKJ r SSK r SSK J r SSK JrJrJr "S S 5r\"S S 55rS \ S\4SjrS\S\S\S\4SjrS*S\S\S\S\4SjjrS\S\S\S\4SjrS+S\ R2S\\S\\S\SS4 SjjrS \S!\S"\S#\S$\ S\\4 S%jrS&\R:S\\4S'jrS\4S(jr \S)5r!g),N)contextmanager) dataclass)ceillog2)Any)PretrainedConfig)FutureRequestState RequestState RequestStatusc\rSrSrSrS SjrS SjrS\\S4S\ RRS-4S jr S\\S4S \ RRSS4S jr S rg)CudaGraphBufferz`A dict for CUDA graphs with a special __del__ method to make sure the graphs are properly reset.returnNc0UlgN_storage)selfs ڿC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\transformers/generation/continuous_batching/utils.py__init__CudaGraphBuffer.__init__s EG cUR(a@URR5upUR5 UR(aM?ggr)rpopitemreset)r_graphs r__del__CudaGraphBuffer.__del__!s1mm}},,.HA KKMmmmrkey.c8URRU5$r)rget)rr!s r get_graphCudaGraphBuffer.get_graph&s}}  %%rrc X RU'grr)rr!rs r set_graphCudaGraphBuffer.set_graph)s" crr)rN)__name__ __module__ __qualname____firstlineno____doc__rrtupleinttorchcuda CUDAGraphr$r'__static_attributes__rrrrsfjH &U38_&1E1E1L&#U38_#UZZ5I5I#d#rrcD\rSrSr%SrSr\\S'Sr\\S'Sr \\S'Sr g) WorkloadHints-zRA tiny dataclass containing hints to help choose good continuous batching defaultsrmax_prompt_lengthmax_generated_length num_requestsr4N) r)r*r+r,r-r8r/__annotations__r9r:r3r4rrr6r6-s%\s !#!L#rr6configrc URS;$)z:Checks if attention mask is needed for the given (config).)z paged|eagerz paged|sdpa)_attn_implementation)r<s rattn_mask_is_neededr?6s  & &*G GGrsize interval_size max_valuecXUS::aU$US:a[X- 5U-OUn[X25$)zQReturn the smallest multiple of (interval_size) >= (size), capped at (max_value).r)rmin)r@rArBpaddeds rpad_to_intervalrF;s5;?!8T$& '- 7F v !!rvalue min_valuec[U[SU55nS[[[U555-n[ X15$)zReturn the smallest power of 2 >= (value), capped at (max_value). If a minimum value is provided, the value is at least padded to that value.r )maxr/rrrD)rGrBrHrEs r pad_to_pow2rLCs: s1i( )E #d4;'( (F v !!rx divide_byalign_tocV[[X- 55nX-(a XX-- - nU$r)r/r)rMrNrOs raligned_dividerQKs, D  A|  && Hrattention_maskcumulative_seqlens_qcumulative_seqlens_ksliding_windowc*[R"UR5Rn[ [ U5S- 5HnXS-X- nX%S-X%- nXg:aUS:aXv- S-nOSn[ XXS-5n [ X%X%S-5n [R"USX4RUURURS9n [R"XS9n US:aXv- U- n U [R"XS9- n XSX4'M g)u~Builds an attention mask inplace using the cumulative seqlens of the query and key. If given a sliding window, it will also apply a sliding window mask on top. The attention mask is not boolean, it uses zeroes and -inf (or its equivalent) so it's more of an attention score bias tensor. The attention mask is a block-diagonal matrix, with each block an attention mask for a single query-key pair. Each of those block is built from a causal mask and, if there is a sliding window, a sliding window mask. An example is represented below, with seqlen_k = 8, seqlen_q = 4 and sliding_window = 6: CAUSAL MASK: █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ █ █ █ █ █ █ █ ░ █ █ █ █ █ █ █ █ SLIDING WINDOW MASK: ┌──────────────────────── seqlen_k - seqlen_q - sliding_window = 8 - 4 - 6 = -2 offset to the left <─┴─> ░ █ | █ █ █ █ █ █ █ █ ░ ░ | █ █ █ █ █ █ █ █ ░ ░ | ░ █ █ █ █ █ █ █ ░ ░ | ░ ░ █ █ █ █ █ █ ATTENTION MASK (sum of causal and sliding window masks): █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ Another example with seqlen_k = 5, seqlen_q = 3 and sliding_window = 2: CAUSAL MASK: █ █ █ ░ ░ █ █ █ █ ░ █ █ █ █ █ SLIDING WINDOW MASK: ┌──────────────────────── seqlen_k - seqlen_q - sliding_window = 5 - 3 - 2 = 0 offset to the left <┴> | ░ █ █ █ █ | ░ ░ █ █ █ | ░ ░ ░ █ █ ATTENTION MASK (sum of causal and sliding window masks): ░ █ █ ░ ░ ░ ░ █ █ ░ ░ ░ ░ █ █ r .)dtypedevice)diagonalN) r0finforWrDrangelenslicefullshaperXtriutril)rRrSrTrUrHiseqlen_qseqlen_kcausal_diagonal query_range key_range minus_infmaskedsliding_diagonals rbuild_attention_maskrkRs-t N00155I 3+,q0 1'A.1E1HH'A.1E1HH  8q=&1A5OO035Ia%5PQ .13GA3NO JJ 3 6 7 = =  &&!((  I@ A '2^C  ejjF FF6A)r)r )"r contextlibr dataclassesrmathrrtypingrr0 transformers.configuration_utilsrrequestsr r r rr6boolr?r/rFrLrQTensorlistrkrQueuerr.rrr4rrrs %! =EE##$  H 0HTH "#"c"c"c""s"s"s"3" c c S S  Q=LLQ=s)Q=s)Q= Q=  Q=h        : u{{ tL/A ##r