jy ddlZddlmZddlmZmZmZddlZddlZddl m Z ddl m cm Z ejddZddlmZddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZdd l m!Z!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+m,Z,ddl-m.Z.ddl/m0Z0m1Z1m2Z2dZ3dZ4dZ5dZ6dZ7dZ8dZ9dZ:dZ;dZd Z?d!Z@d"ZAGd#d$e jBZCGd%d&e jBZDGd'd(e jBZEGd)d*e jBZFd+ejGd,ejGd-ejGd.ejGd/eHejGejGff d0ZIGd1d2e jBZJGd3d4e jBZKGd5d6e jBZLed7Gd8d9e jBZMdYd:ZNGd;de jBZPGd?d@e jBZQee5dABGdCdDe"ZRe5GdEdFe'ZSGdGdHeSZTe5dIBGdJdKeSZUGdLdMe jBZVGdNdOe jBZWGdPdQe jBZXe5GdRdSeSZYee5dTBGdUdVe"ZZGdWdXeSeZ[dS)ZN) dataclass)AnyOptionalUnion USE_BF16_ROPE0) pay_attention)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)FlashAttentionKwargs)BaseModelOutputWithPast ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)PreTrainedModel)Unpack)TransformersKwargsis_torchdynamo_compiling)check_model_inputs) Qwen3VLConfigQwen3VLTextConfigQwen3VLVisionConfigFct|r2t|dkrt|dr |s|dSd}|S)Nrrc|SN)objs NC:\Users\johnn\OneDrive\Documents\AI\models\ideogram4\qwen3_vl_transformers.py decoratorz!auto_docstring..decorator9s )lencallable)argskwargsr$s r#auto_docstringr*5sL D Q8DG#4#4VAw r%cz|dkrdS||zt|dz}tdt||S)Nirr)maxmin)seq_len hidden_sizeintermediate_size chunk_sizes r#_ffn_chunk_sizer3?sD$q;&#.?*C*CCJ sC,, - --r%clt|tr|d}||S|SNr) isinstancelistclear)valuetensors r# _take_tensorr;Fs4%q  Lr%cztj||}t|dr||S)Ndimr8)torchcathasattrr8)tensorsr>outs r# _cat_tensorsrDNs: )G % % %Cw    Jr%c|}|rYtjr;tj|j|jdd}||d|S|ddS)NcpuT)dtypedevice pin_memoryF non_blocking)rHcopy) detachr?cuda is_availableemptyshaperGcopy_to)r:pinnedrCs r#_cache_tensor_cpurUUsz ]]__F %*))++k&,fl5UYZZZ &u --- 99E9 - --r%cn||t|o|jjdkdS)NrNrJ)rRboolrHtype)dstsrcrTs r#_copy_cached_prefixr[^s4IIcV%I 60I J JIKKKKKr%c`|dkr|dn|}|}t|}|dkrdS|d|s||drdS|S)Nrr)r>rWintsumitemany) token_types gen_positionsar_lens r#_static_ar_prefix_lenrebs$/OO$5$5$9$9+a..{K$$&&M =.%%'',,.. / /F {{qWfW!!##vww)?(?'D'D'F'Fr Mr%c |dS|dkr|jd|kr'|jd|kr|ddS|jdd|fvr|jd|kr|S|jddkr'|jd|kr|ddS|dkr|dddddddfS|dkr|ddddddfS|S)Nr)r>rQ transpose)attention_maskq_len num_headss r#!_attention_mask_for_pay_attentionrnms9tq    "i / /N4H4Ku4T4T!++Aq11 1   "q)n 4 49Ma9PTY9Y9Y! !   "a ' 'N,@,Cu,L,L!++Aq11 1q  aaaD!!!m,,q  aaatQQQ.// r%ctj||tj|}||d|d|}|ddddd|f|tj}||zS)NrGrHrrHrG)r?onesrWtril_viewrS)rk query_lengthrHcausalkey_masks r#_build_explicit_text_maskrx~s} Z l%*V T T TF LLNNN [[L!\ : :Faaat]l]:;>>fTYT^>__H H r%ct|}|dkr|S|j\}}}}|dddddddddf|||||}|||||z|SNr)r;rQexpandreshape) hidden_statesn_repbatchr/num_key_value_headshead_dims r#_repeat_kv_for_pay_attentionrs //M zz4A4G1E7'!!!!QQQ4"23::5'K^`egoppM  1Du1Lh W WWr%cvt|}|||}~|Sr )r;rj contiguous)x_listdim0dim1xrCs r#_transpose_contiguousrs6VA ++dD ! ! , , . .C Jr%c|||}||}|jddz}|dd|f}|d|df}|}||||d||||~|S)Nr]rh.r9) unsqueezerQclonemul_addcmul_)rcossin unsqueeze_dimhalfx1x2scratchs r#_apply_rotary_inplacers -- & &C -- & &C 72;! D 3:B 3:BhhjjGGGCLL"c,,,GGCLL'3''' Hr%c|\}}}||jdd\}}||j}|dkr|d}|jddkr|dkr||d}t|}||z } | dkr|||g} dx}x}}t| |dddS|dkr|||g} dx}x}}t| |dd S|} d}| ddd|f} |ddd|f} |ddd|f}| | |g} dx} x} }t| |ddd}| ddd|f |~ ~| dd|df} |} |}| | |g} dx} x} }dx}}t| |dd }| dd|df |~ ~| S) Nrhrrr]Tsdpa) softmax_scalervforce_attention recycle_qFrrvr) r8rQrSrHr>rr{rer rR)qkv_listrbrqkvrr/rdgen_lenattn_qkvrCq_attnk_attnv_attnout_arout_gens r#_pay_attention_token_typesrsZGAq! NNWRaR[NE7..**KA!++A.. q  UQYY!((33 "; / /FG!||q!9AX]4agswxxxx {{q!9AX]5\`aaaa C A GVG_F qqq'6'z]F qqq'6'z]F'H##F#Vf 8=_equ v v vF7F7 O&!!!& FGG_F F F'H##F#Vf LAHM%[_```G677 O'"""' Jr%c$eZdZfdZdZxZS)Qwen3VLVisionMLPc>t|j|_|j|_t j|j|jd|_t j|j|jd|_t|j |_ dSNTbias) super__init__r0r1nnLinear linear_fc1 linear_fc2r hidden_actact_fnselfconfig __class__s r#rzQwen3VLVisionMLP.__init__s !-!'!9)D$4d6LSWXXX)D$:D@VWW ??__\22F[[((F??6** *$l$Pl&8"&=Pt?OPPP1gz22  E ''E3z7U?3S3STTE..I I..I 22I JJr5)/""5 6 6 < thetarNctd|tjd|dtj|z zz }|d|d||_||_dS)N?rrhrrF persistent)rrr?arangefloatregister_bufferr>r)rr>rrrs r#rz%Qwen3VLVisionRotaryEmbedding.__init__ sp %ELC%+$N$N$NQT$TUV ZeDDD r%cd|jtjd|jdtj|jz zz }||_dS)Nrrrhr)rr?rr>rrrrs r#reset_inv_freqz+Qwen3VLVisionRotaryEmbedding.reset_inv_freqs>$*a1EK)X)X)X[_[c)cde  r%seqlenctj||jj|jj}tj||j}|S)Nrq)r?rrrHrGouter)rrseqfreqss r#rz$Qwen3VLVisionRotaryEmbedding.forwards:l6$-*>dmFYZZZ C// r%)rr) rrrr?r__annotations__r^rrrrrrs@r#rrslCD!!!!celr%rcNeZdZddeddffd ZdejdejfdZxZS) Qwen3VLVisionPatchMergerFrrNct|j|jdzz|_||_t j|r|jn|jd|_t j|j|j|_ t j |_ t j|j|j |_ dS)Nrhư>eps)rrr0spatial_merge_sizeuse_postshuffle_normr LayerNormnormrrGELUrout_hidden_sizer)rrrrs r#rz!Qwen3VLVisionPatchMerger.__init__s !-1JA1MN$8!L5I!a!1!1vOagklll )D$4d6FGGgii )D$4f6LMMr%rc@t|}||jr|d|jn|d|j}||}~||}||S)Nr])r;rrrtr0rrrrrrs r#rz Qwen3VLVisionPatchMerger.forward$s OO IId6OVaffR!1222UV W W \ \]_aeaq r r## V$$v&&&r%F) rrrrrr?rrrrs@r#rrsNN2NSWNNNNNN''%,''''''''r%rrrrrrct|}t|}|j}|j}||}}||}}t|||d|}t|||d|}||fS)Nr)r;rGrrrS)rrrr orig_q_dtype orig_k_dtypes r#apply_rotary_pos_emb_visionr-s QAQA7L7L 7799aggiiqAyy{{CIIKKCac2..11,??Aac2..11,??A a4Kr%c eZdZdeddffd Z d dejdejdeejdeeejejfdejf d Z xZ S) Qwen3VLVisionAttentionrrNct|j|_|j|_|j|jz|_d|_tj|j|jdzd|_ tj|j|j|_ |jdz|_ ||_ d|_ d|_dS)NrriTrgF)rrr0r>rmrnum_key_value_groupsrrqkvrscalingrattention_dropout is_causalrs r#rzQwen3VLVisionAttention.__init__<s %)DN2 $%!9TXtx!|$???Idh11 }d*  !$r%r} cu_seqlensrotary_pos_embposition_embeddingsc t|}|jd}|||d|jddddd}~|d\}} } ~|\} } |g} | g}dx}} t| || | \}} |d}| d} | d} |dd|ddz }d}|D]}||z}|dd||f | dd||f | dd||f g}t||j dd}|dd||f |~|}~ ~ ||d }~||}|S) Nrrir]rrhFTr)r;rQrr|rmpermuteunbindrrtolistrr rrRr)rr}rrrr) seq_lengthr query_states key_states value_statesrr query_listkey_listlengthscursorlengthendr chunk_output attn_outputs r#rzQwen3VLVisionAttention.forwardIs"%]33 "(+ hh}%%--j!T^RPPXXYZ\]_`bcdd 14A. j, &S"^ <$(( z#>z8UXZ]#^#^ j#--a00 ))!,, #--a00 abb>JssO3;;==  F6/CQQQs ]+6688111fSj=)4466QQQs ]+6688H )V[gklllL F3J ' - -l ; ; ;FF  "**:r::EEGG ii ,, r%NN) rrrrrr?rrtuplerrrs@r#r r ;s 2 t      "26KO ((|(L(!. ( &eEL%,,F&GH ( ((((((((r%r c eZdZd deddffd Z d dejdejdeejd eeejejfdejf d Z xZ S) Qwen3VLVisionBlockrattn_implementationrNcttj|jd|_tj|jd|_t||_t||_ dS)Nrrr) rrrrr0norm1norm2r attnrmlp)rrr)rs r#rzQwen3VLVisionBlock.__init__uss \&"4$??? \&"4$??? *&999 #6222r%r}rrrc 2t|}||}|g}d}|j|f|||d|}||~||}|g}d}||} || ~ |S)N)rrr)r;r,r.add_r-r/) rr}rrrr) normed_statesnormed_states_listr$ mlp_outputs r#rzQwen3VLVisionBlock.forward|s%]33  =11 +_ di  !) 3       ;'''  =11 +_ XX011 :&&& r%)rr%) rrrstrrr?rrr&rrrs@r#r(r(ts33C3T33333326KO |L!.  &eEL%,,F&GH  r%r(ceZdZUejed<d deffd Zd dZdZ ej e dZ xZ S) Qwen3VLTextRotaryEmbeddingrNrctt|dr(|j!|jdd|_nd|_|j|_|j|_||_ t|j|_ | |j |\}|_ | d|d|j|_|jpidgd|_dS) N rope_scaling rope_typedefaultrFr mrope_section)r>)rrrAr9getr:max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrr rope_init_fnattention_scalingrroriginal_inv_freqr<)rrrHrrs r#rz#Qwen3VLTextRotaryEmbedding.__init__s  6> * * 'v/B/N#044[)LLDNN&DN"("@$*$B! /?+/+<+offsetr!idxs r#apply_interleaved_mropez2Qwen3VLTextRotaryEmbedding.apply_interleaved_mropesx(..""$V1555 5 5KC"3'!+F**C %c3m 4GCH  r%c|jdkr(|dd|jdd}tdkr`|jdddddf|jd|jddd}n_|jdddddf|jd|jddd}|dddddddf}t|jj tr|jj d kr |jj nd }tj |d 5||zdd}~~|||j}|}|||j||jdddn #1swxYwY||j }||j }||fS)NrhN.rirr]1rHrmpsrFF) device_typeenabledr)rr{rQrrrrSrHrEr6rXr5r?autocastrjrMr<rsin_rrDrG) rr position_idsinv_freq_expandedposition_ids_expandedrSrrrs r#rz"Qwen3VLTextRotaryEmbedding.forwards   ! !' 299!\=OPQ=RTVWWL C   $ dD!!!T.A B H H J J M MUVU] M ^ ^ e efgiui{|}i~ACEF!G!G   $ 6tT111d7J K Q Q S S V V^_^f V g g n nopr~sEFGsHJLNO!P!P  ,QQQ4] ; A A C C'1!(-'E'Ek!(-[`J`J`ahmmfk ^ U C C C / /&,,..1F1L1L1N1NNYYZ[]^__E!#8008JKKE))++C JJLLL HHT+ , , , JJt- . . . / / / / / / / / / / / / / / /ff17f##hhQWh%%Cxs5B7H88H<?H<r r)rrrr?rrrrrrMno_gradrrrrs@r#r7r7sl\\0\\\\\\&**** "U]___r%r7RMSNormcTeZdZd deddffd ZdejdejfdZdZxZ S) Qwen3VLTextRMSNormrrrNcttjt j||_||_dS)zA Qwen3VLTextRMSNorm is equivalent to T5LayerNorm N)rrr Parameterr?rrrvariance_epsilon)rr0rrs r#rzQwen3VLTextRMSNorm.__init__sD l5:k#:#:;; #r%r}cvt|}tj||jj|j|jSr )r;Frms_normrrQr`)rr}s r#rzQwen3VLTextRMSNorm.forwards/$]33 z-):DKI^___r%cHt|jjd|jS)Nz, eps=)r&rrQr`rs r# extra_reprzQwen3VLTextRMSNorm.extra_reprs& )**II$2GIIIr%)r) rrrrrr?rrrfrrs@r#r]r]s$$$$$$$$$$`U\`el````JJJJJJJr%r]ct|}t|}t||||}t||||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )r;r)rrrrrWrs r#apply_rotary_pos_embrhsI( QAQAac=99Aac=99A a4Kr%ceZdZdZdedeffd Z ddejd e ejejfd e ejd e e d e ej d e ejde e dededeede eje ejffdZ ddejd e ejejfde dededejf dZxZS)Qwen3VLTextAttentionz=Multi-headed attention from 'Attention Is All You Need' paperr layer_idxc.t||_||_t |d|j|jz|_|j|jz|_ |jdz|_ |j |_ d|_ tj|j|j|jz|j|_tj|j|j|jz|j|_tj|j|j|jz|j|_tj|j|jz|j|j|_t)|j|j|_t)|j|j|_dS)Nrr Trr)rrrrkgetattrr0num_attention_headsrrr rrrrrattention_biasq_projk_projv_projo_projr] rms_norm_epsq_normk_normrrrkrs r#rzQwen3VLTextAttention.__init__s}  " F4F&Jd4dee $*$>&B\$\!}d* !'!9i   :T] JQWQf    i   :T] JQWQf    i   :T] JQWQf    i  & 68JQWQf    )Ft#|ddd| f||d<t#|ddd| f||d <| ||g}~ ~~|t%|||j}n>|du}t)|| d|jj}t/|||j|d }d}|jg| dR}||}||fS) Nr]rrh)rrryrbuiltFrrT)rkrrvr)r;rQrrprtrurjrqrvrrrhupdaterkrrr r?rUrrrnrrnr r|rrs)rr}rrkrxryrbrzr{rdr) input_shape hidden_shaperrrrrrr cache_kwargs value_listrr$rv attn_weightss r#rzQwen3VLTextAttention.forward.sF%]33 #)#2#. 88b8$-88 {{=1166|DD "^  {{:..88A>> [[//44\BB < [[**44Q:: {{=1166|DDNNqRSTT &S"^ <$(( z#7 HcSV#W#W j  &#&snUUL'6'='=j,X\Xfht'u'u $J "^  ,ZA>> < *8Q:: "^  ,ZA>> < 1(D~{[]`d`k`AAN'_c_ktzGKLLLK )k);;;;;;FFHH kk+.. L((r%cbt|}|jd}|jdd}g|d|jR}|||} | g} d} || dd} |||} | g} d} || dd} | ||dd} ~|\}}|dd|df}|dd|df}| g} | g} dx} } t| | ||\} } | g} d} t| dd} | g} d} t| dd} | g}d} t|dd} | g} d} t| |j } | g}d} t||j } ||z}| | jd|| jd| jd}| | jd|| jd| jd}t|ddd|f|d|t|ddd|f|d||dd|df| |dd|df| ~ ~ | ||g}~ ~~t#||jdd }~|jg|dR}||S) Nrr]rhrrirrFTr)r;rQrrprtrurjrqrvrrrhrrr rr[rRr rr|rrs)rr}rrzrdr{ suffix_lenrrrrrrrrrr total_lenkey_full value_fullrr$s r#forward_with_ar_cachez*Qwen3VLTextAttention.forward_with_ar_cachevs%]33 "(+ #)#2#. 88b8$-88 {{=1166|DD "^  {{:..88A>> [[//44\BB < [[**44Q:: {{=1166|DDNNqRSTT &S!!!VWW*o!!!VWW*o"^ <$(( z#7 HcSV#W#W j"^  ,ZA>> < *8Q:: "^  ,ZA>> < 1(Dq,A9lN`abNceqewxyez{{ HQQQZ0(3-QQQJqqq'6'z2HSM?SSSFGG"":...111fgg:$$\222   (J7 (J#HDLQVbfggg )k);;;;;;FFHH {{;'''r%)NNNNFrr)rrr__doc__rr^rr?rr&rr LongTensordictrWrrrrrrs@r#rjrjsGG 0 S      @,059.2#' %F)F)|F)#5<#=>F)!. F) "% F) !!12 F)el+F)4.F)F)F)-.F) u|Xel33 4F)F)F)F)\!& =(=(|=(#5<#=>=( =(  =(  =( =(=(=(=(=(=(=(=(r%rjc*eZdZfdZdZdZxZS)Qwen3VLTextMLPct||_|j|_|j|_t j|j|jd|_t j|j|jd|_t j|j|jd|_ t|j |_ dSNFr) rrrr0r1rr gate_projup_proj down_projr rrrs r#rzQwen3VLTextMLP.__init__s  !-!'!94#3T5KRWXXXy!143IPUVVV 4#94;KRWXXXV./ r%ct|}|||}||}~||~||Sr )r;rrrrr)rrrups r#_forward_chunkzQwen3VLTextMLP._forward_chunks` OOT^^A..// \\!__  B ~~f%%%r%c 6t|}|jdkr |jdn |jd}t||j|j}|dkr|g}d}||S|jg|jdd|jR}td||D]v}| d|t|||z }||}| d||jd |~~w|Sr) r;rrQr3r0r1rrrrr.rR) rrr/r2rrCrrrs r#rzQwen3VLTextMLP.forwards OO!"!!'"++$Wd.>@VWW ??SFA&&v.. .ak:173B3<:)9:::1gz22 ! !EHHRJ%(H(HIIE++E22I JJr5)/""5 6 6 <  ,)%+) 3+       q! m44 !55mDD +_ !344  m44  FQJJx||GU7S7SJ!2=GVG3Lo!^!^HX  $HW r%)NNNFNNFr)rrrrr^rr?rr&rrr rWrrrrrrs@r#rrsMh0hShhhhhh2637+/$)59#' %MM|M#5<#=>M!. M u/0 M "% MD>M!!12M4.MMM+,M MMMMMMMMr%rzJ Base class for Llava outputs, with hidden states and attentions. ) custom_introceZdZUdZdZeejed<dZ ee ed<dZ ee ejed<dZ ee ejed<dZeejed<dZeejed<dZeeed <dS) Qwen3VLModelOutputWithPasta[ past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Nlast_hidden_staterxr} attentions rope_deltasx_pred mid_results)rrrrrrr? FloatTensorrrxr r}r&rrrrrr7r!r%r#rr6s6:x 12999'+OXe_+++8z/Qwen3VLVisionModel.__init__..vs"$]$]$]A%7%?%?$]$]$]r%Frrc2g|]}tdS)Tr)rrs r#rz/Qwen3VLVisionModel.__init__..~s@     )!)-   r%)rrrrspatial_merge_unitr patch_embedr Embeddingnum_position_embeddingsr0 pos_embedr^num_grid_per_sidermrr ModuleListrdepthblocksrmergerdeepstack_visual_indexesr&deepstack_merger_listgradient_checkpointing)rrinputsr)rrs ` r#rzQwen3VLVisionModel.__init__fs|3&333F333"("; +"&"9D>>Jhv>>>J Z???I Z???I!D$!45 BYtUY[\[\[\^bObEccG qqq$!45 BYtUY[_abababObEccGnnXxZPPXXY[\\GnnXxZPPXXY[\\G['7!3<< |dddf|dddf|dddf}}}dtdD}dtdD}t|||D]\}}} tjd|jdz ||jjj} tjd|jdz | |jjj} | } | } | dz |jdz }| dz |jdz }| | z }| | z }| |jz}||jz}|dj | dz |dj |dz |dj | dz |dj |dz g}d|z dj d|z dz d|z dj |dz |dj d|z dz |dj |dz g}tdD]h}|| || || || itj|tj|jjj }tj||jjj|jjj }||}~||dddddf~|d}||d|d|d ~|d t||D}g}|jj}t||||D]~\}}}} ||d}||||z|| |z|d ddd ddd  dd}||~t7|dS)Nrrrhcg|]}gSr!r!rrs r#rzAQwen3VLVisionModel.fast_pos_embed_interpolate..s)))1B)))r%rgcg|]}gSr!r!rs r#rzAQwen3VLVisionModel.fast_pos_embed_interpolate..s,,,ar,,,r%rQ)r-rpricg|] \}}||z Sr!r!)rhws r#rzAQwen3VLVisionModel.fast_pos_embed_interpolate..s 2[2[2[TQ1q52[2[2[r%r]r=)rzipr?linspacerrrrHr^clipTrextendrr:rrGrrr1splitrrrrtrappendrD)rrgrid_tsgrid_hsgrid_wsidx_list weight_listtrrh_idxsw_idxs h_idxs_floor w_idxs_floor h_idxs_ceil w_idxs_ceildhdwbase_h base_h_ceilindicesweightsi idx_tensor weight_tensor pos_embedspatch_pos_embedspatch_pos_embeds_permuterrs r#fast_pos_embed_interpolatez-Qwen3VLVisionModel.fast_pos_embed_interpolates$,QQQTNHQQQTNHQQQPQTN'))a))),,588,,, 7GW55 ; ;GAq!^At'='A1T^MbMijjjF^At'='A1T^MbMijjjF!::<>K,t"44==??+d"33<<>>T"$|D'99BBDDT"${4'88AACC Gb&$!QVTN2;;==b&$!BtH,5577Dq2vtn,5577Dbh&//11 G1XX ; ; ""71:#4#4#6#6777A%%gaj&7&7&9&9:::: ;\(%*T^EZEabbb  t~4:4>CXC_   ^^J//  aaaDj1222 %a=..00jm,,11*Q-@@EEjQRmTTT +112[2[SRYEZEZ2[2[2[\\#% [3 "%&6'"R"R 7 7 Iq!Q!((A..Iq!z/:qJPZ\^__Aq!Q**A  % + +I 6 6 6 6 4!<<<