j<SSKJr SSKrSSKJr SSKJr SSKJr SSK J r SSK J r Jr SSKJrJr SS KJr SS KJrJrJr S S KJrJrJrJrJrJr S S KJ r S SK!J"r"J#r# \RH"\%5r&\"SS9\"SS\ 555r'"SS\5r("SS\"5r)"SS\5r*"SS\5r+"SS\5r,"SS\5r-\"SS \55r."S!S"\5r//S#Qr0g)$)CallableN)strict)initialization)PreTrainedConfig)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)CLIPMLP CLIPAttention CLIPEncoderCLIPEncoderLayerCLIPVisionEmbeddingsCLIPVisionModel)eager_attention_forward)Qwen2_5_VLVisionRotaryEmbeddingapply_rotary_pos_emb_visionz&DeepGlint-AI/mlcd-vit-bigG-patch14-336) checkpointc.\rSrSr%SrSrSrSrSr\ \ S'Sr \ \ S 'S r \ \ S 'S r \ \ S 'Sr\ \ S'Sr\ \ S'Sr\ \\ -\\ \ 4-\ S'Sr\ \\ -\\ \ 4-\ S'Sr\\ S'Sr\\ S'Sr\\ -\ S'Sr\\ S'Sr\\ S'S r\S -\ S!'S"rg )#MLCDVisionConfig)aF num_key_value_groups (`int`, *optional*, defaults to 1): Number of key-value groups used in Attention. Example: ```python >>> from transformers import MLCDVisionConfig, MLCDVisionModel >>> # Initializing a MLCDVisionConfig with DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> configuration = MLCDVisionConfig() >>> # Initializing a MLCDVisionModel (with random weights) from the DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> model = MLCDVisionModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```mlcd_vision_model vision_configaxiali hidden_sizei intermediate_size0num_hidden_layersnum_attention_headsnum_key_value_groupsr num_channelsiP image_size patch_sizegelu hidden_actgh㈵>layer_norm_epsattention_dropoutg{Gz?initializer_rangeg?initializer_factorNrope_parameters)__name__ __module__ __qualname____firstlineno____doc__ model_typebase_config_keydefault_rope_typer!int__annotations__r"r$r&r(r)r*listtupler,r.strr/floatr1r2r3r4dict__static_attributes__r5ڳC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\transformers/models/mlcd/modular_mlcd.pyrr)s &%J%OK!s!s!! !#!L#47Jd3i%S/1746Jd3i%S/16J NE %(us{(#u# ###'OTD['rFrc\rSrSrSrg)MLCDMLPSr5Nr6r7r8r9rEr5rFrGrIrISrFrIc\rSrSrSrg)MLCDRotaryEmbeddingWr5NrKr5rFrGrNrNWrLrFrNcj^\rSrSrS\4U4SjjrS\RS\R4Sjr Sr U=r $)MLCDVisionEmbeddings[configc(>[TU]U5 U?gN)super__init__position_embeddingselfrS __class__s rGrWMLCDVisionEmbeddings.__init__\s   #rF pixel_valuesreturncHURSnURRRnURUR US95nUR S5R SS5nURRUSS5n[R"XT/SS9nU$)Nr)dtyperr'dim) shapepatch_embeddingweightr`toflatten transposeclass_embeddingexpandtorchcat)rZr] batch_size target_dtype patch_embeds class_embeds embeddingss rGforwardMLCDVisionEmbeddings.forward`s!''* ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY ;C rFr5) r6r7r8r9rrWrl FloatTensorTensorrsrE __classcell__r[s@rGrQrQ[s2$/$ E$5$5 %,,  rFrQc^\rSrSrSrS\4U4SjjrS S\RS\ \R\R4S\RS-S \ \ S \ \R\RS-44 S jjr S r U=r$) MLCDAttentionmzMulti-headed attention with RoPE. Refer to papers: - Attention is all you need: https://huggingface.co/papers/1706.03762 - RoFormer: Enhanced Transformer with Rotary Position Embedding: https://huggingface.co/papers/2104.09864 rScT>[TU]U5 URUlSUlg)NF)rVrWr( is_causalrYs rGrWMLCDAttention.__init__us%  $*$?$?!rFN hidden_statesposition_embeddingsattention_maskkwargsr^c URSSupVURU5RXVURUR45nUR U5RXVURUR45nUR U5RXVURUR45n USRS5R5n USRS5R5n [XxX5upxURSSSS5R5nURSSSS5R5nU RSSSS5R5n [R"URR[ 5n U "UUUU U4UR"(dSO UR$UR&UR(S.UD6upU RSSSS5R5n U R+XeS5n UR-U 5n U RSSS5R5n X4$)Nrarr'rrr0)dropoutscalingr})rdq_projreshape num_headshead_dimk_projv_proj unsqueezerCrpermute contiguousr get_interfacerS_attn_implementationrtrainingrscaler}viewout_proj)rZrrrrrn seq_length query_states key_states value_statescossinattention_interface attn_output attn_weightss rGrsMLCDAttention.forwardzs&"/!4!4Sb!9 {{=199:SWSaSacgcpcp:qr [[/77QUQ_Q_aeanan8op {{=199:SWSaSacgcpcp:qr "!$..q1779!!$..q1779#>|Y\#b $++Aq!Q7BBD ''1a3>>@ #++Aq!Q7BBD (?(M(M KK , ,.E) %8      % $}}C$,,JJnn %  % ! "))!Q15@@B !&&zrB mmK0 !))!Q2==? ((rF)r}r(rU)r6r7r8r9r:rrWrlrvrAr r rsrErwrxs@rGrzrzms//3 ,)||,)#5<<#=>,) t+ ,) +, ,) u||U\\D00 1 ,),)rFrzc ^\rSrSrS\4U4SjjrS S\RS\\R\R4S\RS-S\ \ S \\R4 S jjr S r U=r$) MLCDEncoderLayerrScD>[TU]U5 [U5UlgrU)rVrWrz self_attnrYs rGrWMLCDEncoderLayer.__init__s  &v.rFNrrrrr^c UnURU5nUR"SUUUS.UD6upXQ-nUnURU5nURU5nXQ-nU$)ah Args: hidden_states (`torch.FloatTensor`): Input to the layer of shape `(batch, seq_len, embed_dim)`. Represents the hidden states from the previous layer or the input embeddings. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.FloatTensor`): Attention mask of shape `(batch, 1, q_len, k_v_seq_len)` where padding elements are indicated by very large negative values. )rrrr5) layer_norm1r layer_norm2mlp)rZrrrrresidual_s rGrsMLCDEncoderLayer.forwards}$!((7 >> ' 3)    !0  ((7 /  0 rF)rrU)r6r7r8r9rrWrlrvrAr r rursrErwrxs@rGrrs////3 "||"#5<<#=>" t+ " +, " u  ! ""rFrc ^\rSrSrSrS\4U4SjjrS S\RS\ \R\R4S\RS-S \ \ S \ \ -4 S jjrS rU=r$) MLCDEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`MLCDEncoderLayer`]. Args: config: MLCDVisionConfig rSc$>[TU]U5 g)z3Overwrite dummy `MLCDConfig` to `MLCDVisionConfig`.N)rVrWrYs rGrWMLCDEncoder.__init__s  rFN inputs_embedsrrrr^c TUnURHnU"UUU40UD6nM [US9$)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) )last_hidden_state)layersr)rZrrrrr encoder_layers rGrsMLCDEncoder.forwardsF,& ![[M)# M)+  rFr5rU)r6r7r8r9r:rrWrlrurArvr r rrsrErwrxs@rGrrs!/!/3 ! ((! #5<<#=>!  t+ ! +, !  ! ! rFrc^\rSrSr%\\S'SrS/rSrSr Sr Sr Sr Sr \\S.r\R$"5U4Sj5rS rU=r$) MLCDPreTrainedModelirS vision_modelrTF)r attentionsc>[TU]U5 URRn[ U[ 5(aURRn[ R"URSURS-U-S9 [ R"URRURRU-S9 [ R"UR[R "URR"S5R%S55 g[ U[&5(aURRnURS-SURR(-S--U-nURS-U-n[ R"UR*RUS9 [ R"UR,RUS9 [ R"UR.RUS9 [ R"UR0RUS9 g[ U[25(aURRnURR4S-SURR(-S--U-nSURR4-S-U-n[ R"UR6RUS9 [ R"UR8RUS9 g[ U[:5(amURRnURR4URR<-S-S-U-n[ R"UR>SUS9 gg) zInitialize the weightsr0g)meanstd)rra)r'rarN) rV _init_weightsrSr3 isinstancerQinitnormal_rj embed_dimrerfr2copy_ position_idsrlarangerdrkrzr$rrrrrIr!fc1fc2MLCDVisionModelr& class_pos_emb)rZmodulefactor in_proj_std out_proj_stdfc_std pos_emb_stdr[s rGr!MLCDPreTrainedModel._init_weightss f%// f2 3 3[[33F LL//cv?O?OQU?UX^?^ _ LL//66FMM<[<[^d[TU]U5 [U5Ul[R "[ R"SURUR-S-55Ul g)Nr'r) rVrWrNvision_rotary_embeddingnn Parameterrlrandnr!r&rrYs rGrWMLCDVisionModel.__init__3sP  ':6'B$\\%++a9K9KvOiOi9imn9n*oprFNr]rr^c Uc [S5eURU5nURU5nURSURR -nURSURR -n[ R"XARS9RS5RSU5n[ R"XQRS9RS5RUS5n[ R"UR5UR5/SS9nURX85n [ R"URUR/SS9n [ R"U R!5U S/SS9[ R"U R#5U S/SS9/n UR$"S UU S .UD6n U Sn U SS2SSS24n UR'U 5n [)U U S 9$) ac Example: ```python >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> from transformers import AutoProcessor, MLCDVisionModel >>> model = MLCDVisionModel.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> processor = AutoProcessor.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs, output_attentions=True) >>> features = outputs.last_hidden_state >>> print(f"Extracted features shape: {features.shape}") >>> print(f"Number of attention layers: {len(outputs.attentions)}") >>> print(f"Attention shape: {outputs.attentions[0].shape}") ```Nz You have to specify pixel_valuesra)devicer'rrb)rr)r pooler_outputr5) ValueErrorrr pre_layrnormrdrSr,rlrrrrkstackrhrrmrrrencoderpost_layernormr )rZr]rrnum_patches_heightnum_patches_widthhpos_idswpos_idspos_idsrrencoder_outputsr pooled_outputs rGrsMLCDVisionModel.forward8s:  ?@ @ 5 ))-8 )//3t{{7M7MM(..r2dkk6L6LL LL+4G4G H R RST U \ \]_ar s  LL*3F3F G Q QRS T [ [\npr s ++x//183C3C3EFBO"::=R 4#5#5t7I7I"JPRS II}((*,?,BC K II}((*,?,BC K  ,, ' 3   ,A.)!Q'2 ++M: )/'  rF)rrrU)r6r7r8r9rrWrlrur r rAr rsrErwrxs@rGrr2sYq/q26@ ''$.@ +,@  + + @ @ rFr)rrr)1collections.abcrrltorch.nnrhuggingface_hub.dataclassesrrrconfiguration_utilsrmodeling_outputsrr modeling_utilsr r processing_utilsr utilsr rrclip.modeling_cliprrrrrrllama.modeling_llamarqwen2_5_vl.modeling_qwen2_5_vlrr get_loggerr6loggerrrIrNrQrzrrrr__all__r5rFrGrs% .&3KF&@@;i   H %CD%('%(E%(P g  9 /$9)M9)x'''T. +. b*J/*J*JZF oF R rF