j<SSKJr SSKrSSKJr SSKJr SSKJr SSK J r SSK J r SSKJr SS KJr SS KJrJrJr SS KJrJrJrJrJr SS KJr SS KJrJ r SSK!J"r" SSK#J$r$J%r% SSK&J'r'J(r(J)r) SSK*J+r+J,r,J-r-J.r.J/r/J0r0J1r1 SSK2J3r3 SSK4J5r5 SSK6J7r7J8r8 SSK9J:r:J;r; SSKJ?r? SSK@JArAJBrBJCrCJDrDJErE SSKFJGrG SSKHJIrIJJrJJKrKJLrLJMrM SSKNJOrO SSKPJQrQ SSKRJSrS \/R"\U5rV/SQrW\-"S S!9\"S"S#\Q555rX\-"S S!9\"S$S%\G555rY\-"S S!9\"S&S'\ 555rZ"S(S)\'S*S+9r[\-"S,S-\55r\"S.S/\?5r]"S0S1\M5r^"S2S3\L5r_"S4S5\5r`"S6S7\S5ra\-"S8S9\%55rb\-"S:S;\b55rc"S<S=\C5rd"S>S?\I5re"S@SA\;5rf\0"SBSC\:55rg"SDSE\K5rh"SFSG\J5ri"SHSI\A5rj"SJSK\B5rk"SLSM\=5rl"SNSO\85rm"SPSQ\75rn\-"SRSS\(55rog)T)CallableN)strict)initialization)Cache)PreTrainedConfig)TorchvisionBackend) BatchFeature)divide_to_patchesgroup_images_by_shapereorder_images)OPENAI_CLIP_MEANOPENAI_CLIP_STD ImageInputPILImageResamplingSizeDict)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)ROPE_INIT_FUNCTIONS)ALL_ATTENTION_FUNCTIONSPreTrainedModel) ImagesKwargsProcessorMixinUnpack) TensorTypeTransformersKwargsauto_docstringcan_return_tupleloggingno_inherit_decorator torch_int)capture_outputs)get_vision_position_ids)$DeepseekOcr2ForConditionalGenerationDeepseekOcr2Model)DeepseekV4Experts DeepseekV4MLP)Gemma3TextModel)Kimi_K25VisionRotaryEmbedding)LagunaAttentionLagunaDecoderLayerLagunaRotaryEmbeddingapply_rotary_pos_embeager_attention_forward)MiniMaxM3VLTextConfig)MiniMaxM3VLRMSNormMiniMaxM3VLSparseMoeBlockMiniMaxM3VLTopKRouterMiniMaxM3VLVisionAttentionMiniMaxM3VLVisionMLP)apply_rotary_pos_emb_vision)SiglipVisionConfig)SiglipVisionEmbeddings) Step3p7ForConditionalGeneration Step3p7ModelStep3p7PreTrainedModelStep3p7TextModelStep3p7VisionModelStep3p7VisionConfigStep3p7TextConfig Step3p7ConfigStep3p7ImageProcessorStep3p7Processorzstepfun-ai/Step-3.7-Flash) checkpointc\rSrSr%SrSrSrSr\\ S'Sr \\ S'S r \\ S 'S r \\ S 'S r \\ S'Sr\\ S'Sr\\ S'Sr\\ S'Sr\\ S'Sr\S-\ S'Sr\\ S'SrSrg)r?Xstep3p5_vision vision_configaxiali hidden_size/num_hidden_layersnum_attention_heads image_size patch_size quick_gelu hidden_acth㈵>layer_norm_epsgUUUUUU@ mlp_ratiog?layer_scale_init_valueNrope_parametersi max_position_embeddingsc URSUR5UlURSUR5UlURSUR5UlURSUR5Ul[ R "U40UD6 [URUR-5Ul g)Nwidthlayersheads ls_init_value) poprJrLrNrXr __post_init__intrWintermediate_size)selfkwargss ڹC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\transformers/models/step3p7/modular_step3p7.pyra!Step3p7VisionConfig.__post_init__ms!::gt/?/?@!'Hd6L6L!M#)::gt7O7O#P &,jj$B]B]&^#&&t6v6!$T%5%5%F!G)rJrcrXrNrL)__name__ __module__ __qualname____firstlineno__ model_typebase_config_keydefault_rope_typerJrb__annotations__rLrNrPrRrTstrrVfloatrWrXrYdictrZra__static_attributes__rhrfr?r?Xs"J%OKs!!JJ"J" NE "Iu"$'E'#'OTD['#'S'Hrhr?c ^\rSrSr%SrSrSSSSSSS .rS rS rS r S S S S SSSSS.r \ "5r \ "5r \ "5r\ "5r\ "5r\ "5r\ "5r\ "5r\ "5r\ "5r\ "5r\ "5r\ "5r\ "5rSr\\S'Sr\\S'Sr\\S'Sr\\S'Sr \\S'Sr!\\S'Sr"\#\S'S r$\\S!'S"r%\\S'Sr&\\S'S r'\\S'S#r(\S#-\S$'S#r)\S#-\S%'S&r*\\S''S#r+\S#-\S('S#r,\\-\-S#-\S)'S r.\/\S*'S#r0\\#-S#-\S+'S,r1\#\S-'S r2\/\S.'S#r3\-\#\-S#-S#-\S/'S#r4\-\#\-S#-S#-\S0'S#r5\-\6S#-\S1'S#r7\-\6S#-\S2'U4S3jr8S4r9U=r:$)5r@va mlp_layer_types (`list[str]`, *optional*): Per-layer MLP type: `"sparse"` (MoE) or `"dense"`. If not provided, derived from the legacy `moe_layers_enum` hub-config kwarg (comma-separated string or list of MoE layer indices), defaulting to all layers from index 3 onward being MoE. n_routed_experts (`int`, *optional*, defaults to 288): Total number of routed experts. Accessible as `num_local_experts` via `attribute_map`. share_expert_dim (`int`, *optional*, defaults to 1280): Intermediate size of the always-active shared expert. num_sliding_attention_heads (`int`, *optional*): Attention head count for `"sliding_attention"` layers, if different from `num_attention_heads`. Defaults to the legacy `attention_other_setting` hub-config kwarg's `num_attention_heads` entry. Applied via a `per_layer_config` override (see `PreTrainedConfig`), not a per-layer list field. query_pre_attn_scalar (`int` or `float`, *optional*): `Step3p7Attention.__init__` hook point: defaults to `head_dim`, giving standard `head_dim ** -0.5` scaling; overridable per released checkpoint variant. moe_router_scaling_factor (`float`, *optional*, defaults to 1.0): Scaling factor applied to the MoE block's routed-expert output (`routed_scaling_factor` in `Step3p7SparseMoeBlock`). swiglu_limits (`list[float | None]`, *optional*): Per-layer gate/up clamping bound; `None` means no clamping. swiglu_limits_shared (`list[float | int | None]`, *optional*): Per-layer gate/up clamping bound for the always-active shared expert; `None` means no clamping. mtp_layer_types (`list[str]`, *optional*): Per-MTP-layer attention type; split off `layer_types`'s legacy trailing pad instead of being discarded, so `Step3p7Config.get_mtp_config()` can build the MTP layers for `generate(use_mtp=True)`. mtp_mlp_layer_types (`list[str]`, *optional*): Per-MTP-layer MLP type, analogous to `mtp_layer_types` for `mlp_layer_types`. step3p5n_routed_expertsnum_key_value_headsnum_experts_per_tokshare_expert_dimnum_nextn_predict_layers)num_local_expertsnum_attention_groupsmoe_num_experts moe_top_kshare_expert_dimsnum_mtp_layersg@TFcolwise_gather_outputrowwise_split_inputpacked_colwiserowwisemoe_tp_experts)zlayers.*.self_attn.q_projzlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.g_projzlayers.*.self_attn.o_projz!layers.*.mlp.experts.gate_up_projzlayers.*.mlp.experts.down_projzlayers.*.mlp.expertsirJi,rc-rLirZi/ vocab_sizerU rms_norm_epsimoe_intermediate_sizei Nsliding_windownum_sliding_attention_heads pad_token_id bos_token_id eos_token_idattention_biasquery_pre_attn_scalarg?moe_router_scaling_factormlp_bias swiglu_limitsswiglu_limits_sharedmtp_layer_typesmtp_mlp_layer_typesc >^URSS5nX lURURU-pCU(aSHUupV[X5n[ U[ 5(dM'[ U5U:XdM8[XUSU5 [XXsU5 MW SHEn[X5n[ U[ 5(dM%[ U5U:XdM6[XUSU5 MG SHCnURU5n[ U[ 5(dM+[ U5U:XdM<USUX'ME URcS/U-Ul U(a S/U-Ul URcURSS5n U bd[ U [5(aU RS5OU n U V s1sH.n [U 5R5(dM#[U 5iM0 n n O[![#S U55n [#U5V s/sH oU ;aS OS PM n n U SUUl U(a XUUlUR&cLURS S5nU(a"URS UR(5UlOUR(UlUR+S[-UR5VVs0sHunnUS:XdMUS UR&0_M snn5 UR.cUR0UlURSUR25nURSS5nURSS5n[4TU]l"S0UD6 [!UR5nUR8=(d 0m[;U4SjU55(aUVs0sH nUTU_M snUlg[ U[ 5(dU/[ UR5-n0UlUHDnURR=U5n SUU S.nU(aUU US'UUR8U'MF U(a0SUR8;aUR8SR?U5 gggs sn fs sn fs snnfs snf)Nr}r)) layer_typesr)mlp_layer_typesr)rr) rope_thetapartial_rotary_factorsfull_attentionmoe_layers_enum,rsparsedenseattention_other_settingrNper_layer_configsliding_attentionrr rope_scalingc3b># UH$n[TRU5[5v M& g7fN) isinstancegetrs).0 layer_typerps rf 2Step3p7TextConfig.__post_init__..,s$Un z"&&,d33ns,/default) rope_typerpartial_rotary_factorru) r`r}rLgetattrrlistlensetattrrrrrrqsplitstriprbsetrangerrrN setdefault enumeraterhead_dim default_thetasuperrarYallindexupdate)rdrer}npaddedfield mtp_fieldvaluekeyritemsimoe_setrr layer_idxrrrrlayer_type_setparamsr __class__s @rfraStep3p7TextConfig.__post_init__s+ $*::.H!#L (@%**D,B,BE],]6 #%  ,eT**s5zV/CDr3DUV_=%C,eT**s5zV/CDr3C@ 3eT**s5zV/C"')FK@    # 01A5D '(8'9Q_#`D j$//(\C0@0@,AA #%D , $$**:6'0 1 N)6LQ6OF2339$$Z0 -  0D4H4H H$$%56==lK!I|sD]& 0$as$#"Q Q=Q ) Q% :Q% Q+)rrrrr}rrrY);rirjrkrl__doc__rm attribute_maprgatinguse_bidirectional_attentionbase_model_tp_planAttributeErrordense_intermediate_sizeshared_intermediate_size rotary_dim swiglu_alpha swiglu_limit index_n_headsindex_head_dimindex_block_sizeindex_topk_blocksindex_local_blocksoutput_router_logitsrouted_scaling_factorrouter_aux_loss_coefrouter_jitter_noiserJrbrprcrzrLrZrrrrrryr{r|rrrrrrrboolrrrrrrrqrrart __classcell__rs@rfr@r@vs<<J/ 5-*/4 MM F"' &=%<%<%<%:-=*3 0 -.-/!J!#L!#L"$M#%N%'&('))+*,)+(*K"s"  s#)S)JL%!%3%c   c !%NC$J%.2t2L##L#*#+/L#S /D(/ ND 043;-4'*u*Hd59M4 d*+d29<@$us{T12T9@(,OT#Y%,,0cT)0_L_Lrhr@c^\rSrSr%Sr\\S.rSr\ \ -S-\ S'Sr \ \ -S-\ S'Sr \\ S'S r\\ S 'U4S jrS rU=r$) rAi<step3p7)rH text_configNrHrFprojector_biasiPimage_token_idc B>URc[5UlOd[UR[5(aE[S0URR 5VVs0sHup#US:wdM X#_M snnD6UlUR c[ 5UlOd[UR [5(aE[ S0UR R 5VVs0sHup#US:wdM X#_M snnD6Ul[TU] "S0UD6 gs snnfs snnf)Nrmru) rHr?rrsrrr@rra)rdrekvrs rfraStep3p7Config.__post_init__Gs    %!4!6D  **D 1 1!4"$($6$6$<$<$>T$>DA!|BS414$>T"D     #02D  (($ / /0tTEUEUE[E[E]3sE]TQabfrarDADE]3stD  ''U 4ts" D2D# D3D)rrH)rirjrkrlrmr?r@ sub_configsrHrsrrprrrrrbrartrrs@rfrArA<s]J$7HYZK48M4**T1826K((4/6 ND  NC  ( (rhrAc.\rSrSr%Sr\\S'\\S'Srg)Step3p7ImageProcessorKwargsiWa  patch_size (`int`, *optional*, defaults to 504): Target size (height = width) for each local patch crop. max_image_size (`int`, *optional*, defaults to 3024): Images larger than this (on their longest side) are scaled down uniformly before patch planning. rRmax_image_sizeruN)rirjrkrlrrbrprtrurhrfrrWsOrhrF)totalc>^\rSrSr%Sr\R rSSS.rSr \ \ S'Sr Sr \r\\\ S'\r\\\ S 'Sr\r/S QrS r\ \ S 'S r\ \ S'Sr\ \ S'\S\ S\ S\4Sj5rS\ S\ S\ S\ S\\\ \ 4\\ \ 4\ \ \ \44 SjrS*S\ S\ S\ 4Sjjr S\!RDS\ S\ SSS\\!RD\\!RD\ \ 44 Sjr#S\SS\S \S!\S\\\-S-S \\\-S-SSS"\$S\ S#\S-S$S%S\%4S&jr&\'S\(S'\)\S\%4U4S(jj5r*S)r+U=r,$)+rBidz Image processor for Step-3.7-Flash. Each input image is split into a global down-scaled view plus zero or more local patch crops via a sliding-window strategy, then every sub-image is resized and normalised independently. rOheightr\irRT image_mean image_std) pixel_valuespixel_values_localnum_local_patchesi rrQvision_patch_sizedownsampler_strider\rreturncR[X5S:=(a [X- X- 5S:$)zH`True` for near-degenerate images (min side < 32px, aspect ratio > 4:1). rminmax)r\rs rf_is_extreme_aspect(Step3p7ImageProcessor._is_extreme_aspects*5!B&R3u~v~+NQR+RRrhrPc^ URX5nU(a [X5=p[X!5UR:a1UR[X!5- n[X-5[X&-5p![ X!5[X!5pX::a X- S:aUOSm OX- S:a [ Xt5m OUm T S:Xa X4X4SSSU4$U 4SjX45up[SU T -5n [SU T -5n X4X4T XU4$)u%Compute the sliding-window patch layout for one image. Unlike models with a fixed tile size (Idefics3, LLaVA-NeXT), Step3p7 adapts the window size to the image's aspect ratio, so this cannot be reduced to a simple ``ceil(h / tile) × ceil(w / tile)`` formula. Step 1 = normalise extreme inputs: - extreme-aspect images (min_side < 32, ratio > 4) are squared - images larger than ``max_image_size`` are scaled down uniformly Step 2 — choose window size from the normalised aspect ratio: - fits in global view (long_side ≤ image_size): tile only if elongated (long_side / short_side > 1.5), using short_side as the window - very elongated (ratio > 4): ``min(short_side, patch_size)`` - standard case: ``patch_size`` Step 3 — snap each dimension to the nearest window multiple (snap up when the remainder exceeds 20 % of the window size). Returns: global_width_height: ``(width, height)`` to resize the global view to before squaring crop_width_height: ``(crop_width, crop_height)`` snapped dimensions for patch extraction window_size: tile side length (``0`` → no local patches) num_patches_x: number of patches along the width num_patches_y: number of patches along the height needs_square_pad: whether the raw image must be zero-padded to a square before resizing g?rrc3\># UH!nUT:aTUT-UT-ST-:--OUv M# g7f)g?Nru)rdim window_sizes rfr6Step3p7ImageProcessor._plan_patches..sL# '+%sk1S;5F{IZ5Z[\'s),r)r r rrbr)rdr\rrPrRneeds_square_padscale short_side long_side crop_width crop_height num_patches_x num_patches_yrs @rf _plan_patches#Step3p7ImageProcessor._plan_patchess> 225A  / /E v  3 3 3''#f*<(D*!K  #a 'j5K$K ! ?UOQ1>NN N#  #  Az[89 A{k9:  9; fvvvrhNcU=(d 0nURSUR5nUSnURSUR5nURX!XV5SSupxXx-$)zBReturn the number of local patches for an image of the given size.sizerrRr)rrrRr) rdrr\ images_kwargsrrPrRrrs rfget_number_of_image_patches1Step3p7ImageProcessor.get_number_of_image_patchessh%+   3(^ "&&|T__E '+'9'9%'`abcd'e$ ,,rhimgresamplercURupVnURXvX#5uupupn n nnU(a'[Xv5nURU/[ UUS9S9SnUR S5nUR U[ XS9US9nUR U[ X"S9US9RS5nU S:XaU/X4$UR U[ XS9US9RS5n[UU S9nUUX4$)aJStep3p7-specific cropping: square-pad extreme aspect ratios, resize the global view, and slice out raw (pre-final-resize) local-patch tiles per `_plan_patches`'s layout. Returns the resized global view, the list of raw local-patch tensors (still at `window_size`, not yet resized to `patch_size`), and the patch grid dimensions. r)pad_sizerr")rR) shaperr padr unsqueezeresizesqueezer )rdr!rPrRr"_rr\ global_width global_heightrrrrrrside img_batch global_img img_for_croppatchess rf_get_image_patches(Step3p7ImageProcessor._get_image_patchess995   uj E )\ %Z     u%D((C584t+L(MaPCMM!$ [[HM,^iq[r [[XZ-Zem[nvv   ! r=? ?{{ x{EPX# '!* $L[I7M@@rhimagesz torch.Tensor do_rescalerescale_factor do_normalizerdisable_groupingreturn_tensorszstr | TensorType | Nonec URn ////4upnnUHnURUXU5unnnnURU5 UR[U55 URU5 UR[ U5VVs/sH-n[ U5HnUUS- :H=(a UUS- :PM M/ snn5 M UR [ R"U5X#XEU5nUUS.n[UU S9n[USS9nU(a[USU S9unnUR5H5unn URU [XS9US 9n!UR U!X#XEU5UU'M7 [UUSS 9n"[ R"U"V#V$s/sHn#U#Hn$U$PM M sn$n#5US 'UV%s/sHn%U%S /U[U%5- --PM sn%US 'U$s snnfs sn$n#fs sn%f)Nr)rr)data tensor_typer)rT) is_nestedr9rr%)r>rFpatch_newline_masks)rr3appendrrrescale_and_normalizetorchstackr r r rr)rr )&rdr5r6r7r8rrr"rrRr9r:rerP global_imagesnested_patchesrr?r!r0r2rrrowcol global_stackr<result max_patchesgrouped_patches grouped_indexr&stacked_patchesresizednested_pixel_values_localper_image_patchespatchmasks& rf _preprocess!Step3p7ImageProcessor._preprocesssB[[ PRTVXZ\^P^M '8:MC@D@W@WZXA =J    ,  $ $S\ 2  ! !' *  & & %]33$]3=1,,H}q7H1HH3I3 "11 KK & L^g )!2 4^D+Q7 .C$AQ. *O]+:*?*?*A&++#XZ%R]e&*.)C)CZS\*& +B)7 ae(f %,1;;0Ii0I,WheWh0Ii,F' ( H[-GZtw+D "9::GZ-F( ) ]Pj-s 4G G -G rec &>[TU]"U40UD6$r)r preprocess)rdr5rers rfrV Step3p7ImageProcessor.preprocessFsw!&3F33rhrur)-rirjrkrlrrBILINEARr"rrRrbrpr6r8rrrrrrrdo_convert_rgbr valid_kwargsmodel_input_namesrrr staticmethodrr tuplerrrBTensorr3rr rSrrrrVrtrrs@rfrBrBdsW"**HC (DJJL.JU .,ItE{,N.LSNC sS#SsStSS>w>w"%>w36>wDG>w uS#Xc3hc3D E>w@-#-c-Z]-+A \\+A+A +A ' +A u||T%,,/c9 : +AZI^$II I  I DK'$. I4;&-I'III+I2I IV44v>Y7Z4_k44rhrBc\rSrSrSrSrg)Step3p7VisionRotaryEmbeddingiNcPURS5n[R"X"4SS9$)zM Recompose the frequencies into the final spatial layout used per each grid. rr)flattenrBcat)rdfreqfreq_hws rfrecomposition_frequencies6Step3p7VisionRotaryEmbedding.recomposition_frequenciesOs%,,q/yy'+44rhruN)rirjrkrlrhrtrurhrfr`r`Ns5rhr`c\rSrSrSrg)Step3p7VisionMLPiWruNrirjrkrlrtrurhrfrkrkWrhrkc\rSrSrSrg)Step3p7VisionAttentioni[ruNrlrurhrfroro[rmrhroc ^\rSrSrS\4U4SjjrS S\RS\RS-S\\R\R4S-S\R4S jjr S r U=r $) Step3p7VisionEncoderLayeri_configc>>[TU]5 Xl[U5Ul[ U5Ul[R"URURS9Ul [R"URURS9Ul [R"UR[R "UR5-5Ul[R"UR[R "UR5-5Ulg)Neps)r__init__rrro self_attnrkmlpnn LayerNormrJrVlayernorm_beforelayernorm_after ParameterrXrBoneslambda_1lambda_2rdrrrs rfrv"Step3p7VisionEncoderLayer.__init__`s  /7#F+ " V-?-?VEZEZ [!||F,>,>FDYDYZ V%B%BUZZPVPbPbEc%cd  V%B%BUZZPVPbPbEc%cd rhN hidden_statesattention_maskposition_embeddingsrc UnURU5nUR"U4UUS.UD6upURU-nX-nUnURU5nUR U5nUR U-nX-nU$)N)rr)r{rwrr|rxr)rdrrrreresidualr+s rfforward!Step3p7VisionEncoderLayer.forwardjs!--m< >>  ) 3     5 %0  ,,]; /  5 %0 rh)rrrrr|r{rxrwNN) rirjrkrlr?rvrBr^r]rrtrrs@rfrqrq_sre2e/3HL || t+#5<<#=>E   rhrqcr^\rSrSrS\4U4SjjrS\RS\S\S\R4Sjr S r U=r $) Step3p7VisionEmbeddingsirrc>[TU]U5 [R"URUR UR UR SS9Ulg)NF) in_channels out_channels kernel_sizestridebias)rrvryConv2d num_channelsrJrRpatch_embeddingrs rfrv Step3p7VisionEmbeddings.__init__sH  !yy++++))$$  rh embeddingsrr\rcURRRSnX R-nX0R-n[ US-5n[ R R5(d/XW:Xa*Xg:Xa%URRRS5$URRRS5nURSn URSXwU 5nURSSSS5n[RRUXV4SSS 9nURSSSS5RSSU 5nU$) Nrg?rbrrr%bilinearF)rmode align_corners)position_embeddingweightr&rRr"rBjit is_tracingr(reshapepermutery functional interpolateview) rdrrr\ num_positions new_height new_widthsqrt_num_positionspatch_pos_embedrs rfinterpolate_pos_encoding0Step3p7VisionEmbeddings.interpolate_pos_encodings-//66< >1188BB1Er")11!5G]`a)11!Q1=--33 ( 4  *11!Q1=BB1b#Nrh)r) rirjrkrlr?rvrBr^rbrrtrrs@rfrrsB 2 5<<UX]b]i]irhrc^\rSrSr%\\S'SrSrSS/rS/r Sr Sr Sr Sr Sr\R "5U4S j5rS rU=r$) r<irrmodelTrqStep3p7DecoderLayerpast_key_valuesFc>[TU]U5 [U[5(aO[R "UR [R"UR5RS55 g[U[5(a}[RRURURR 5 [RRUR"URR 5 g[U[$5(aUR&HnUR(nUR*US:wa[,UR*UnU"URUS9upE[R "[/XS35U5 [R "[/XS35U5 M g[U[05(aX[/URSS5n[R2"UR4SUS 9 [R2"UR6SUS 9 g[U[85(aX[/URSS5n[R2"UR:SUS 9 [R<"UR>5 g[U[@5(a![R<"UR:5 gg) N)rrbr)r _inv_freq_original_inv_freqinitializer_rangeg{Gz?)meanstd)!r _init_weightsrrinitcopy_ position_idsrBarangerexpandrqry constant_rrrrXrStep3p7RotaryEmbeddingrcompute_default_rope_parametersrrrStep3p7Expertsnormal_ gate_up_proj down_projStep3p7TopKRouterrzeros_e_score_correction_biasStep3p7RMSNorm)rdmoduler rope_init_fn curr_inv_freqr+rrs rfr$Step3p7PreTrainedModel._init_weightss f% f5 6 6 JJv**ELL9M9M,N,U,UV],^ _  9 : : GG  foov}}/S/S T GG  foov}}/S/S T  6 7 7$00 %EE ##J/9<#6v7G7G 7S#TL#/ *#U  76\+CDmT 76\9K+LM}] 1 / /$++':DAC LL,,3C @ LL)) =  1 2 2$++':DAC LLSc : KK66 7  / / KK &0rhru)rirjrkrlrArpbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_static_cache_supports_attention_backendrBno_gradrrtrrs@rfr<r<s_ &*#46KL#4"5 N!"& ]]_''rhr<c^\rSrSr%Sr\\S'\\S.r S\4U4Sjjr \ "SS9\ S\ RS \\S \4S j55rS rU=r$) r>iu!Vision encoder: patch embeddings → 2-D RoPE transformer layers → conv downsampler. The rotary embedding (``self.rotary_emb``) and layer stack (``self.layers``) are held directly on this module, following the Gemma4 convention of not wrapping them in a separate ``Encoder`` submodule. rrr attentionscP>[TU]U5 [U5Ul[R "UR URS9Ul[U5Ul [R"[UR5Vs/sHn[U5PM sn5Ul[R "UR UR S-SSSS9Ul[R "UR S-UR S-SSSS9UlUR'5 gs snf)Nrtr%rr)rrpaddingr)rrvrrryrzrJrV pre_layernormr` rotary_emb ModuleListrrLrqr]r downsampler1 downsampler2 post_init)rdrrr+rs rfrvStep3p7VisionModel.__init__s  1&9\\&*<*<&BWBWX6v>mmPUV\VnVnPo$pPo1%>v%FPo$pq IIf&8&8&:L:Lq:P^_histuII    "F$6$6$:RS]^   %qsD#F)tie_last_hidden_statesrrerc rURup4pVXPRR-nX`RR-nURUSS9n URU 5n [R "SXx//U R S9n [U SS9n URX5n URHn U "U 4SU 0UD6n M U RSnU RSS S5RX>Xx5n URU 5n URU 5n [U RS 5RSS S5S 9$) NT)rr)device)spatial_merge_sizerrbrr%)last_hidden_state)r&rrRrrBtensorrr$rr]rrrrrrd)rdrre batch_sizer+rr\grid_hgrid_w hidden_stategrid_thwrrlayerchannelss rfrStep3p7VisionModel.forwards5(4'9'9$ v??555//444|dS )),7 <Q>QR.xAN "oolI[[E aCVaZ`aL! %%b)#++Aq!499*PV_ ((6 ((6 1E1Ea1H1P1PQRTUWX1YZZrh)rrrr]rr)rirjrkrlrr?rprqro_can_record_outputsrvr#rrBr^rrrrrtrrs@rfr>r>sq 2, 2 E2[ELL[FCUc\rSrSrSrg)riruNrlrurhrfrrs rhrc\rSrSrSrg)ri ruNrlrurhrfrr rmrhrc2^\rSrSrSU4SjjrSrSrU=r$) Step3p7MLPi c>[TU]U5 U(a URO URUlUR(aURUOS=(d [ S5Ulg)Nrinf)rrvr|rcrrrlimit)rdrrris_shared_expertrs rfrvStep3p7MLP.__init__sV  >!,-33 3C \\!_ " " { " C~~di((rh)rcr)F)rirjrkrlrvrrtrrs@rfrr st))rhrcj^\rSrSrSrSU4SjjrS\RS\R4SjrSr U=r $) riTc|>[TU]U5 URUlUc[ S5UlgUUlg)Nr)rrvrintermediate_dimrrr)rdrrrrs rfrvStep3p7Experts.__init__s4  & < <%1%9U5\ | rhgate_uprcURSSS9up#URU5RURS9nURUR*URS9nX#-$)Nr%rbrcrr)chunkrrr)rdrrrs rf _apply_gateStep3p7Experts._apply_gate$sZ===+{{4 &&4::&6 XX4::+4::X 6yrh)rrr) rirjrkrl#_fp8_experts_clamp_after_activationrvrBr^r rtrrs@rfrrs/*.'L 5<<ELLrhrc\rSrSrSrg)ri+ruNrlrurhrfrr+rmrhrc\rSrSrSrSrg)Step3p7SparseMoeBlocki/c[RRU5 UR(aURU=(d SOSn[ U5Ul[ XS9Ul[XSS9Ul URUl g)N)rT)r) ryModulervrrrrexpertsrshared_expertsrr)rdrrrrs rfrvStep3p7SparseMoeBlock.__init__0sn 4 EKDXDX,,Y7?4^b %f- %fH (TR%+%E%E"rh)rrrrNrirjrkrlrvrtrurhrfrr/sFrhrc^\rSrSrS\S\4U4SjjrSS\RS\ \R\R4S\RS-S \ S-S \ \ S \ \R\RS-44 S jjr S rU=r$)Step3p7Attentioni;rrrcf>URn[TU] XU5 URS-Ulg)Ng)rNrrvrscaling)rdrrr num_headsrs rfrvStep3p7Attention.__init__<s/..  I633T9 rhNrrrrrerc 6URSSn/UQSPURP7nURURU5R U55R SS5nUR URU5R U55R SS5n URU5R U5R SS5n URU5n Uup[XX5upUbURXUR5up[R"URR ["5nU"UUU U U4UR$(dSO UR&UR(UR*S.UD6unnUR,"/UQSP76nURnUR"/USSQUR.PURP76nXR1S5R35-nUR"U6nUR5U5nUU4$)Nrbrr%r)dropoutrr)r&rq_normq_projr transposek_normk_projv_projg_projr/rrr get_interfacerr_attn_implementationr0trainingattention_dropoutrrrrr(sigmoido_proj)rdrrrrre input_shape hidden_shape query_states key_states value_states gate_statescossinattention_interface attn_output attn_weights output_shapes rfrStep3p7Attention.forwardAs$))#2. 88b8$--8 {{4;;}#=#B#B<#PQ[[\]_`a [[]!;!@!@!NOYYZ[]^_ {{=166|DNNqRST kk-0 &#7 RU#[  &'6'='=jX\XfXf'g $J(?(M(M KK , ,.E) %8      % $}}C$2H2HLL.. %  % ! \"));;;; "(( !&&Y Sb(9Y4>>Y4==Y !$9$9"$=$E$E$GG !&& 5 kk+. L((rh)rr)rirjrkrlr@rbrvrBr^r]rrrrrtrrs@rfrr;s:0:S:)- *)||*)#5<<#=>*) t+ *)  *) +, *) u||U\\D00 1*)*)rhrc\rSrSrSrSrg)rinc[RRU5 URUlURUn[ X25UlXR lURUUl URUS:Xa[O[nU"X5Ul [URURS9Ul[URURS9Ulg)Nrrt)ryrrvrJrrrwrrrattention_typerrrrxrrinput_layernormpost_attention_layernorm)rdrrr layer_config mlp_classs rfrvStep3p7DecoderLayer.__init__os 4 !--..y9 ),B &$00;.4-C-CI-NRZ-Z)`j V/-f.@.@fFYFYZ(6v7I7IvObOb(c%rh)r;rJr<rxr=rwNrrurhrfrrnsdrhrcF^\rSrSr%\\S'\\S.rS\4U4Sjjr Sr U=r $)r=irrrc >[TU]U5 [R"URUR UR 5Ul[R"[UR5Vs/sHn[X5PM sn5Ul [UR URS9Ul[!US9UlUR$(ai[URURUR$-5nUVs1sH nSUS3iM nn['UR(=(d /5U-UlUR+5 gs snfs snf)Nrt)rrz(^|\.)layers\.z\.)rrvry EmbeddingrrJ padding_idx embed_tokensrrrLrr]rrnormrrr}r"_keys_to_ignore_on_load_unexpectedr)rdrrr mtp_layerspatternsrs rfrvStep3p7TextModel.__init__s  LL):):F!B/HD69$:a:a:geg6hks6sD 3 %nEs 3EE)rGrEr]rFr) rirjrkrlr@rprrrrvrtrrs@rfr=r=s) ,& 0rhr=c \rSrSr%\\S'S\4Sjr\\S S\ RS\ RS-S\ \ S-S\ \-4S jj55rS rg) r;irrc[RX5 [UR5Ul[ UR 5UlUR RUl[R"URRS-UR RURS9Ul URUlUR!5 g)Nr)r)r<rvr>rH vision_modelr=rlanguage_modelrryLinearrJrmulti_modal_projectorrimage_placeholder_token_idr)rdrrs rfrvStep3p7Model.__init__s''5.v/C/CD.v/A/AB ,,77&(YY  , ,q 0&2D2D2P2PW]WlWl& "+1*?*?' rhNrrrrc UR"U40UD6nURUR5nUb2URUR"U400UESS0ED6R5OSnUcS/URS-n/nSn [ U5Hup/n U S:a4XyX-n U R U R SU RS55 U R XjR SURS55 X- n UR [U 5S:a[R"U 5OU S5 M [URUURURS9$)Noutput_hidden_statesFrrbr)r pooler_outputrr) rMrPrr&rr@rrrBrerrr)rdrrrre vision_outputimage_featurespatch_image_featuresmerged cur_patch_idxr num_patch cur_feature patch_slices rfget_image_featuresStep3p7Model.get_image_featuress{)),A&A 33M4S4ST "-  & &!!"4b8a68aCY[`8abtt    $!"n&:&:1&= =  %&78LAK1}2=C\] "";#3#3B 8I8I"8M#NO   ~055b.:N:Nr:RS T  &M MMC 4Dq4H%))K0kZ[n ]9*+== '55$//   rh)rQrNrPrMrr)rirjrkrlrArprvrrrBr^rrbr]rr^rtrurhrfr;r;sz  } 37.2 # ll# "LL4/#  9t+ # + + # # rhr;c \rSrSr%\\S'Srg)r:irrruN)rirjrkrlrArprtrurhrfr:r:s rhr:ch^\rSrSrSrS S U4Sjjjr\S\\4Sj5r S\ S\ S\4Sjr S r U=r$) rCiaAProcessor for Step-3.7-Flash. Uses :class:`ProcessorMixin.__call__` for the standard image-token expansion flow: the image processor splits each image into global + local patch crops, then :meth:`replace_image_token` builds the per-image replacement string that ``get_text_with_replacements`` substitutes into the text. rc >SUlUbURUR5OSUlURUR-nUR SU-S-UlURU-S-UlURUR -Ul URUR-Ul [TU]0"SXUS.UD6 g)Nz rr%)image_processor tokenizer chat_templateru) image_tokenconvert_tokens_to_idsrrrrnum_image_feature_sizerRnum_patch_feature_sizeimage_feature_placeholderpatch_feature_placeholderrrv)rdrcrdrererrs rfrvStep3p7Processor.__init__s'S\Shi==d>N>NOnr 22_5W5WW'6';';H'E'OTU&U#'6'A'AV'KPQ&Q#)-)9)9Dz zz z)rrrkrj) rdrprqre num_patchesr?patch_newline_maskreplrs rfreplace_image_token$Step3p7Processor.replace_image_tokens"#67 B *../DE?R?^0;dh{#A mD$B$B#C;O OD!!&8&;&;))$ *T;;rrrrrrrrr=r;r:rCrurhrfrs% .& 3;2XXff9K6FDD63jO4FNE<;   H % 67H,H8H867AL-AL8ALH67($(8(2 ,e c4.c4c4R5#@5 +  7 " :"J4@('_('('V/[//[/[d 2  '  ) ) &   -  F5 F0)0)fd,d*45 $5 p&J"~""rh