j+nSrSSKrSSKJr SSKJrJrJr SSKJr SSK J r SSK J r SS KJrJrJrJrJr SS KJr SS KJrJr S S KJr \R6"\5r"SS\R<5r"SS\R<5r \RBRDS5r#\RBRDS5r$\RBRDS5r%\RBRDS5r&\RBRDS\RNS\(4Sj5r)\RBRDS\RNS\RN4Sj5r*\RBRDS\RNS\RNS\(4Sj5r+\RBRDS\RNS\RN4Sj5r,"SS\R<5r-"S S!\R<5r."S"S#\R<5r/"S$S%\R<5r0"S&S'\R<5r1"S(S)\ 5r2"S*S+\R<5r3\"S,S-\55r4\"S.S/\455r5"S0S1\R<5r6"S2S3\R<5r7"S4S5\R<5r8"S6S7\R<5r9"S8S9\R<5r:\"S:S;\455r;"S<S=\R<5r<\"S>S?9"S@SA\455r=\"SBSC\455r>\"SDSE\455r?/SFQr@g)GzPyTorch DeBERTa model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)auto_docstringlogging) DebertaConfigc6^\rSrSrSrSU4SjjrSrSrU=r$)DebertaLayerNorm&z2LayerNorm module (epsilon inside the square root).c>[TU]5 [R"[R "U55Ul[R"[R"U55UlX l gN) super__init__r Parametertorchonesweightzerosbiasvariance_epsilon)selfsizeeps __class__s ںC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\transformers/models/deberta/modeling_deberta.pyrDebertaLayerNorm.__init__)sF ll5::d#34 LLT!23 #cHURnUR5nURSSS9nX- RS5RSSS9nX- [R "X@R -5- nURU5nURU-UR-nU$)NT)keepdim) dtypefloatmeanpowrsqrtr"torr!)r# hidden_states input_typer0varianceys r'forwardDebertaLayerNorm.forward/s"(( %++- !!"d!3!(--a055b$5G&-HG\G\<\1]] %((4 KK- '$)) 3r))r!r"r)g-q= __name__ __module__ __qualname____firstlineno____doc__rr8__static_attributes__ __classcell__r&s@r'rr&s<$ r)rc.^\rSrSrU4SjrSrSrU=r$)DebertaSelfOutput:c>[TU]5 [R"URUR5Ul[ URUR5Ul[R"UR5Ul gr) rrrLinear hidden_sizedenserlayer_norm_eps LayerNormDropouthidden_dropout_probdropoutr#configr&s r'rDebertaSelfOutput.__init__;s\ YYv1163E3EF )&*<*S>STzz&"<"<= r)cpURU5nURU5nURX-5nU$rrIrNrKr#r4 input_tensors r'r8DebertaSelfOutput.forwardA5 =1  ]3 }'CD r))rKrIrNr;r<r=r>rr8r@rArBs@r'rDrD:s> r)rDcURS5nURS5n[R"U[RURS9n[R"U[RURS9nUSS2S4UR SS5R US5- nUSU2SS24nURS5nU$)a Build relative position according to the query and key We assume the absolute position of query \(P_q\) is range from (0, query_size) and the absolute position of key \(P_k\) is range from (0, key_size), The relative positions from query to key is \(R_{q \rightarrow k} = P_q - P_k\) Args: query_size (int): the length of query key_size (int): the length of key Return: `torch.LongTensor`: A tensor with shape [1, query_size, key_size] r.deviceNrr+r)r$rarangelongr\viewrepeat unsqueeze) query_layer key_layer query_sizekey_sizeq_idsk_ids rel_pos_idss r'build_relative_positionriHs$!!"%J~~b!H LL5::k>P>P QE LLI>;++A. 0@0@0C[EUEUVWEXZfZkZklnZop qqr)cURURS5URS5URS5URS5/5$)NrrrZrk)rmrbrcs r'p2c_dynamic_expandrqjsG >>;++A. 0@0@0CY^^TVEWYbYgYghjYkl mmr)cURUR5SSURS5URS54-5$)Nr-rZrk) pos_indexp2c_attrcs r'pos_dynamic_expandruos=   GLLN2A.)..2DinnUWFX1YY ZZr)rb scale_factorc[R"[R"URS5[RS9U-5$)Nr+r.)rr2tensorr$r/)rbrvs r'scaled_size_sqrtrzws0 ::ell;#3#3B#7u{{KlZ [[r)rccdURS5URS5:wa [X5$U$NrZ)r$ri)rbrcrns r' build_rposr}|s/y~~b11&{>>r)max_relative_positionsc [R"[[UR S5UR S55U55$r|)rryminmaxr$)rbrcr~s r'compute_attention_spanrs4 <<C 0 0 4innR6HIKab ccr)c URS5URS5:wa>USS2SS2SS2S4RS5n[R"US[ X@U5S9$U$)NrZrr+r-dimindex)r$rargatherru)rtrbrcrnrss r'uneven_size_correctedrs\y~~b11 Aq!,66r: ||G2DYYb2cddr)c^\rSrSrSrU4SjrSrSS\RS\RS\ S \RS-S \RS-S \RS-S \ \R\RS-44S jjr S\RS\RS \RS \RS\ 4 Sjr SrU=r$)DisentangledSelfAttentionz Disentangled self-attention module Parameters: config (`str`): A model config class instance with the configuration to build a new model. The schema is similar to *BertConfig*, for more details, please refer [`DebertaConfig`] cn>[TU]5 URUR-S:wa&[ SURSURS35eURUl[ URUR- 5UlURUR -Ul[R"URURS-SS9Ul [R"[R"UR[RS95Ul[R"[R"UR[RS95UlUR"b UR"O/Ul[%US S5Ul[%US S5UlUR((a_[R"URURSS9Ul[R"URURSS9UlOSUlSUlUR&(a[%US S 5UlUR.S :aUR0Ul[R2"UR45UlSUR";a/[R"URURSS9UlSUR";a0[R"URUR5Ul[R2"UR<5Ulg)NrzThe hidden size (z6) is not a multiple of the number of attention heads ()rFr!rxrelative_attention talking_headr~r+rc2pp2c) rrrHnum_attention_heads ValueErrorintattention_head_size all_head_sizerrGin_projrrr r/q_biasv_bias pos_att_typegetattrrrhead_logits_projhead_weights_projr~max_position_embeddingsrLrM pos_dropoutpos_proj pos_q_projattention_probs_dropout_probrNrOs r'r"DisentangledSelfAttention.__init__sm     : : :a ?#F$6$6#78 445Q8 $*#=#= #&v'9'9FP>PW\ ] )))"$))F,>,>@R@R"Szz&"E"EF r)cUR5SSURS4-nURU5nURSSSS5$)Nr+rr-rr)r$rr_permute)r#x new_x_shapes r'transpose_for_scores.DisentangledSelfAttention.transpose_for_scoressFffhsmt'?'?&DD FF; yyAq!$$r)Nr4attention_maskoutput_attentions query_statesrnrel_embeddingsreturnc JUc5URU5nURU5RSSS9upn GOzURRRURS-SS9n [ S5V V s/sHDn [ R"[ UR5V s/sH oU S-U -PM sn SS9PMF nn n [ R"USUR5RUSRS95n[ R"USUR5RUSRS95n [ R"USUR5RUSRS95nXU4Vs/sHnURU5PM snupn XRURSSSS245-nXRURSSSS245-n SnS[UR5-n[!UU5nUURURS9- n[ R"XR#SS 55nUR$(a*Ub'Ub$UR'U5nUR)XXVU5nUbUU-nUR*b5UR+UR-SSSS55R-SSSS5nUR/5nUR1U)[ R2"UR5R45n[6R8R;USS9nUR=U5nUR>b5UR?UR-SSSS55R-SSSS5n[ R"UU 5nUR-SSSS5RA5nURC5SS S -nUREU5nU(dUS4$UU4$s sn fs sn n fs snf) a1 Call the module Args: hidden_states (`torch.FloatTensor`): Input states to the module usually the output from previous layer, it will be the Q,K and V in *Attention(Q,K,V)* attention_mask (`torch.BoolTensor`): An attention mask matrix of shape [*B*, *N*, *N*] where *B* is the batch size, *N* is the maximum sequence length in which element [i,j] = *1* means the *i* th token in the input can attend to the *j* th token. output_attentions (`bool`, *optional*): Whether return the attention matrix. query_states (`torch.FloatTensor`, *optional*): The *Q* state in *Attention(Q,K,V)*. relative_pos (`torch.LongTensor`): The relative position encoding between the tokens in the sequence. It's of shape [*B*, *N*, *N*] with values ranging in [*-max_relative_positions*, *max_relative_positions*]. rel_embeddings (`torch.FloatTensor`): The embedding of relative distances. It's a tensor of shape [\(2 \times \text{max_relative_positions}\), *hidden_size*]. Nrr+rrrxrr-rZ)r+)#rrchunkrrrangercatmatmultr3r.rrlenrrz transposerrdisentangled_att_biasrrbool masked_fillfinforr functionalsoftmaxrNr contiguousr$r_)r#r4rrrrnrqprbrc value_layerwskiqkvwqvrrel_attrvscaleattention_scoresattention_probs context_layernew_context_layer_shapes r'r8!DisentangledSelfAttention.forwardsL  m,B262K2KB2O2U2UVW]_2U2` /KK$$**4+C+Ca+GQ*OBhmnohpqhpcdEIIeDX$YY !$=$=dkk$PTVW->X$YY 3t0011  l;!EHH;3D3DH$EE  << 5H5HR5PQ  " "~'AlF^!--n=N00gstG  /'9   ,#445E5M5MaQRTUWX5YZbbcdfgijlmn ',,.+77.8I5;;WbWhWhKiKmKmn--//0@b/I,,7  ! ! -"44_5L5LQPQSTVW5XYaabcefhiklmO _kB %--aAq9DDF "/"4"4"6s";e"C%**+BC  !4( (//UWq3ds8-P%P9 PP Prbrcrvc Uc[XUR5nUR5S:Xa!URS5RS5nOVUR5S:XaURS5nO0UR5S:wa[ SUR535e[ XUR 5nUR5nUUR U- UR U-2SS24RS5nSnSUR;aURU5nURU5n[R"XRSS 55n [R"X6-SUS-S- 5n [R"U S[!XU5S 9n Xy- nS UR;aUR#U5n URU 5n U [%X5-n ['UUU5n [R"U *U-SUS-S- 5n [R"X+RSS 5R)UR*S 95n[R"US[-XU5S 9RSS 5n[/XX#5nX~- nU$) Nr-rrrz2Relative position ids must be of dim 2 or 3 or 4. rr+rZrrrx)rir\rrarrr~r^rrrrrrclamprrorrzr}r3r.rqr)r#rbrcrnrrvatt_spanscore pos_key_layerc2p_attrmpos_query_layerr_posp2c_posrts r'r/DisentangledSelfAttention.disentangled_att_bias"sq  2;;K]K]^L     "'11!4>>qAL    1 $'11!4L    1 $QR^RbRbRdQefg g)+$B]B]^#((* '  ' '( 2T5P5PS[5[ []^ ^ )A,  D%% % MM.9M 55mDMll;0G0GB0OPGkk,"91hlQ>NOGll7:LWco:pqG  E D%% %"oon=O"77HO /N NOE kk5&8"3Q1 q8HIGll9.G.GB.O.R.RYbYhYh.R.ijGllR'9'PY'ZiB ,G)ZG  E r))rrrNrrrr~rrrrrrrrrFNNN)r;r<r=r>r?rrrTensorrtupler8rrr@rArBs@r'rrs$GL%#(,0,0.2U0||U0 U0 U0 llT) U0 llT) U0 t+U0 u||U\\D00 1U0n6\\6<<6ll 6  6  66r)rc6^\rSrSrSrU4SjrSSjrSrU=r$)DebertaEmbeddingsi[zGConstruct the embeddings from word, position and token_type embeddings.c>[TU]5 [USS5n[USUR5Ul[ R "URURUS9Ul[USS5Ul UR(dSUl O0[ R "URUR5Ul URS:a1[ R "URUR5Ul OSUl URUR:wa0[ R"URURSS9UlOSUl[!URUR"5Ul[ R&"UR(5UlXl[ R."[0R2"UR5R5S 5SS 9Ulg) N pad_token_idrembedding_size) padding_idxposition_biased_inputTFrrr+) persistent)rrrrHrr Embedding vocab_sizeword_embeddingsrposition_embeddingsrtype_vocab_sizetoken_type_embeddingsrG embed_projrrJrKrLrMrNrPBufferrr]rl position_ids)r#rPrr&s r'rDebertaEmbeddings.__init__^sp v~q9 %f.>@R@RS!||F,=,=t?R?R`lm%,V5Ld%S"))'+D $')||F4R4RTXTgTg'hD $  ! !A %)+f6L6LdNaNa)bD &)-D &   &"4"4 4 ii(;(;V=O=OV[\DO"DO)&*<*S>STzz&"<"<=  IIell63Q3Q&R&Y&YZa&botur)cUbUR5nOUR5SSnUSnUcURSS2SU24nUc8[R"U[RURR S9nUcUR U5nURb URUR 55nO[R"U5nUn UR(aX-n URbURU5n X-n URbURU 5n URU 5n UbUR5U R5:waEUR5S:Xa URS5RS5nURS5nUR!U R"5nX-n UR%U 5n U $)Nr+rr[rr-)r$rrr r^r\rr zeros_likerrrrKrsqueezerar3r.rN) r# input_idstoken_type_idsrmask inputs_embeds input_shape seq_lengthr embeddingsrs r'r8DebertaEmbeddings.forward{s  #..*K',,.s3K ^  ,,Q  ^$>~$N !#;J ?? &4J^^J/  xxzZ^^--88:?<<?2215D~~a(77:++,D#*J\\*- r)) rKrPrNrrrrrrr)NNNNNr:rBs@r'rr[sQv:,,r)rc~^\rSrSrU4SjrSS\S\\R\RS-44Sjjr Sr U=r $) DebertaAttentionicn>[TU]5 [U5Ul[ U5UlXlgr)rrrr#rDoutputrPrOs r'rDebertaAttention.__init__s+ -f5 '/  r)Nrrc vURUUUUUUS9upxUcUnURXt5n U(aX4$U S4$)N)rrnr)r#r) r#r4rrrrnr self_output att_matrixattention_outputs r'r8DebertaAttention.forwards_#'))   %%) #,#    (L;;{A $1 1$d+ +r))rPrr#r r;r<r=r>rrrrrr8r@rArBs@r'rrsK#(, , u||U\\D00 1,,r)rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)DebertaIntermediateic>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) rrrrGrHintermediate_sizerI isinstance hidden_actstrr intermediate_act_fnrOs r'rDebertaIntermediate.__init__s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r)r4rcJURU5nURU5nU$rrIr r#r4s r'r8DebertaIntermediate.forwards& =1 00? r)r r;r<r=r>rrrr8r@rArBs@r'rrs(9U\\ellr)rc.^\rSrSrU4SjrSrSrU=r$) DebertaOutputic">[TU]5 [R"URUR 5Ul[UR UR5Ul [R"UR5Ul Xl gr)rrrrGr rHrIrrJrKrLrMrNrPrOs r'rDebertaOutput.__init__sa YYv779K9KL )&*<*S>STzz&"<"<=  r)cpURU5nURU5nURX-5nU$rrSrTs r'r8DebertaOutput.forwardrWr))rKrPrIrNrXrBs@r'rrsr)rc~^\rSrSrU4SjrSS\S\\R\RS-44Sjjr Sr U=r $) DebertaLayeric>[TU]5 [U5Ul[ U5Ul[ U5Ulgr)rrr attentionr intermediaterrrOs r'rDebertaLayer.__init__s3 )&1/7#F+ r)Nrrc URUUUUUUS9upxURU5n URX5n U(aX4$U S4$)Nrrrnrrrr) r#r4rrrnrrrrintermediate_output layer_outputs r'r8DebertaLayer.forwardsg(,~~  /%%) (6( $#//0@A{{#6I  - - $' 'r)r")NNNFrrBs@r'rrsK,"'( ( u||U\\D00 1((r)rc ^\rSrSrSrU4SjrSrSrSSjrSS\ RS\ RS \ S \ S \ 4 S jjr S r U=r$)DebertaEncoderi z8Modified BertEncoder with relative position bias supportc>[TU]5 [R"[ UR 5Vs/sHn[ U5PM sn5Ul[USS5Ul UR(af[USS5Ul URS:aURUl [R"URS-UR5UlSUlgs snf)NrFr~r+rr-)rrr ModuleListrnum_hidden_layersrlayerrrr~rrrHrgradient_checkpointing)r#rP_r&s r'rDebertaEncoder.__init__s ]]%H`H`Ba#bBaQL$8Ba#bc ")&2F"N  " "*1&:RTV*WD '**Q..4.L.L+"$,,t/J/JQ/NPVPbPb"cD &+#$csC)c\UR(aURRnU$SnU$r)rrr)r#rs r'get_rel_embedding DebertaEncoder.get_rel_embeddings07;7N7N,,33UYr)cUR5S::aDURS5RS5nX"RS5RS5-nU$UR5S:XaURS5nU$)Nr-rrZr+r)rrar)r#rextended_attention_masks r'get_attention_mask!DebertaEncoder.get_attention_masks    1 $&4&>&>q&A&K&KA&N #47V7VWY7Z7d7deg7hhN   !Q &+55a8Nr)cdUR(aUcUb [X!5nU$[X5nU$r)rri)r#r4rrns r' get_rel_posDebertaEncoder.get_rel_pos's:  " "|';'6|S  7}T r)r4routput_hidden_statesr return_dictc ~URU5nURXU5nU(aU4OSnU(aSOSn Un UR5n [UR5H4upU "U UUUU US9upU(aX4-nUbUnOUn U(dM/X4-n M6 U(d[ SXU 455$[ XU S9$)N)rrnrrc3.# UH ocMUv M g7frr<).0rs r' )DebertaEncoder.forward..Xsh$Vq$Vs last_hidden_stater4 attentions)r4r7r0 enumerater+rr )r#r4rr9rrrnr:all_hidden_statesall_attentionsnext_kvrr layer_moduleatt_ms r'r8DebertaEncoder.forward/s00@'' \R L`8Hfj0d//1(4OA#/))-"3 $ M$$58H$H!', '  !/(!:' 5*h]~$Vhh h+Yg  r))r,r+r~rr)NN)TFNNT)r;r<r=r>r?rr0r4r7rrrr8r@rArBs@r'r'r' smB ,&*"' , ||,  , # ,  , , , r)r'cj^\rSrSr%\\S'SrS/rSr\ R"5U4Sj5r Sr U=r $)DebertaPreTrainedModeli^rPdebertarTc*>[TU]U5 [U[5(aA[R "UR 5 [R "UR5 g[U[[45(a![R "UR5 g[U[5(a\[R"UR[R"URR S5R#S55 gg)zInitialize the weights.r+rN)r _init_weightsr rinitzeros_rrLegacyDebertaLMPredictionHeadDebertaLMPredictionHeadr!rcopy_rrr]shaperl)r#moduler&s r'rO$DebertaPreTrainedModel._init_weightses f% f7 8 8 KK & KK & !>@W X Y Y KK $  1 2 2 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h3r)r<)r;r<r=r>r__annotations__base_model_prefix"_keys_to_ignore_on_load_unexpectedsupports_gradient_checkpointingrno_gradrOr@rArBs@r'rLrL^s8 !*?)@&&*# ]]_ i ir)rLc^\rSrSrU4SjrSrSr\SS\RS-S\RS-S\RS-S \RS-S \RS-S \ S-S \ S-S \ S-S\ \ -4Sjj5r SrU=r$) DebertaModelirc>[TU]U5 [U5Ul[ U5UlSUlXlUR5 gNr) rrrrr'encoderz_stepsrP post_initrOs r'rDebertaModel.__init__ts>  +F3%f-    r)c.URR$rrrr#s r'get_input_embeddings!DebertaModel.get_input_embeddings~s...r)c$XRlgrrfr#new_embeddingss r'set_input_embeddings!DebertaModel.set_input_embeddingss *8'r)Nrrrrrrr9r:rc UbUOURRnUbUOURRnUbUOURRnUbUb [ S5eUb"UR X5 UR 5n O"UbUR 5SSn O [ S5eUb URO URn Uc[R"XS9nUc$[R"U [RU S9nURUUUUUS9n URU USUUS9n U S nURS :aUS n[UR5Vs/sHnURR SPM nnUSnURR#5nURR%U5nURR'U 5nUS SHnU"UUS UUUS 9nUR)U5 M! USnU(dU4X(aS S-$S S-$[+UU(a U R,OSU R.S9$s snf)NzDYou cannot specify both input_ids and inputs_embeds at the same timer+z5You have to specify either input_ids or inputs_embeds)r\r[)rrrrrT)r9rr:rrZFr!r-rA)rPrr9r:r%warn_if_padding_and_no_attention_maskr$r\rrr r^rrarbrr+r0r4r7appendr r4rC)r#rrrrrrr9r:kwargsrr\embedding_outputencoder_outputsencoded_layersr4r-layersrrrel_posr+sequence_outputs r'r8DebertaModel.forwardst2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U%.%:!! @T@T  !"ZZ CN  !"[[EJJvVN??)%' + ,,  !%/# ' )+ <R8\(]] ]XY8\(]] ]-;O/77UY&11  +Ks#I)rPrrarb)NNNNNNNN)r;r<r=r>rrhrmrrrrrr r8r@rArBs@r'r^r^rs/9*..2.2,0-1)-,0#'O <<$&O  t+O  t+ O llT) O ||d* O  $;O #TkO D[O   O O r)r^c.^\rSrSrU4SjrSrSrU=r$)$LegacyDebertaPredictionHeadTransformic>[TU]5 [USUR5Ul[ R "URUR5Ul[UR[5(a[URUl OURUl [ R"URURS9Ul g)Nr)r%)rrrrHrrrGrIr r r r transform_act_fnrKrJrOs r'r-LegacyDebertaPredictionHeadTransform.__init__s %f.>@R@RSYYv1143F3FG f'' - -$*6+<+<$=D !$*$5$5D !d&9&9v?T?TUr)clURU5nURU5nURU5nU$r)rIr}rKrs r'r8,LegacyDebertaPredictionHeadTransform.forwards4 =1 --m< }5 r))rKrIrr}rXrBs@r'r{r{s Vr)r{c.^\rSrSrU4SjrSrSrU=r$)rRicJ>[TU]5 [U5Ul[ USUR 5Ul[R"UR URSS9Ul [R"[R"UR55Ulg)NrTr)rrr{ transformrrHrrrGrdecoderrrr r!rOs r'r&LegacyDebertaLMPredictionHead.__init__ss =fE%f.>@R@RSyy!4!4f6G6GdS LLV->->!?@ r)cJURU5nURU5nU$r)rrrs r'r8%LegacyDebertaLMPredictionHead.forwards$}5  ]3 r))r!rrrrXrBs@r'rRrRs Ar)rRcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)LegacyDebertaOnlyMLMHeadicB>[TU]5 [U5Ulgr)rrrR predictionsrOs r'r!LegacyDebertaOnlyMLMHead.__init__s 8@r)rxrc(URU5nU$rr)r#rxprediction_scoress r'r8 LegacyDebertaOnlyMLMHead.forwards ,,_=  r)rrrBs@r'rrs)A!u||! !!r)rc2^\rSrSrSrU4SjrSrSrU=r$)rSizMhttps://github.com/microsoft/DeBERTa/blob/master/DeBERTa/deberta/bert.py#L270c>[TU]5 [R"URUR5Ul[ UR[5(a[URUl OURUl [R"URURSS9Ul [R"[R"UR 55Ulg)NT)r%elementwise_affine)rrrrGrHrIr r r r r}rKrJrrr rr!rOs r'r DebertaLMPredictionHead.__init__ s YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>ShlmLLV->->!?@ r)cURU5nURU5nURU5n[R"XR R 55UR-nU$r)rIr}rKrrrrr!)r#r4rs r'r8DebertaLMPredictionHead.forwardsb =1 --m<    ]4J4J4L4L4NORVR[R[[ r))rKr!rIr}r:rBs@r'rSrSsW Ar)rSc.^\rSrSrU4SjrSrSrU=r$)DebertaOnlyMLMHeadi"cB>[TU]5 [U5Ulgr)rrrSlm_headrOs r'rDebertaOnlyMLMHead.__init__#s .v6 r)c(URX5nU$rr)r#rxrrs r'r8DebertaOnlyMLMHead.forward(s LLJ  r)rrXrBs@r'rr"s7 !!r)rcD^\rSrSrSSS.rU4SjrSrSr\SS \ RS-S \ RS-S \ RS-S \ RS-S \ RS-S\ RS-S\ S-S\ S-S\ S-S\ \ -4Sjj5rSrU=r$)DebertaForMaskedLMi-zcls.predictions.bias)deberta.embeddings.word_embeddings.weight)zcls.predictions.decoder.biaszcls.predictions.decoder.weightc>[TU]U5 URUl[U5UlUR(a[ U5UlOSS0Ul[U5Ul UR5 g)Nzlm_predictions.lm_head.weightr) rrlegacyr^rMrcls_tied_weights_keysrlm_predictionsrcrOs r'rDebertaForMaskedLM.__init__4sg  mm #F+ ;;/7DH01\'D ##5V"   ' ' -/=/B/BD   ' ' ,r)Nrrrrrlabelsrr9r:rc U bU OURRn URUUUUUUUU S9n U Sn UR(aUR U 5n O/UR XRR R5n SnUbF[5nU"U RSURR5URS55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrrrrrr9r:rr+rlosslogitsr4rC)rPr:rMrrrrrrr_rr r4rC)r#rrrrrrrr9r:rroutputsrxrmasked_lm_lossloss_fctrs r'r8DebertaForMaskedLM.forwardQs*&1%rrrrrrrrrr r8r@rArBs@r'rr-s)?*U 5 C*..2.2,0-1&*)-,0#'5 <<$&5  t+5  t+ 5 llT) 5 ||d* 5  t#5  $;5 #Tk5 D[5   5 5 r)rc>^\rSrSrU4SjrSr\S5rSrU=r $) ContextPooleric>[TU]5 [R"URUR5Ul[R "UR5UlXl gr) rrrrGpooler_hidden_sizerIrLpooler_dropoutrNrPrOs r'rContextPooler.__init__sG YYv88&:S:ST zz&"7"78  r)cUSS2S4nURU5nURU5n[URR"U5nU$r`)rNrIr rPpooler_hidden_act)r#r4 context_token pooled_outputs r'r8ContextPooler.forwardsM&ad+  ]3  =1 t{{<<=mL r)c.URR$r)rPrHrgs r' output_dimContextPooler.output_dims{{&&&r))rPrIrN) r;r<r=r>rr8propertyrr@rArBs@r'rrs! ''r)rz DeBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introc:^\rSrSrU4SjrSrSr\SS\RS-S\RS-S\RS-S \RS-S \RS-S \RS-S \ S-S \ S-S\ S-S\ \ -4Sjj5r SrU=r$) DebertaForSequenceClassificationic>[TU]U5 [USS5nX l[ U5Ul[ U5UlURRn[R"X25Ul [USS5nUcURROUn[R"U5UlUR!5 g)N num_labelsr- cls_dropout)rrrrr^rMrpoolerrrrG classifierrPrMrLrNrc)r#rPrrdrop_outr&s r'r)DebertaForSequenceClassification.__init__s  V\15 $#F+ #F+ [[++ ))J;6=$76>6F4;;22Hzz(+  r)c6URR5$r)rMrhrgs r'rh5DebertaForSequenceClassification.get_input_embeddingss||0022r)c:URRU5 gr)rMrmrks r'rm5DebertaForSequenceClassification.set_input_embeddingss )).9r)Nrrrrrrrr9r:rc $U bU OURRn URUUUUUUUU S9n U Sn URU 5n UR U 5n UR U 5nSnUGbURR GcURS:XaX[R"5nURS5RUR5nU"XRS55nGOhUR5S:XdURS5S:XGaUS:R5nUR!5nURS5S:a["R$"USUR'URS5URS555n["R$"USURS55n[)5nU"URSUR5R+5URS55nGOM["R,"S5RU5nGO&[R."S5nU"U5U-R1S5R35*nOURR S:XaI[5nURS:Xa&U"UR55UR555nOU"X5nOURR S:Xa=[)5nU"URSUR5URS55nO,URR S:Xa[75nU"X5nU (dU4U SS-nUbU4U-$U$[9XU R:U R<S 9$) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrrrrr9r:rrr+ regressionsingle_label_classificationmulti_label_classificationr)rPr:rMrrNr problem_typerrrr_r3r.rr$nonzeror^rrrlrr/ry LogSoftmaxsumr0rrrr4rC)r#rrrrrrrr9r:rrr encoder_layerrrrloss_fn label_indexlabeled_logitsr log_softmaxrs r'r8(DebertaForSequenceClassification.forwards(&1%#)Q;"7"7"9K#[[]F"''*Q.)."A{'9'9+:J:J1:Mv{{[\~'^*"'fa9I9I"9M!N#3#5'(;(;B(P(V(V(XZ`ZeZefhZij$||A11&9"$--"3K)&1F:??CIIKKD))\9"9??a'#FNN$4fnn6FGD#F3D))-JJ+- B @&++b/R))-II,./Y,F)-)9TGf$ Ev E'G4I4IV]VhVh  r))rrMrNrrr)r;r<r=r>rrhrmrrrrrrr8r@rArBs@r'rrs$3:*..2.2,0-1&*)-,0#'N <<$&N  t+N  t+ N llT) N ||d* N  t#N  $;N #TkN D[N  ) )N N r)rc.^\rSrSrU4Sjr\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \S-S \S-S \S-S \ \ -4Sjj5r Sr U=r $)DebertaForTokenClassificationic0>[TU]U5 URUl[U5Ul[ R "UR5Ul[ R"URUR5Ul UR5 gr) rrrr^rMrrLrMrNrGrHrrcrOs r'r&DebertaForTokenClassification.__init__si   ++#F+ zz&"<"<= ))F$6$68I8IJ r)Nrrrrrrrr9r:rc U bU OURRn URUUUUUUUU S9n U Sn URU 5n UR U 5n SnUb<[ 5nU"U R SUR5UR S55nU (dU 4U SS-nUbU4U-$U$[XU RU RS9$)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr+rr) rPr:rMrNrrr_rrr4rC)r#rrrrrrrr9r:rrrrxrrrrs r'r8%DebertaForTokenClassification.forwards$&1%rrrrrrrr8r@rArBs@r'rrs *..2.2,0-1&*)-,0#'. <<$&.  t+.  t+ . llT) . ||d* .  t#.  $;. #Tk. D[.  & &. . r)rcN^\rSrSrU4Sjr\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \RS-S \S-S \S-S \S-S\ \ -4Sjj5r Sr U=r $)DebertaForQuestionAnsweringiPc>[TU]U5 URUl[U5Ul[ R "URUR5UlUR5 gr) rrrr^rMrrGrH qa_outputsrcrOs r'r$DebertaForQuestionAnswering.__init__RsS   ++#F+ ))F$6$68I8IJ r)Nrrrrrstart_positions end_positionsrr9r:rc "U bU OURRn URUUUUUUU U S9n U Sn URU 5nUR SSS9unnUR S5R 5nUR S5R 5nSnUbUb[UR55S:aUR S5n[UR55S:aUR S5nURS5nURSU5nURSU5n[US9nU"X5nU"UU5nUU-S- nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS9$) Nrrrr+r) ignore_indexr-)r start_logits end_logitsr4rC)rPr:rMrsplitrrrr$rrr r4rC)r#rrrrrrrrr9r:rrrrxrrr total_loss ignored_indexr start_lossend_lossrs r'r8#DebertaForQuestionAnswering.forward\s&1%rrrrrrr r8r@rArBs@r'rrPs*..2.2,0-1/3-1)-,0#'= <<$&=  t+=  t+ = llT) = ||d* = ,= ||d*=  $;= #Tk= D[=  - -= = r)r)rrrrr^rL)Ar?rrtorch.nnrrrrrP activationsr modeling_layersr modeling_outputsr r r rrmodeling_utilsrutilsrrconfiguration_debertar get_loggerr;loggerModulerrDjitscriptrirorqrurrrzr}rrrrrrrrr'rLr^r{rRrrSrrrrrr__all__r<r)r'r se AA&!9.,0   H %ryy(  8rrnn[[\%,,\c\\ELLU\\d ddgjdd C CLL L^,ryy,F "))  BII (-(BO RYYO di_ii&a )a a H299&BII&!ryy!bii6!!Y /Y Y x'BII', h '=h  h V; $:; ; |I "8I I X r)