ju{SrSSKrSSKJr SSKJrJrJr SSKJr SSK J r SSK J r SSKJrJrJrJrJrJrJr SS KJr SS KJr SS KJrJrJr SS KJr S SK J!r!J"r"J#r#J$r$J%r% SSK&J'r' \RP"\)5r*"SS\"5r+"SS\%5r,"SS\!5r-"SS\#5r.\"SS\55r/"SS\$5r0\"SS9"SS \/\ 55r1\"S!S"\/55r2"S#S$\Rf5r4\"S%S9"S&S'\/55r5\"S(S)\/55r6\"S*S+\/55r7"S,S-\Rf5r8\"S.S/\/55r9/S0Qr:g)1zPyTorch RoBERTa model.N)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)gelu)GenerationMixin),BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuple)BertCrossAttentionBertEmbeddings BertLayer BertModelBertSelfAttention) RobertaConfigc ^\rSrSrU4SjrS S\R S-S\R S-S\R S-S\RS-S\4 S jjr \ S 5r \ SS j5r S r U=r$)RobertaEmbeddings,c>[TU]U5 U?U?URUl[ R "URURURS9Ulg)N) padding_idx) super__init__ pad_token_idposition_embeddingsr#nn Embeddingmax_position_embeddings hidden_sizeselfconfig __class__s ڹC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\transformers/models/roberta/modular_roberta.pyr%RobertaEmbeddings.__init__-sT     $!..#%<<  * *F,>,>DL\L\$  N input_idstoken_type_ids position_ids inputs_embedspast_key_values_lengthcUc;UbURXRU5nOURX@R5nUbUR5nOUR5SSnUupxUc[ US5(ajUR R UR5RURSS5n [R"U SUS9n U RXx5nO8[R"U[RURRS9nUcURU5nUR!U5n XJ-n UR#U5n X-n UR%U 5n UR'U 5n U $)Nr4rr)dimindexdtypedevice)"create_position_ids_from_input_idsr#&create_position_ids_from_inputs_embedssizehasattrr4tor>expandshapetorchgatherzeroslongr5word_embeddingstoken_type_embeddingsr' LayerNormdropout) r-r3r4r5r6r7 input_shape batch_size seq_lengthbuffered_token_type_idsrK embeddingsr's r0forwardRobertaEmbeddings.forward8st  $#FF//1G   $JJ=ZjZjk  #..*K',,.s3K!,  !t-..*.*=*=*@*@ATAT*U*\*\]i]o]opq]rtv*w'*/,,7NTU]i*j'!8!?!? !W!&[ SWSdSdSkSk!l   00;M $ : :> J": "66|D5 ^^J/ \\*- r2cUR5SSnUSn[R"US-X1-S-[RURS9nUR S5R U5$)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr9rr<r)rArFarangerIr> unsqueezerD)r6r#rNsequence_lengthr5s r0r@8RobertaEmbeddings.create_position_ids_from_inputs_embedshsn$((*3B/ %a.|| !O_:Q>ejjYfYmYm %%a(// <[TU]U5 [U[5(a![R "UR 5 g[U[5(a|[R"UR[R"URRS5RS55 [R "UR5 gg)zInitialize the weightsr9)rr9N)r$ _init_weights isinstance RobertaLMHeadinitzeros_biasr copy_r5rFrVrErDr4)r-moduler/s r0r$RobertaPreTrainedModel._init_weightss f% fm , , KK $  1 2 2 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- .3r2rp)rcrdrerfr config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrwrnrt_can_record_outputsrFno_gradrrjrkrls@r0rzrzsV L!&*#N"&%*1  ]]_//r2rzc,^\rSrSrSU4SjjrSrU=r$) RobertaModelc$>[TU]X5 gN)r$r%)r-r.add_pooling_layerr/s r0r%RobertaModel.__init__s &r2rp)T)rcrdrerfr%rjrkrls@r0rrs ''r2rzS RoBERTa Model with a `language modeling` head on top for CLM fine-tuning. ) custom_introc^\rSrSrSSS.rU4SjrSrSr\\ SS \ RS-S \ RS-S \ RS-S \ RS-S \ RS-S\ RS-S\ RS-S\ RS-S\ \ \ RS-S\S-S\\ R -S\\S\ \ R \-4Sjj55rSrU=r$)RobertaForCausalLM)roberta.embeddings.word_embeddings.weight lm_head.biaszlm_head.decoder.weightzlm_head.decoder.biasc>[TU]U5 UR(d[R S5 [ USS9Ul[U5UlUR5 g)NzOIf you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`Fr r$r% is_decoderloggerwarningrr|rlm_head post_initr,s r0r%RobertaForCausalLM.__init__sL    NNl m#FeD $V,  r2c.URR$rrdecoderr-s r0get_output_embeddings(RobertaForCausalLM.get_output_embeddings||###r2c$XRlgrrr-new_embeddingss r0set_output_embeddings(RobertaForCausalLM.set_output_embeddings - r2Nr3attention_maskr4r5r6encoder_hidden_statesencoder_attention_masklabelspast_key_values use_cachelogits_to_keepkwargsreturnc UbSn UR"U4UUUUUUU U SS. U D6n U Rn[U [5(a [ U *S5OU nUR USS2USS245nSnUb)UR "SUXRRS.U D6n[UUU RU RU RU RS9$)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, RobertaForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base") >>> config = AutoConfig.from_pretrained("FacebookAI/roberta-base") >>> config.is_decoder = True >>> model = RobertaForCausalLM.from_pretrained("FacebookAI/roberta-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NFT) rr4r5r6rrrr return_dict)logitsr vocab_size)lossrrr}r~rrp)r|last_hidden_staterr]slicer loss_functionr.rr rr}r~r)r-r3rr4r5r6rrrrrrroutputsr} slice_indicesrrs r0rSRobertaForCausalLM.forwards`  I@D  A ))%'"7#9+ A  A  11 8B>SV8W8W~ot4]k mA}a,?@A  %%pVF{{OeOepiopD0#33!//))$55   r2rr|) NNNNNNNNNNr)rcrdrerf_tied_weights_keysr%rrrrrFrgrhtupleboolr]Tensorrrr rSrjrkrls@r0rrs#N . $..237260426:>;?*.BF!%-.O ##d*O ))D0O ((4/ O &&- O ((4/ O  %0047O !& 1 1D 8O   4'O uU%6%6784?O $;O ell*O +,O  u|| @ @O O r2rc^\rSrSrSSS.rU4SjrSrSr\\ SS \ RS-S \ RS-S \ RS-S \ RS-S \ RS-S\ RS-S\ RS-S\ RS-S\ \S\\ R \-4Sjj55rSrU=r$)RobertaForMaskedLMi'rrrc>[TU]U5 UR(a[R S5 [ USS9Ul[U5UlUR5 g)NznIf you want to use `RobertaForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Frrr,s r0r%RobertaForMaskedLM.__init__.sR     NN1  $FeD $V,  r2c.URR$rrrs r0r(RobertaForMaskedLM.get_output_embeddings=rr2c$XRlgrrrs r0r(RobertaForMaskedLM.set_output_embeddings@rr2Nr3rr4r5r6rrrrrc pUR"U4UUUUUUSS.U D6n U Sn URU 5n Sn UbaURU R5n[ 5nU"U R SUR R5UR S55n [U U U RU RS9$)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rr4r5r6rrrrNr9rrr}r~) r|rrCr>rviewr.rr r}r~)r-r3rr4r5r6rrrrrsequence_outputprediction_scoresmasked_lm_lossloss_fcts r0rSRobertaForMaskedLM.forwardCs:,,   ))%'"7#9    "!* LL9  YY0778F')H%&7&<&;?*.5 ##d*5 ))D05 ((4/ 5 &&- 5 ((4/ 5  %00475 !& 1 1D 85   4'5 +,5  u|| ~ -5 5 r2rc2^\rSrSrSrU4SjrSrSrU=r$)ri}z*Roberta Head for masked language modeling.c>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"URUR5Ul [R"[R"UR55Ulg)N)eps)r$r%r(Linearr+denserLlayer_norm_eps layer_normrr ParameterrFrHrr,s r0r%RobertaLMHead.__init__s YYv1163E3EF ,,v'9'9v?T?TUyy!3!3V5F5FG LLV->->!?@ r2c URU5n[U5nURU5nURU5nU$r)rrrrr-featuresrxs r0rSRobertaLMHead.forwards; JJx  G OOA  LLOr2)rrrr rcrdrerf__doc__r%rSrjrkrls@r0rr}s4Ar2rz RoBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c8^\rSrSrU4Sjr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \ \ S \ \R\-4S jj55rS rU=r$) RobertaForSequenceClassificationic>[TU]U5 URUlXl[ USS9Ul[ U5UlUR5 gNFr) r$r% num_labelsr.rr|RobertaClassificationHead classifierrr,s r0r%)RobertaForSequenceClassification.__init__sH   ++ #FeD 3F; r2Nr3rr4r5r6rrrc UR"U4UUUUSS.UD6nUSn URU 5n Sn UGbURU R5nURR cUR S:XaSURlOoUR S:aNUR[R:XdUR[R:XaSURlOSURlURR S:XaI[5n UR S:Xa&U "U R5UR55n OU "X5n OURR S:Xa=[5n U "U RS UR 5URS 55n O,URR S:Xa[5n U "X5n [!U U UR"UR$S 9$) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trr4r5r6rrNr regressionsingle_label_classificationmulti_label_classificationr9r)r|rrCr>r. problem_typerr=rFrIr]rsqueezerrrrr}r~ r-r3rr4r5r6rrrrrrrs r0rS(RobertaForSequenceClassification.forwards6,,  ))%'   "!*1  YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#F3D))-JJ+- B @&++b/R))-II,./'!//))   r2)rr.rr|NNNNNN)rcrdrerfr%rrrFrgrhrrrrrrSrjrkrls@r0rrs .237260426*.C ##d*C ))D0C ((4/ C &&- C ((4/ C   4'C +,C  u|| 7 7C C r2rc8^\rSrSrU4Sjr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \ \ S \ \R\-4S jj55rS rU=r$)RobertaForMultipleChoiceic>[TU]U5 [U5Ul[R "UR 5Ul[R"URS5Ul UR5 g)Nr) r$r%rr|r(Dropouthidden_dropout_probrMrr+rrr,s r0r%!RobertaForMultipleChoice.__init__sV  #F+ zz&"<"<= ))F$6$6: r2Nr3r4rrr5r6rrc *UbURSOURSnUb!URSURS55OSn Ub!URSURS55OSn Ub!URSURS55OSn Ub!URSURS55OSn Ub1URSURS5URS55OSn UR"U 4U U U U SS.UD6nUSnUR U5nUR U5nURSU5nSnUb.UR UR5n[5nU"UU5n[UUURURS9$)aO input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nrr9T)r5r4rr6rr) rErrAr|rMrrCr>rr r}r~)r-r3r4rrr5r6r num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsr pooled_outputrreshaped_logitsrrs r0rS RobertaForMultipleChoice.forwardsV-6,Aiooa(}GZGZ[\G] CLCXINN2,>?^bLXLdL--b,2C2CB2GHjnR`Rln11"n6I6I"6MNrvR`Rln11"n6I6I"6MNrv(   r=#5#5b#9=;M;Mb;Q R  ,,  *..,      ]3 / ++b+6  YY556F')HOV4D("!//))   r2)rrMr|r)rcrdrerfr%rrrFrgrhrrrrr rSrjrkrls@r0rrs.22637*.0426P ##d*P ((4/P ))D0 P   4' P &&- P ((4/P +,P  u|| 8 8P P r2rc8^\rSrSrU4Sjr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \ \ S \ \R\-4S jj55rS rU=r$)RobertaForTokenClassificationiNcd>[TU]U5 URUl[USS9UlUR b UR O UR n[R"U5Ul [R"URUR5Ul UR5 gr)r$r%rrr|classifier_dropoutrr(rrMrr+rrr-r.rr/s r0r%&RobertaForTokenClassification.__init__Ps   ++#FeD )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ r2Nr3rr4r5r6rrrc zUR"U4UUUUSS.UD6nUSn URU 5n URU 5n Sn UbWURU R5n[ 5n U "U R SUR5UR S55n [U U URURS9$)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrrNr9r) r|rMrrCr>rrrrr}r~rs r0rS%RobertaForTokenClassification.forward^s2,,  ))%'   "!*,,71  YYv}}-F')HFKKDOO[TU]5 [R"URUR5UlUR b UR O URn[R"U5Ul [R"URUR5Ul gr) r$r%r(rr+rrrrrMrout_projrs r0r%"RobertaClassificationHead.__init__s YYv1163E3EF )/)B)B)NF % %TZTnTn zz"45  &"4"4f6G6GH r2c USS2SSS24nURU5nURU5n[R"U5nURU5nUR U5nU$)Nr)rMrrFtanhrrs r0rS!RobertaClassificationHead.forwardsY Q1W  LLO JJqM JJqM LLO MM! r2)rrMrrrls@r0rrs7Ir2rcX^\rSrSrU4Sjr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \RS-S \ \ S \ \R\-4S jj55rSrU=r$)RobertaForQuestionAnsweringic>[TU]U5 URUl[USS9Ul[ R "URUR5UlUR5 gr) r$r%rrr|r(rr+ qa_outputsrr,s r0r%$RobertaForQuestionAnswering.__init__sU   ++#FeD ))F$6$68I8IJ r2Nr3rr4r5r6start_positions end_positionsrrc UR"U4UUUUSS.UD6n U Sn URU 5n U RSSS9upU RS5R 5n U RS5R 5n SnUbUb[ UR 55S:aURS5n[ UR 55S:aURS5nU R S5nURSU5nURSU5n[US9nU"X5nU"X5nUU-S - n[UU U U RU RS 9$) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) Trrrr9r[N) ignore_indexr)r start_logits end_logitsr}r~) r|r%splitr contiguouslenrAclamprrr}r~)r-r3rr4r5r6r'r(rrrrr+r, total_loss ignored_indexr start_lossend_losss r0rS#RobertaForQuestionAnswering.forwardsx0,,  ))%'   "!*1#)<<r<#: #++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J :H$x/14J+%!!//))   r2)rr%r|)NNNNNNN)rcrdrerfr%rrrFrgrhrrrrrrSrjrkrls@r0r#r#s.2372604263715> ##d*> ))D0> ((4/ > &&- > ((4/ > ))D0> ''$.> +,>  u|| ; ;> > r2r#)rrrr#rrrrz);rrFtorch.nnr(rrrrr activationsr generationr modeling_outputsr r r r rrrmodeling_utilsrprocessing_utilsrutilsrrr utils.genericrbert.modeling_bertrrrrrconfiguration_robertar get_loggerrcrr rnrtrwrzrrrModulerrrrrr#__all__rpr2r0rDs AA&).&@@-ll0   H %\8\8~ ,  .  9 /_//2'9'  i /i  i XR /R R jBII, Q '=Q  Q h] 5] ] @C $:C C L ,K "8K K \ r2