j@ S SK JrJrJrJr S SKJr SSKJrJ r J
r
J r J r J
r
JrJrJrJr SSKJr SSKJrJr \R. " \5 rSSS S
.rS r " S S
\5 r \ r S
S/r g) ) Tokenizerdecoderspre_tokenizers
processors)BPE )
ENCODE_KWARGS_DOCSTRING
AddedToken
BatchEncoding EncodedInputPaddingStrategyPreTokenizedInput
TensorType TextInput
TextInputPairTruncationStrategy)TokenizersBackend)add_end_docstringsloggingz
vocab.jsonz
merges.txtztokenizer.json)
vocab_file merges_filetokenizer_filea<
add_special_tokens (`bool`, *optional*, defaults to `True`):
Whether or not to encode the sequences with the special tokens relative to their model.
padding (`bool`, `str` or [`~tokenization_utils_base.PaddingStrategy`], *optional*, defaults to `False`):
Activates and controls padding. Accepts the following values:
- `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
sequence if provided).
- `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
acceptable input length for the model if that argument is not provided.
- `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different
lengths).
truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`):
Activates and controls truncation. Accepts the following values:
- `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or
to the maximum acceptable input length for the model if that argument is not provided. This will
truncate token by token, removing a token from the longest sequence in the pair if a pair of
sequences (or a batch of pairs) is provided.
- `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the
maximum acceptable input length for the model if that argument is not provided. This will only
truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided.
- `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the
maximum acceptable input length for the model if that argument is not provided. This will only
truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided.
- `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths
greater than the model maximum admissible input size).
max_length (`int`, *optional*):
Controls the maximum length to use by one of the truncation/padding parameters. If left unset or set to
`None`, this will use the predefined model maximum length if a maximum length is required by one of the
truncation/padding parameters. If the model has no specific maximum input length (like XLNet)
truncation/padding to a maximum length will be deactivated.
stride (`int`, *optional*, defaults to 0):
If set to a number along with `max_length`, the overflowing tokens returned when
`return_overflowing_tokens=True` will contain some tokens from the end of the truncated sequence
returned to provide some overlap between truncated and overflowing sequences. The value of this
argument defines the number of overlapping tokens.
is_split_into_words (`bool`, *optional*, defaults to `False`):
Whether or not the input is already pretokenized (e.g. split into words). Set this to `True` if you are
passing pretokenized inputs to avoid additional tokenization.
pad_to_multiple_of (`int`, *optional*):
If set will pad the sequence to a multiple of the provided value. This is especially useful to enable
the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta).
return_tensors (`str` or [`~tokenization_utils_base.TensorType`], *optional*):
If set, will return tensors instead of list of python integers. Acceptable values are:
- `'pt'`: Return PyTorch `torch.Tensor` objects.
- `'np'`: Return Numpy `np.ndarray` objects.
c , ^ \ rS rSrSr\r/ SQr\r S6S\
\ \
\ 4 - \
\\
\4 - S- S\
\
\
- S- 4U 4S jjjrS
r\" \\5 S7S \\- \
\ - \
\ - S
\\
\ - S- S\
\
\ \
\
\
\ - S- S\
\ \
\
\ - S- S\S\\
- \- S\\
- \- S- S\ S- S\ S\S\ S- S\
S- S\
\- S- S\S- S\S- S\S \S \S \S \S \4*S! jj5 r \" \\5 S8S"\
\ \
\ - \
\ - S#\S- S\
\
\
\ S- S\
\ \
\
\ - S- S\S\\
- \- S\\
- \- S- S\ S- S\ S\ S- S\
S- S\
\- S- S\S- S\S- S\S \S \S \S \S \4(S$ jj5 r S9S \
S%\
S- S\S \
\
4S& jjr \" \\5 S8S \\- S
\S- S\
\
\ S- S\
\ S- S\S\\
- \- S\\
- \- S- S\ S- S\ S\ S- S\
S- S\
\- S- S\S- S\S- S\S \S \S \S \S \4(S' jj5 r SSSS\RB \RD SS SSSSSSSSSS4S"\
\ \
\ - \
\ - S#\S- S\
\
\
\ S- S\
\
\ S- S\S(\S)\S\ S- S\ S\ S- S\
S- S\
S- S\S- S\S- S\S \S \S \S \S \4(S* jjr#SSSS\RB \RD SS SSSSSSSSSS4S \\- S
\S- S\
\
\ S- S\
\ S- S\S(\S)\S\ S- S\ S\ S- S\
S- S\S- S\S- S\S- S\S \S \S \S \S \4(S+ jjr$S\RB SSS4S,\ \
\%4 \- S\ S- S(\S\ S- S\
S- S\S- S \ 4S- jjr& S:S.\
\ S/\
\ S- S \
\ 4S0 jjr' S:S.\
\ S/\
\ S- S \
\ 4S1 jjr(S:S2\
S3\
S- S \\
4S4 jjr)S5r*U =r+$ );MarkupLMTokenizer[ a9
Construct a MarkupLM tokenizer. Based on byte-level Byte-Pair-Encoding (BPE).
[`MarkupLMTokenizer`] can be used to turn HTML strings into to token-level `input_ids`, `attention_mask`,
`token_type_ids`, `xpath_tags_seq` and `xpath_tags_seq`. This tokenizer inherits from [`TokenizersBackend`] which
contains most of the main methods and ensures a `tokenizers` backend is always instantiated.
Users should refer to this superclass for more information regarding those methods.
Args:
vocab (`str` or `dict[str, int]`, *optional*):
Custom vocabulary dictionary. If not provided, the vocabulary is loaded from `vocab_file`.
merges (`str` or `list[str]`, *optional*):
Custom merges list. If not provided, merges are loaded from `merges_file`.
errors (`str`, *optional*, defaults to `"replace"`):
Paradigm to follow when decoding bytes to UTF-8. See
[bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
mask_token (`str`, *optional*, defaults to `""`):
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.
add_prefix_space (`bool`, *optional*, defaults to `False`):
Whether or not to add an initial space to the input. This allows to treat the leading word just as any
other word. (RoBERTa tokenizer detect beginning of words by the preceding space).
) input_idstoken_type_idsattention_maskNFTvocabmergesc > [ U[ 5 ( a [ USSS9OUn[ U[ 5 ( a [ USSS9OUn[ U[ 5 ( a [ USSS9OUn[ U[ 5 ( a [ USSS9OUn[ U [ 5 ( a [ U SSS9OU n [ U
[ 5 ( a [ U
SSS9OU
n
[ U [ 5 ( a [ U SSS9OU n Uc9 [ U
5 S[ U 5 S[ U5 S[ U5 S[ U 5 S0nU=( d / n[ [ UUS S S SS
95 n[
R " U S 9Ul [ R " 5 Ul X l
X0l UU l [ TU ]8 " S 0 S U_S
U_SU_SU_SU _SU_SU_SU
_SU _SU _SU_SU
_SU_SU_SU_SU_UD6 [ U5 n[ U5 nU R nU R n[" R$ " U S U 3U S U S U 3UU4UU4/S 9U R l Xl Xl Xl Xl [1 U R( 5 U l U R2 S- U l U R4 /U R* - U l U R. /U R* - U l UU l UU l g ) NF)lstriprstripTr r )r r dropoutcontinuing_subword_prefixend_of_word_suffixfuse_unk)add_prefix_space tags_dicterrors bos_token eos_token unk_token sep_token cls_token pad_token
mask_tokenr, trim_offsets max_depth max_width pad_widthpad_token_labelonly_label_first_subwordz $A z $B )singlepairspecial_tokens )
isinstancestrr
r r r ByteLevel
pre_tokenizerr decoder_vocab_merges
_tokenizersuper__init__ cls_token_id sep_token_idr TemplateProcessingpost_processorr- r7 r8 r9 len
unk_tag_id
pad_tag_idpad_xpath_tags_seqpad_xpath_subs_seqr: r; )selfr- r r r. r/ r0 r2 r3 r1 r4 r5 r, r7 r8 r9 r: r; r6 kwargs tokenizer
sep_token_str
cls_token_strrJ rK __class__s C:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\transformers/models/markuplm/tokenization_markuplm.pyrI MarkupLMTokenizer.__init__ s , JTT]_bIcIcJyuEir IST]_bIcIcJyuEir IST]_bIcIcJyuEir IST]_bIcIcJyuEir IST]_bIcIcJyuEir IST]_bIcIcJyuEir JTU_adJeJeZ
4Fku
=IIIIJ E 2 *,#%
#1":":L\"] $ . . 0 #
"
.
&
,
&>#
& I
I
(( (( )3)F)F#_D 8!?$}oT- I - - *
& # #"" dnn- //A-#'??"3dnn"D #'>>"2T^^"C . (@% c / n/ nUR S5 nU H nUR 5 ( d M UR 5 R S5 nUS n[ U5 S:X a SO[ US SS 5 nUR U R
R
XpR 5 5 UR [ U R U5 5 M USU R nUSU R nX R /U R [ U5 -
- -
nX0R /U R [ U5 -
- -
nX#4$ )z
Given the xpath expression of one particular node (like "/html/body/div/li[1]/div/span[2]"), return a list of
tag IDs and corresponding subscripts, taking into account max depth.
/[r r$ N)
splitstriprN intappendr- getrO minr8 r7 rP r9 ) rS xpathxpath_tags_listxpath_subs_list xpath_unitsunit name_substag_namesubs rY
get_xpath_seq MarkupLMTokenizer.get_xpath_seq s
kk#& D::<<
**3/I |H9~*!IaL"4E0FC " "4>>#5#5h#P Q " "3t~~s#; < **:DNN;)*:DNN;OO,_AU0UVVNN+t~~O@T/TUU //r[ r text text_pairxpaths node_labelsadd_special_tokenspadding
truncation
max_lengthstrideis_split_into_wordspad_to_multiple_of padding_sidereturn_tensorsreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping
return_lengthverbosereturnc ^ Sn[ U[ 5 ( a [ U5 nUb [ U[ 5 ( a [ U5 nUGc U
( Gd Uc UOUn[ U[ 5 ( a [ U5 nUn[ U[ 5 ( a UR 5 nO[ U[ 5 ( a U( aU [ US [ 5 ( a= [ S U 5 5 nU( a! U Vs/ s H nUR 5 PM nnO=UnO:U( a3 [ US [ 5 ( a U Vs/ s H n[ U5 PM nnUc UnOUn[ U[ 5 ( aE U( a> [ US [ [ 45 ( a U Vs/ s H nU/[
U5 - PM nnO$[ US5 ( a [
U5 OSn U/U - nS n UbU U " U5 ( d [ S5 e[ U[ [ 45 ( d [ S5 e[ U[ [ 45 n Oe[ U[ [ 45 ( d [ S5 e[ U[ [ 45 =( a" U=( a [ US [ [ 45 n Uc UOUn Uc S 5 eU ( aU [
U 5 [
U5 :X d S
5 e[ U U5 H% u n n![
U 5 [
U!5 :X a M S 5 e O [
U 5 [
U5 :X d S 5 eU ( a Ub< [
U5 [
U5 :w a$ [ S [
U5 S
[
U5 S35 eUb [ [ X5 5 OUn"[ USL5 n#U R " S$0 SU"_SU#_SU_SU_SU_SU_SU_SU_SU _SU _SU _SU
_SU_S U_S U_S U_S U_S U_S!U_UD6$ U R " S$0 S"U_S#U_SU_SU_SU_SU_SU_SU_SU _SU _SU _SU
_SU_S U_S U_S U_S U_S U_S!U_UD6$ s snf s snf s snf )%a
Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of
sequences with nodes, xpaths and optional labels.
Args:
text (`str`, `list[str]`, `list[list[str]]`):
The sequence or batch of sequences to be encoded. Each sequence can be a string, a list of strings
(words of a single example or questions of a batch of examples) or a list of list of strings (batch of
words).
text_pair (`list[str]`, `list[list[str]]`):
The sequence or batch of sequences to be encoded. Each sequence should be a list of strings
(pretokenized string).
xpaths (`list[list[int]]`, `list[list[list[int]]]`):
Node-level xpaths. Each bounding box should be normalized to be on a 0-1000 scale.
node_labels (`list[int]`, `list[list[int]]`, *optional*):
Node-level integer labels (for token classification tasks).
is_split_into_words (`bool`, *optional*):
Set to `True` if the inputs are already provided as pretokenized word lists.
/document/nodeNr c 3 , # U H
nS U; v M g7f) Nr? ).0entrys rY -MarkupLMTokenizer.__call__..T s (P<% ._is_valid_text_inputg s !S!!!dE]++q6Q; adC(( adT5M22 qt9>EZ!Q-EEr[ zStext input must of type `str` (single example) or `list[str]` (batch of examples). zwNodes must be of type `list[str]` (single pretokenized example), or `list[list[str]]` (batch of pretokenized examples).z%You must provide corresponding xpathszAYou must provide nodes and xpaths for an equal amount of examplesz2You must provide as many nodes as there are xpathszbatch length of `text`: z- does not match batch length of `text_pair`: .batch_text_or_text_pairsis_pairrr rs rt ru rv rw rx rz r{ r| r} r~ r r r r r rp rq r? )
r@ r r rA r` anyrN hasattr
ValueErrorzipboolbatch_encode_plus encode_plus)$rS rp rq rr rs rt ru rv rw rx ry rz r{ r| r} r~ r r r r r rT placeholder_xpath nodes_sourceprocessed_nodesrequires_splitr samplelengthr
is_batchednodes
nodes_examplexpaths_exampler r s$ rY __call__MarkupLMTokenizer.__call__ s Z - dE " ":D Z 5%A%A YI >"5#,#44)L,.. #L 1 *O,,,"."4"4"6L$// J|A$D$D%((P<(P%PN%FR*SlU5;;=l*S*6!j a %&H&HBN&O,tF|,O&O & + /400_TcdeTfimothuIvIvJYZ/,-F ;/Z18 )1T1T_-Z[+,v5
'-- !vww i$ 77 M $D4-8J dT5M22 M $D4-8hThjQUVWQX[_afZgFhJ!)y!J#JJ! u:V, q.q q,14UF1C-
~=)S -@@vBvv@ 2D u:V, b.b b, $Tc)n)D .s4yk :I'q* FOEZtC,@'A`d $9D01G)) )A (
$6
& & $6 * . '< '< +D ,F!" (>#$ ,%& )
. ## # (
$6
& & $6 * . '< '< +D ,F!" (>#$ ,%& )
q +T 'P [s 2P 5P%P*r r c U R " SUUUU
US.UD6u nnnnU R " S0 SU_SU_SU_SU_SU_SU_SU_S U_S
U _S U
_S U _S
U _SU
_SU_SU_SU_SU_SU_SU_UD6$ )Nru rv rw rz r r r rr rs rt padding_strategytruncation_strategyrw rx rz r{ r| r} r~ r r r r r r? )"_get_padding_truncation_strategies_batch_encode_plus)rS r r rr rs rt ru rv rw rx rz r{ r| r} r~ r r r r r rT r r s rY r #MarkupLMTokenizer.batch_encode_plus s 2 EIDkDk E
!! 1 E
E
A-z6 &&
%=
$
2
.
!4
"
2
&
*
#8
#8
'@
(B!
" $:#
$ (%
& )
r[ r= c | U( a X4/OU/nU R R " U4USS.UD6nUS R $ )NFrt is_pretokenizedr )rG encode_batchtokens)rS rp r= rt rT
batched_input encodingss rY tokenizeMarkupLMTokenizer.tokenize sM *.$ TF
OO00
.@RW
[a
|"""r[ c U R " SUUUU
US.UD6u nnnnU R " S0 SU_SU_SU_SU_SU_SU_SU_S U_S
U _S U
_S U _S
U _SU
_SU_SU_SU_SU_SU_SU_UD6$ )a"
Tokenize and prepare for the model a sequence or a pair of sequences. .. warning:: This method is deprecated,
`__call__` should be used instead.
Args:
text (`str`, `list[str]`, `list[list[str]]`):
The first sequence to be encoded. This can be a string, a list of strings or a list of list of strings.
text_pair (`list[str]` or `list[int]`, *optional*):
Optional second sequence to be encoded. This can be a list of strings (words of a single example) or a
list of list of strings (words of a batch of examples).
r rp rr rq rs rt r r rw rx rz r{ r| r} r~ r r r r r r? )r _encode_plus)rS rp rq rr rs rt ru rv rw rx rz r{ r| r} r~ r r r r r rT r r s rY r MarkupLMTokenizer.encode_plus s J EIDkDk E
!! 1 E
E
A-z6
$
2
.
!4
"
2
&
*
#8
#8
'@
(B!
" $:#
$ (%
& )
r[ r r c [ U[ 5 ( d [ S[ U5 S35 eU R UUUU U
U S9 U( a / nU H u nn[ U[
5 ( a [ U5 n[ U[ 5 ( a U/n[ U[
5 ( a [ U5 n[ U[ 5 ( a U/nUR UU45 M UnOV/ nU HL n[ U[
5 ( a [ U5 n[ U[ 5 ( a U/nUR U5 MN UnU R R UUSS9nU Vs/ s H nU R UU
UUUUb SOUUUS9PM nn0 nUS S H. nU V V V s/ s H u n n U U H n U PM M n n n n U UU' M0 U V V V s/ s H u n n U H n U PM M n n n n U( a4 / n![ U5 H u n"u n#n U!U"/[ U#S 5 - -
n!M U!US ' US H n$U R U$UU5 M / n%/ n&[ [ US 5 5 GHd n'U( a US U' n(OU'n(/ n)/ n*[ US U' U U' R U U' R" 5 H u n+n,n-U-b U( a> U,S:X a8 U)R U R$ 5 U*R U R& 5 MO U R) UU( U- 5 u n.n/U)R+ U./5 U*R+ U//5 M U+U R, U R. U R0 4; a8 U)R U R$ 5 U*R U R& 5 M [3 S
5 e U%R U)5 U&R U*5 GMg U%US ' U&US ' UGb / n0[ [ US 5 5 H n'U( a US U' n(OU'n(/ n1[ US U' US
U' U U' R" 5 H u n+n2n-U-bi U R4 ( a? U2S S:X a U1R UU( U- 5 M= U1R U R6 5 MZ U1R UU( U- 5 Ms U1R U R6 5 M U0R U15 M U0US' U( d US
[9 UU U S9$ s snf s sn n n f s sn n n f )Nz/batch_text_or_text_pairs has to be a list (got ))r r rw rx rz r{ Tr )encodingr} r~ r r r r r r r overflow_to_sample_mappingzId not recognizedxpath_tags_seqxpath_subs_seqoffset_mappinglabels) tensor_type) r@ r TypeErrortypeset_truncation_and_paddingr rA rc rG r _convert_encoding enumeraterN &_eventual_warn_about_too_long_sequenceranger sequence_idsword_idsrQ rR rn extendrJ rK pad_token_idr r; r: r )3rS r r rr rs rt r r rw rx rz r{ r| r} r~ r r r r r processed_inputsrp rq r r tokens_and_encodingssanitized_tokenskeyitem_estacksanitized_encodingsr itoksr r r batch_indexoriginal_indexxpath_tags_seq_examplexpath_subs_seq_exampleid sequence_idword_idrg rh r labels_exampleoffsets3 rY r $MarkupLMTokenizer._batch_encode_plusK sb , 2D99 MdSkNlMmmn op p
'' - 3! 1%
(
! #;i dE** :D dC(( 6D i// $YI i--!* I ''y(9: $< (8 $ ! 0 dE** :D dC(( 6D ''- 1 (8 $OO00 $ 1 1
, &
&
" "!&;&;*C+E * (,+ +
#
&
, '*1-C&:N&:74DIqQIQ&:EN$) S ! . 1E S0DWQdqdq0D S %)+ & )*> ? 9D!*qcC[8I4J.JJ* !@=W 9 :)+6I 7 7 :w W 7 %5k%B!CDK(!12N!OP[!\!,%' "%' ",/ -k:#K0==#K099-(K
& ;!#3.55d6M6MN.55d6M6MN;?;M;MfUcNdelNm;n8 .556GH.556GHd / /1B1BDDUDU VV.55d6M6MN.55d6M6MN ()< ==%-&
! !"8 9 ! !"8 97 E: .<)*-;)* "F $S)9+)F%G H ,%56R%ST_%`N%0N!#+.$[1+>$%56{C' 4==,'B
* 88 %ayA~ . 5 5k.6QRY6Z [ . 5 5d6J6J K * 1 1+n2Mg2V W&--d.B.BC ,
n-- I0 *0 X &)$%56 -/BP^__U
0 O Ss %S S
;S'c Sn[ U[ 5 ( a [ U5 nUb [ U[ 5 ( a [ U5 nUc UOUnUn[ U[ 5 ( a UR 5 nO6[ U[ 5 ( a! U( a [ US [ 5 ( a UnUc UnOUnUc$ [ US5 ( a [
U5 OSnU/U- nU( a X4/OU/nU/nUb U/OS nU R " U40 S[ US L5 _SU_SU_SU_SU_S U_S
U_S U _S U
_S
U _SU _SU
_SU_SU_SU_SU_SU_SU_UD6n U cn U( dg [ U R 5 V V s0 s H5 u n n U [
U 5 S: a [ U S [ 5 ( a U S OU _M7 sn n U R 5 n U R U S UU5 U $ s sn n f )Nr r r r rr rs rt r r rw rx rz r{ r| r} r~ r r r r r r )
r@ r r rA r` r rN r r r itemsr r ) rS rp rq rr rs rt r r rw rx rz r{ r| r} r~ r r r r r rT r nodes_singler r r batched_xpathsbatched_node_labelsbatched_outputr values rY r MarkupLMTokenizer._encode_plus s[ . - dE " ":D Z 5%A%A YI ( 0ti & lC ( ( * 0 0 2O
d
+
+ LYZO]`AaAa *O "D'I >-4_i-P-PS )VWF'(61F 09$*+tf
/:/F{mD 00
$./
"
,
2
.
!4
"
2
&
*
#8
#8
'@
(B!
" $:#
$ (%
& )
2 !*C * '5&:&:&<&<
U SZ!^
58T8R8RqX]]&< (( N
33N;4OQ[]de s )= 7.5` (Volta).
padding_side:
The side on which the model should have padding applied. Should be selected between ['right', 'left'].
Default value is picked from the class attribute of the same name.
return_attention_mask:
(optional) Set to False to avoid returning attention mask (default: set to model specifics)
r r r$ rightr r r r special_tokens_maskleftzInvalid padding strategy:)
model_input_namesr
LONGESTrN
DO_NOT_PADr{ pad_token_type_idrQ rR r: r r rA )
rS r rw r rz r{ r~ required_inputneeds_to_be_padded
differences
rY _padMarkupLMTokenizer._padV s B ! ($48N8N$N !'(>(>q(AB 6 6 6 ^,J !&8&D*JimnJn%;q@DVVJ - 1K1K K qPSTbPcgqPq !%5^%K01sS 5H/HN+ , #c.&99J+7+C<IZIZLw&(7EFV7W[\Z]`jZj7jN#34#~5&'78DC* X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`list[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
)rJ rK )rS r r clsseps rY build_inputs_with_special_tokens2MarkupLMTokenizer.build_inputs_with_special_tokens s` " %%&48I8I7JJ J ! ! 3&4s::r[ c U R /nU R /nUc [ XA- U- 5 S/- $ [ XA- U- U- U- 5 S/- $ )a{
Create a mask from the two sequences passed to be used in a sequence-pair classification task. RoBERTa does not
make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of zeros.
r )rK rJ rN )rS r r r r s rY $create_token_type_ids_from_sequences6MarkupLMTokenizer.create_token_type_ids_from_sequences sd ! ! s(3./1#5 53$s*[83>?1#EEr[ save_directoryfilename_prefixc ^ U R R R XS9n[ U5 $ )N)name)rG modelsaver )rS r r filess rY save_vocabulary!MarkupLMTokenizer.save_vocabulary s) %%**>*PU|r[ )
rF rG rE r7 r8 r; rP r: r9 rR rQ r- rO )NNreplace