| Server IP : 46.17.175.229 / Your IP : 216.73.216.57 Web Server : LiteSpeed System : Linux lt-bnk-web1163.main-hosting.eu 4.18.0-553.109.1.lve.el8.x86_64 #1 SMP Thu Mar 5 20:23:46 UTC 2026 x86_64 User : u528038015 ( 528038015) PHP Version : 8.2.33 Disable Function : system, exec, shell_exec, passthru, mysql_list_dbs, ini_alter, dl, symlink, link, chgrp, leak, popen, apache_child_terminate, virtual, mb_send_mail MySQL : OFF | cURL : ON | WGET : ON | Perl : OFF | Python : OFF | Sudo : OFF | Pkexec : OFF Directory : /opt/alt/python37/lib/python3.7/site-packages/html5lib/__pycache__/ |
Upload File : |
B
63)Z+ � @ s� d dl mZmZmZ d dlmZ d dlmZ ddl m
Z
ddl m Z ddl m Z m
Z
ddl mZmZmZ dd l mZmZ dd
l mZ dd lmZ dd lmZ ee �ZG d
d� de�ZdS )� )�absolute_import�division�unicode_literals)�unichr)�deque� )�spaceCharacters)�entities)� asciiLetters�asciiUpper2Lower)�digits� hexDigits�EOF)�
tokenTypes�
tagTokenTypes)�replacementCharacters)�HTMLInputStream)�Triec sd e Zd ZdZd�� fdd� Zdd� Zdd� Zd�d
d �Zd d
� Zdd� Z dd� Z
dd� Z dd� Z dd� Z
dd� Zdd� Zd d � Zd d � Zd d!� Zd"d#� Zd$d%� Zd&d'� Zd(d)� Zd*d+� Zd,d-� Zd.d/� Zd0d1� Zd2d3� Zd4d5� Z d6d7� Z d8d9� Z d:d;� Z d<d=� Z d>d?� Z!d@dA� Z"dBdC� Z#dDdE� Z$dFdG� Z%dHdI� Z&dJdK� Z'dLdM� Z(dNdO� Z)dPdQ� Z*dRdS� Z+dTdU� Z,dVdW� Z-dXdY� Z.dZd[� Z/d\d]� Z0d^d_� Z1d`da� Z2dbdc� Z3ddde� Z4dfdg� Z5dhdi� Z6djdk� Z7dldm� Z8dndo� Z9dpdq� Z:drds� Z;dtdu� Z<dvdw� Z=dxdy� Z>dzd{� Z?d|d}� Z@d~d� ZAd�d�� ZBd�d�� ZCd�d�� ZDd�d�� ZEd�d�� ZFd�d�� ZGd�d�� ZHd�d�� ZId�d�� ZJd�d�� ZKd�d�� ZL� ZMS )��
HTMLTokenizera This class takes care of tokenizing HTML.
* self.currentToken
Holds the token that is currently being processed.
* self.state
Holds a reference to the method to be invoked... XXX
* self.stream
Points to HTMLInputStream object.
Nc sF t |f|�| _|| _d| _g | _| j| _d| _d | _t t
| �� � d S )NF) r �stream�parserZ
escapeFlagZ
lastFourChars� dataState�state�escape� currentToken�superr �__init__)�selfr r �kwargs)� __class__� �D/opt/alt/python37/lib/python3.7/site-packages/html5lib/_tokenizer.pyr " s zHTMLTokenizer.__init__c c s\ t g �| _xL| �� rVx&| jjr:td | jj�d�d�V qW x| jrR| j�� V q>W q W dS )z� This is where the magic happens.
We do our usually processing through the states and when we have a token
to return we yield the token which pauses processing until the next token
is requested.
�
ParseErrorr )�type�dataN)r �
tokenQueuer r �errorsr �pop�popleft)r r r r! �__iter__1 s
zHTMLTokenizer.__iter__c % C s t }d}|rt}d}g }| j�� }x(||krJ|tk rJ|�|� | j�� }q$W td�|�|�}|tkr�t| }| j �t
d dd|id�� �nbd| kr�d ks�n |d
kr�d }| j �t
d dd|id�� �n d | kr�d
k�s�n d| kr�dk�s�n d| k�rdk�s�n d| k�r0dk�s�n |t ddddddddd d d d d d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d
g#�k�r�| j �t
d dd|id�� y t |�}W n> t
k
�r� |d6 }t d|d? B �t d7|d8@ B � }Y nX |d9k�r| j �t
d d:d;�� | j�|� |S )<z�This function returns either U+FFFD or the character based on the
decimal or hexadecimal representation. It also discards ";" if present.
If not present self.tokenQueue.append({"type": tokenTypes["ParseError"]}) is invoked.
�
� � r" z$illegal-codepoint-for-numeric-entity� charAsInt)r# r$ �datavarsi � i�� i�� u �r � � � � � i� i� � i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i��
i��
i�� i�� i�� i�� i��
i��
i�� i�� i�� i�� i�� i i � i� �;z numeric-entity-without-semicolon)r# r$ )r r
r �charr �append�int�joinr r% r � frozenset�chr�
ValueError�unget) r ZisHexZallowed�radix� charStack�cr- r6 �vr r r! �consumeNumberEntityA s`
&
z!HTMLTokenizer.consumeNumberEntityFc C s� d}| j �� g}|d tksB|d tddfksB|d k rV||d krV| j �|d � �n |d dk�r d}|�| j �� � |d dkr�d}|�| j �� � |r�|d tks�|s�|d tkr�| j �|d � | �|�}n4| j �t
d d
d �� | j �|� � � dd � |� }�njx8|d tk �rDt
�d � |���s0P |�| j �� � �qW y$t
�d � |d d� ��}t|�}W n tk
�r� d }Y nX |d k �rB|d d
k�r�| j �t
d dd �� |d d
k�r|�r|| tk�s�|| tk�s�|| dk�r| j �|� � � dd � |� }n.t| }| j �|� � � |d � ||d � �7 }n4| j �t
d dd �� | j �|� � � dd � |� }|�r�| jd d d |7 < n*|tk�r�d}nd}| j �t
| |d �� d S )N�&r �<�#F���)�x�XTr" zexpected-numeric-entity)r# r$ r, r5 z named-entity-without-semicolon�=zexpected-named-entityr$ r �SpaceCharacters�
Characters)r r6 r r r= r7 r
r rB r% r r' r9 � entitiesTrieZhas_keys_with_prefixZlongest_prefix�len�KeyErrorr
r r ) r � allowedChar�
fromAttribute�outputr? �hexZ
entityNameZ entityLengthZ tokenTyper r r! �
consumeEntity� sf
zHTMLTokenizer.consumeEntityc C s | j |dd� dS )zIThis method replaces the need for "entityInAttributeValueState".
T)rO rP N)rS )r rO r r r! �processEntityInAttribute� s z&HTMLTokenizer.processEntityInAttributec C s� | j }|d tkrp|d �t�|d< |d td krp|d rR| j�td dd�� |d rp| j�td d d�� | j�|� | j| _d
S ) z�This method is a generic handler for emitting the tags. It also sets
the state to "data" because that's what's needed after a token has been
emitted.
r# �name�EndTagr$ r" zattributes-in-end-tag)r# r$ � selfClosingz self-closing-flag-on-end-tagN) r r � translater r r% r7 r r )r �tokenr r r! �emitCurrentToken� s
z HTMLTokenizer.emitCurrentTokenc C s� | j �� }|dkr | j| _n�|dkr.| j| _n�|dkrd| j�td dd�� | j�td dd�� n`|tkrpdS |t kr�| j�td || j �
t d
� d�� n&| j �
d �}| j�td || d�� d
S ) NrC rD � r" zinvalid-codepoint)r# r$ rK FrJ T)rC rD r[ ) r r6 �entityDataStater � tagOpenStater% r7 r r r �
charsUntil)r r$ �charsr r r! r � s&
zHTMLTokenizer.dataStatec C s | � � | j| _dS )NT)rS r r )r r r r! r\ s z HTMLTokenizer.entityDataStatec C s� | j �� }|dkr | j| _n�|dkr.| j| _n�|tkr:dS |dkrp| j�td dd�� | j�td d d�� nT|t kr�| j�td
|| j �
t d � d�� n&| j �
d �}| j�td || d�� d S )
NrC rD Fr[ r" zinvalid-codepoint)r# r$ rK u �rJ T)rC rD r[ ) r r6 �characterReferenceInRcdatar �rcdataLessThanSignStater r% r7 r r r^ )r r$ r_ r r r! � rcdataState s&
zHTMLTokenizer.rcdataStatec C s | � � | j| _dS )NT)rS rb r )r r r r! r` 1 s z(HTMLTokenizer.characterReferenceInRcdatac C s� | j �� }|dkr | j| _nh|dkrR| j�td dd�� | j�td dd�� n2|tkr^dS | j �d �}| j�td || d�� d
S ) NrD r[ r" zinvalid-codepoint)r# r$ rK u �F)rD r[ T) r r6 �rawtextLessThanSignStater r% r7 r r r^ )r r$ r_ r r r! � rawtextState6 s
zHTMLTokenizer.rawtextStatec C s� | j �� }|dkr | j| _nh|dkrR| j�td dd�� | j�td dd�� n2|tkr^dS | j �d �}| j�td || d�� d
S ) NrD r[ r" zinvalid-codepoint)r# r$ rK u �F)rD r[ T) r r6 �scriptDataLessThanSignStater r% r7 r r r^ )r r$ r_ r r r! �scriptDataStateH s
z HTMLTokenizer.scriptDataStatec C sr | j �� }|tkrdS |dkrL| j�td dd�� | j�td dd�� n"| j�td || j �d� d�� dS ) NFr[ r" zinvalid-codepoint)r# r$ rK u �T)r r6 r r% r7 r r^ )r r$ r r r! �plaintextStateZ s
z HTMLTokenizer.plaintextStatec C s | j �� }|dkr | j| _n�|dkr.| j| _n�|tkrVtd |g ddd�| _| j| _n�|dkr�| j �
td dd �� | j �
td
d d �� | j | _nt|d kr�| j �
td d
d �� | j � |� | j
| _n@| j �
td dd �� | j �
td
dd �� | j � |� | j | _dS )N�!�/ZStartTagF)r# rU r$ rW ZselfClosingAcknowledged�>r" z'expected-tag-name-but-got-right-bracket)r# r$ rK z<>�?z'expected-tag-name-but-got-question-markzexpected-tag-namerD T)r r6 �markupDeclarationOpenStater �closeTagOpenStater
r r � tagNameStater% r7 r r= �bogusCommentState)r r$ r r r! r] i s6
zHTMLTokenizer.tagOpenStatec C s� | j �� }|tkr0td |g dd�| _| j| _n�|dkrX| j�td dd�� | j | _nn|t
kr�| j�td dd�� | j�td d
d�� | j | _n0| j�td d d |id
�� | j � |� | j | _dS )NrV F)r# rU r$ rW rj r" z*expected-closing-tag-but-got-right-bracket)r# r$ z expected-closing-tag-but-got-eofrK z</z!expected-closing-tag-but-got-charr$ )r# r$ r. T)
r r6 r
r r rn r r% r7 r r r= ro )r r$ r r r! rm � s(
z HTMLTokenizer.closeTagOpenStatec C s� | j �� }|tkr | j| _n�|dkr.| �� n~|tkrV| j�t d dd�� | j
| _nV|dkrh| j | _nD|dkr�| j�t d dd�� | j d d 7 < n| j d |7 < d
S ) Nrj r" zeof-in-tag-name)r# r$ ri r[ zinvalid-codepointrU u �T)
r r6 r �beforeAttributeNameStater rZ r r% r7 r r �selfClosingStartTagStater )r r$ r r r! rn � s"
zHTMLTokenizer.tagNameStatec C sP | j �� }|dkr"d| _| j| _n*| j�td dd�� | j �|� | j | _dS )Nri r, rK rD )r# r$ T)
r r6 �temporaryBuffer�rcdataEndTagOpenStater r% r7 r r= rb )r r$ r r r! ra � s
z%HTMLTokenizer.rcdataLessThanSignStatec C sX | j �� }|tkr*| j|7 _| j| _n*| j�td dd�� | j � |� | j
| _dS )NrK z</)r# r$ T) r r6 r
rr �rcdataEndTagNameStater r% r7 r r= rb )r r$ r r r! rs � s
z#HTMLTokenizer.rcdataEndTagOpenStatec C s | j o | j d �� | j�� k}| j�� }|tkrT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | �
� | j | _nH|t kr�| j|7 _n0| j
�td d| j d �� | j�|� | j| _d
S ) NrU rV F)r# rU r$ rW ri rj rK z</)r# r$ T)r �lowerrr r r6 r r rp r rq rZ r r
r% r7 r= rb )r � appropriater$ r r r! rt � s2
z#HTMLTokenizer.rcdataEndTagNameStatec C sP | j �� }|dkr"d| _| j| _n*| j�td dd�� | j �|� | j | _dS )Nri r, rK rD )r# r$ T)
r r6 rr �rawtextEndTagOpenStater r% r7 r r= rd )r r$ r r r! rc � s
z&HTMLTokenizer.rawtextLessThanSignStatec C sX | j �� }|tkr*| j|7 _| j| _n*| j�td dd�� | j � |� | j
| _dS )NrK z</)r# r$ T) r r6 r
rr �rawtextEndTagNameStater r% r7 r r= rd )r r$ r r r! rw � s
z$HTMLTokenizer.rawtextEndTagOpenStatec C s | j o | j d �� | j�� k}| j�� }|tkrT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | �
� | j | _nH|t kr�| j|7 _n0| j
�td d| j d �� | j�|� | j| _d
S ) NrU rV F)r# rU r$ rW ri rj rK z</)r# r$ T)r ru rr r r6 r r rp r rq rZ r r
r% r7 r= rd )r rv r$ r r r! rx s2
z$HTMLTokenizer.rawtextEndTagNameStatec C sx | j �� }|dkr"d| _| j| _nR|dkrJ| j�td dd�� | j| _n*| j�td dd�� | j � |� | j
| _dS ) Nri r, rh rK z<!)r# r$ rD T) r r6 rr �scriptDataEndTagOpenStater r% r7 r �scriptDataEscapeStartStater= rf )r r$ r r r! re s
z)HTMLTokenizer.scriptDataLessThanSignStatec C sX | j �� }|tkr*| j|7 _| j| _n*| j�td dd�� | j � |� | j
| _dS )NrK z</)r# r$ T) r r6 r
rr �scriptDataEndTagNameStater r% r7 r r= rf )r r$ r r r! ry , s
z'HTMLTokenizer.scriptDataEndTagOpenStatec C s | j o | j d �� | j�� k}| j�� }|tkrT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | �
� | j | _nH|t kr�| j|7 _n0| j
�td d| j d �� | j�|� | j| _d
S ) NrU rV F)r# rU r$ rW ri rj rK z</)r# r$ T)r ru rr r r6 r r rp r rq rZ r r
r% r7 r= rf )r rv r$ r r r! r{ 7 s2
z'HTMLTokenizer.scriptDataEndTagNameStatec C sJ | j �� }|dkr2| j�td dd�� | j| _n| j �|� | j| _dS )N�-rK )r# r$ T) r r6 r% r7 r � scriptDataEscapeStartDashStater r= rf )r r$ r r r! rz S s
z(HTMLTokenizer.scriptDataEscapeStartStatec C sJ | j �� }|dkr2| j�td dd�� | j| _n| j �|� | j| _dS )Nr| rK )r# r$ T) r r6 r% r7 r � scriptDataEscapedDashDashStater r= rf )r r$ r r r! r} ] s
z,HTMLTokenizer.scriptDataEscapeStartDashStatec C s� | j �� }|dkr2| j�td dd�� | j| _n�|dkrD| j| _nn|dkrz| j�td dd�� | j�td dd�� n8|tkr�| j | _n&| j �
d �}| j�td || d�� d
S ) Nr| rK )r# r$ rD r[ r" zinvalid-codepointu �)rD r| r[ T) r r6 r% r7 r �scriptDataEscapedDashStater �"scriptDataEscapedLessThanSignStater r r^ )r r$ r_ r r r! �scriptDataEscapedStateg s"
z$HTMLTokenizer.scriptDataEscapedStatec C s� | j �� }|dkr2| j�td dd�� | j| _n�|dkrD| j| _nn|dkr�| j�td dd�� | j�td dd�� | j| _n0|t kr�| j
| _n | j�td |d�� | j| _d S )
Nr| rK )r# r$ rD r[ r" zinvalid-codepointu �T) r r6 r% r7 r r~ r r� r� r r )r r$ r r r! r { s"
z(HTMLTokenizer.scriptDataEscapedDashStatec C s� | j �� }|dkr*| j�td dd�� n�|dkr<| j| _n�|dkrd| j�td dd�� | j| _nn|dkr�| j�td dd�� | j�td d d�� | j| _n0|t kr�| j
| _n | j�td |d�� | j| _d
S ) Nr| rK )r# r$ rD rj r[ r" zinvalid-codepointu �T) r r6 r% r7 r r� r rf r� r r )r r$ r r r! r~ � s&
z,HTMLTokenizer.scriptDataEscapedDashDashStatec C s� | j �� }|dkr"d| _| j| _n\|tkrT| j�td d| d�� || _| j | _n*| j�td dd�� | j �
|� | j | _dS )Nri r, rK rD )r# r$ T) r r6 rr � scriptDataEscapedEndTagOpenStater r
r% r7 r � scriptDataDoubleEscapeStartStater= r� )r r$ r r r! r� � s
z0HTMLTokenizer.scriptDataEscapedLessThanSignStatec C sP | j �� }|tkr"|| _| j| _n*| j�td dd�� | j � |� | j
| _dS )NrK z</)r# r$ T) r r6 r
rr � scriptDataEscapedEndTagNameStater r% r7 r r= r� )r r$ r r r! r� � s
z.HTMLTokenizer.scriptDataEscapedEndTagOpenStatec C s | j o | j d �� | j�� k}| j�� }|tkrT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | �
� | j | _nH|t kr�| j|7 _n0| j
�td d| j d �� | j�|� | j| _d
S ) NrU rV F)r# rU r$ rW ri rj rK z</)r# r$ T)r ru rr r r6 r r rp r rq rZ r r
r% r7 r= r� )r rv r$ r r r! r� � s2
z.HTMLTokenizer.scriptDataEscapedEndTagNameStatec C s� | j �� }|ttd�B krR| j�td |d�� | j�� dkrH| j | _
q�| j | _
nB|t kr�| j�td |d�� | j|7 _n| j �
|� | j | _
dS )N)ri rj rK )r# r$ �scriptT)r r6 r r: r% r7 r rr ru � scriptDataDoubleEscapedStater r� r
r= )r r$ r r r! r� � s
z.HTMLTokenizer.scriptDataDoubleEscapeStartStatec C s� | j �� }|dkr2| j�td dd�� | j| _n�|dkrZ| j�td dd�� | j| _nt|dkr�| j�td dd�� | j�td dd�� n>|tkr�| j�td d d�� | j | _n| j�td |d�� d
S ) Nr| rK )r# r$ rD r[ r" zinvalid-codepointu �zeof-in-script-in-scriptT)
r r6 r% r7 r � scriptDataDoubleEscapedDashStater �(scriptDataDoubleEscapedLessThanSignStater r )r r$ r r r! r� � s$
z*HTMLTokenizer.scriptDataDoubleEscapedStatec C s� | j �� }|dkr2| j�td dd�� | j| _n�|dkrZ| j�td dd�� | j| _n�|dkr�| j�td dd�� | j�td dd�� | j| _nF|t kr�| j�td d d�� | j
| _n | j�td |d�� | j| _d
S ) Nr| rK )r# r$ rD r[ r" zinvalid-codepointu �zeof-in-script-in-scriptT) r r6 r% r7 r �$scriptDataDoubleEscapedDashDashStater r� r� r r )r r$ r r r! r� s(
z.HTMLTokenizer.scriptDataDoubleEscapedDashStatec C s | j �� }|dkr*| j�td dd�� n�|dkrR| j�td dd�� | j| _n�|dkrz| j�td dd�� | j| _n�|dkr�| j�td dd�� | j�td d d�� | j| _nF|t kr�| j�td d
d�� | j
| _n | j�td |d�� | j| _d S ) Nr| rK )r# r$ rD rj r[ r" zinvalid-codepointu �zeof-in-script-in-scriptT) r r6 r% r7 r r� r rf r� r r )r r$ r r r! r� s,
z2HTMLTokenizer.scriptDataDoubleEscapedDashDashStatec C sP | j �� }|dkr8| j�td dd�� d| _| j| _n| j �|� | j | _dS )Nri rK )r# r$ r, T)
r r6 r% r7 r rr � scriptDataDoubleEscapeEndStater r= r� )r r$ r r r! r� 0 s
z6HTMLTokenizer.scriptDataDoubleEscapedLessThanSignStatec C s� | j �� }|ttd�B krR| j�td |d�� | j�� dkrH| j | _
q�| j | _
nB|t kr�| j�td |d�� | j|7 _n| j �
|� | j | _
dS )N)ri rj rK )r# r$ r� T)r r6 r r: r% r7 r rr ru r� r r� r
r= )r r$ r r r! r� ; s
z,HTMLTokenizer.scriptDataDoubleEscapeEndStatec C s0 | j �� }|tkr$| j �td� �n|tkrJ| jd �|dg� | j| _n�|dkr\| � � n�|dkrn| j
| _n�|dkr�| j �t d dd �� | jd �|dg� | j| _n�|d
kr�| j �t d d d �� | jd �d dg� | j| _nF|t
k�r| j �t d d
d �� | j| _n | jd �|dg� | j| _dS )NTr$ r, rj ri )�'�"rI rD r" z#invalid-character-in-attribute-name)r# r$ r[ zinvalid-codepointu �z#expected-attribute-name-but-got-eof)r r6 r r^ r
r r7 �attributeNameStater rZ rq r% r r r )r r$ r r r! rp K s6
z&HTMLTokenizer.beforeAttributeNameStatec C s� | j �� }d}d}|dkr&| j| _�n.|tkr\| jd d d || j �td� 7 < d}n�|dkrjd}n�|tkr|| j| _n�|dkr�| j | _n�|d kr�| j
� t d
d d �� | jd d d d
7 < d}n�|dk�r | j
� t d
dd �� | jd d d |7 < d}nH|t
k�r6| j
� t d
dd �� | j| _n | jd d d |7 < d}|�r�| jd d d �t�| jd d d< xP| jd d d� D ]:\}}| jd d d |k�r�| j
� t d
dd �� P �q�W |�r�| �� dS )NTFrI r$ rF r rj ri r[ r" zinvalid-codepoint)r# r$ u �)r� r� rD z#invalid-character-in-attribute-namezeof-in-attribute-namezduplicate-attribute)r r6 �beforeAttributeValueStater r
r r^ r �afterAttributeNameStaterq r% r7 r r r rX r rZ )r r$ ZleavingThisStateZ emitTokenrU �_r r r! r� i sR
&
z HTMLTokenizer.attributeNameStatec C sD | j �� }|tkr$| j �td� �n |dkr8| j| _�n|dkrJ| �� n�|tkrp| jd � |dg� | j
| _n�|dkr�| j | _n�|dkr�| j � t
d d d
�� | jd � d dg� | j
| _n�|d kr�| j � t
d d
d
�� | jd � |dg� | j
| _nF|tk�r$| j � t
d dd
�� | j| _n | jd � |dg� | j
| _dS )NTrI rj r$ r, ri r[ r" zinvalid-codepoint)r# r$ u �)r� r� rD z&invalid-character-after-attribute-namez expected-end-of-tag-but-got-eof)r r6 r r^ r� r rZ r
r r7 r� rq r% r r r )r r$ r r r! r� � s:
z%HTMLTokenizer.afterAttributeNameStatec C sh | j �� }|tkr$| j �td� �n@|dkr8| j| _�n,|dkrX| j| _| j �|� �n |dkrj| j| _n�|dkr�| j �
t d dd�� | � � n�|d kr�| j �
t d d
d�� | j
d d d
d7 < | j| _n�|dk�r| j �
t d dd�� | j
d d d
|7 < | j| _nL|tk�rB| j �
t d dd�� | j| _n"| j
d d d
|7 < | j| _dS )NTr� rC r� rj r" z.expected-attribute-value-but-got-right-bracket)r# r$ r[ zinvalid-codepointr$ rF r u �)rI rD �`z"equals-in-unquoted-attribute-valuez$expected-attribute-value-but-got-eof)r r6 r r^ � attributeValueDoubleQuotedStater �attributeValueUnQuotedStater= � attributeValueSingleQuotedStater% r7 r rZ r r r )r r$ r r r! r� � s>
z'HTMLTokenizer.beforeAttributeValueStatec C s� | j �� }|dkr | j| _n�|dkr0| �d� n�|dkrj| j�td dd�� | jd d d d
7 < nN|t kr�| j�td d d�� | j
| _n&| jd d d || j � d � 7 < d
S )Nr� rC r[ r" zinvalid-codepoint)r# r$ r$ rF r u �z#eof-in-attribute-value-double-quote)r� rC r[ T) r r6 �afterAttributeValueStater rT r% r7 r r r r r^ )r r$ r r r! r� � s
z-HTMLTokenizer.attributeValueDoubleQuotedStatec C s� | j �� }|dkr | j| _n�|dkr0| �d� n�|dkrj| j�td dd�� | jd d d d
7 < nN|t kr�| j�td d d�� | j
| _n&| jd d d || j � d � 7 < d
S )Nr� rC r[ r" zinvalid-codepoint)r# r$ r$ rF r u �z#eof-in-attribute-value-single-quote)r� rC r[ T) r r6 r� r rT r% r7 r r r r r^ )r r$ r r r! r� � s
z-HTMLTokenizer.attributeValueSingleQuotedStatec C s | j �� }|tkr | j| _n�|dkr0| �d� n�|dkrB| �� n�|dkr|| j�t d dd�� | j
d d d |7 < n�|d
kr�| j�t d d d�� | j
d d d d 7 < nV|t kr�| j�t d d
d�� | j | _n.| j
d d d || j �
td�tB � 7 < dS )NrC rj )r� r� rI rD r� r" z0unexpected-character-in-unquoted-attribute-value)r# r$ r$ rF r r[ zinvalid-codepointu �z eof-in-attribute-value-no-quotes)rC rj r� r� rI rD r� r[ T)r r6 r rp r rT rZ r% r7 r r r r r^ r: )r r$ r r r! r� s,
z)HTMLTokenizer.attributeValueUnQuotedStatec C s� | j �� }|tkr | j| _n�|dkr.| �� np|dkr@| j| _n^|tkrt| j� t
d dd�� | j � |� | j | _n*| j� t
d dd�� | j � |� | j| _dS )Nrj ri r" z$unexpected-EOF-after-attribute-value)r# r$ z*unexpected-character-after-attribute-valueT)
r r6 r rp r rZ rq r r% r7 r r= r )r r$ r r r! r� s"
z&HTMLTokenizer.afterAttributeValueStatec C s� | j �� }|dkr&d| jd< | �� n^|tkrZ| j�td dd�� | j �|� | j | _
n*| j�td dd�� | j �|� | j | _
dS )Nrj TrW r" z#unexpected-EOF-after-solidus-in-tag)r# r$ z)unexpected-character-after-solidus-in-tag) r r6 r rZ r r% r7 r r= r r rp )r r$ r r r! rq 4 s
z&HTMLTokenizer.selfClosingStartTagStatec C sD | j �d�}|�dd�}| j�td |d�� | j �� | j| _dS )Nrj r[ u ��Comment)r# r$ T) r r^ �replacer% r7 r r6 r r )r r$ r r r! ro F s
z HTMLTokenizer.bogusCommentStatec C s� | j �� g}|d dkrR|�| j �� � |d dkrPtd dd�| _| j| _dS n�|d dkr�d}x.dD ]&}|�| j �� � |d |krhd }P qhW |r�td
dd d dd �| _| j| _dS n�|d d k�rF| jd k �rF| jj j
�rF| jj j
d j | jj j k�rFd}x2d
D ]*}|�| j �� � |d |k�rd }P �qW |�rF| j
| _dS | j�td dd�� x |�rx| j �|�� � �q^W | j| _dS )NrF r| r� r, )r# r$ T)�d�D))�o�O)r@ �C)�t�T)�y�Y)�p�P)�e�EFZDoctype)r# rU �publicId�systemId�correct�[)r� r� �Ar� r� r� r" zexpected-dashes-or-doctype)r r6 r7 r r �commentStartStater � doctypeStater ZtreeZ openElements� namespaceZdefaultNamespace�cdataSectionStater% r= r' ro )r r? �matched�expectedr r r! rl U sP
z(HTMLTokenizer.markupDeclarationOpenStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� | j�| j� | j| _nP|t kr�| j�td d
d�� | j�| j� | j| _n| jd |7 < | j
| _d S ) Nr| r[ r" zinvalid-codepoint)r# r$ r$ u �rj zincorrect-commentzeof-in-commentT) r r6 �commentStartDashStater r% r7 r r r r � commentState)r r$ r r r! r� � s(
z HTMLTokenizer.commentStartStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� | j�| j� | j| _nT|t kr�| j�td d
d�� | j�| j� | j| _n | jd d| 7 < | j
| _d S ) Nr| r[ r" zinvalid-codepoint)r# r$ r$ u -�rj zincorrect-commentzeof-in-commentT) r r6 �commentEndStater r% r7 r r r r r� )r r$ r r r! r� � s(
z#HTMLTokenizer.commentStartDashStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < nT|tkr�| j�td dd�� | j�| j� | j | _n | jd || j �
d � 7 < d
S ) Nr| r[ r" zinvalid-codepoint)r# r$ r$ u �zeof-in-comment)r| r[ T) r r6 �commentEndDashStater r% r7 r r r r r^ )r r$ r r r! r� � s
zHTMLTokenizer.commentStatec C s� | j �� }|dkr | j| _n�|dkrV| j�td dd�� | jd d7 < | j| _nT|t kr�| j�td dd�� | j�| j� | j
| _n | jd d| 7 < | j| _d S )
Nr| r[ r" zinvalid-codepoint)r# r$ r$ u -�zeof-in-comment-end-dashT) r r6 r� r r% r7 r r r� r r )r r$ r r r! r� � s
z!HTMLTokenizer.commentEndDashStatec C s, | j �� }|dkr*| j�| j� | j| _n�|dkrd| j�td dd�� | jd d7 < | j| _n�|dkr�| j�td d d�� | j | _n�|d
kr�| j�td d d�� | jd |7 < nj|t
kr�| j�td d d�� | j�| j� | j| _n4| j�td d
d�� | jd d| 7 < | j| _dS )Nrj r[ r" zinvalid-codepoint)r# r$ r$ u --�rh z,unexpected-bang-after-double-dash-in-commentr| z,unexpected-dash-after-double-dash-in-commentzeof-in-comment-double-dashzunexpected-char-in-commentz--T) r r6 r% r7 r r r r r� �commentEndBangStater )r r$ r r r! r� � s6
z HTMLTokenizer.commentEndStatec C s� | j �� }|dkr*| j�| j� | j| _n�|dkrN| jd d7 < | j| _n�|dkr�| j�td dd�� | jd d 7 < | j | _nT|t
kr�| j�td d
d�� | j�| j� | j| _n | jd d| 7 < | j | _d S ) Nrj r| r$ z--!r[ r" zinvalid-codepoint)r# r$ u --!�z eof-in-comment-end-bang-stateT) r r6 r% r7 r r r r� r r� r )r r$ r r r! r� � s(
z!HTMLTokenizer.commentEndBangStatec C s� | j �� }|tkr | j| _nj|tkr\| j�td dd�� d| j d< | j�| j � | j
| _n*| j�td dd�� | j � |� | j| _dS )Nr" z!expected-doctype-name-but-got-eof)r# r$ Fr� zneed-space-after-doctypeT) r r6 r �beforeDoctypeNameStater r r% r7 r r r r= )r r$ r r r! r� s
zHTMLTokenizer.doctypeStatec C s� | j �� }|tkrn�|dkrT| j�td dd�� d| jd< | j�| j� | j| _n�|dkr�| j�td dd�� d | jd
< | j | _nR|t
kr�| j�td d d�� d| jd< | j�| j� | j| _n|| jd
< | j | _d S )
Nrj r" z+expected-doctype-name-but-got-right-bracket)r# r$ Fr� r[ zinvalid-codepointu �rU z!expected-doctype-name-but-got-eofT) r r6 r r% r7 r r r r �doctypeNameStater )r r$ r r r! r� s.
z$HTMLTokenizer.beforeDoctypeNameStatec C s | j �� }|tkr2| jd �t�| jd< | j| _n�|dkrh| jd �t�| jd< | j� | j� | j
| _n�|dkr�| j� t d dd�� | jd d7 < | j | _nh|t
kr�| j� t d dd�� d | jd
< | jd �t�| jd< | j� | j� | j
| _n| jd |7 < d S ) NrU rj r[ r" zinvalid-codepoint)r# r$ u �zeof-in-doctype-nameFr� T)r r6 r r rX r �afterDoctypeNameStater r% r7 r r r� r )r r$ r r r! r� 6 s,
z HTMLTokenizer.doctypeNameStatec C sL | j �� }|tkr�n2|dkr8| j�| j� | j| _�n|tkr�d| jd< | j � |� | j�t
d dd�� | j�| j� | j| _n�|dkr�d}x$d D ] }| j �� }||kr�d}P q�W |r�| j | _dS nF|d
k�rd}x$d D ] }| j �� }||kr�d}P q�W |�r| j | _dS | j � |� | j�t
d d d
|id�� d| jd< | j
| _dS )Nrj Fr� r" zeof-in-doctype)r# r$ )r� r� T))�u�U)�b�B)�l�L)�i�I)r@ r� )�s�S))r� r� )r� r� )r� r� )r� r� )�m�Mz*expected-space-or-right-bracket-in-doctyper$ )r# r$ r. )r r6 r r% r7 r r r r r= r � afterDoctypePublicKeywordState� afterDoctypeSystemKeywordState�bogusDoctypeState)r r$ r� r� r r r! r� O sP
z#HTMLTokenizer.afterDoctypeNameStatec C s� | j �� }|tkr | j| _n�|dkrP| j�td dd�� | j �|� | j| _nT|t kr�| j�td dd�� d| j
d< | j�| j
� | j | _n| j �|� | j| _dS ) N)r� r� r" zunexpected-char-in-doctype)r# r$ zeof-in-doctypeFr� T) r r6 r �"beforeDoctypePublicIdentifierStater r% r7 r r= r r r )r r$ r r r! r� � s"
z,HTMLTokenizer.afterDoctypePublicKeywordStatec C s� | j �� }|tkrn�|dkr0d| jd< | j| _n�|dkrLd| jd< | j| _n�|dkr�| j�t d dd�� d | jd
< | j�| j� | j
| _nh|t kr�| j�t d d d�� d | jd
< | j�| j� | j
| _n(| j�t d d d�� d | jd
< | j | _d
S )Nr� r, r� r� rj r" zunexpected-end-of-doctype)r# r$ Fr� zeof-in-doctypezunexpected-char-in-doctypeT)
r r6 r r �(doctypePublicIdentifierDoubleQuotedStater �(doctypePublicIdentifierSingleQuotedStater% r7 r r r r� )r r$ r r r! r� � s4
z0HTMLTokenizer.beforeDoctypePublicIdentifierStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d
| jd < | j�| j� | j| _nR|t kr�| j�td d d�� d
| jd < | j�| j� | j| _n| jd |7 < d
S )Nr� r[ r" zinvalid-codepoint)r# r$ r� u �rj zunexpected-end-of-doctypeFr� zeof-in-doctypeT)
r r6 �!afterDoctypePublicIdentifierStater r% r7 r r r r )r r$ r r r! r� � s*
z6HTMLTokenizer.doctypePublicIdentifierDoubleQuotedStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d
| jd < | j�| j� | j| _nR|t kr�| j�td d d�� d
| jd < | j�| j� | j| _n| jd |7 < d
S )Nr� r[ r" zinvalid-codepoint)r# r$ r� u �rj zunexpected-end-of-doctypeFr� zeof-in-doctypeT)
r r6 r� r r% r7 r r r r )r r$ r r r! r� � s*
z6HTMLTokenizer.doctypePublicIdentifierSingleQuotedStatec C s | j �� }|tkr | j| _n�|dkr<| j�| j� | j| _n�|dkrn| j�t d dd�� d| jd< | j
| _n�|dkr�| j�t d dd�� d| jd< | j | _nh|t kr�| j�t d d d�� d
| jd < | j�| j� | j| _n(| j�t d dd�� d
| jd < | j
| _d S )
Nrj r� r" zunexpected-char-in-doctype)r# r$ r, r� r� zeof-in-doctypeFr� T)r r6 r �-betweenDoctypePublicAndSystemIdentifiersStater r% r7 r r r �(doctypeSystemIdentifierDoubleQuotedState�(doctypeSystemIdentifierSingleQuotedStater r� )r r$ r r r! r� � s6
z/HTMLTokenizer.afterDoctypePublicIdentifierStatec C s� | j �� }|tkrn�|dkr4| j�| j� | j| _n�|dkrPd| jd< | j| _n�|dkrld| jd< | j | _nh|t
kr�| j�t d dd�� d | jd
< | j�| j� | j| _n(| j�t d d d�� d | jd
< | j | _d S )
Nrj r� r, r� r� r" zeof-in-doctype)r# r$ Fr� zunexpected-char-in-doctypeT)
r r6 r r% r7 r r r r� r� r r r� )r r$ r r r! r� s.
z;HTMLTokenizer.betweenDoctypePublicAndSystemIdentifiersStatec C s� | j �� }|tkr | j| _n�|dkrP| j�td dd�� | j �|� | j| _nT|t kr�| j�td dd�� d| j
d< | j�| j
� | j | _n| j �|� | j| _dS ) N)r� r� r" zunexpected-char-in-doctype)r# r$ zeof-in-doctypeFr� T) r r6 r �"beforeDoctypeSystemIdentifierStater r% r7 r r= r r r )r r$ r r r! r� s"
z,HTMLTokenizer.afterDoctypeSystemKeywordStatec C s� | j �� }|tkrn�|dkr0d| jd< | j| _n�|dkrLd| jd< | j| _n�|dkr�| j�t d dd�� d | jd
< | j�| j� | j
| _nh|t kr�| j�t d d d�� d | jd
< | j�| j� | j
| _n(| j�t d dd�� d | jd
< | j | _d S )
Nr� r, r� r� rj r" zunexpected-char-in-doctype)r# r$ Fr� zeof-in-doctypeT)
r r6 r r r� r r� r% r7 r r r r� )r r$ r r r! r� / s4
z0HTMLTokenizer.beforeDoctypeSystemIdentifierStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d
| jd < | j�| j� | j| _nR|t kr�| j�td d d�� d
| jd < | j�| j� | j| _n| jd |7 < d
S )Nr� r[ r" zinvalid-codepoint)r# r$ r� u �rj zunexpected-end-of-doctypeFr� zeof-in-doctypeT)
r r6 �!afterDoctypeSystemIdentifierStater r% r7 r r r r )r r$ r r r! r� L s*
z6HTMLTokenizer.doctypeSystemIdentifierDoubleQuotedStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d
| jd < | j�| j� | j| _nR|t kr�| j�td d d�� d
| jd < | j�| j� | j| _n| jd |7 < d
S )Nr� r[ r" zinvalid-codepoint)r# r$ r� u �rj zunexpected-end-of-doctypeFr� zeof-in-doctypeT)
r r6 r� r r% r7 r r r r )r r$ r r r! r� d s*
z6HTMLTokenizer.doctypeSystemIdentifierSingleQuotedStatec C s� | j �� }|tkrn~|dkr4| j�| j� | j| _n^|tkrt| j�t d dd�� d| jd< | j�| j� | j| _n | j�t d dd�� | j
| _dS ) Nrj r" zeof-in-doctype)r# r$ Fr� zunexpected-char-in-doctypeT) r r6 r r% r7 r r r r r r� )r r$ r r r! r� | s
z/HTMLTokenizer.afterDoctypeSystemIdentifierStatec C sZ | j �� }|dkr*| j�| j� | j| _n,|tkrV| j �|� | j�| j� | j| _n dS )Nrj T) r r6 r% r7 r r r r r= )r r$ r r r! r� � s
z HTMLTokenizer.bogusDoctypeStatec C s� g }x�|� | j�d�� |� | j�d�� | j�� }|tkr@P q|dksLt�|d dd � dkrx|d d d� |d< P q|� |� qW d�|�}|�d�}|dkr�x&t|�D ]}| j � t
d d
d �� q�W |� dd �}|r�| j � t
d
|d �� | j | _
dS )N�]rj rF ���z]]r, r[ r r" zinvalid-codepoint)r# r$ u �rK T)r7 r r^ r6 r �AssertionErrorr9 �count�ranger% r r� r r )r r$ r6 Z nullCountr� r r r! r� � s0
z HTMLTokenizer.cdataSectionState)N)NF)N�__name__�
__module__� __qualname__�__doc__r r) rB rS rT rZ r r\ rb r` rd rf rg r] rm rn ra rs rt rc rw rx re ry r{ rz r} r� r r~ r� r� r� r� r� r� r� r� r� rp r� r� r� r� r� r� r� rq ro rl r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� �
__classcell__r r )r r! r s� H
P #
6 "- 3 r N)�
__future__r r r �sixr r; � collectionsr � constantsr r r
r r r
r r r r Z _inputstreamr Z_trier rL �objectr r r r r! �<module> s