Systematic parsing of X.509: Eradicating security issues with a parse tree

2.1. Parser generation

Given a finite set of symbols V t , known as alphabet, a language L is defined as a set of elements, named sentences, obtained by concatenating zero or more symbol of V t . Conventionally, the empty sentence is denoted as ε, while a portion of a sentence is known as a factor.

Given a language it is also possible to describe it via a generative formalism, i.e., a grammar. A grammar is a quadruple G : ( V t , V n , P , S ) , with V t the alphabet of the generated language, V n a finite set of symbols named nonterminals, P a set of productions defined as pairs of strings obtained by concatenating elements of V = V t ∪ V n , and S ∈ V n the axiom of the grammar. The productions are denoted as usual as α → β ; α ∈ V n + , β ∈ V ∗ , where V + represents the closure with respect to concatenation over V, and V ∗ = V + ∪ { ε } . A sentence of a language L is generated by the grammar G if it is possible to obtain it starting from a string u ∈ V ∗ made of the grammar axiom alone (i.e., the nonterminal S), and iteratively substituting a portion of u appearing as the left hand side of a production in P with the right hand side of the said production, until the result is made only of elements of V t . The grammar is said to be ambiguous, if it is possible to generate a sentence of the language through two different sequences of substitutions. Languages are classified according to the Chomsky hierarchy [18], depending on which constraints are holding on the productions of the grammar generating them. Language families generated from grammars with stricter constraints are included in all the families with weaker constraints. In the following, we describe the language families starting from the least constrained one. Grammars with no restrictions on the left and right sides of its productions are denoted as unrestricted grammars, and generate context sensitive with erasure (CS-E) languages. Determining if a generic string over V t belongs to a CS-E language is not decidable.

Restricting the productions of a grammar so that the sequence of substitutions does not allow the erasure of symbols yields context sensitive (CS) grammars, which generate CS languages. Determining if a generic string over V t belongs to a CS language is decidable, i.e., it is always possible to state whether a string over V t can be generated by a CS grammar. Restricting the productions of a grammar to have a single element of V n on their left hand side yields the so-called context-free (CF) grammars, which generate CF languages. Deterministic CF (DCF) languages can be recognized by a Deterministic PushDown Automaton (DPDA) and their generating grammars constitute a proper subset of non ambiguous CF grammars. Finally, restricting the productions of a grammar either to the ones of the form { A → a , A → a A } or to the ones of the form { A → a , A → A a } , where A ∈ V n , a ∈ V t ∪ { ε } , yields right- or left-linear grammars, which generate regular (REG) languages. We will denote such grammars as linear whenever the direction of the recursion in their productions is not fundamental.

Given a grammar G generating the language L and a generic terminal string w ∈ V t ∗ , the process of parsing consists both in determining if w ∈ L and in computing the sequence(s) of applications of the productions of G which generate(s) w. The sequence(s) of productions are depicted as a tree, called either parse tree or abstract syntax tree, with the axiom of the grammar being the root and the terminal symbols being the leaves. There exists a straightforward parsing algorithm of a CS language which is complete and correct, but its running time is exponential in the length of the input. As a consequence, a number of higher efficiency automated parser generation techniques were defined for subsets of the CS grammar family. CF grammars are the mainstay of parser generation as it is possible to automatically generate a recognizer automaton for any language generated by them. In particular, given a generic DCF grammar it is possible to automatically generate a deterministic parser for the strings of the corresponding language [18]; such parser enjoys worst-case linear space and time requirements in the length of input. Linear grammars are optimal from a parsing standpoint, as it is possible to derive from them a parser which runs with constant memory and linear time requirements in the length of input. Moreover, a parser for a REG language can be generated starting from the definition of a regular expression [6].

We aim at obtaining a grammar definition for the X.509 language such that the parsing is decidable. In addition, we require that such grammar is expressed as much as possible in terms of right-linear productions so that the parser will only require a constant amount of memory (besides the one to contain the input). In the remainder of this work, the terminal alphabet V t will be the set of 256 values which can be taken by a byte, unless pointed out otherwise. Each terminal symbol will be denoted by two hexadecimal digits (e.g., the byte taking the decimal value 42 will be denoted as 2A). We will also employ the Extended Backus-Naur Form (EBNF) to write the right hand sides of grammar productions.

2.2. ASN.1 overview

The ASN.1 meta-syntax was introduced by the ITU to specify structured data as abstract data types (ADTs), and it is described in the set of ITU Recommendations (ITU-R) X.680-X.683 [21]. The encoding schemes for the ADTs are specified in ITU-R X.690–X.696 [21]. The ASN.1 provides the means to express both the syntax of the ADT at hand, in a form akin to a grammar, and some semantic constraints concerning the values taken by an instance of such an ADT. As our systematic parsing strategy will rely on a parser generated from a grammar specification for the X.509 ADT, we first provide a mapping between the syntactic-structure-specifying keywords of ASN.1 and the corresponding productions of an EBNF grammar. Subsequently, we highlight how the remaining, non purely syntactic features of ASN.1 act as constraints on the language of the instances of the described ADT in terms of semantics and describe the meta-language facilities exposed to ease the definition of a non ambiguous specification.

Syntactic elements. An ASN.1 ADT can be regarded as a construct equivalent to a single EBNF grammar generating all the possible concrete data type instances as its language. The user-defined name of the ADT corresponds to the axiom of the EBNF grammar, while the productions are represented as structured data definitions with the ::= operator separating the left and right hand side, in lieu of the common →.

The structure of an ADT may either be a single element, in case the type is primitive, or a composition of other types employing ASN.1 constructs in case it is constructed.

Primitive types in ASN.1 represent terminal rules of an EBNF grammar, i.e., rules where α → β , α ∈ V n + , β ∈ V t ∗ . The right hand side of a primitive type definition is described by a single line ended by a specific keyword (e.g., INTEGER, BOOLEAN, OBJECT IDENTIFIER, OCTET STRING, BIT STRING), specifying completely its nature. An OBJECT IDENTIFIER is a sequence of decimal numbers separated by dots, of which a single decimal number is known as an arc.

Constructed types may either have a single user-defined name appearing on the right hand side of their definition, in which case they act as the copy rules of an EBNF grammar (i.e., rules where α → β , α , β ∈ V n ), or an arbitrary ASN.1 syntactic construct may be used. The only exception to the aforementioned constructed type definition is the possibility of turning a primitive OCTET STRING or BIT STRING type into a constructed one through appending the keyword CONTAINING to it, followed by the description of its contents. Deriving a constructed type from an OCTET STRING forces its value in an ADT instance to be byte-aligned, and allows the designer to enforce the choice of the encoding rules to be employed for it with the ENCODED BY keywords followed by the encoding name (see ITU-R X.682) [21]. Table 1 shows a mapping between the ASN.1 structures appearing on the right hand side of the data types which are considered in this work and their matching EBNF notation, expressed with a set of sample ASN.1 types u, v, x and identically named strings u , v , x ∈ ( V t ∪ V n ) ∗ in EBNF.

The ASN.1 notation specifies the common concatenation and alternative choice via the SEQUENCE and CHOICE keywords, respectively, while the syntactic constraint indicated by the SET keyword mandates that a set of ADTs may appear in any order, without repetitions. Postfixing an ADT appearing in a right hand side of a definition with the OPTIONAL keyword allows it to be either missing or present only once. Given an ASN.1 ADT u, the syntactic constraints imposed by the SEQUENCE OF and SET OF constructs, indicate a concatenation of zero or more instances of u, matching the star operator in EBNF. The ternary range operator in ASN.1, having the syntax t(low .. high), where t is an ADT and low, high the range boundaries, is employed with two purposes: specifying the range of possible values of the instances of the primitive ADT to which they are appended, or indicating the concatenation of any number low < n < high of instances of the user-defined ADT preceding them. The bounds of a range operator are either constant values or the keywords MIN and MAX, which indicate that the minimum (resp., maximum) of the given range, is interpreted as the smallest (resp., greatest) value which can be taken by the ADT on their left. ASN.1 allows to specify size constraints through the use of the keyword SIZE, followed by a range of allowed sizes. A common idiom in ASN.1 ADT declarations is to employ MAX as an upper bound for SIZEs to indicate that there is no upper bound on the size. As a consequence, the two common ASN.1 SEQUENCE SIZE(1 .. MAX) OF and SET SIZE(1 .. MAX) OF idioms in Table 1 are representing a concatenation of one or more instances of the involved ADT u, corresponding to the EBNF cross construct. Finally, the ASN.1 keyword ANY allows to delegate the definition of the structure of a given ADT to another document, potentially not expressed in ASN.1. Specifying further the effect of the ANY construct with the DEFINED BY keywords, followed by the name of an ADT, enforces the fact that the specification should be expressed in ASN.1.

Semantic elements and disambiguation constructs. ASN.1 allows to describe semantic information on instances of an ADT either specifying a constant value for a given primitive type element, appending such value between round brackets on the line where the type appears, or specifying a so-called DEFAULT value. Appending the DEFAULT keyword allows to indicate that, in case an element is missing in an instance of the ADT, the recognizer should assume its presence nonetheless, and assign the semantic value present in the ASN.1 specification to it. The expressive power of the ASN.1 allows the designer to specify an ADT corresponding to an inherently ambiguous language, i.e., a language for which no unambiguous grammar exists. An illustrative example of such a case is the following ADT t:

which has its instances belonging to the intrinsically ambiguous language L = { u i v i x j ∨ u j v i x i s.t. u , v , x ∈ V t , i ⩾ 1 , j ⩾ 1 } . To provide a convenient way to cope with ambiguities, ASN.1 introduces the so-called user-defined tag elements. A tag is a syntactic element, denoted as a decimal number enclosed in square brackets, which is prefixed to an ADT appearing in the right hand side of a data description. Proper use of tags minimally alters the language of accepted ADT instances, while effectively curbing ambiguities. The inherent ambiguity of the sentences of the language of the previously shown ADT t can be eliminated adding two tags to its description:

It is worth noting that there is no algorithmic way to check whether a given set of tags introduced in an ASN.1 specification is enough to suppress all parsing ambiguities, as the expressive power of ASN.1 is sufficient to define a generic CF grammar, for which determining if it is ambiguous is a well known undecidable problem [18].

The encoding rules for an ASN.1 data type instance (see ITU-R X.690–X.696 [21]) define several formats portable across different architectures by mandating bit and byte value conventions and ordering of the encoded contents. We tackle the Distinguished Encoding Rules (DER), as a constructed typed field in the X.509 standard certificate ADT requires its DER encoding via the ENCODED BY keyword. Consequentially, DER is the encoding rule employed in the overwhelming majority of X.509 certificate instances found. DER encoded material may be further mapped to the fully printable base64 encoding, resulting in the so-called Privacy-enhanced Electronic Mail (PEM) format [27]. The DER encoding strategy represents an ASN.1 ADT instance as a stream of bytes which is logically split up into three fields: identifier octets, length octets, contents octets.

The identifier octets field is employed to encode the ASN.1 tag value and whether the ADT instance at hand is a primitive or a constructed ASN.1 type. The tag value may be either the disambiguating user-defined one present in the ASN.1 ADT definition, or a so-called universal tag assigned by the DER standard to all ASN.1 primitive types and to the SEQUENCE, SEQUENCE OF, SET, SET OF constructs. If a user-defined tag is present, its encoding is stored in the identifier octets field, while the encoding of the tagged ADT is stored in the contents octets field. To provide a more succinct encoding, ASN.1 allows to specify that a given user-defined tag is IMPLICIT, i.e., that it should replace the tag of the tagged ADT in the encoding in the identifier octets field. On the other hand, the EXPLICIT keyword states that a user-defined tag should be encoded according to the default behavior.

The contents octets field contains the actual encoding of the ADT instance at hand, while the length octet one stores the size of the content field as a number of bytes. The number of bytes constituting the length octets field varies in the 1 to 126 range. The encoding conventions for the length value are stated in the ITU-R X.690 [21]. A short form and a long form for the encoding of a length value are possible. The short form mandates the encoding of the length field as a single octet in which the most significant bit is 0 and the remaining ones encode the size of the contents octet field from 0 to 127 bytes. The long form consists of one initial octet followed by one or more subsequent octets containing the actual value of the length octet field. In the initial octet, the most significant bit is 1, while the remaining ones encode the number of length field octets that follow as an integer varying from 1 to 126. Thus, the number of bytes for the contents octet field is at most 2 126 · 8 . The length octet field value equal to 128 encoded as a single byte is forbidden in DER, while in other standard encoding rules it is reserved to indicate that an indefinite number of bytes will follow. It is worth noting that the requirement to validate correctly the information in the length octets field against the actual length of the contents octets one can be done via a regular language matcher, as the lengths have an upper bound. Nevertheless, the corresponding minimal finite state automaton (FSA) has a number of states which cannot be represented (see Section 4).

OPEN IN VIEWER

Fig. 1.

Portion of the description of the X.509 Certificate ADT and its fields.

The structure of an X.509 certificate is described as an ASN.1 ADT in the ITU-R X.509 [22], and in the RFC 4210 [1] and its complements [8,10,19,26,37,38]. Its structure evolved over time, as its first version dates back to 1988. In this section, we provide a synthetic description of its contents.

Certificate abstract data type. Fig. 1 reports a shortened version of the X.509 standard, defining the main Certificate ADT. In the figure, field names start with a lowercase letter, while ASN.1 user-defined ADT names start with capital letters. The Certificate ADT is a concatenation of three fields: the material to be signed typed as TBSCertificate, the identification data for the signature algorithm typed as AlgorithmIdentifier, and a BIT STRING field containing the actual signature value.

TBSCertificate abstract data type. Considering the contents of the TBSCertificate ADT, the first two fields contain a version number typed as Version (a value constrained INTEGER), and an integer value which must be unique among all the certificates signed by the same certification authority (CA), which is typed as CertSerialNumber. The third field, typed as an AlgorithmIdentifier, contains the information to uniquely identify the cryptographic primitive employed to sign the certificate.

The AlgorithmIdentifier ADT is a concatenation of two fields: an ASN.1 OBJECT IDENTIFIER (OID) typed field algorithm and an optional parameters field, typed as ANY DEFINED BY AlgorithmP. The OID value allows to uniquely label the signature algorithm to allow the description of the structure of its parameters, done in [10,19,26,37,38] for a set of standardized signature algorithms. The issuer, validity and subject fields contain information on the CA issuing the certificate and the subject to whom it has been issued, together with the validity time interval of the certificate. issuer and subject fields are typed as a Name ADT: a SEQUENCE OF of SET OF structures containing a concatenation of two fields typed as OID and ANY, respectively. Despite the quite baroque definition, the Name ADT is indeed employed to represent a list of names for both the issuer and the subject which are typically expressed as printable strings prefixed with a standardized OID value stating their meaning (e.g., organization, state).

The subjectPublicKeyInfo field provides both the public key binded to the subject identity, and information on the employed cryptographic primitive in the form of a BIT STRING and an AlgorithmIdentifier typed field, respectively. Following the subjectPublicKeyInfo field, the TBSCertificate ADT includes two deprecated extra optional fields, containing further information about the issuer and the subject. These fields are tagged with tags [1] and [2] respectively, preventing a possible parsing ambiguity arising from only one of them being present.

OPEN IN VIEWER

Fig. 2.

X.509 ASN.1 description reporting the definition of the last field of the TBSCertificate ADT.

Extension abstract data type. The extensions field concludes the definition of the TBSCertificate ADT. Most of the information of modern certificates is contained in it, and its presence is mandatory in the current version (v3) of X.509 certificates. As reported in Fig. 2 the Extensions ADT is a sequence of one or more Extension typed fields. Each Extension ADT is composed of an OID typed field identifying it unambiguously, and a critical field typed as BOOLEAN indicating, if True, that the certificate validation should fail in case the application either does not recognize or cannot process the information contained in the subsequent extnValue field. An example of information contained in the extnValue field is the so-called KeyUsage, i.e., information stating which is the legitimate purpose of the subject public key in the certificate at hand (e.g., signature validation or encryption).

3.1. Summing up hindrances to X.509 parsing

In the following statements we provide a synthesis of the undecidability, context sensitiveness and ambiguity problems arising from the X.509 certificate standard and recall the issue about the size of the grammar representation reported in Section 2.3.

In addition to the aforementioned issues, some of the portions of the X.509 language which can be defined with a linear grammar are still unpractical to be fully specified.

We consider the aforementioned issues to be among the main causes of the current approach to X.509 parsing, which sees fully handcrafted implementations of the parser as an alternative to the development of an automatically generated one. However, given the complexity of the X.509 certificate standard specification, and the impossibility of methodically checking the correctness of a human-implemented recognizer (due to context-sensitive portions of the X.509 language), the current state-of-the-art parser implementations are exhibiting a non-trivial amount of security critical errors in the recognition of X.509 sentences [23,29,30].

4.1. Coping with undecidability, context sensitiveness and ambiguity

The first step to obtain a useful grammar description of the X.509 standard is to cope with its parsing undecidability problem. We restricted the AlgorithmIdentifier ADT descriptions to the ones which are explicitly and fully described in the standards [10,19,26,37,38]. As a consequence we remove the possibility of having an unrestricted syntactic structure of their fields allowed by the ASN.1 keyword ANY in the standard. This decision implies that we will recognize certificates employing only algorithms for which a standardized AlgorithmIdentifier description exists, which is typically a sign of wide acceptance of the cryptographic security margin offered by them. Examining all the standardized AlgorithmIdentifier descriptions we note that their structures are described by regular languages. This restriction may prevent correct recognition of valid X.509 certificates found in the wild. During our practical evaluation campaign, we observed just a negligible percentage of certificates being affected by this restriction, as detailed in Section 5. We tackled the similar problem caused by the ANY-typed extnValue field by parsing the syntactic structure of all the standard defined extensions [8] strictly.

However, in contrast with the decision taken for the non standard AlgorithmIdentifier structures, we accept user-defined extensions, considering them opaque byte sequences, and performing only a syntactic check on the correctness of the OID field and the length of the unspecified structure field. Indeed, in case of a custom extension for which the Boolean critical field is set, a validation failure must be signaled by the application logic, in case it is not able to handle the contents of the extnValue field [22]. Taking into account the described restrictions, the resulting X.509 specification is a CS language L X .509 − r 1 where the string membership problem is decidable.

The second step in producing a grammar description of X.509 is to tackle its context-sensitiveness. To this end, we consider the language L X .509 − r 1 as the intersection of three languages L X .509 − r 1 = L AId − match ∩ L SI − match ∩ L CF , where L AId − match is the language containing sentences with two matching AlgorithmIdentifier type instances, L SI − match is the one where whenever the certificate subject and issuer match, the keyIdentifier field of the AuthorityKeyIdentifier extension ADT should be present, and L CF is the CF language where all the remaining X.509 linguistic constraints are holding on the sentences. Thus, parsing L X .509 − r 1 can be done analyzing the candidate string (i.e., a certificate instance) with three parsers, each one recognizing one of the three aforementioned languages. We choose to start with the parser A CF for L CF , as it can be automatically generated from its grammar description. The parse tree produced by A CF is then employed by a handwritten string matcher, which checks the conditions required for a string to belong both to L AId − match and to L SI − match . The code auditing of such a simple handwritten string matcher can be confidently performed by direct inspection to check its correctness. Indeed, these pieces of code are small and definitely simpler than a full hand-written parser, dramatically decreasing the complexity of the code auditing process. Therefore, spotting any vulnerabilities in these hand-written parsers is expected to be easier. Finally, we remark that this code has no influence on the automatic generated parser for A CF , since these matchings are performed when the parse tree for A CF has already been built. Consequently, it is not possible that a subtle vulnerability in this code affects the correctness of the A CF parser.

The last step to achieve unambiguous, decidable recognition of the X.509 certificates is to cope with the ambiguity issues stemming from the primitive encoding forced by DER on constructed OCTET STRINGs in the extnValue field and the BIT STRINGs containing signature and public key material specified by the standard to be primitive. Concerning the issue of the extnValue field, we choose to eliminate the ambiguity relying on the fact that the standards define the contents of the constructed type binding it to a specific OID: this in turn allows to parse unambiguously an ADT instance. Concerning the BIT STRING fields, we eliminate the parsing ambiguity recognizing them as constructed or primitive types according to the individual cryptographic primitive needs as specified in the standards [10,19,26,37,38]. Such an approach will indeed pick only a single way of interpreting the data contained in the field, preventing security critical parsing issues such as the ones in [2].

Once such ambiguities are removed from L CF , the resulting language is indeed regular and will be denoted as L REG from now on, while the resulting restricted X.509 language will be denoted as L X .509 − r 2 = L AId − match ∩ L SI − match ∩ L REG .

Lemma 1.

The proposed parser for L X .509 − r 2 terminates on any input string.

Proof.

Since L X .509 − r 2 = L AId − match ∩ L SI − match ∩ L REG is the result of the intersection of two CS languages and a REG language, the closure property of the CS family w.r.t. set intersection implies that L X .509 − r 2 is CS, a necessary condition for the termination of its parsing process. To prove the termination guarantee of the proposed parser, it is sufficient to observe that all the three parsers, sequentially executed to recognize L X .509 − r 2 , have guarantees on their termination, namely: the parser for L REG is a FSA, while the string matchers act on arbitrarily large, but not infinite strings. □

Lemma 2.

L X .509 − r 2 is non intrinsically ambiguous, and all its sentences can be parsed unambiguously.

Proof.

The only source of ambiguity in the original X.509 specification is the one caused by interpretation issues of the X.509 standard, which are no longer present in L X .509 − r 2 . The first stage of the proposed parser matches the sentences of L X .509 − r 2 = ( ( L REG ∩ L AId − match ) ∩ L SI − match ) by means of a deterministic finite state recognizer, which can only have a single recognition path for each one of them. The CS recognizer stages simply discard some of the strings deemed valid by the first stage, and thus do not introduce ambiguity. □

4.2. Managing grammar representation and recognizer generation

Having dealt with the linguistic issues which could either have prevented the parsing (i.e., the undecidability) or made its result unusable (i.e., the ambiguity), we now tackle the issues concerning the practical construction of a grammar representation for X.509 and the generation of its recognizer. Among the recognizers required to match L X .509 − r 2 = ( ( L REG ∩ L AId − match ) ∩ L SI − match ) the last two suffer from no issues either in their syntax definition or in their recognizer generation.

In order to tackle the issues coming from Statement 2 in Section 3.1, we consider L REG as the intersection of three regular languages: L REG = L REG − 1 ∩ L REG − 2 ∩ L REG − 3 . L REG − 1 is the language of byte strings containing sequences of Extensions instances where each Extension instance appears at most once, L REG − 2 is the language of DER encoded ASN.1 fields with the correct constraints holding between the length octet field and the contents octets field, and L REG − 3 contains all the remaining regular constraints to obtain a sentence in L REG when considered in intersection with L REG − 1 and L REG − 2 .

A straightforward definition of L REG − 1 with a linear grammar requires 17 ! rules, a number which can be reduced to a little over 2 17 through careful rewriting. We deem this number of productions too high to be described and checked reliably, thus we match a relaxation of L REG − 1 at first, which allows the Extensions to be duplicated, and perform a subsequent uniqueness check on the result of the parsing action. This choice allows us to retain the description of the structure of the individual Extension instance as a purely syntactic one, and perform the check for duplicates on their parsed OID fields. We implemented the duplicate checking logic as an handwritten portion of our parser, as we deem the code to be small and simple enough to allow effective code auditing of it.

OPEN IN VIEWER

Fig. 3.

Strawman example of finite state recognizer accepting strings of a characters appended to their lengths expressed as two radix-4 digits. The initial state q 0 accepts the first digit, and memorizes it by means of the finite state memory, and moves to one of the four s i states, which in turn read the second digit of the length and transition to an r i state. r i states are linked by a countdown chain of transitions, where i indicates the amount of remaining a characters to be recognized before acceptance.

Providing an effective and efficient grammar definition for L REG − 2 is similarly difficult. Indeed, it is enough to consider a simpler but structurally equivalent language L REG − 2 − P , where only the length constraints on DER encoded ASN.1 primitive types are enforced, to obtain an unmanageable number of productions. L REG − 2 − P is composed of strings of the form l v , where l is a byte string containing a unique encoding of a length in [ 0 , 2 126 · 8 ] and v a string of arbitrary bytes with the given length. A straightforward grammar representation of it requires at least a grammar production for each valid value of the byte string l. Such a number of productions is well beyond the realm of feasibility in realization. The first observation concerning this issue is that the possible values taken by the l string in practice will be significantly less than the possible ones, as the sensible lengths for an X.509 certificate are typically far smaller than 2 126 · 8 B. As a consequence, we deemed reasonable to restrict the set of valid represented lengths (both in L REG − 2 − P and in L REG − 2 ) to the [ 0 , 2 32 − 1 ] interval, which entails accepting certificates of size up to 4 Gib. Note that, restricting further the encoded length values to a number which allows handwriting of the corresponding linear grammar by a human designer (i.e., in the hundreds range at most) would critically reduce the usefulness of the parser, as it would be able to match only contents smaller than a couple of hundreds bytes. We chose the aforementioned conservative upper bound of 2 32 to prevent a parser that preallocates memory for the contents of the soon-to-be-recognized string in advance from being subject to easy Denial-of-Service attacks, via incorrect certificates overstating their lengths. Since the bound imposed by the practical usefulness on the number of rules of the grammar is still insufficient to allow a manageable specification of the grammar itself, we resort to analyze the parser automaton of L REG − 2 − P to derive a more manageable implementation of the recognizer, and adapt it to recognize the entire L REG − 2 .

To illustrate how the problem of implementing the recognizer for L REG − 2 − P is solved in practice, we first provide a toy example, recognizing the language of words l v , where the content length l is expressed by two, radix-4 digits, and is followed by the content v which is a sequence of a.

The toy Finite State Automaton (FSA) recognizer operating on the set of bytes Σ = { 0 , 1 , 2 , 3 , a } is defined as A REG − 2 − P : ( Σ , Q , δ , q 0 , F ) , where Q is the set of states of the automaton, q 0 the initial state, F the set of final states, and δ : Q × Σ → Q the transition function. A depiction of the recognizer FSA is provided in Fig. 3. The key idea underlying its functioning is count the a symbols in v employing a chain of states, of which only the last is an accepting one (depicted in figure as r i , i ∈ [ 0 , 15 ] ). The recognition of the length l is performed while decoding which one is the state of the chain of counting states where the computation should start recognizing the content v. The decoding is performed memorizing the number read digit by digit, yielding a single sequence of transitions starting from the initial state q 0 and ending in a counting state for each one of the latter.

The main issue in implementing an automaton with the structure depicted in Fig. 3 is to provide a way to implement the transition function δ different from the common tabulated form, which would require an amount of space proportional to the maximum length of the represented string multiplied by the alphabet size; i.e., a number of entries ⩾ 2 32 · 256 for the recognizer of A REG − 2 − P .

Our approach to cope with this issue is to employ a computational form of the said δ, where the states are encoded as unsigned integers, and the destination state can be computed in closed form. In the toy example, the δ function, which has its description as δ ( q 0 , t ) = s t , t ∈ { 0 , 1 , 2 , 3 } ; δ ( s i , t ) = r 4 i + t , t ∈ { 0 , 1 , 2 , 3 } ; δ ( r i , t ) = r i − 1 , t = a , i ∈ [ 1 , 15 ] , can be transformed in computational form encoding the r i states with the unsigned integer i, the s i states with 2 4 + i and q 0 as 2 5 , and computing it as δ ( q 0 , t ) = s t , t ∈ { 0 , 1 , 2 , 3 } → δ ( q , t ) = 2 4 + t , t ∈ { 0 , 1 , 2 , 3 } , q = 2 5 δ ( s i , t ) = r 4 i + t , t ∈ { 0 , 1 , 2 , 3 } → δ ( q , t ) = ( q − 2 4 ) · 4 + t , t ∈ { 0 , 1 , 2 , 3 } , q = 2 4 + i , i ∈ [ 0 , 3 ] δ ( r i , t ) = r i − 1 , i ∈ [ 1 , 15 ] → δ ( i , t ) = i − 1 , t = a , i ∈ [ 1 , 15 ] Following the same line of reasoning employed to obtain a computational form for the delta of our toy example automaton, we devised the representation of the transition function for the recognizer A REG − 2 − P . In particular, such a length function deals with the possible variable length encoding of the length field dedicating a specific transition path for each one of the lengths: δ ( q 0 , a ) = a with 0 ⩽ a ⩽ 127 2 32 with a = 129 2 33 with a = 130 2 34 with a = 131 2 35 with a = 132 This split in the parsing paths also allows us to encode in the state the actual value of the length counter, accumulating it in case it is longer than a single byte. All the length-field recognition paths terminate with a transition moving to the state encoding the corresponding number of content octets field bytes to be read, upon reading the last byte of the length field, following the same structure of the toy automaton. Such a transition function is shown in Fig. 4.

OPEN IN VIEWER

Fig. 4.

Transition function for the recognizer A REG − 2 − P .

Finally, the remaining portion of the transition function takes care of counting the content octet field bytes while they are being read simply decrementing the counter encoded in the state as ∀ a ∈ V t , δ ( q , a ) = q − 1 with 1 ⩽ q ⩽ 2 32 − 1 until the only final state of the automaton F = { 0 } is reached. All the unspecified transitions are leading to the error state. This δ can be conveniently implemented in any programming language employing a single 64-bit unsigned integer as storage for the current state.

In order to recognize L REG − 2 to its full extent, we build on A REG − 2 − P a machine able to take into account ASN.1 constructed types. Since the content of a constructed type is a list of other ASN.1 types, a nesting structure is introduced, requiring different counters depending on the nesting level. In our case, it is still possible to do so, while still employing a FSA recognizer as the maximum nesting depth is bounded, as a consequence of the capping imposed on the value of the outermost counter. As a consequence, we represent the state of A REG − 2 as a vector of states of A REG − 2 − P , employing one for each nesting level, and transitioning from one to the other upon recognizing the beginning and end of a constructed type. It is thus possible to implement the recognizer FSA for L REG taking care of performing the product among the automata A REG − 1 , A REG − 2 and A REG − 3 employing any parser generator allowing to do so (e.g., multistate lexers in GNU Flex [36], or predicate grammars in ANTLR [34]).

Willing to implement the proposed recognizer by means of a tool generating the actual code from a synthetic-grammar based description of the L X .509 − r 2 language, we choose the formalism of predicated grammars [35] which can be employed as an input to the ANTLR parser generator to obtain an efficient LL(*) parser [34]. The LL(*) grammars are obtained as an extension of the common subset of the DCF grammars known as LL(k) grammars, i.e., the ones amenable to recursive descent parsing without backtracking [18]. Our choice of an LL(*) predicated grammar is justified by the expressiveness of the formalism, which allows us to combine the computational implementations of the transition functions of the length validating automaton A REG − 2 , together with the two CS checks and the generated portion of the parser via the so-called semantic predicates. We note that, although it is possible in principle to provide an equally functional description of the X.509 employing a GNU Flex generated parser, exploiting the multi-state lexer functionality to combine the different transition functions and through a generous use of the semantic actions, such a description is less manageable as it would be made of a huge regular expression defining L REG . Moreover, LL(*) parsers inherit from LL(k) ones the capability of providing a useful syntax error reporting, which is useful to diagnose the nature of certificate faults.

Predicated grammars are defined as augmented LL(k) grammars adding to the usual quadruple (see Section 2) G : ( V t , V n , P , S , Π , M ) two sets: the set of side-effect free semantic predicates Π and the set of mutators M. Semantic predicates are employed as a prefix to the right hand side of a production, so that such a production is considered during the corresponding recursive descent parsing action only if the predicate evaluated on the state of the parser is true [35]. Mutators are the way predicated grammars formalize semantic actions acting on the state of the parser, which should be taken upon the matching of the right hand side of a rule [34].

The parser generation process of ANTLR produces a modified recursive descent LL parser, where in lieu of the common sets of finite-length lookaheads the decisions are taken relying on the fact that the language of the lookaheads (i.e., the language of the suffixes of the strings) can be split into a regular partition. This allows the LL(*) parser to employ FSAs to recognize the lookaheads, effectively enhancing its recognition power, although at the possible cost of examining the entire remainder of the string for each decision to be taken. Therefore, while this tends to happen quite rarely, it is possible for a parsing action to take quadratic time in the size of the input.

In case the input grammar cannot be recognized by an LL(*) automaton, ANTLR falls back to an exponential-time backtracking based strategy for the generated parser. ANTLR also supports semantic actions to manipulate the results of the parsing action, after a successful match of the right hand side of a rule, in the same style as GNU Flex [36] and GNU Bison [9].

We implemented the recognizer described in the previous sections, employing semantic actions to perform the CS checks required to match L AId − match and L SI − match , introducing them as soon as the portion of the parse tree matched contains the required information. The implementation of the δ functions for A REG − 1 and A REG − 2 FSAs and the product with the FSA portion of the LL(*) matcher automaton are performed exploiting semantic predicates for A REG − 2 with the aim to forbid the transitions which would be valid according to the matcher of A REG − 3 alone, and performing the uniqueness check for A REG − 1 in an appropriate semantic action, upon matching of the required portion of the parse tree. We also employed semantic predicates to obtain an efficient implementation of the choice of the recognition path in the parser, when we employed the value of the OIDs to disambiguate whether an OCTET STRING contained in an extnValue field of the Extension ADT is indeed composite or primitive as described in Section 4.1. The same strategy was applied to disambiguate BIT STRING encodings in the subjectPublicKeyInfo and the signatureValue fields. The recognizer generation process of ANTLR confirmed that no fallback to the backtracking based parser is made, and that the grammar is indeed LL(*) and non ambiguous.

LL(*) parsers achieve the same correctness guarantees of classical automatically generated parsers: that is, they recognize the same language generated by the grammar from which the parser was derived. Therefore, we can claim our parser is correct, given that the grammar specification is compliant with the standard [8]. We validate the compliance of the grammar we propose by classifying the parsing errors reported, exploiting the accurate information obtainable with an LL(*) recognizer, and by manually inspecting the certificate to verify that there is actually the syntactic failure reported by our parser. We are able to report that no syntactic error detected by other libraries escapes our parser on the entire corpus of 11M X.509 certificates in use for TLS.

Table 4 reports the outcomes of our certificate parsing on the set of 468 , 052 certificates deemed differentially valid by all the other libraries (i.e., on the intersection of the sets accounted for in the 4th column of Table 3(b)), while the data reported in Table 5 represents the outcomes of our parser on certificates belonging to a chain correctly validated by all the considered libraries (i.e., on the set of 468 , 052 certificates obtained as the intersection of the sets accounted for in the penultimate column of Table 3(b)). Despite the reduction in the variety of errors reported in Table 5 with respect to the ones in Table 4, we note that the errors appearing only in the latter are not guaranteed to be safely detected by each library, while the ones in Table 5 are definitely evading detection by all the libraries.1

Both tables categorize the errors splitting them in two groups, according to their potential in generating exploitable security vulnerabilities. In the following, we detail the possible exploitation for each of these security critical errors, providing a rationale for such criticality. While we deem relevant the detailed investigation of each of the highlighted security flaws reported in this work, we provide only a single full proof-of-concept attack employing them, as the purpose of this work is to build a sound X.509 parser preventing all of them altogether. We spur the detailed investigation in this regard by releasing publicly our parser implementation [28].

The first and the third entry in Table 4 report issues on the keyCertSign bit being improperly set (either missing the BasicConstraints extension or in a leaf certificate) potentially allowing their subject to act as a malicious CA. We actually prove this attack to be feasible against some of the libraries, using certificates which exhibit this kind of error. Character set violations and strings of a type different from the one expected may be exploited in the same way as the already discussed issues about malformed DNS, URI or email addresses. Therefore, these issues may lead to impersonation attacks arising from different interpretations of the same string. Lexing errors state an incorrect DER encoding of ASN.1 structures which may lead to a variety of attacks depending on the type of violation. Examining more closely some of them, we discovered that some implementations consider null character of byte strings such as 0x02 0x01 0x03 0x00 in the DER encoded INTEGER (tag 2) of length 1 with value 3, which suggest such implementations may be employing the outcome of the strlen C library function with the value of the actual length field, or employ C string based input reading functions. More serious issues such as flawed checks on length fields may lead to forgery attacks such as the ones reported in [11,12]. The errors named as “Empty Issuer Distinguished Names” imply that a certificate has no issuer, which may lead to security issues depending on how the issuer of the certificate is retrieved by the implementation. The presence of duplicated extensions may lead to impersonation attacks, since implementations may employ the contents of one duplicate only, at their choice. A critical case for such a behavior is the one of a certificate with two basicConstraints extensions, one with ca flag set to true while the other one set to false. In such a case, it depends on an implementation-related choice whether to take into account only at the first extension, and thus considering the subject a CA, or to consider the second extension valid deeming the subject an end entity. Such misinterpretation may lead to powerful attacks where an end entity acts as a malicious CA. Unexpected NULL in algorithmP means that there are no parameters for an algorithm, even if they are expected, potentially weakening the security guarantees provided by some primitives (e.g., missing elliptic curve parameters in ECDSA may lead to backtracking to an unsafe default choice). OID arc overflows indicate that a single arc of an OID is indeed exceeding the maximum value set by the standard. While mishandling of arcs due to a short integer representation is known to lead to practical attacks in flawed implementations [23], we note that an OID arc overflow is potentially more dangerous as even standard abiding libraries will likely fail to manage it. Wrong Algorithm errors imply that a non standard algorithm is employed. While the employed algorithm may still be a sound one, we note that standardized algorithms have usually undergone a higher level of scrutiny. Finally, we report that some certificates have non-existing dates in their validity specification (e.g., the 29th of February 2022), potentially resulting in an erroneous expiration check.

Following the quantitative analysis on how many certificates are affected by potentially security threatening syntactic flaws, we want to analyze the effect of such flaws when reflected onto the hosts which are employing the said certificates. To this end, we analyzed the contents of the Name ADT of the certificates, which usually contains one or more URLs of the host for which the certificate is valid. Moreover, there is a particular standardized extension, called Subject Alternative Names, which contains a further set of names (we consider DNS, email addresses, URLs and URIs) which are related to the subject of the certificate. To evaluate the practical impact of flaws in the certificates we report in Table 6 the amount of names bound to flawed certificates which are not detected by a library, but deemed incorrect by our parser. As emerging from the data, on average 4 common names are present for each certificate, thus pointing to a four-fold increase in the number of impacted hosts with respect to the rough estimate which can be provided considering each certificate as used by a single host.

5.3. Parsing vulnerability exploitation

Willing to validate the practical exploitation of the security issues emerged we focus on the syntactic problem of a certificate having no BasicConstraints (BC) extension, while having the keyCertSign bit set in the KeyUsage extension (there are 1,369 such instances as reported in Table 4). Refer to Fig. 7 for the detailed structure of these two extensions.

The BasicConstraints extension contains a single boolean field indicating whether the subject is a CA or not, and an optional constraint on the maximum length of the certification path. The KeyUsage extension is a 9-bit string storing flags which indicate the legitimate uses for the public key of the certificate subject. Semantically, the keyCertSign flag allows the public key of the subject to be used to verify certificate signatures, but the subject must be a CA. Such an information is contained in the cA Boolean field of BC, which has a default value of FALSE. Thus, if the BasicConstraints extension is missing, the subject cannot be considered a CA and the certificate must be rejected. An incorrect behavior in this case was reported in the recent OpenSSL bug report [24]: indeed OpenSSL versions prior to 1.0.2d allowed such syntactically flawed certificates to be used as CA ones. The version of OpenSSL employed in our experimental evaluation includes the bug fix reported in [24]; however such a fix is not addressing the root cause of the issue, as a number of syntactically incorrect certificates are still deemed valid.

OPEN IN VIEWER

Fig. 7.

KeyUsage and BasicConstraints extensions.

We reproduced the issue at hand in a certificate of which we own the private key, employing the said private key to sign a forged certificate for paypal.com. The syntactically flawed certificate was signed with a private key of a CA bootstrapped by us, and such a CA certificate was added to the trusted storages of the verifying clients, completing the reproduction of the situation found in the wild. We provided the certification path of our forged paypal.com certificate to OpenSSL both via the programming API, and via command-line client, employing the default validation options, succeeding in getting it accepted as a valid certificate. An interesting remark is that such attack is mitigated if the x509_strict option is set for the validation algorithm. However, this option is not enabled by default, leaving all entities relying on default OpenSSL settings (which are expected to be the majority) vulnerable to this serious attack. We were able to get the said certification path to be accepted also by BoringSSL, while the other libraries discard the chain. Nonetheless, we confirm the practicality of the threat on two of the most used TLS implementations, including the one employed by Chrome/Chromium. As a consequence, any one among the owners of the 1,369 certificates of the dataset with the said vulnerability are likely to be able to exploit it successfully, as such items have been signed by a legitimate and trusted root CA. An inspection of such certificates show that some have been generated with the Open Directory framework of Mac OS X Server, providing a pointer to a real world TLS implementation generating syntactically incorrect certificates. Such implementations can be exploited by an attacker who could require a legitimate certificate for its own identity to such implementations. The generated certificate would suffer the mentioned syntactic issue and could be used as an intermediate certificate to sign other certificates for an arbitrary identity. Such certificates can later be used to perform a Man-In-The-Middle attack where a TLS session is successfully established impersonating the subject of the fake certificate.

OPEN IN VIEWER

Fig. 8.

Parsing times for all the certificates present in our dataset. The blue line marks the linear interpolation of the obtained data points, with equation Time = 0.778 · Size − 0.203 .

Finally, although our work focuses on the effectiveness of the proposed certificate parsing approach, we provide some efficiency results of our parser. Figure 8 reports the performance results obtained running our parser on a 3.2 GHz Intel i5-6500 based desktop, running Gentoo Linux 13.0 (x86_64). The reported timings show how the vast majority of the certificates can be recognized in less than 10 ms, a timing we deem acceptable for practical use, and which may be improved employing a more efficient C code generation backend with respect to the current one in ANTLR3. In addition to the absolute timings reported, it is interesting to note that the practical parsing complexity appears to be substantially linear, despite the theoretical quadratic worst case of LL(*) parsers. Such a fact represents a validation of our claim that the lookahead which the LL(*) parser requires is indeed far shorter than the entirety of the remaining input, indeed resulting in near optimal (i.e., linear) parsing complexity.