oss-security - CVE-2016-9015: Python urllib3 1.17 and 1.18 certificate verification failure

Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [day] [month] [year] [list]
Message-Id: <7FA392D7-7065-4BE4-80BD-E3118A003F79@lukasa.co.uk>
Date: Thu, 27 Oct 2016 12:26:20 +0100
From: Cory Benfield <cory@...asa.co.uk>
To: oss-security@...ts.openwall.com
Subject: CVE-2016-9015: Python urllib3 1.17 and 1.18 certificate verification
 failure

Versions 1.17 and 1.18 of the Python urllib3 library suffer from a vulnerability that can cause them, in certain configurations, to not correctly validate TLS certificates. This places users of the library with those configurations at risk of man-in-the-middle and information leakage attacks. This vulnerability affects users using versions 1.17 and 1.18 of the urllib3 library, who are using the optional PyOpenSSL support for TLS instead of the regular standard library TLS backend, and who are using OpenSSL 1.1.0 via PyOpenSSL. This is an extremely uncommon configuration, so the security impact of this vulnerability is low.

Affected users should upgrade to urllib3 1.18.1, which has been published today and contains only the mitigation for this vulnerability on top of the changes in 1.18. If unable to upgrade, users should downgrade their OpenSSL version or temporarily stop injecting PyOpenSSL into urllib3 until they are able to upgrade. A more lengthy description of the vulnerability follows.

—

This vulnerability was introduced in a substantial refactor of the PyOpenSSL contrib module. During this refactor, a branch of code that mapped the Python standard library certificate verification constants (ssl.CERT_NONE, ssl.CERT_OPTIONAL, ssl.CERT_REQUIRED) to OpenSSL verification mode flags (SSL_VERIFY_NONE, SSL_VERIFY_PEER, etc.) was accidentally lost. This meant that Python standard library constants would be passed directly to OpenSSL via the SSL_CTX_set_verify function’s mode argument.

Unfortunately, these Python standard library constants are Python wrappers around the values of a C enumerated type, declared like this:

    enum py_ssl_cert_requirements {
        PY_SSL_CERT_NONE,
        PY_SSL_CERT_OPTIONAL,
        PY_SSL_CERT_REQUIRED
    };

Per the standard C enumerated type rules, these constants have the values 0, 1, and 2 respectively. These integers do not all map to their OpenSSL verification mode flag equivalents. While PY_SSL_CERT_NONE and SSL_VERIFY_NONE have the same value (0) and PY_SSL_CERT_OPTIONAL and SSL_VERIFY_PEER have the same value (1), PY_SSL_CERT_REQUIRED has the value 2, which maps to SSL_VERIFY_FAIL_IF_NO_PEER_CERT. This flag is defined by the OpenSSL manual page as being meaningless on its own, requiring SSL_VERIFY_PEER to also be set in order to have any effect. Additionally, the manual page declares that SSL_VERIFY_FAIL_IF_NO_PEER_CERT has no effect in client mode.

In OpenSSL versions prior to 1.1.0, an implementation detail in the OpenSSL codebase would mean that if any nonzero value was passed to the mode argument of SSL_CTX_set_verify this would implicitly have the same effect as setting SSL_VERIFY_PEER. Essentially, the only value of mode in OpenSSL versions prior to 1.1.0 that would cause certificate validation to be disabled was 0 (SSL_VERIFY_NONE).

In the work done for OpenSSL 1.1.0, this implementation detail was changed to check for the SSL_VERIFY_PEER bit directly. As the OpenSSL flags are a bit mask, passing PY_SSL_CERT_REQUIRED (2) would *not* have the SSL_VERIFY_PEER bit (1) set, which means that OpenSSL would act act as though SSL_VERIFY_PEER was not set. Thus, if PY_SSL_CERT_REQUIRED is passed to OpenSSL directly in client mode, OpenSSL 1.0.2 and earlier treat it as equivalent to SSL_VERIFY_PEER, whereas OpenSSL 1.1.0 and later treat it as equivalent to SSL_VERIFY_NONE.

This is unquestionably an application error. The OpenSSL documentation is clear that the SSL_VERIFY_FAIL_IF_NO_PEER_CERT flag is both meaningless by itself and in client mode. However, OpenSSL’s unexpected change in behaviour, combined with the fact that it has no way to report an invalid flag combination to SSL_CTX_set_verify, means that this application error leads to a catastrophic silent security failure when used with OpenSSL 1.1.0.

The fix for urllib3 is to reintroduce a mapping between the Python standard library enumerated type and the OpenSSL flags. Other applications should audit and confirm that they always successfully pass SSL_VERIFY_PEER to SSL_CTX_set_verify. The OpenSSL team have been notified of this behaviour and have concluded that it is not a security vulnerability in OpenSSL. However, they are considering reverting this change regardless, on the principle that it is better to fail closed than to fail open: https://github.com/openssl/openssl/pull/1793

Fortunately, urllib3’s test suite caught this failure, which led to this investigation. Unfortunately, due to the relative scarcity of OpenSSL 1.1.0, two released versions of urllib3 had passed before anyone attempted to run urllib3 with OpenSSL 1.1.0.

The urllib3 team have contacted downstream redistributors for Red Hat and Debian: both distributions are not using versions of urllib3 later than 1.16, and so are unaffected. Additionally, the Python Requests library is using urllib3 version 1.16 and is also not affected.
Please check out the Open Source Software Security Wiki, which is counterpart to this mailing list.
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.