Jump to the The Short Version below.

When working to implement the REST API of an existing service, there are many avenues to understanding what is going on. Ideally, documentation would be complete enough to implement to the APIs. If the documentation is not present or is incomplete, examining an existing SDK (possibly in a different language) is an option. In reality, things are not always that simple and other sources are needed.

On a recent project that required implementing the Azure KeyVault REST APIs for the Apache JClouds Project ambiguities were encountered. To resolve the issues encountered, a method was needed to get more information. SDKs do exist in a number of languages, yet that requires writing code to use the SDKs.

For Azure, another option exists, the Azure CLI. The Azure CLI is a tool for interacting with most of the Azure services. It is written in Python and leverages modules for each of the various components within the Azure Cloud. Through various commands, one can trigger the desired REST APIs to complete the desired task. The code driving the REST APIs is already written.

There is one hitch, however, the Azure CLI interacts with Azure over a secure connection, HTTPS. How does one examine the traffic?

Given communication between the computer where the Azure CLI commands are issued and the Azure is encrypted, a mechanism is necessary to spy on the traffic. A class of tools exists that can help listen in on this traffic, they are referred to as “Man in the Middle” (MitM) proxies.

What is a MitM Proxy?

By it’s name, a MitM proxy is a software agent that sits in between two pieces of software attempting to communicate with one another. If successfully setup, the proxy has access to the information being transmitted.

How does one use the MitM Proxy?

First, one must obtain, install, and run the proxy. There are many out there depending on the platform. Some choices include:

Once installed and running, proxies listen on the local system on a specific port. For example, 8080 and 8888 are two common ports.

Second, one must configure the application or the environment it is running in (will depend upon the application) to use the proxy. As mentioned previously, the Azure CLI is written in the Python language. Most of it’s modules leverage either the requests module or a lower level Python HTTP library urllib3.

In this case, Python allows one to configure proxy settings using simple environment variables. Assume the proxy is running on localhost and port 8888. On Linux, setting those environment variables would be as follows:

$ export https_proxy=https://localhost:8888

Once the proxy environment variable is set, but the proxy is not running, one can verify that the proxy setting is respected by attempting an operation that will make a network request. For instance, attempt to login:

$ az login
Please ensure you have network connection. Error detail: HTTPSConnectionPool(host='login.microsoftonline.com',
port=443): Max retries exceeded with url: /common/oauth2/devicecode?api-version=1.0 (Caused by
a ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection
object at 0x7f405b6eea58>: Failed to establish a new connection: [Errno 111] Connection refused',)))

If one examines the error above, the proxy can not be connected to. As it is not running, this is as expected.

Next, enable the proxy (the specific approach depends upon the proxy you have chosen). It should be noted that, it may be necessary to configure which domains the proxy should pay attention to. As that depends on the proxy, it is outside the scope of this tutorial. With the proxy enabled, attempt to login again:

$ az login

Please ensure you have network connection. Error detail: HTTPSConnectionPool(host='login.microsoftonline.com',
port=443): Max retries exceeded with url: /common/oauth2/devicecode?api-version=1.0 (Caused by
SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate',
'certificate verify failed')],)",),))

The connection is being proxied, but the Azure CLI (specifically the piece that interacts with Azure Active Directory) does not like the certificate presented. This is because the Azure CLI is first connecting to the proxy which will then connect to the desired site. In this case the site is login.microsoftonline.com. The error is saying that the certificate the proxy is presenting is not trusted. There are typically two approaches to handling this, assuming one can not modify the source code for the application:

  • An environment variable which disables certificate verification
  • Installing the proxy certificate into the trusted store used by the program (this will vary both on the application as well as the system being used)

It should be noted installing a proxy in between applications should not be a normal practice or adding self signed or other locally generated certificates to a trust store as both break the intended security of the handshake between the local machine and the remote host/website/etc.

Getting the Proxy to Work With the Azure CLI

For the Azure Active Directory, it is possible to set an environment variable to disable the certificate checking used by the Python Active Directory Authentication Library (ADAL). The environment variable is ADAL_PYTHON_SSL_NO_VERIFY. By setting the variable and attempting to run az login again we see the following:

$ export ADAL_PYTHON_SSL_NO_VERIFY=1
$ az login
/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning:
Unverified HTTPS request is being made. Adding certificate verification is strongly advised.
See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code
B62UZNSR9 to authenticate.
/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning:
Unverified HTTPS request is being made. Adding certificate verification is strongly advised.
See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)

In this case, az login is ignoring the proxy certificate and polling awaiting the a successful login to the devicelogin site. Note within the warnings, the lack of certificate verification is explicitly called out.

Continue with the device login and very likely you will see something like:

$ az login
/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning:
Unverified HTTPS request is being made. Adding certificate verification is strongly advised.
See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code
BY9KKGMJR to authenticate.
/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning:
Unverified HTTPS request is being made. Adding certificate verification is strongly advised.
See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
...
Error occurred in request., SSLError: HTTPSConnectionPool(host='management.azure.com', port=443):
Max retries exceeded with url: /tenants?api-version=2016-06-01 (Caused by
SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate',
'certificate verify failed')],)",),))
Traceback (most recent call last):
File "/opt/az/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 441, in wrap_socket
  cnx.do_handshake()
File "/opt/az/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1806, in do_handshake
  self._raise_ssl_error(self._ssl, result)
File "/opt/az/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1546, in _raise_ssl_error
  _raise_current_error()
File "/opt/az/lib/python3.6/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
  raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
During handling of the above exception, another exception occurred:

The TLS handshake is failing again, previously we handled things for login.microsoftonline.com which the Python ADAL uses. However, the Azure CLI typically uses a multistage handshake (assuming you have already logged in):

  1. Authenticate to login.microsoftonline.com and request a token.
  2. Use that token to perform an operation on one of the Azure services. In this case the service being accessed is management.azure.com.

The Azure CLI is built upon various Azure Python SDKs. Active Directory Authentication Libraries (ADAL) is one of those libraries. As we saw previously, there is a specific flag that needs to be set for that library in order to ignore certificate authentication. Many of the other Azure Python SDKs are built upon a set of common Microsoft libraries, thus it should be able to disable the verification of certificates when accessing management.azure.com.

The environment variable to disable the certificate check is AZURE_CLI_DISABLE_CONNECTION_VERIFICATION. Like ADAL_PYTHON_SSL_NO_VERIFY it is set as an environment variable. Setting the variable and attempting the az login shuffle again results in:

$ export AZURE_CLI_DISABLE_CONNECTION_VERIFICATION=1
$ az login
/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning:
Unverified HTTPS request is being made. Adding certificate verification is strongly advised.
See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code
BZ8VJNH9G to authenticate.
/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning:
Unverified HTTPS request is being made. Adding certificate verification is strongly advised.
See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
Connection verification disabled by environment variable AZURE_CLI_DISABLE_CONNECTION_VERIFICATION
[
  {
    "cloudName": "AzureCloud",
    "id": "68ff811e-11aa-4b5c-9c62-a2f28b517123",
    "isDefault": true,
    "name": "Jims-Azure-Playground",
    "state": "Enabled",
    "tenantId": "72f5434a-86f1-52b1-91ab-2d7cd0118fc3",
    "user": {
      "name": "noemail@foobar.com",
      "type": "user"
    }
  }
]

If one takes a look at what is happening on the proxy, you would see the following calls:

  1. POST https://login.microsoftonline.com/common/oauth2/devicelogin?api-version=1.0
  2. POST https://login.microsoftonline.com/common/oauth2
  3. POST https://login.microsoftonline.com/72f5434a-86f1-52b1-91ab-2d7cd0118fc3/oauth2/token
  4. GET https://management.azure.com/subscriptions?api-version=2016-06-01

What these calls do are:

  1. Initiate the devicelogin process
  2. Poll to see if the devicelogin process has completed
  3. Once logged in, request a token to interact with management.azure.com
  4. Request account information for the logged in user from management.azure.com

What Does the Traffic Look Like?

What do the contents of each look like? Since we are using a proxy, we have access to the details of each request.

POST https://login.microsoftonline.com/common/oauth2/devicelogin?api-version=1.0“

Request:
  Headers:
    host:                      login.microsoftonline.com
    User-Agent:                python-requests/2.18.4
    Accept-Encoding:           gzip, deflate
    Accept:                    */*
    Connection:                keep-alive
    content-type:              application/x-www-form-urlencoded
    Accept-Charset:            utf-8
    client-request-id:         2ad6b57d-7f3f-4004-af7a-be523b98d91d
    return-client-request-id:  true
    x-client-SKU:              Python
    x-client-Ver:              0.4.7
    x-client-OS:               linux
    x-client-CPU:              x64
    Content-Length:            100
  Body:
    client_id=04b07795-8ddb-461a-bbee-02f9e1bf7b46&resource=https%3A%2F%2Fmanagement.core.windows.net%2F

Response:
  Headers:
    Cache-Control:              no-cache, no-store
    Pragma:                     no-cache
    Content-Type:               application/json; charset=utf-8
    Expires:                    -1
    Server:                     Microsoft-IIS/8.5
    Strict-Transport-Security:  max-age=31536000; includeSubDomains
    X-Content-Type-Options:     nosniff
    client-request-id:          2ad6b57d-7f3f-4004-af7a-be523b98d91d
    x-ms-request-id:            90ce3067-0039-4fd8-8bd6-8af50cbf2b00
    x-ms-clitelem:              1,0,0,,
    P3P:                        CP="DSP CUR OTPi IND OTRi ONL FIN"
    Set-Cookie:                 esctx=...; domain=.login.microsoftonline.com; path=/; secure; HttpOnly
    Set-Cookie:                 x-ms-gateway-slice=008; path=/; secure; HttpOnly
    Set-Cookie:                 stsservicecookie=ests; path=/; secure; HttpOnly
    X-Powered-By:               ASP.NET
    Date:                       Wed, 27 Dec 2017 05:05:33 GMT
    content-length:             441
  Body:
    {
      "user_code": "BZ8VJNH9G",
      "device_code": "...",
      "verification_url": "https://aka.ms/devicelogin",
      "expires_in": "900",
      "interval": "5",
      "message": "To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code BZ8VJNH9G to authenticate."
    }

POST https://login.microsoftonline.com/common/oauth2“

Request:
  Headers:
    host:                      login.microsoftonline.com
    User-Agent:                python-requests/2.18.4
    Accept-Encoding:           gzip, deflate
    Accept:                    */*
    Connection:                keep-alive
    content-type:              application/x-www-form-urlencoded
    Accept-Charset:            utf-8
    client-request-id:         7c369b38-ae75-4cf6-99e3-f0c18b0c13c2
    return-client-request-id:  true
    x-client-SKU:              Python
    x-client-Ver:              0.4.7
    x-client-OS:               linux
    x-client-CPU:              x64
    Content-Length:            314
  Body:
    grant_type=device_code&client_id=04b07795-8ddb-461a-bbee-02f9e1bf7b46&resource=https%3A%2F%2Fmanagement.core.windows.net%2F&code=...

Response:
  Headers:
    Cache-Control:              no-cache, no-store
    Pragma:                     no-cache
    Content-Type:               application/json; charset=utf-8
    Expires:                    -1
    Server:                     Microsoft-IIS/8.5
    Strict-Transport-Security:  max-age=31536000; includeSubDomains
    X-Content-Type-Options:     nosniff
    client-request-id:          2ad6b57d-7f3f-4004-af7a-be523b98d91d
    x-ms-request-id:            68bc6671-9a74-4d88-8bd1-8f704d112b00
    x-ms-clitelem:              1,0,0,,
    P3P:                        CP="DSP CUR OTPi IND OTRi ONL FIN"
    Set-Cookie:                 esctx=...; domain=.login.microsoftonline.com; path=/; secure; HttpOnly
    Set-Cookie:                 x-ms-gateway-slice=003; path=/; secure; HttpOnly
    Set-Cookie:                 stsservicecookie=ests; path=/; secure; HttpOnly
    X-Powered-By:               ASP.NET
    Date:                       Wed, 27 Dec 2017 05:05:51 GMT
    content-length:             3798
  Body:
  {
    "access_token": "...",
    "expires_in": "3599",
    "expires_on": "1514354752",
    "ext_expires_in": "262800",
    "id_token": "...",
    "not_before": "1514350852",
    "refresh_token": "...",
    "resource": "https://management.core.windows.net/",
    "scope": "user_impersonation",
    "token_type": "Bearer"
  }

POST https://login.microsoftonline.com/72f5434a-86f1-52b1-91ab-2d7cd0118fc3/oauth2/token“

Request:
  Headers:
    host:                      login.microsoftonline.com
    User-Agent:                python-requests/2.18.4
    Accept-Encoding:           gzip, deflate
    Accept:                    */*
    Connection:                keep-alive
    content-type:              application/x-www-form-urlencoded
    Accept-Charset:            utf-8
    client-request-id:         2942f1bf-b96e-45f6-ad63-0737f8cc4662
    return-client-request-id:  true
    x-client-SKU:              Python
    x-client-Ver:              0.4.7
    x-client-OS:               linux
    x-client-CPU:              x64
    Content-Length:            1050
  Body:
    grant_type=refresh_token&client_id=04b07795-8ddb-461a-bbee-02f9e1bf7b46&resource=https%3A%2F%2Fmanagement.core.windows.net%2F&refresh_token=...

Response:
  Headers:
    Cache-Control:              no-cache, no-store
    Pragma:                     no-cache
    Content-Type:               application/json; charset=utf-8
    Expires:                    -1
    Server:                     Microsoft-IIS/8.5
    Strict-Transport-Security:  max-age=31536000; includeSubDomains
    X-Content-Type-Options:     nosniff
    client-request-id:          2942f1bf-b96e-45f6-ad63-0737f8cc4662
    x-ms-request-id:            51557a24-b5fe-4bbf-9ffe-83a336662e00
    x-ms-clitelem:              1,0,0,3782.6494,
    P3P:                        CP="DSP CUR OTPi IND OTRi ONL FIN"
    Set-Cookie:                 esctx=...; domain=.login.microsoftonline.com; path=/; secure; HttpOnly
    Set-Cookie:                 x-ms-gateway-slice=005; path=/; secure; HttpOnly
    Set-Cookie:                 stsservicecookie=ests; path=/; secure; HttpOnly
    X-Powered-By:               ASP.NET
    Date:                       Wed, 27 Dec 2017 05:05:57 GMT
    content-length:             3093
  Body:
    {
      "access_token": "...",
      "expires_in": "3599",
      "expires_on": "1514354757",
      "ext_expires_in": "262800",
      "not_before": "1514350857",
      "refresh_token": "...",
      "resource": "https://management.core.windows.net/",
      "scope": "user_impersonation",
      "token_type": "Bearer"
    }

GET https://management.azure.com/subscriptions?api-version=2016-06-01“

Request:
  Headers:
    host:                   management.azure.com
    User-Agent:             python/3.6.1 (Linux-4.4.0-104-generic-x86_64-with-debian-stretch-sid) requests/2.18.4
                            msrest/0.4.21 msrest_azure/0.4.19 subscriptionclient/1.2.1 Azure-SDK-For-Python
    Accept-Encoding:        gzip, deflate
    Accept:                 application/json
    Connection:             keep-alive
    Authorization:          Bearer ...
    Content-Type:            application/json; charset=utf-8
    x-ms-client-request-id:  9ddb919e-eac3-11e7-8a2b-001c424e0939
    accept-language:         en-US

Response:
  Headers:
    Cache-Control:                  no-cache
    Pragma:                         no-cache
    Transfer-Encoding:              chunked
    Content-Type:                   application/json; charset=utf-8
    Content-Encoding:               gzip
    Expires:                        -1
    Vary:                           Accept-Encoding
    x-ms-ratelimit-remaining-tenan  14999
    t-reads:
    x-ms-request-id:                d8a2d282-86bc-45a2-b31e-62ec1b4c1edb
    x-ms-correlation-request-id:    d8a2d282-86bc-45a2-b31e-62ec1b4c1edb
    x-ms-routing-request-id:        EASTUS:20171227T050556Z:d8a2d282-86bc-45a2-b31e-62ec1b4c1edb
    Strict-Transport-Security:      max-age=31536000; includeSubDomains
    Date:                           Wed, 27 Dec 2017 05:05:55 GMT
    content-length:                 493
  Body:
    {
      [
        {
          "authorizationSource": "RoleBased",
          "displayName": "Jims-Azure-Playground",
          "id": "/subscriptions/68ff811e-11aa-4b5c-9c62-a2f28b517123",
          "state": "Enabled",
          "subscriptionId": "68ff811e-11aa-4b5c-9c62-a2f28b517123",
          "subscriptionPolicies": {
              "locationPlacementId": "Internal_2014-09-01",
              "quotaId": "Internal_2014-09-01",
              "spendingLimit": "Off"
          }
        }
      ]
    }

What Happens is Proxy Environment Variables Don’t Work?

As mentioned previously, the Azure CLI is built upon multiple Azure Python SDKs. ADAL required it’s own environment variable separate from the CLI itself in order for the encrypted connections to ignore the certificate verification. There are other APIs called by the Azure SDK where these environment variables don’t work. Further, hunting down these environment variables is non-trivial even with decent searching.

In this case, what does one do? Recall that a second method for listening in on the traffic is to somehow add the proxy certificate to the trust store of the application, in this case the Azure CLI.

For a Python application, one can try and figure out all the possible places that the application may use or store a certificate. A number of systems have a default trust store that may enable adding certificates. Doing so is outside the scope of this post. In the case of figuring out how to handle cases that don’t respect the previously mentioned two environment variables, a number of approaches were tried (and failed).

It turns out, the Azure CLI uses a Python library certifi. certifi is a library that contains a number of trusted root certificates and plays nicely with the requests library.

In order to add the proxy certificate such that certify will support it, one needs to find where the library is installed. On Ubuntu Linux, using the Microsoft package repository, the Azure CLI installs itself into /opt/az. Within this directory, the certifi library can be found in /opt/az/lib/python3.6/site-packages/certifi. Within this directory is a file cacert.pem.

To enable the Azure CLI to support the proxy certificate, it need simply be appended to the cacert.pem file. As follows:

$ cd /opt/az/lib/python3.6/site-packages/certifi
$ cat ~/.mitmproxy/mitmproxy-ca-cert.pem >> cacert.pem

Attempt to create an Azure Key Vault. Doing so makes a request to the Azure Active Directory Graph API. These calls fail if relying on the environment variables. So, with cacert.pem updated per above, if we attempt to create the Key Vault:

$ az keyvault create --name jmskvtest1 -g jmskvtest1
Connection verification disabled by environment variable AZURE_CLI_DISABLE_CONNECTION_VERIFICATION
/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning:
Unverified HTTPS request is being made. Adding certificate verification is strongly advised.
See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
{
  "id": "/subscriptions/3fee811e-11bf-4b5c-9c62-a2f28b517724/resourceGroups/jmskvtest1/providers/Microsoft.KeyVault/vaults/jmskvtest1",
  "location": "westus2",
  "name": "jmskvtest1",
  "properties": {
    "accessPolicies": [
      {
        "applicationId": null,
        "objectId": "86a607ff-039e-497e-bab1-92247bc5ed02",
        "permissions": {
          "certificates": [
            "get",
            "list",
            "delete",
            "create",
            "import",
            "update",
            "managecontacts",
            "getissuers",
            "listissuers",
            "setissuers",
            "deleteissuers",
            "manageissuers",
            "recover"
          ],
          "keys": [
            "get",
            "create",
            "delete",
            "list",
            "update",
            "import",
            "backup",
            "restore",
            "recover"
          ],
          "secrets": [
            "get",
            "list",
            "set",
            "delete",
            "backup",
            "restore",
            "recover"
          ],
          "storage": [
            "get",
            "list",
            "delete",
            "set",
            "update",
            "regeneratekey",
            "setsas",
            "listsas",
            "getsas",
            "deletesas"
          ]
        },
        "tenantId": "72f988bf-86f1-41af-91ab-2d7cd011db47"
      }
    ],
    "createMode": null,
    "enableSoftDelete": null,
    "enabledForDeployment": false,
    "enabledForDiskEncryption": null,
    "enabledForTemplateDeployment": null,
    "sku": {
      "name": "standard"
    },
    "tenantId": "72f988bf-86f1-41af-91ab-2d7cd011db47",
    "vaultUri": "https://jmskvtest1.vault.azure.net"
  },
  "resourceGroup": "jmskvtest1",
  "tags": {},
  "type": "Microsoft.KeyVault/vaults"
}

If one were to unset the environment variables ADAL_PYTHON_SSL_NO_VERIFY and AZURE_CLI_DISABLE_CONNECTION_VERIFICATION and rerun the commands, you would notice the commands would work through the proxy without any warnings. However, as was mentioned, adding an untrusted certificate to the trust store is not a solution for anything other than when debugging is needed or in a developer setup.

The Short Version

In order to use a proxy with the Azure CLI, you need to:

  1. Install and configure your proxy
  2. Tell Python / Azure CLI to use the proxy. On Linux this is something like export https_proxy=localhost:8888
  3. Disable certificate verification (one of):
    • Set ADAL_PYTHON_SSL_NO_VERIFY and AZURE_CLI_DISABLE_CONNECTION_VERIFICATION
    • Add proxy certificate to certifi trust store
  4. Run Azure CLI commands and see transactions in the proxy