envoy, docker and websockets – debugging and configuration

By Maximilian Batz | 2019-05-18

Websockets are an exciting technology, allowing you to upgrade a HTTP connection to a long-running persistent binary connection, which you can use to send bi-directional messages.

As an aside, the MQTT protocol can be transported using websockets – which is the only (?) way for a JavaScript client delivered by the website, for instance.

In any case, as websockets run on the same ports as the normal HTTP & HTTPS traffic (80 / 443), corporate networks are more likely to let them pass.

Envoy and websockets

Envoy supports websockets. There are some gotchas:

Unable to parse JSON as proto (INVALID_ARGUMENT:(route_config.virtual_hosts[3].routes[0].route) use_websocket: Cannot find field.):

Envoy used to support websockets with an old directive, “use_websocket”. On a current envoy installation (e.g. I currently use envoy 1.10.0-dev for testing purposes) this directive is gone and it has been replaced.

The idea is to have more generic upgrade requests / possibilities.

The correct syntax is now using upgrade_configs. Have a look at my envoy.yaml:

static_resources:
   listeners:
   – address:
       socket_address:
         address: 0.0.0.0
         port_value: 80
     filter_chains:
     – filters:
       – name: envoy.http_connection_manager
         config:
           upgrade_configs:
             – upgrade_type: websocket
           codec_type: auto
           stat_prefix: ingress_http
           use_remote_address: true
           xff_num_trusted_hops: 0
           route_config:
             virtual_hosts:
             – name: debug
               domains: [“debug.local:80”]
               routes:
                 – match: { prefix: “/” }
                   route:
                     cluster: target_dwebsocket
             – name: backend
               domains: [“morpheus.local”]
               routes:
               – match: { prefix: “/” }
                 redirect:
                   path_redirect: “/”
                   https_redirect: true

(snip)

– address:
     socket_address:
       address: 0.0.0.0
       port_value: 443
   filter_chains:
   – tls_context:
       common_tls_context:
         tls_certificates:
         – certificate_chain: { filename: “/etc/envoy/example.crt” }
           private_key: { filename: “/etc/envoy/example.key” }
         alpn_protocols: [ “h2,http/1.1” ]
     filters:
     – name: envoy.http_connection_manager
       config:
         upgrade_configs:
           – upgrade_type: websocket
         stat_prefix: ingress_https
         use_remote_address: true
         xff_num_trusted_hops: 0
         route_config:
           virtual_hosts:
           – name: debug
             domains: [“debug.local:443”]
             routes:
               – match: { prefix: “/” }
                 route:
                   cluster: target_dwebsocket


(snip)

clusters:
– name: target_dwebsocket
   connect_timeout: 0.25s
   type: strict_dns
   lb_policy: round_robin
   hosts:
   – socket_address:
       address: dwse
       port_value: 8080

You can use this upgrade_configs directive in two places, according to the documentation. on http_connection_manager (as in my example above), and on individual routes.

          upgrade_configs:
            – upgrade_type: websocket

Routes not matching

The other important thing to note, if you get “404s” ( error: Unexpected server response: 404 ): the :authority header will be, for some strange reason, set including the port – and will therefore not match if you supply just the domain.

Look at this route:

[“debug.local:80”]

The port is specified after the domain in this instance. In my tests – only if the port was specified, was I able to connect through envoy to the websocket server I have set up. This is a major stumbling block.

Next up, I will discuss, how to debug envoy in cases like this.

Debugging envoy, websockets, Docker:

Making envoy output verbose

Here’s my docker-compose.yaml:

version: ‘3.7’

services:
   envoy:
     build:
       context: ./
       dockerfile: Dockerfile
     container_name: penvoyage-morpheus-envoy
     ports:
       – “80:80”
       – “443:443”
     volumes:
       – type: volume
         source: penvoyage_volume
         target: /etc/envoy
     networks:
       – envoy_net
     #user: “2000:2000”
     user: “root:root”
     restart: unless-stopped
     environment:
       loglevel: debug

volumes:
   penvoyage_volume:
     external:
       name: penvoyage_volume

networks:
   envoy_net:
     external:
       name: my-bridge-network

Note that an environment variable “loglevel: debug” is set.

loglevel can be one of:

  • trace
  • debug
  • info
  • warn

Starting the container will now yield a lot more output:

docker-compose up

Note that we do not deattach from the container, to directly see it’s output. (Once you’re satisfied, you can run it using docker-compose up –d)

Your output should now look something like this:

penvoyage-morpheus-envoy | [2019-05-18 14:20:32.093][1][debug][main] [source/server/server.cc:143] flushing stats

on a connection from a websocket client, you will get the following output:

penvoyage-morpheus-envoy | [2019-05-18 14:21:14.915][23][debug][main] [source/server/connection_handler_impl.cc:257] [C0] new connection
penvoyage-morpheus-envoy | [2019-05-18 14:21:14.916][23][debug][http] [source/common/http/conn_manager_impl.cc:210] [C0] new stream
penvoyage-morpheus-envoy | [2019-05-18 14:21:14.918][23][debug][http] [source/common/http/conn_manager_impl.cc:548] [C0][S4222457489265630919] request headers complete (end_stream=false):
penvoyage-morpheus-envoy | ‘:authority’, ‘debug.local:80’
penvoyage-morpheus-envoy | ‘:path’, ‘/mqtt_drinks_are_free’
penvoyage-morpheus-envoy | ‘:method’, ‘GET’
penvoyage-morpheus-envoy | ‘sec-websocket-version’, ’13’
penvoyage-morpheus-envoy | ‘sec-websocket-key’, ‘HQFiCTWiFMktGDPFXwzrjQ==’
penvoyage-morpheus-envoy | ‘connection’, ‘Upgrade’
penvoyage-morpheus-envoy | ‘upgrade’, ‘websocket’
penvoyage-morpheus-envoy | ‘sec-websocket-extensions’, ‘permessage-deflate; client_max_window_bits’
penvoyage-morpheus-envoy |
penvoyage-morpheus-envoy | [2019-05-18 14:21:14.918][23][debug][router] [source/common/router/router.cc:321] [C0][S4222457489265630919] cluster ‘target_dwebsocket’ match for URL ‘/mqtt_drinks_are_free’
penvoyage-morpheus-envoy | [2019-05-18 14:21:14.918][23][debug][router] [source/common/router/router.cc:379] [C0][S4222457489265630919] router decoding headers:
penvoyage-morpheus-envoy | ‘:authority’, ‘debug.local:80’
penvoyage-morpheus-envoy | ‘:path’, ‘/mqtt_drinks_are_free’
penvoyage-morpheus-envoy | ‘:method’, ‘GET’
penvoyage-morpheus-envoy | ‘:scheme’, ‘http’
penvoyage-morpheus-envoy | ‘sec-websocket-version’, ’13’
penvoyage-morpheus-envoy | ‘sec-websocket-key’, ‘HQFiCTWiFMktGDPFXwzrjQ==’
penvoyage-morpheus-envoy | ‘connection’, ‘Upgrade’
penvoyage-morpheus-envoy | ‘upgrade’, ‘websocket’
penvoyage-morpheus-envoy | ‘sec-websocket-extensions’, ‘permessage-deflate; client_max_window_bits’
penvoyage-morpheus-envoy | ‘content-length’, ‘0’
penvoyage-morpheus-envoy | ‘x-forwarded-for’, ‘192.168.1.2’
penvoyage-morpheus-envoy | ‘x-forwarded-proto’, ‘http’
penvoyage-morpheus-envoy | ‘x-envoy-internal’, ‘true’
penvoyage-morpheus-envoy | ‘x-request-id’, ‘ca8a765c-e549-4c45-988c-58b6c853df7b’
penvoyage-morpheus-envoy | ‘x-envoy-expected-rq-timeout-ms’, ‘15000’
penvoyage-morpheus-envoy |
penvoyage-morpheus-envoy | [2019-05-18 14:21:14.918][23][debug][pool] [source/common/http/http1/conn_pool.cc:82] creating a new connection
penvoyage-morpheus-envoy | [2019-05-18 14:21:14.919][23][debug][client] [source/common/http/codec_client.cc:26] [C1] connecting
penvoyage-morpheus-envoy | [2019-05-18 14:21:14.919][23][debug][connection] [source/common/network/connection_impl.cc:638] [C1] connecting to 172.18.0.8:8080
penvoyage-morpheus-envoy | [2019-05-18 14:21:14.919][23][debug][connection] [source/common/network/connection_impl.cc:647] [C1] connection in progress
penvoyage-morpheus-envoy | [2019-05-18 14:21:14.919][23][debug][pool] [source/common/http/conn_pool_base.cc:20] queueing request due to no available connections
penvoyage-morpheus-envoy | [2019-05-18 14:21:14.919][23][debug][connection] [source/common/network/connection_impl.cc:516] [C1] connected
penvoyage-morpheus-envoy | [2019-05-18 14:21:14.919][23][debug][client] [source/common/http/codec_client.cc:64] [C1] connected
penvoyage-morpheus-envoy | [2019-05-18 14:21:14.919][23][debug][pool] [source/common/http/http1/conn_pool.cc:236] [C1] attaching to next request
penvoyage-morpheus-envoy | [2019-05-18 14:21:14.919][23][debug][router] [source/common/router/router.cc:1122] [C0][S4222457489265630919] pool ready
penvoyage-morpheus-envoy | [2019-05-18 14:21:14.920][23][debug][router] [source/common/router/router.cc:669] [C0][S4222457489265630919] upstream headers complete: end_stream=false
penvoyage-morpheus-envoy | [2019-05-18 14:21:14.920][23][debug][http] [source/common/http/conn_manager_impl.cc:1234] [C0][S4222457489265630919] encoding headers via codec (end_stream=false):
penvoyage-morpheus-envoy | ‘:status’, ‘101’
penvoyage-morpheus-envoy | ‘upgrade’, ‘websocket’
penvoyage-morpheus-envoy | ‘connection’, ‘Upgrade’
penvoyage-morpheus-envoy | ‘sec-websocket-accept’, ‘2W9caJQU0JKW3MhWV6T8FHychjk=’
penvoyage-morpheus-envoy | ‘date’, ‘Sat, 18 May 2019 14:21:14 GMT’
penvoyage-morpheus-envoy | ‘server’, ‘envoy’
penvoyage-morpheus-envoy | ‘content-length’, ‘0’
penvoyage-morpheus-envoy |

Note the :authority header as I mentioned above:

‘:authority’, ‘debug.local:80’

Here’s an authority header for a normal website request:

penvoyage-morpheus-envoy | ‘:method’, ‘GET’
penvoyage-morpheus-envoy | ‘:authority’, ‘morpheus.local’
penvoyage-morpheus-envoy | ‘:scheme’, ‘https’

Notice the presence of the port in the first case (request via websocket), and the absence of the port in the second case.

Testcontainer

Please note, that https://hub.docker.com/r/dataferret/websocket-echo the dataferret/websocket-echo image seems to be broken!

There seems to be some incompatibility in twisted / Autobahn for the particular version which was used, and I therefore decided to just roll my own:

We’re going to build a test container which will echo the input which we send to it as a websocket.

Here’s the Dockerfile:

FROM python:stretch
COPY assets/websocketd /usr/bin/websocketd
RUN chmod +x /usr/bin/websocketd
COPY assets/run.py /opt/run.py
WORKDIR /opt

EXPOSE 8080
ENTRYPOINT [“websocketd”, “–port=8080”, “python3”, “/opt/run.py”]

Note that using ENTRYPOINT, when the container is started with CMD from docker-compose.yml, the CMD value will be passed as a parameter to the ENTRYPOINT command.

websocketd is a websocket daemon, you can obtain the standalone executable from here:

http://websocketd.com/

I downloaded https://github.com/joewalnes/websocketd/releases/download/v0.3.0/websocketd-0.3.0-linux_amd64.zip and unzipped it into the assets folder I created for my docker container.

The general idea of this nifty application is to provide a wrapper around your application to be able to serve up websockets.

All your application has to do is listen on stdin, and write the responses / any communication on stdout.

run.py – this is a simple script which will echo everything which is written to it:

#!/usr/bin/python
import sys

while True:
     read_co = sys.stdin.readline()
     print(read_co, end=””)
     sys.stdout.flush()

Notice the usage of readline(), which will return the string immediately upon a newline. Otherwise you have to deal with buffers / wait for the input to accumulate.

Also notice the sys.stdout.flush()

Finally, here’s the docker-compose.yml:

version: ‘3.6’

services:
   dwse:
     build:
       context: ./
       dockerfile: Dockerfile
     container_name: dwse
     hostname: dwse
     ports:
       – “8078:8080”
     networks:
       – envoy_net
     restart: always
     command: “80”
     #user: “root:root”

networks:
   envoy_net:
     external:
       name: my-bridge-network

Notice that I put it in the same network “my-bridge-network” as envoy above.

the command here is a random “80” which is not interpreted by the application, but you could create more elaborate command lines, etc. using this.

debugging

Finally, we want a client to connect and be able to see the messages we send to our test websocket service, which is proxied through envoy.

I suggest wscat for this.

wscat is an npm (node.js) package. Therefore you have to install npm first:

apt install npm

Then you can install wscat:

npm install -g wscat

Test wscat with a public websocket echo service:

wscat -c ws://echo.websocket.org

image

get a list of command line options for wscat:

wscat –help

image

To check a secure websocket connection through envoy with self-signed certificates do:

wscat -c ws://debug.local/mqtt_drinks_are_free –no-check

image

Hints

Unexpected server response

error: Unexpected server response: 404

Have a look at your route: do you put the port in the domain to match, as discussed above? Apparently it seems to be necessary for websocket clients.

When in doubt, include a match all domain at the end of your virtual hosts, to make debugging easier. I suggest to set a fixed path matching on this, instead of prefix – by changing the path you use in wscat, this way, you can verify which rule was matched:

– name: matcheverything
   domains: [“*”]
   routes:
     – match: { path: “/mqtt” }
       route: { cluster: target_dwebsocket }

Unable to parse JSON / cannot find field:

Unable to parse JSON as proto (INVALID_ARGUMENT:(route_config.virtual_hosts[3].routes[0].route) use_websocket: Cannot find field.):

As discussed above, this is a legacy style of configuration which has been superseded by the new configuration:

          upgrade_configs:
            – upgrade_type: websocket

no cluster match for URL

[source/common/router/router.cc:278] [C0][S5067510418337690596] no cluster match for URL ‘/’

See above – the URL is matched including the port for Websockets apparently ( in my tests), therefore include a rule to match all domains, and debug from there / use the syntax I have provided above:

          route_config:
            virtual_hosts:
            – name: debug
              domains: [“debug.local:80”]
              routes:
                – match: { prefix: “/” }
                  route:
                    cluster: target_dwebsocket

self signed certificate

error: self signed certificate

Run wscat with the –no-check command line parameter (this is a double dash which WordPress will mess up):

image

Websocket connection failed:

pcp-code.js:15 WebSocket connection to ‘wss://key:[email protected]/mqtt’ failed: Error during WebSocket handshake: Unexpected response code: 403

See information above: most likely, this javascript error message is due to a misconfiguration of the domain match (see above).

autobahn error:

“To use txaio, you must first select a framework ” exceptions.RuntimeError: To use txaio, you must first select a framework with .use_twisted() or .use_txaio()

This is the error I got when running the dataferret/websocket-echo image. I suggest to use my code which I provided above as an alternative.

Ref: https://stackoverflow.com/questions/34157314/autobahn-websocket-issue-while-running-with-twistd-using-tac-file

References:

Posted in Development, Docker and tagged , , , , , , , ,