envoy not connecting to VerneMQ for MQTT over websockets, error code 503

Some quick background information for anyone running into the same issues, and how to debug them.

Error code 503

This is the error code I am getting:


Error during WebSocket handshake: Unexpected response code: 503.

Set envoy logging to debug

Here’s my docker-compose.yaml for envoy:

version: ‘3.7’

       context: ./
       dockerfile: Dockerfile
     container_name: penvoyage-morpheus-envoy
       – “80:1080”
       – “443:1443”
       – type: volume
         source: penvoyage_volume
         target: /etc/envoy
       – envoy_net
     user: “2000:2000”
     #user: “root:root”
     restart: unless-stopped
       loglevel: debug

       name: penvoyage_volume

       name: my-bridge-network

Show the VerneMQ listeners

on the VerneMQ container, show the listeners and the ip’s:

vmq-admin listener show

ip –4 addr show


Notice how mqttws is bound to on port 8080.

There are two IPs present in the system:


this is due to the VerneMQ container being part of two internal docker networks.

Look at the envoy log output

Here are some excerpts to illustrate a websocket opening request from the MQTT client:

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][http] [source/common/http/conn_manager_impl.cc:210] [C85] new stream

penvoyage-morpheus-envoy | ‘:authority’, ‘picockpit.local’

penvoyage-morpheus-envoy | ‘:path’, ‘/mqtt’

penvoyage-morpheus-envoy | ‘:method’, ‘GET’

penvoyage-morpheus-envoy | ‘connection’, ‘Upgrade’

[C85][S10028475751570510266] cluster ‘target_verne’ match for URL ‘/mqtt’

[C85][S10028475751570510266] router decoding headers:

penvoyage-morpheus-envoy | ‘:authority’, ‘picockpit.local’

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][pool] [source/common/http/http1/conn_pool.cc:82] creating a new connection

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug]

[source/common/http/codec_client.cc:26] [C86] connecting

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:638] [C86] connecting to

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:647] [C86] connection in progress

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][pool] [source/common/http/conn_pool_base.cc:20] queueing request due to no available connections

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:525] [C86] delayed connection error: 111
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:183] [C86] closing socket: 0
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug]

[source/common/http/codec_client.cc:82] [C86] disconnect. resetting 0 pending requests
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][pool] [source/common/http/http1/conn_pool.cc:123] [C86] client disconnected
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][router] [source/common/router/router.cc:532] [C85][S10028475751570510266] upstream reset

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][http] [source/common/http/conn_manager_impl.cc:1234] [C85][S10028475751570510266] encoding headers via codec (end_stream=false):

penvoyage-morpheus-envoy | ‘:status’, ‘503’

The reason for the 503 error

In this case the reason for the 503 error is that envoy is not able to reach VerneMQ on IP port 8080 – VerneMQ is listening on the other IP!

Therefore envoy returns 503, that the upstream server is not available – and that this condition might be fixed once the upstream server becomes available again.