Some quick background information for anyone running into the same issues, and how to debug them.

Error code 503

This is the error code I am getting:

Error during WebSocket handshake: Unexpected response code: 503.

Set envoy logging to debug

Here’s my docker-compose.yaml for envoy:

version: ‘3.7’

services:
   envoy:
     build:
       context: ./
       dockerfile: Dockerfile
     container_name: penvoyage-morpheus-envoy
     ports:
       – “80:1080”
       – “443:1443”
     volumes:
       – type: volume
         source: penvoyage_volume
         target: /etc/envoy
     networks:
       – envoy_net
     user: “2000:2000”
     #user: “root:root”
     restart: unless-stopped
     environment:
       loglevel: debug

volumes:
   penvoyage_volume:
     external:
       name: penvoyage_volume

networks:
   envoy_net:
     external:
       name: my-bridge-network

Show the VerneMQ listeners

on the VerneMQ container, show the listeners and the ip’s:

vmq-admin listener show

ip –4 addr show

Notice how mqttws is bound to 172.21.0.2 on port 8080.

There are two IPs present in the system:

• 172.21.0.2
• 172.18.0.7

this is due to the VerneMQ container being part of two internal docker networks.

Look at the envoy log output

Here are some excerpts to illustrate a websocket opening request from the MQTT client:

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][http] [source/common/http/conn_manager_impl.cc:210] [C85] new stream

penvoyage-morpheus-envoy | ‘:authority’, ‘picockpit.local’

penvoyage-morpheus-envoy | ‘:path’, ‘/mqtt’

penvoyage-morpheus-envoy | ‘:method’, ‘GET’

penvoyage-morpheus-envoy | ‘connection’, ‘Upgrade’

[C85][S10028475751570510266] cluster ‘target_verne’ match for URL ‘/mqtt’

[C85][S10028475751570510266] router decoding headers:

penvoyage-morpheus-envoy | ‘:authority’, ‘picockpit.local’

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][pool] [source/common/http/http1/conn_pool.cc:82] creating a new connection

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug]

[source/common/http/codec_client.cc:26] [C86] connecting

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:638] [C86] connecting to 172.18.0.7:8080

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:647] [C86] connection in progress

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][pool] [source/common/http/conn_pool_base.cc:20] queueing request due to no available connections

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:525] [C86] delayed connection error: 111
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:183] [C86] closing socket: 0
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug]

[source/common/http/codec_client.cc:82] [C86] disconnect. resetting 0 pending requests
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][pool] [source/common/http/http1/conn_pool.cc:123] [C86] client disconnected
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][router] [source/common/router/router.cc:532] [C85][S10028475751570510266] upstream reset

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][http] [source/common/http/conn_manager_impl.cc:1234] [C85][S10028475751570510266] encoding headers via codec (end_stream=false):

penvoyage-morpheus-envoy | ‘:status’, ‘503’

The reason for the 503 error

In this case the reason for the 503 error is that envoy is not able to reach VerneMQ on IP 172.18.0.7 port 8080 – VerneMQ is listening on the other IP!

Therefore envoy returns 503, that the upstream server is not available – and that this condition might be fixed once the upstream server becomes available again.