envoy not connecting to VerneMQ for MQTT over websockets, error code 503

Some quick background information for anyone running into the same issues, and how to debug them.

Error code 503

This is the error code I am getting:

image

Error during WebSocket handshake: Unexpected response code: 503.

Set envoy logging to debug

Here’s my docker-compose.yaml for envoy:

version: ‘3.7’

services:
   envoy:
     build:
       context: ./
       dockerfile: Dockerfile
     container_name: penvoyage-morpheus-envoy
     ports:
       – “80:1080”
       – “443:1443”
     volumes:
       – type: volume
         source: penvoyage_volume
         target: /etc/envoy
     networks:
       – envoy_net
     user: “2000:2000”
     #user: “root:root”
     restart: unless-stopped
     environment:
       loglevel: debug

volumes:
   penvoyage_volume:
     external:
       name: penvoyage_volume

networks:
   envoy_net:
     external:
       name: my-bridge-network

Show the VerneMQ listeners

on the VerneMQ container, show the listeners and the ip’s:

vmq-admin listener show

ip –4 addr show

image

Notice how mqttws is bound to 172.21.0.2 on port 8080.

There are two IPs present in the system:

  • 172.21.0.2
  • 172.18.0.7

this is due to the VerneMQ container being part of two internal docker networks.

Look at the envoy log output

Here are some excerpts to illustrate a websocket opening request from the MQTT client:

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][http] [source/common/http/conn_manager_impl.cc:210] [C85] new stream

penvoyage-morpheus-envoy | ‘:authority’, ‘picockpit.local’

penvoyage-morpheus-envoy | ‘:path’, ‘/mqtt’

penvoyage-morpheus-envoy | ‘:method’, ‘GET’

penvoyage-morpheus-envoy | ‘connection’, ‘Upgrade’

[C85][S10028475751570510266] cluster ‘target_verne’ match for URL ‘/mqtt’

[C85][S10028475751570510266] router decoding headers:

penvoyage-morpheus-envoy | ‘:authority’, ‘picockpit.local’

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][pool] [source/common/http/http1/conn_pool.cc:82] creating a new connection

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug]

[source/common/http/codec_client.cc:26] [C86] connecting

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:638] [C86] connecting to 172.18.0.7:8080

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:647] [C86] connection in progress

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][pool] [source/common/http/conn_pool_base.cc:20] queueing request due to no available connections

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:525] [C86] delayed connection error: 111
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:183] [C86] closing socket: 0
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug]

[source/common/http/codec_client.cc:82] [C86] disconnect. resetting 0 pending requests
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][pool] [source/common/http/http1/conn_pool.cc:123] [C86] client disconnected
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][router] [source/common/router/router.cc:532] [C85][S10028475751570510266] upstream reset

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][http] [source/common/http/conn_manager_impl.cc:1234] [C85][S10028475751570510266] encoding headers via codec (end_stream=false):

penvoyage-morpheus-envoy | ‘:status’, ‘503’

The reason for the 503 error

In this case the reason for the 503 error is that envoy is not able to reach VerneMQ on IP 172.18.0.7 port 8080 – VerneMQ is listening on the other IP!

Therefore envoy returns 503, that the upstream server is not available – and that this condition might be fixed once the upstream server becomes available again.