Some quick background information for anyone running into the same issues, and how to debug them.

Error code 503

This is the error code I am getting:

image

Error during WebSocket handshake: Unexpected response code: 503.

Set envoy logging to debug

Here’s my docker-compose.yaml for envoy:

version: ‚3.7‘

services:
   envoy:
     build:
       context: ./
       dockerfile: Dockerfile
     container_name: penvoyage-morpheus-envoy
     ports:
       – „80:1080“
       – „443:1443“
     volumes:
       – type: volume
         source: penvoyage_volume
         target: /etc/envoy
     networks:
       – envoy_net
     user: „2000:2000“
     #user: „root:root“
     restart: unless-stopped
     environment:
       loglevel: debug

volumes:
   penvoyage_volume:
     external:
       name: penvoyage_volume

networks:
   envoy_net:
     external:
       name: my-bridge-network

Show the VerneMQ listeners

on the VerneMQ container, show the listeners and the ip’s:

vmq-admin listener show

ip –4 addr show

image

Notice how mqttws is bound to 172.21.0.2 on port 8080.

There are two IPs present in the system:

  • 172.21.0.2
  • 172.18.0.7

this is due to the VerneMQ container being part of two internal docker networks.

Look at the envoy log output

Here are some excerpts to illustrate a websocket opening request from the MQTT client:

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][http] [source/common/http/conn_manager_impl.cc:210] [C85] new stream

penvoyage-morpheus-envoy | ‚:authority‘, ‚picockpit.local‘

penvoyage-morpheus-envoy | ‚:path‘, ‚/mqtt‘

penvoyage-morpheus-envoy | ‚:method‘, ‚GET‘

penvoyage-morpheus-envoy | ‚connection‘, ‚Upgrade‘

[C85][S10028475751570510266] cluster ‚target_verne‘ match for URL ‚/mqtt‘

[C85][S10028475751570510266] router decoding headers:

penvoyage-morpheus-envoy | ‚:authority‘, ‚picockpit.local‘

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][pool] [source/common/http/http1/conn_pool.cc:82] creating a new connection

penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug]

    • [source/common/http/codec_client.cc:26] [C86] connecting

    penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:638] [C86] connecting to 172.18.0.7:8080

    penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:647] [C86] connection in progress

    penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][pool] [source/common/http/conn_pool_base.cc:20] queueing request due to no available connections

    penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:525] [C86] delayed connection error: 111
    penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:183] [C86] closing socket: 0
    penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug]

    • [source/common/http/codec_client.cc:82] [C86] disconnect. resetting 0 pending requests
    penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][pool] [source/common/http/http1/conn_pool.cc:123] [C86] client disconnected
    penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][router] [source/common/router/router.cc:532] [C85][S10028475751570510266] upstream reset

    penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][http] [source/common/http/conn_manager_impl.cc:1234] [C85][S10028475751570510266] encoding headers via codec (end_stream=false):

    penvoyage-morpheus-envoy | ‚:status‘, ‚503‘

    The reason for the 503 error

    In this case the reason for the 503 error is that envoy is not able to reach VerneMQ on IP 172.18.0.7 port 8080 – VerneMQ is listening on the other IP!

    Therefore envoy returns 503, that the upstream server is not available – and that this condition might be fixed once the upstream server becomes available again.