Some quick background information for anyone running into the same issues, and how to debug them.
Error code 503
This is the error code I am getting:
Error during WebSocket handshake: Unexpected response code: 503.
Set envoy logging to debug
Here’s my docker-compose.yaml for envoy:
version: ‚3.7‘
services:
envoy:
build:
context: ./
dockerfile: Dockerfile
container_name: penvoyage-morpheus-envoy
ports:
– „80:1080“
– „443:1443“
volumes:
– type: volume
source: penvoyage_volume
target: /etc/envoy
networks:
– envoy_net
user: „2000:2000“
#user: „root:root“
restart: unless-stopped
environment:
loglevel: debugvolumes:
penvoyage_volume:
external:
name: penvoyage_volumenetworks:
envoy_net:
external:
name: my-bridge-network
Show the VerneMQ listeners
on the VerneMQ container, show the listeners and the ip’s:
vmq-admin listener show
ip –4 addr show
Notice how mqttws is bound to 172.21.0.2 on port 8080.
There are two IPs present in the system:
- 172.21.0.2
- 172.18.0.7
this is due to the VerneMQ container being part of two internal docker networks.
Look at the envoy log output
Here are some excerpts to illustrate a websocket opening request from the MQTT client:
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][http] [source/common/http/conn_manager_impl.cc:210] [C85] new stream
penvoyage-morpheus-envoy | ‚:authority‘, ‚picockpit.local‘
penvoyage-morpheus-envoy | ‚:path‘, ‚/mqtt‘
penvoyage-morpheus-envoy | ‚:method‘, ‚GET‘
penvoyage-morpheus-envoy | ‚connection‘, ‚Upgrade‘
[C85][S10028475751570510266] cluster ‚target_verne‘ match for URL ‚/mqtt‘
[C85][S10028475751570510266] router decoding headers:
penvoyage-morpheus-envoy | ‚:authority‘, ‚picockpit.local‘
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][pool] [source/common/http/http1/conn_pool.cc:82] creating a new connection
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug]
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:638] [C86] connecting to 172.18.0.7:8080
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:647] [C86] connection in progress
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][pool] [source/common/http/conn_pool_base.cc:20] queueing request due to no available connections
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:525] [C86] delayed connection error: 111
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][connection] [source/common/network/connection_impl.cc:183] [C86] closing socket: 0
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug]
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][pool] [source/common/http/http1/conn_pool.cc:123] [C86] client disconnected
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][router] [source/common/router/router.cc:532] [C85][S10028475751570510266] upstream reset
penvoyage-morpheus-envoy | [2019-05-19 11:14:33.024][22][debug][http] [source/common/http/conn_manager_impl.cc:1234] [C85][S10028475751570510266] encoding headers via codec (end_stream=false):
penvoyage-morpheus-envoy | ‚:status‘, ‚503‘
The reason for the 503 error
In this case the reason for the 503 error is that envoy is not able to reach VerneMQ on IP 172.18.0.7 port 8080 – VerneMQ is listening on the other IP!
Therefore envoy returns 503, that the upstream server is not available – and that this condition might be fixed once the upstream server becomes available again.