mattermost websockets

Hi All,

I hope you can help me. I have the following situation.

kopano server with mattermost
nginx reverse proxy (to proxy requests to my backend servers)

If i am on my company network i can access the mattermost website, however websocket is not working. If i am outside my work company websockets is working fo my site.

I have checked the proxy server at work. it is supporting websockets. If i go to the demo page of mattermost websockets are working fine. (so i would expect that websockets for my site should also work)

i have read the mattermost manual. When i am checking the logs of mattermost i see the follwing error when accessing mattermost from my company network.

{“level”:“error”,“ts”:1588973100.5914853,“caller”:“mlog/log.go:172”,“msg”:“Failed to upgrade websocket connection”,“path”:"/api/v4/websocket",“request_id”:“ju33redxj3y1dmtdjkcwg951gr”,“ip_addr”:“1.2.3.4”,“user_id”:“hgkpf4yhxfguzr5o54y4foboqe”,“method”:“GET”,“err_where”:“connect”,“http_code”:500,“err_details”:""

This is my nginx config file for mattermost:

upstream backend {
   server 192.168.2.8:8065;
   keepalive 32;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

server {
   listen 80;
   server_name    mattermost.ckruijntjens.nl;
 return         301 https://$server_name$request_uri;
}

server {                                                     
listen 443;                                            
server_name mattermost.ckruijntjens.nl;
ssl on;
ssl_certificate /etc/letsencrypt/live/ckruijntjens.nl/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/ckruijntjens.nl/privkey.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers         HIGH:!aNULL:!MD5;

   location ~ /api/v[0-9]+/(users/)?websocket$ {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Host $http_host;
        proxy_headers_hash_max_size 51200;
            proxy_headers_hash_bucket_size 6400;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       client_body_timeout 60;
       send_timeout 300;
       lingering_timeout 5;
       proxy_connect_timeout 90;
       proxy_send_timeout 300;
       proxy_read_timeout 90s;
       proxy_http_version 1.1;
       proxy_pass http://backend;
   }


   location / {
       client_max_body_size 50M;
       proxy_set_header Connection "";
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_cache mattermost_cache;
       proxy_cache_revalidate on;
       proxy_cache_min_uses 2;
       proxy_cache_use_stale timeout;
       proxy_cache_lock on;
       proxy_pass http://192.168.2.8:8065;
   }
}


       # Optional headers if Mattermost should be accesible from within Kopano WebApp
       add_header X-Frame-Options "ALLOW-FROM 'https://webmail.ckruijntjens.nl'";
       add_header Content-Security-Policy "frame-ancestors 'self' https://webmail.ckruijntjens.nl";
       proxy_hide_header X-Frame-Options;
       proxy_hide_header Content-Security-Policy;

fbartels

@ckruijntjens said in mattermost websockets:

If i am on my company network i can access the mattermost website, however websocket is not working. If i am outside my work company websockets is working fo my site.
I have checked the proxy server at work. it is supporting websockets.

Does the software there only support websockets or is it also configured to allow them? Just a few weeks ago I was helping a partner setup meet and his Sophos UTM was breaking websocket connections one its in transit virus scanning was activated.

@fbartels

HI Thank you for your reply. If i go to the demo website of mattermost that works. (same company network, same laptop, same software)

Could it still be this is blocking the websockets?

fbartels

Hard to say without access to the environment. But since it works everywhere, but your office connection it feels like it has to do with that environment.

I’d recommend to reach out to the person managing your office it to have a closer look.

Hi fbartels,

I understand.

Thanks for your reply.

@fbartels

hi i did some more research on this toppic. what i did is i compared the demo site with my site and see a difference in the https certificate.

The demo site of mattermost (oss.mattermost.com) uses a letsencrypt certificate. My site uses also a letsencrypt certificate.

The only difference is this when i check the certificates.

My certificate:

issued by: Company Secure Web Gateway

and from oss.mattermost.com

issued by: Let’s Encrypt Authority X3

Could this be the problem why websockets are not working? If i check the certificates are installed correct.

fbartels

@ckruijntjens said in mattermost websockets:

Could this be the problem why websockets are not working?

Do you mean: “It does not work simply because it shows a different name for the issuer”? If so then, no I don’t think so.

But the issuer name implies that at your company a custom ca is installed on workstations. This can be used to decrypt traffic. And maybe the system that decrypts traffic also mangles it and breaks the websocket connection.

Again: I would recommend to follow up with the it responsible at your company because this cannot be meaningfully debugged without access to the environment or knowledge about configuration/product details only your it knows.