CORS issues behind nginx ingress after updating to JupyterHub 2.1.1 – JupyterHub

Hi there, I noticed multiple issues after upgrading to JupyterHub 2.1.1 from 2.0.0.

  1. I stopped seeing spawn progress when starting the singleuser server. Logs:
[I 2022-03-04 20:57:47.436 JupyterHub log:189] 302 GET /hub/spawn -> /hub/spawn-pending/konstantin.taletskiy@labshare.org (konstantin.taletskiy@labshare.org@192.168.15.108) 1006.02ms
[I 2022-03-04 20:57:47.524 JupyterHub pages:405] konstantin.taletskiy@labshare.org is pending spawn
[I 2022-03-04 20:57:47.528 JupyterHub log:189] 200 GET /hub/spawn-pending/konstantin.taletskiy@labshare.org (konstantin.taletskiy@labshare.org@192.168.15.108) 9.75ms
[W 2022-03-04 20:57:47.746 JupyterHub base:94] Blocking Cross Origin API request.  Referer: https://<redacted_url>/hub/spawn-pending/konstantin.taletskiy@labshare.org, Host: <redacted_url>, Host URL: http://<redacted_url>/hub/
[W 2022-03-04 20:57:47.747 JupyterHub scopes:501] Not authorizing access to /hub/api/users/konstantin.taletskiy@labshare.org/server/progress. Requires any of [read:servers], not derived from scopes []
[W 2022-03-04 20:57:47.747 JupyterHub web:1787] 403 GET /hub/api/users/konstantin.taletskiy@labshare.org/server/progress (192.168.15.108): Action is not authorized with current scopes; requires any of [read:servers]
[W 2022-03-04 20:57:47.747 JupyterHub log:189] 403 GET /hub/api/users/konstantin.taletskiy@labshare.org/server/progress (@192.168.15.108) 4.93ms
  1. I can’t stop the singleuser server anymore. Logs:
[W 2022-03-04 23:07:26.200 JupyterHub base:94] Blocking Cross Origin API request.  Referer: https://<redacted_url>/hub/home, Host: <redacted_url>, Host URL: http://<redacted_url>/hub/
[W 2022-03-04 23:07:26.200 JupyterHub scopes:501] Not authorizing access to /hub/api/users/konstantin.taletskiy%40labshare.org/server. Requires any of [delete:servers], not derived from scopes []
[W 2022-03-04 23:07:26.200 JupyterHub web:1787] 403 DELETE /hub/api/users/konstantin.taletskiy%40labshare.org/server (192.168.3.68): Action is not authorized with current scopes; requires any of [delete:servers]
[W 2022-03-04 23:07:26.201 JupyterHub log:189] 403 DELETE /hub/api/users/konstantin.taletskiy%40labshare.org/server (@192.168.3.68) 8.96ms
  1. Admin page is empty

My deployment is on K8s with Helm, using heavily modified zero-to-jupyterhub-k8s chart. You could see the chart here. I am using nginx ingress, which is not described in JupyterHub reverse proxy docs, but is used in zero-to-jupyterhub-k8s, for example. I found similar issues for users running Apache and nginx reverse proxies:

The relevant parts of my config are provided below

  1. JupyterHub Docker file
FROM frolvlad/alpine-miniconda3:python3.7@sha256:bf5c88b66776e27b6745a7489104263d4ec05748eb447e7a292a0c4b5ed6c074
LABEL maintainer="Labshare <konstantin.taletskiy@labshare.org>"

# Track semantic versioning
COPY VERSION /

# Copy cull-idle script
COPY cull-idle-servers.py /srv/jupyterhub/config/cull-idle-servers.py

RUN conda install --yes -c conda-forge 
      git 
      sqlalchemy 
      tornado 
      jinja2 
      traitlets 
      requests 
      pycurl 
      nodejs=12 
      configurable-http-proxy 
      escapism=1.0.1 
      jupyterhub=2.1.1 
      python-kubernetes=22.6.0 
      jupyterhub-kubespawner=2.0.1 && 
    pip install 
      psycopg2-binary==2.9.1  
      oauthenticator==14.2.0

RUN mkdir -p /srv/jupyterhub/
COPY config-wrapper.py /srv/jupyterhub/config-wrapper.py
WORKDIR /srv/jupyterhub/
EXPOSE 8000

LABEL org.jupyter.service="jupyterhub"

CMD ["python", "/srv/jupyterhub/config-wrapper.py"]
  1. Nginx ingress resource template from the Helm chart:
{{- if .Values.hub.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
  name: {{ include "jupyterhub.ingress.fullname" . }}
spec:
  rules:
    - host: {{ .Values.hub.ingress.hostName }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: jupyterhub
                port:
                  number: 80
            
{{- end }}
  1. nginx configmap
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  name: nginx-configuration
  namespace: ingress-nginx
data:
  location-snippet: |
    more_set_input_headers "X-Forwarded-Proto: https";
  proxy-body-size: "0"
  use-forwarded-headers: "true"
  use-proxy-protocol: "false"
  1. JupyterHub Portion of the nginx config generated by nginx ingress (/etc/nginx/nginx.conf file in the nginx-ingress-controller-xxxx-zzzz pod)
server {
		server_name <redacted_url> ;
		
		listen 80  ;
		listen 443  ssl http2 ;
		
		set $proxy_upstream_name "-";
		
		ssl_certificate_by_lua_block {
			certificate.call()
		}
		
		location / {
			
			set $namespace      "<jupyterhub_namespace>";
			set $ingress_name   "<helm_deployment_full_name>";
			set $service_name   "jupyterhub";
			set $service_port   "80";
			set $location_path  "https://discourse.jupyter.org/";
			
			rewrite_by_lua_block {
				lua_ingress.rewrite({
					force_ssl_redirect = false,
					ssl_redirect = true,
					force_no_ssl_redirect = false,
					use_port_in_redirects = false,
				})
				balancer.rewrite()
				plugins.run()
			}
			
			# be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
			# will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
			# other authentication method such as basic auth or external auth useless - all requests will be allowed.
			#access_by_lua_block {
			#}
			
			header_filter_by_lua_block {
				lua_ingress.header()
				plugins.run()
			}
			
			body_filter_by_lua_block {
			}
			
			log_by_lua_block {
				balancer.log()
				
				monitor.call()
				
				plugins.run()
			}
			
			port_in_redirect off;
			
			set $balancer_ewma_score -1;
			set $proxy_upstream_name "<k8s_namespace-service_name-port>;
			set $proxy_host          $proxy_upstream_name;
			set $pass_access_scheme  $scheme;
			
			set $pass_server_port    $server_port;
			
			set $best_http_host      $http_host;
			set $pass_port           $pass_server_port;
			
			set $proxy_alternative_upstream_name "";
			
			client_max_body_size                    0;
			
			proxy_set_header Host                   $best_http_host;
			
			# Pass the extracted client certificate to the backend
			
			# Allow websocket connections
			proxy_set_header                        Upgrade           $http_upgrade;
			
			proxy_set_header                        Connection        $connection_upgrade;
			
			proxy_set_header X-Request-ID           $req_id;
			proxy_set_header X-Real-IP              $remote_addr;
			
			proxy_set_header X-Forwarded-For        $remote_addr;
			
			proxy_set_header X-Forwarded-Host       $best_http_host;
			proxy_set_header X-Forwarded-Port       $pass_port;
			proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
			
			proxy_set_header X-Scheme               $pass_access_scheme;
			
			# Pass the original X-Forwarded-For
			proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
			
			# mitigate HTTPoxy Vulnerability
			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
			proxy_set_header Proxy                  "";
			
			# Custom headers to proxied server
			
			proxy_connect_timeout                   5s;
			proxy_send_timeout                      60s;
			proxy_read_timeout                      60s;
			
			proxy_buffering                         off;
			proxy_buffer_size                       4k;
			proxy_buffers                           4 4k;
			
			proxy_max_temp_file_size                1024m;
			
			proxy_request_buffering                 on;
			proxy_http_version                      1.1;
			
			proxy_cookie_domain                     off;
			proxy_cookie_path                       off;
			
			# In case of errors try the next upstream server before returning an error
			proxy_next_upstream                     error timeout;
			proxy_next_upstream_timeout             0;
			proxy_next_upstream_tries               3;
			
			# Custom code snippet configured in the configuration configmap
			more_set_input_headers "X-Forwarded-Proto: https";
			
			proxy_pass http://upstream_balancer;
			
			proxy_redirect                          off;
			
		}
		
	}

The only change that I did from the working configuration was to change jupyterhub=2.0.0 to jupyterhub=2.1.1 in the Docker file and upgrade the Helm Chart. Could anyone suggest the possible solutions to fix the problem?

So far, I looked into adding config snippet to my JupyterHub ingress definitions to make it look more like the one provided in the documentation:

nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_set_header Host $http_host;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection upgrade;

That didn’t help. Please help.

Read more here: Source link