Failed to start the single-user server in JupyterHub – Zero to JupyterHub on Kubernetes

Hi, I recently encountered an error indicating spawn failure due to the python module issue. From the error log it suggested /opt/conda/bin/jupyterhub-singleuser is trying to load the ‘alembic’ package from /home/jovyan/.local instead of from /opt/conda/lib.

What is strange to me is that all the other people can successfully launch their server except that specific user – seem all the other users have no ‘alembic’ package installed under their home directory at all (/home/jovyan). I believe this error might be caused by the incorrect version of ‘alembic’ under /home/jovyan.

I further examine ‘sys.path’ under a successfully spawned server and it showed the following:

import sys
print(sys.path)
[‘’, ‘/opt/conda/lib/python38.zip’, ‘/opt/conda/lib/python3.8’, ‘/opt/conda/lib/python3.8/lib-dynload’, ‘/home/jovyan/.local/lib/python3.8/site-packages’, ‘/home/jovyan/.local/lib/python3.8/site-packages/Swiss_Re_Analytics-0.0.5-py3.8.egg’, ‘/opt/conda/lib/python3.8/site-packages’]

From the output I can see /home/jovyan has higher priority over /opt/conda in terms of module search order. I believe the jupyterhub-singleuser process is supposed to load ‘alembic’ from /opt/conda instead of from /home/jovyan.

Now the question is how can I change the order in sys.path? I’ve tried to modify sys.path in the config.yaml file (leveraged by Helm) but got no luck.

P.S. The platform is deployed on a K8S cluster and releases are managed by Helm.

Traceback (most recent call last):
File “/opt/conda/bin/jupyterhub-singleuser”, line 7, in
from jupyterhub.singleuser import main
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser/init.py”, line 5, in
from .app import main
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser/app.py”, line 10, in
from .mixins import make_singleuser_app
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser/mixins.py”, line 43, in
from …log import log_request
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/log.py”, line 15, in
from .handlers.pages import HealthCheckHandler
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/handlers/init.py”, line 1, in
from . import base
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/handlers/base.py”, line 32, in
from … import orm
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/orm.py”, line 11, in
import alembic.command
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/init.py”, line 3, in
from . import context # noqa
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/context.py”, line 1, in
from .runtime.environment import EnvironmentContext
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/runtime/environment.py”, line 1, in
from .migration import MigrationContext
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/runtime/migration.py”, line 15, in
from … import ddl
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/ddl/init.py”, line 1, in
from . import mssql # noqa
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/ddl/mssql.py”, line 8, in
from .base import AddColumn
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/ddl/base.py”, line 11, in
from …util import sqla_compat
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/util/init.py”, line 26, in
from .sqla_compat import has_computed # noqa
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/util/sqla_compat.py”, line 12, in
from sqlalchemy.sql.expression import _BindParamClause
ImportError: cannot import name ‘_BindParamClause’ from ‘sqlalchemy.sql.expression’ (/home/jovyan/.local/lib/python

Read more here: Source link