I am trying to run pytorch 1.12.1 with cuda 11.6. I need this specific version to run on my schools HPC. I already got the conda environment working but I tried to install another package which changed things. I realized I did not need the package anymore so I tried setting things back to normal which only included me changing the packages torch and torchvision. Now when I submit and sbatch command that I normally would do the out put of my error file is as followed. I saw similar questions but am a little afraid to try the solutions because I do not want to mess up things more.

cat centering26_activations_250_0.0511.err
/home/az_peer/dials/conda_base/envs/pytorch_older_2/lib/python3.8/site-packages/google/protobuf/internal/api_implementation.py:109: UserWarning: Selected implementation cpp is not available.
warnings.warn(
Traceback (most recent call last):
File “/home/az_peer/training_data/centering_2/az_peer/environment/train.py”, line 13, in
from torch.utils.tensorboard import SummaryWriter
File “/home/az_peer/dials/conda_base/envs/pytorch_older_2/lib/python3.8/site-packages/torch/utils/tensorboard/init.py”, line 12, in
from .writer import FileWriter, SummaryWriter # noqa: F401
File “/home/az_peer/dials/conda_base/envs/pytorch_older_2/lib/python3.8/site-packages/torch/utils/tensorboard/writer.py”, line 9, in
from tensorboard.compat.proto.event_pb2 import SessionLog
File “/home/az_peer/dials/conda_base/envs/pytorch_older_2/lib/python3.8/site-packages/tensorboard/compat/proto/event_pb2.py”, line 8, in
from google.protobuf import descriptor as _descriptor
File “/home/az_peer/dials/conda_base/envs/pytorch_older_2/lib/python3.8/site-packages/google/protobuf/descriptor.py”, line 51, in
from google.protobuf.pyext import _message
ImportError: /opt/ohpc/pub/compiler/gcc/5.4.0/lib64/libstdc++.so.6: version `GLIBCXX_3.4.29′ not found (required by /home/az_peer/dials/conda_base/envs/pytorch_older_2/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so)

I am not sure how to solve the error. I tried looking it up but found little help. Here is my conda list and pip list if this helps clarify. Any help would mean the world.

Conda list:
conda list

packages in environment at /home/az_peer/dials/conda_base/envs/pytorch_older_2:

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_kmp_llvm conda-forge
absl-py 2.1.0 pyhd8ed1ab_0 conda-forge
aiohttp 3.9.3 py38h01eb140_0 conda-forge
aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge
anyio 4.3.0 pypi_0 pypi
argon2-cffi 23.1.0 pypi_0 pypi
argon2-cffi-bindings 21.2.0 pypi_0 pypi
argparse 1.4.0 pypi_0 pypi
arrow 1.3.0 pypi_0 pypi
asttokens 2.4.1 pypi_0 pypi
astunparse 1.6.3 pyhd8ed1ab_0 conda-forge
async-lru 2.0.4 pypi_0 pypi
async-timeout 4.0.3 pyhd8ed1ab_0 conda-forge
attrs 23.2.0 pyh71513ae_0 conda-forge
babel 2.14.0 pypi_0 pypi
backcall 0.2.0 pypi_0 pypi
beautifulsoup4 4.12.3 pypi_0 pypi
blas 2.121 mkl conda-forge
blas-devel 3.9.0 21_linux64_mkl conda-forge
bleach 6.1.0 pypi_0 pypi
blinker 1.7.0 pyhd8ed1ab_0 conda-forge
brotli-python 1.1.0 py38h17151c0_1 conda-forge
bzip2 1.0.8 hd590300_5 conda-forge
c-ares 1.27.0 hd590300_0 conda-forge
ca-certificates 2024.2.2 hbcca054_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cachetools 5.3.3 pyhd8ed1ab_0 conda-forge
certifi 2024.2.2 pyhd8ed1ab_0 conda-forge
cffi 1.16.0 py38h6d47a40_0 conda-forge
charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge
click 8.1.7 unix_pyh707e725_0 conda-forge
colorama 0.4.6 pypi_0 pypi
comm 0.2.1 pypi_0 pypi
contourpy 1.1.1 pypi_0 pypi
cryptography 42.0.5 py38hcdda232_0 conda-forge
cuda-version 11.6 hca96458_2 conda-forge
cudatoolkit 11.6.2 hfc3e2af_13 conda-forge
cudnn 8.8.0.121 hcdd5f01_4 conda-forge
cycler 0.12.1 pypi_0 pypi
debugpy 1.8.1 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
defusedxml 0.7.1 pypi_0 pypi
exceptiongroup 1.2.0 pypi_0 pypi
executing 2.0.1 pypi_0 pypi
fastjsonschema 2.19.1 pypi_0 pypi
ffmpeg 4.3 hf484d3e_0 pytorch
filelock 3.14.0 pypi_0 pypi
flatbuffers 22.12.06 hcb278e6_2 conda-forge
fonttools 4.49.0 pypi_0 pypi
fqdn 1.5.1 pypi_0 pypi
freetype 2.12.1 h267a509_2 conda-forge
frozenlist 1.4.1 py38h01eb140_0 conda-forge
fsspec 2024.3.1 pypi_0 pypi
gast 0.4.0 pyh9f0ad1d_0 conda-forge
giflib 5.2.1 h0b41bf4_3 conda-forge
gmp 6.3.0 h59595ed_0 conda-forge
gnutls 3.6.13 h85f3911_1 conda-forge
google-auth 2.28.1 pyhca7485f_0 conda-forge
google-auth-oauthlib 1.0.0 pypi_0 pypi
google-pasta 0.2.0 pyh8c360ce_0 conda-forge
grpcio 1.62.0 pypi_0 pypi
h11 0.14.0 pypi_0 pypi
h5py 3.10.0 nompi_py38h2c1edd7_101 conda-forge
hdf5 1.14.3 nompi_h4f84152_100 conda-forge
httpcore 1.0.3 pypi_0 pypi
httpx 0.26.0 pypi_0 pypi
icu 70.1 h27087fc_0 conda-forge
idna 3.6 pyhd8ed1ab_0 conda-forge
importlib-metadata 7.0.1 pyha770c72_0 conda-forge
importlib-resources 6.1.1 pypi_0 pypi
ipykernel 6.29.2 pypi_0 pypi
ipython 8.12.3 pypi_0 pypi
ipywidgets 8.1.2 pypi_0 pypi
isoduration 20.11.0 pypi_0 pypi
jedi 0.19.1 pypi_0 pypi
jinja2 3.1.3 pypi_0 pypi
jpeg 9e h0b41bf4_3 conda-forge
json5 0.9.17 pypi_0 pypi
jsonpointer 2.4 pypi_0 pypi
jsonschema 4.21.1 pypi_0 pypi
jsonschema-specifications 2023.12.1 pypi_0 pypi
jupyter-client 8.6.0 pypi_0 pypi
jupyter-core 5.7.1 pypi_0 pypi
jupyter-events 0.9.0 pypi_0 pypi
jupyter-lsp 2.2.2 pypi_0 pypi
jupyter-server 2.12.5 pypi_0 pypi
jupyter-server-terminals 0.5.2 pypi_0 pypi
jupyterlab 4.1.2 pypi_0 pypi
jupyterlab-pygments 0.3.0 pypi_0 pypi
jupyterlab-server 2.25.3 pypi_0 pypi
jupyterlab-widgets 3.0.10 pypi_0 pypi
keras 2.11.0 pyhd8ed1ab_0 conda-forge
keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.5 pypi_0 pypi
krb5 1.21.2 h659d440_0 conda-forge
lame 3.100 h166bdaf_1003 conda-forge
lcms2 2.15 hfd0df8a_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libabseil 20220623.0 cxx17_h05df665_6 conda-forge
libaec 1.1.2 h59595ed_1 conda-forge
libblas 3.9.0 21_linux64_mkl conda-forge
libcblas 3.9.0 21_linux64_mkl conda-forge
libcurl 8.5.0 hca28451_0 conda-forge
libdeflate 1.17 h0b41bf4_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 hd590300_2 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.2.0 h807b86a_5 conda-forge
libgfortran-ng 13.2.0 h69a702a_5 conda-forge
libgfortran5 13.2.0 ha4646dd_5 conda-forge
libgomp 13.2.0 h807b86a_5 conda-forge
libgrpc 1.51.1 h4fad500_1 conda-forge
libhwloc 2.9.1 hd6dc26d_0 conda-forge
libiconv 1.17 hd590300_2 conda-forge
liblapack 3.9.0 21_linux64_mkl conda-forge
liblapacke 3.9.0 21_linux64_mkl conda-forge
libnghttp2 1.58.0 h47da74e_1 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libpng 1.6.42 h2797004_0 conda-forge
libprotobuf 3.21.12 hfc55251_2 conda-forge
libsqlite 3.45.1 h2797004_0 conda-forge
libssh2 1.11.0 h0841786_0 conda-forge
libstdcxx-ng 13.2.0 h7e041cc_5 conda-forge
libtiff 4.5.0 h6adf6a1_2 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libwebp-base 1.3.2 hd590300_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libxml2 2.10.3 hca2bb57_4 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
lightning-utilities 0.11.2 pypi_0 pypi
llvm-openmp 17.0.6 h4dfa4b3_0 conda-forge
markdown 3.5.2 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.5 py38h01eb140_0 conda-forge
matplotlib 3.7.5 pypi_0 pypi
matplotlib-inline 0.1.6 pypi_0 pypi
mistune 3.0.2 pypi_0 pypi
mkl 2024.0.0 ha957f24_49657 conda-forge
mkl-devel 2024.0.0 ha770c72_49657 conda-forge
mkl-include 2024.0.0 ha957f24_49657 conda-forge
mpmath 1.3.0 pypi_0 pypi
multidict 6.0.5 py38h01eb140_0 conda-forge
nbclient 0.9.0 pypi_0 pypi
nbconvert 7.16.1 pypi_0 pypi
nbformat 5.9.2 pypi_0 pypi
nccl 2.20.3.1 h0800d71_0 conda-forge
ncurses 6.4 h59595ed_2 conda-forge
nest-asyncio 1.6.0 pypi_0 pypi
nettle 3.6 he412f7d_0 conda-forge
networkx 3.1 pypi_0 pypi
notebook 7.1.0 pypi_0 pypi
notebook-shim 0.2.4 pypi_0 pypi
numpy 1.24.4 py38h59b608b_0 conda-forge
nvidia-cublas-cu12 12.1.3.1 pypi_0 pypi
nvidia-cuda-cupti-cu12 12.1.105 pypi_0 pypi
nvidia-cuda-nvrtc-cu12 12.1.105 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.1.105 pypi_0 pypi
nvidia-cudnn-cu12 8.9.2.26 pypi_0 pypi
nvidia-cufft-cu12 11.0.2.54 pypi_0 pypi
nvidia-curand-cu12 10.3.2.106 pypi_0 pypi
nvidia-cusolver-cu12 11.4.5.107 pypi_0 pypi
nvidia-cusparse-cu12 12.1.0.106 pypi_0 pypi
nvidia-nccl-cu12 2.20.5 pypi_0 pypi
nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi
nvidia-nvtx-cu12 12.1.105 pypi_0 pypi
oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge
openh264 2.1.1 h780b84a_0 conda-forge
openjpeg 2.5.0 hfec8fc6_2 conda-forge
openssl 3.3.0 hd590300_0 conda-forge
opt_einsum 3.3.0 pyhc1e730c_2 conda-forge
overrides 7.7.0 pypi_0 pypi
packaging 23.2 pyhd8ed1ab_0 conda-forge
pandocfilters 1.5.1 pypi_0 pypi
parso 0.8.3 pypi_0 pypi
pexpect 4.9.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pillow 9.4.0 py38hde6dc18_1 conda-forge
pip 24.0 pyhd8ed1ab_0 conda-forge
pkgutil-resolve-name 1.3.10 pypi_0 pypi
platformdirs 4.2.0 pyhd8ed1ab_0 conda-forge
pooch 1.8.1 pyhd8ed1ab_0 conda-forge
pretty-errors 1.2.25 pypi_0 pypi
prometheus-client 0.20.0 pypi_0 pypi
prompt-toolkit 3.0.43 pypi_0 pypi
protobuf 4.25.3 pypi_0 pypi
psutil 5.9.8 pypi_0 pypi
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.2 pypi_0 pypi
pyasn1 0.5.1 pyhd8ed1ab_0 conda-forge
pyasn1-modules 0.3.0 pyhd8ed1ab_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pygments 2.17.2 pypi_0 pypi
pyjwt 2.8.0 pyhd8ed1ab_1 conda-forge
pyopenssl 24.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.1.1 pypi_0 pypi
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.8.18 hd12c33a_1_cpython conda-forge
python-dateutil 2.8.2 pypi_0 pypi
python-flatbuffers 23.5.26 pyhd8ed1ab_0 conda-forge
python-json-logger 2.0.7 pypi_0 pypi
python_abi 3.8 4_cp38 conda-forge
pytorch-mutex 1.0 cuda pytorch
pytz 2024.1 pypi_0 pypi
pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge
pyyaml 6.0.1 pypi_0 pypi
pyzmq 25.1.2 pypi_0 pypi
re2 2023.02.01 hcb278e6_0 conda-forge
readline 8.2 h8228510_1 conda-forge
referencing 0.33.0 pypi_0 pypi
requests 2.31.0 pyhd8ed1ab_0 conda-forge
requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge
rfc3339-validator 0.1.4 pypi_0 pypi
rfc3986-validator 0.1.1 pypi_0 pypi
rpds-py 0.18.0 pypi_0 pypi
rsa 4.9 pyhd8ed1ab_0 conda-forge
scipy 1.10.1 py38h59b608b_3 conda-forge
send2trash 1.8.2 pypi_0 pypi
setuptools 69.1.0 pyhd8ed1ab_1 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.10 h9fff704_0 conda-forge
sniffio 1.3.0 pypi_0 pypi
soupsieve 2.5 pypi_0 pypi
stack-data 0.6.3 pypi_0 pypi
sympy 1.12 pypi_0 pypi
tbb 2021.9.0 hf52228f_0 conda-forge
tensorboard 2.14.0 pypi_0 pypi
tensorboard-data-server 0.7.2 pypi_0 pypi
tensorboard-plugin-wit 1.8.1 pyhd8ed1ab_0 conda-forge
tensorboardx 2.6.2.2 pypi_0 pypi
tensorflow 2.11.0 cuda112py38hded6998_0 conda-forge
tensorflow-base 2.11.0 cuda112py38h7d4ee78_0 conda-forge
tensorflow-estimator 2.11.0 cuda112py38hb656414_0 conda-forge
termcolor 2.4.0 pyhd8ed1ab_0 conda-forge
terminado 0.18.0 pypi_0 pypi
tinycss2 1.2.1 pypi_0 pypi
tk 8.6.13 noxft_h4845f30_101 conda-forge
tomli 2.0.1 pypi_0 pypi
torch 1.12.1+cu116 pypi_0 pypi
torchaudio 0.12.1 py38_cu116 pytorch
torchvision 0.13.1+cu116 pypi_0 pypi
tornado 6.4 pypi_0 pypi
tqdm 4.66.2 pypi_0 pypi
traitlets 5.14.1 pypi_0 pypi
triton 2.3.0 pypi_0 pypi
types-python-dateutil 2.8.19.20240106 pypi_0 pypi
typing-extensions 4.9.0 hd8ed1ab_0 conda-forge
typing_extensions 4.9.0 pyha770c72_0 conda-forge
uri-template 1.3.0 pypi_0 pypi
urllib3 2.2.1 pyhd8ed1ab_0 conda-forge
wcwidth 0.2.13 pypi_0 pypi
webcolors 1.13 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
websocket-client 1.7.0 pypi_0 pypi
werkzeug 3.0.1 pyhd8ed1ab_0 conda-forge
wheel 0.42.0 pyhd8ed1ab_0 conda-forge
widgetsnbextension 4.0.10 pypi_0 pypi
wrapt 1.16.0 py38h01eb140_0 conda-forge
xorg-libxau 1.0.11 hd590300_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yarl 1.9.4 py38h01eb140_0 conda-forge
zipp 3.17.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 hd590300_5 conda-forge
zstd 1.5.5 hfc55251_0 conda-forge

Pip List:
pip list
WARNING: Ignoring invalid distribution -orch (/home/az_peer/dials/conda_base/envs/pytorch_older_2/lib/python3.8/site-packages)
Package Version

absl-py 2.1.0
aiohttp 3.9.3
aiosignal 1.3.1
anyio 4.3.0
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.3.0
asttokens 2.4.1
astunparse 1.6.3
async-lru 2.0.4
async-timeout 4.0.3
attrs 23.2.0
Babel 2.14.0
backcall 0.2.0
beautifulsoup4 4.12.3
bleach 6.1.0
blinker 1.7.0
Brotli 1.1.0
cached-property 1.5.2
cachetools 5.3.3
certifi 2024.2.2
cffi 1.16.0
charset-normalizer 3.3.2
click 8.1.7
colorama 0.4.6
comm 0.2.1
contourpy 1.1.1
cryptography 42.0.5
cycler 0.12.1
debugpy 1.8.1
decorator 5.1.1
defusedxml 0.7.1
exceptiongroup 1.2.0
executing 2.0.1
fastjsonschema 2.19.1
filelock 3.14.0
flatbuffers 23.5.26
fonttools 4.49.0
fqdn 1.5.1
frozenlist 1.4.1
fsspec 2024.3.1
gast 0.4.0
google-auth 2.28.1
google-auth-oauthlib 0.4.6
google-pasta 0.2.0
grpcio 1.51.1
h11 0.14.0
h5py 3.10.0
httpcore 1.0.3
httpx 0.26.0
idna 3.6
importlib-metadata 7.0.1
importlib-resources 6.1.1
ipykernel 6.29.2
ipython 8.12.3
ipywidgets 8.1.2
isoduration 20.11.0
jedi 0.19.1
Jinja2 3.1.3
json5 0.9.17
jsonpointer 2.4
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
jupyter_client 8.6.0
jupyter_core 5.7.1
jupyter-events 0.9.0
jupyter-lsp 2.2.2
jupyter_server 2.12.5
jupyter_server_terminals 0.5.2
jupyterlab 4.1.2
jupyterlab_pygments 0.3.0
jupyterlab_server 2.25.3
jupyterlab_widgets 3.0.10
keras 2.11.0
Keras-Preprocessing 1.1.2
kiwisolver 1.4.5
lightning-utilities 0.11.2
Markdown 3.5.2
MarkupSafe 2.1.5
matplotlib 3.7.5
matplotlib-inline 0.1.6
mistune 3.0.2
mpmath 1.3.0
multidict 6.0.5
nbclient 0.9.0
nbconvert 7.16.1
nbformat 5.9.2
nest-asyncio 1.6.0
networkx 3.1
notebook 7.1.0
notebook_shim 0.2.4
numpy 1.24.4
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.20.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.1.105
oauthlib 3.2.2
opt-einsum 3.3.0
overrides 7.7.0
packaging 23.2
pandocfilters 1.5.1
parso 0.8.3
pexpect 4.9.0
pickleshare 0.7.5
Pillow 9.4.0
pip 24.0
pkgutil_resolve_name 1.3.10
platformdirs 4.2.0
pooch 1.8.1
pretty-errors 1.2.25
prometheus_client 0.20.0
prompt-toolkit 3.0.43
protobuf 4.21.12
psutil 5.9.8
ptyprocess 0.7.0
pure-eval 0.2.2
pyasn1 0.5.1
pyasn1-modules 0.3.0
pycparser 2.21
Pygments 2.17.2
PyJWT 2.8.0
pyOpenSSL 24.0.0
pyparsing 3.1.1
PySocks 1.7.1
python-dateutil 2.8.2
python-json-logger 2.0.7
pytz 2024.1
pyu2f 0.1.5
PyYAML 6.0.1
pyzmq 25.1.2
referencing 0.33.0
requests 2.31.0
requests-oauthlib 1.3.1
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rpds-py 0.18.0
rsa 4.9
scipy 1.10.1
Send2Trash 1.8.2
setuptools 69.1.0
six 1.16.0
sniffio 1.3.0
soupsieve 2.5
stack-data 0.6.3
sympy 1.12
tensorboard 2.11.2
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
tensorboardX 2.6.2.2
tensorflow 2.11.0
tensorflow-estimator 2.11.0
termcolor 2.4.0
terminado 0.18.0
tinycss2 1.2.1
tomli 2.0.1
torch 1.12.1+cu116
torchaudio 0.12.1
torchvision 0.13.1+cu116
tornado 6.4
tqdm 4.66.2
traitlets 5.14.1
triton 2.3.0
types-python-dateutil 2.8.19.20240106
typing_extensions 4.9.0
uri-template 1.3.0
urllib3 2.2.1
wcwidth 0.2.13
webcolors 1.13
webencodings 0.5.1
websocket-client 1.7.0
Werkzeug 3.0.1
wheel 0.42.0
widgetsnbextension 4.0.10
wrapt 1.16.0
yarl 1.9.4
zipp 3.17.0