Troubleshooting Linking Errors

When you run into errors along the lines of:

/my/program: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /my/program)

...or:

/my/program: error while loading shared libraries: somelibrary.so: cannot open shared object file: No such file or directory

...you're dealing with a linking issue--the binary is failing since it can't find a library that it relies on at runtime.

ldd

Start your debugging journey with ldd:

$ ldd /my/program
/my/program: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ./wavesolve_openmp)
        linux-vdso.so.1 =>  (0x00007fff5276d000)
        somelibrary.so => not found
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007efc97a93000)
        libm.so.6 => /lib64/libm.so.6 (0x00007efc97791000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007efc9757b000)
        libc.so.6 => /lib64/libc.so.6 (0x00007efc971ad000)
        /lib64/ld-linux-x86-64.so.2 (0x00007efc97fb7000))

Complaints about libstdc++.so.6 almost always result from compiling with, but not linking against, a newer compiler than is included on the system by default--/my/program finds a libstdc++.so.6, but not the correct one. Load the compiler you built with and echo $LD_RUN_PATH to figure out where the appropriate libstdc++.so.6 (or similar) is.

Libraries that ldd shows as being "not found" do exist--the binary did compile against them, after all--but the binary doesn't know where to find them. Libraries are found via information stored in the binary itself (RPATH and RUNPATH), default search locations like /usr/lib64, and the environment variable LD_LIBRARY_PATH.

Rebuilding

If you built your software from source, you can usually solve the problem by setting the environment variables LD_RUN_PATH and LDFLAGS appropriately then rebuilding. LD_RUN_PATH indicates what should be included in the RUNPATH of binaries built in environments where it is set, and LDFLAGS drives the point home to build systems that erroneously ignore LD_RUN_PATH. This means that you'll need to know where the libraries that your executable relies on are located. We try to set LD_RUN_PATH as needed in modules we provide, though, so the search may be trivial--just check each path in LD_RUN_PATH for the missing libraries, only resetting LD_RUN_PATH:

export LD_RUN_PATH="$LD_RUN_PATH:/my/libary/directory"

...if the libraries you need aren't already there.

If you needed to reset LD_RUN_PATH, it's worth trying to rebuild. If not, or if that doesn't work, try setting LDFLAGS to tell the linker explicitly about the RUNPATH you want:

export LDFLAGS="-Wl,--enable-new-dtags,-rpath,$LD_RUN_PATH"

Patching

If rebuilding fails, or if you didn't compile the code, you can try modifying the binary itself with patchelf. It's a good idea to see what was already there before overwriting it:

patchelf --print-rpath /my/program

Combine that path (if it existed) with the paths to the library directories you need to add (delimited with colons) and update the RUNPATH:

patchelf --set-rpath /the/new:/runpath /my/program

Again, you can sometimes use LD_RUN_PATH as a starting point after loading the modules you need at build time or run time.

Deeper problems

Linking errors can propagate to arbitrary depth: if /my/program is linked correctly, but links to the correctly linked libGood.so which links to the incorrectly linked libBad.so, /my/program won't run. This often results from libraries erroneously relying on LD_LIBRARY_PATH, which is only for debugging and should NOT be set in anything resembling a permanent fashion. Chase down the original error with ldd:

$ patchelf --print-rpath /my/program
/libs/good
$ ldd /my/program | grep "not found"
        libGood.so => not found
$ ls /libs/good
libGood.so
$ patchelf --print-rpath /libs/good/libGood.so
/libs/bad
$ ldd /libs/good/libGood.so | grep "not found"
        libBad.so => not found
$ ldd /libs/bad/libBad.so | grep "not found"
/libs/bad/libBad.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /libs/bad/libBad.so)

...and either recompile the trouble library, or fix it with patchelf as outlined above, e.g.:

$ module load gcc/14.1
$ echo $LD_RUN_PATH
/apps/gcc/14.1.0/lib64
$ patchelf --print-rpath /libs/bad/libBad.so
/custom/libs:$ORIGIN
$ patchelf --set-rpath '/custom/libs:$ORIGIN:/apps/gcc/14.1.0/lib64' /libs/bad/libBad.so