Problems Upgrading CSW Packages on Solaris 10

On a few different occasions, I have run into strange problems when upgrading open source (CSW) packages on my Solaris 10 boxes. The preconditions usually go something like this: I have a package that I want to upgrade, and so I use pkg-get to do so:

 
  >> pkg-get upgrade apache2

If the package is something complicated, like apache2 or mysql, then usually several other dependencies will also be upgraded at the same time, by pkg-get. It's even easy to lose track of which packages have been modified, as you're frantically hitting the y key as pkg-get tells you that everything on your system is woefully out-of-date. Hence, there's a good chance that the package openssl gets upgraded, whether you realize it or not.

In the past, when I've gone through such an upgrade process, I start seeing application failures like the follwing: after upgrading my web server, I am no longer able to use Outlook® on my PC to pickup email from my pop server. Outlook tells me that the server has returned this error message:

Task '[hostname] - Receiving' reported error (0x800CCC0F) : 'The connection to the server was interrupted. If this problem continues, contact your server administrator or Internet service provider (ISP). The server responded: ld.so.1: ipop3d: fatal: libssl.so.0.9.7: open failed: No such file or directory

It turns out that the executable, running on the mail server (Solaris 10 box), depends on the libssl.so library, and this library has been updated by the recent pkg-get invocation. However, since there are typically several versions of libraries like libssl.so, your server will probably have several of them collocated in a lib directory (e.g. /usr/lib or /opt/csw/lib), with the "proper" version soft-linked to. Unfortunately, the pkg-get upgrade process usually does not correctly handle this soft link. So, you need to!

In this particular case, I was able to see that in /opt/csw/lib, I had softlinks to the older versions of libssl and libcrypto. So, I remade the links:

root@hostname:/opt/csw/lib >>rm libcrypto.so.0.9.7
root@hostname:/opt/csw/lib >>rm libcrypto.so.0.9.8
root@hostname:/opt/csw/lib >>ln -s libcrypto.so.0.9.8.20080621182105.9968 libcrypto.so.0.9.8
root@hostname:/opt/csw/lib >>ln -s libcrypto.so.0.9.7.20080621182105.9968 libcrypto.so.0.9.7

The producers of the shared object (.so) libraries are nice enough to name them with an obvious timestamp. So, it should be easy to determine which version, for example, should be your system's current implementation of libcrypto.so.0.9.8 and libcrypto.so.0.9.7, which is what other binaries will be looking for.

I repeated the same process for libssl.so.*, which lives in the same location as libcrypto.so.*. It's possible that these libraries exist in different places on your system, so you may have to look around. /usr/lib and /usr/sfw/lib may be other places to look. But, if you're seeing complaints about libssl or libcrypto after a pkg-get upgrade, this is your likely culprit. Although the symptoms are virtually unlimited, here is another problem I encountered after using pkg-get upgrade, that required the same fix:

[ Oct 25 21:49:05 Executing start method ("/opt/csw/lib/svc/method/svc-cswapache2 start") ]
httpd: Syntax error on line 256 of /opt/csw/apache2/etc/httpd.conf: 
Cannot load /opt/csw/apache2/libexec/mod_ssl.so into server: ld.so.1: httpd: fatal: 
relocation error: file /opt/csw/apache2/libexec/mod_ssl.so: symbol SSL_CTX_sess_set_new_cb: 
referenced symbol not found

The above error message was displayed in the log file for the cswapache2 service (/var/svc/log/network-http:cswapache2.log).