Posted on by packetcat

Notes from my server upgrades to Ubuntu 22.04 LTS

Almost two years after I wrote my notes from my server upgrades to Ubuntu 20.04 LTS I am here again to write about my upgrades to the current LTS which is 22.04 (Jammy Jellyfish).

As is good practice for such upgrades I waited till the .1 release of the LTS was out to rule out any initial issues from the release. This time around I staggered out and planned out the upgrades over multiple weekends and I left the most difficult servers for last. I also kept notes of all the various issues/quirks I encountered when I did upgrades inside my Obsidian vault.

So let’s get into it.

apt-key deprecation

The first thing I noticed as I started upgrading my servers is that apt was throwing a warning message about legacy keyring deprecation:

W: https://packages.icinga.com/ubuntu/dists/icinga-jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

An example of such an warning message from running apt-get update. In this case the man page section mentioned is actually useful and provides the transition method to the new keyring system:

DEPRECATION
Except for using apt-key del in maintainer scripts, the use of apt-key is deprecated. This section shows how to replace existing use of apt-key.

If your existing use of apt-key add looks like this:

wget -qO- https://myrepo.example/myrepo.asc | sudo apt-key add -

Then you can directly replace this with (though note the recommendation below):

wget -qO- https://myrepo.example/myrepo.asc | sudo tee /etc/apt/trusted.gpg.d/myrepo.asc

Make sure to use the "asc" extension for ASCII armored keys and the "gpg" extension for the binary OpenPGP format (also known as "GPG key public ring"). The binary OpenPGP format works for all apt versions, while the ASCII armored format works for apt version >= 1.4.

Recommended: Instead of placing keys into the /etc/apt/trusted.gpg.d directory, you can place them anywhere on your filesystem by using the Signed-By option in your sources.list and pointing to the filename of the key. See sources.list(5) for details. Since APT 2.4, /etc/apt/keyrings is provided as the recommended location for keys not managed by packages. When using a deb822-style sources.list, and with apt version >= 2.4, the Signed-By option can also be used to include the full ASCII armored keyring directly in the sources.list without an additional file.

So in my case for the Icinga apt repository, the command that needs to be run is:

wget -qO- https://packages.icinga.com/icinga.key | sudo tee /etc/apt/trusted.gpg.d/icinga.asc

Some pieces of software (like weechat) provided instructions on their documentation on how to obtain the GPG key and configure the signed-by method instead like below:

$ sudo mkdir -p /usr/share/keyrings
$ sudo gpg --no-default-keyring --keyring /usr/share/keyrings/weechat-archive-keyring.gpg --keyserver hkps://keys.openpgp.org --recv-keys 11E9DE8848F2B65222AA75B8D1820DB22A11534E
$ echo "deb [signed-by=/usr/share/keyrings/weechat-archive-keyring.gpg] https://weechat.org/ubuntu jammy main" | sudo tee /etc/apt/sources.list.d/weechat.list
$ echo "deb-src [signed-by=/usr/share/keyrings/weechat-archive-keyring.gpg] https://weechat.org/ubuntu jammy main" | sudo tee -a /etc/apt/sources.list.d/weechat.list

systemd-networkd transition

Some of my servers were still using the ‘old’ method of Debian network configuration, which is ifupdown and the /etc/network/interfaces file. I wanted to transition all of these servers to the new systemd-networkd method that newer servers were using for the sake of configuration consistency.

The basic steps (for me):

  • If you are using Linode for your servers like I am, you will want to enable network-helper in your Linode server’s configuration from their management portal. This very helpfully generates a systemd-networkd configuration file for you upon the next reboot. If you aren’t using Linode, you should refer to your distro’s network configuration documentation for configuring systemd-networkd.
  • Enable the systemd-networkd service using systemctl
  • Remove ifupdown
  • Reboot to verify everything is working

A note about removing ifupdown – I found that removing this package while on 20.04 disables network access so I would recommend doing it from a out-of-band console and make sure you know your root password before you do so you can actually log in from the console.

Alternatively, you can remove ifupdown after you have completed do-release-upgrade’s process for upgrading to 22.04. This way network is not interrupted and you can perform the process from SSH. If you are really paranoid, I recommend having OOB access ready just to be safe.

PHP upgrades

Like the last time we did these Ubuntu upgrades, one of the things we get with the upgrades is a new PHP version. In this case, we are going from PHP 7.4.x to PHP 8.1.x. I didn’t expect any breakages from the PHP version upgrade from the various web apps I run/ran – WordPress, DokuWiki, and tt-rss.

However as is always with such things, there were a couple issues at least one DokuWiki instance was broken due to a incompatibility with the gallery plugin, it was resolved by updating the instance to the latest hotfix release 2022-07-31a “Igor” and updating the gallery plugin to the latest version.

tt-rss was also broken, which was solved by a git pull in the application directory (which updates the application). I only needed tt-rss to work for a moment so I could export my feeds as a OPML and set up a Miniflux instance as my replacement RSS feed reader. I am no longer running tt-rss.

My WordPress installs did not see any breakages, yay!

Apache2 + php-fpm

If you are using Apache2 as your httpd, you will get this message:

NOTICE: Not enabling PHP 8.1 FPM by default.
NOTICE: To enable PHP 8.1 FPM in Apache2 do:
NOTICE: a2enmod proxy_fcgi setenvif
NOTICE: a2enconf php8.1-fpm
NOTICE: You are seeing this message because you have apache2 package installed.

As far as I can tell, the upgrade process does this all of that automatically, but just in case it doesn’t the above serves as a reference for what to do.

On my LibreNMS server, I also had to configure PHP-FPM to run as the librenms user to make sure PHP processes had the right accesses. This also involves configuring Apache2 to use the new PHP-FPM pool. The LibreNMS project’s recommended setup for a webserver nowadays is using nginx instead of Apache2, I may at some point in the future switch to that setup instead.

nginx + php-fpm

The only note to make about my nginx web server was that I needed to update my nginx configuration to use the new socket that PHP 8.1 FPM was using. Just a one line change.

PostgreSQL upgrades

For the most part, PostgreSQL cluster upgrades (12 -> 14) went without issue using the following steps:

  • Rename new versions cluster
  • Upgrade old cluster to new version
  • Restart and check if the upgrade is good
  • Drop old version’s cluster
  • Rebuild pgbackrest and set it up for new version’s cluster (set new data directory and upgrade stanza)

On the server with the largest PostgreSQL database (the Mastodon server), running pg_upgradecluster resulted in the server running out of disk space. I noted the disk usage the last time I did this. This time around however, the disk usage using the default “dump + restore” method that pg_upgradecluster does was not a good idea.

Instead I realized after I had upgraded the Linode server to add more disk space that I could simply use the the “link” method instead to upgrade the PostgreSQL server without worrying about disk space or extended downtime. The real lesson here is to read the documentation for the update tool before running it.

Mastodon

The Mastodon server itself had a couple other issues that needed resolving.

The Node.js version installed from the Nodesource repository was 12.x which was no longer supported on Ubuntu 22.04. I changed the repository to the LTS version of node.js which was 16.x. Instructions for that can be found here.

The second big issue I noticed was that due to OpenSSL version upgrades, the webauthn gem that Mastodon defines in its Gemfile no longer installs as the openssl gem version (openssl ~> 2.0) it pulls in is no longer compatible with the OpenSSL version in 22.04. The webauthn gem version defined in the Gemfile is 3.0.0alpha1 which for some reason is two years (!) older than the most current version…2.5.2. I am not entirely sure what’s going on with the versioning there but that is the case.

After I set the Gemfile to use webauthn 2.5.2, the bundle installation completed without issue.

Conclusions

Overall, I would say that the upgrades went reasonably well. There were no major issues that resulted in a rollback. I took Linode backup snapshots of each server before I started the upgrades as a safety measure, thankfully none of those were needed. For my LibreNMS server running inside a VMWare VM at home, I took a snapshot using VMWare Professional’s snapshot tool instead.

I am also glad I spread these out over multiple weekends because while each individual upgrade is not that time/energy consuming, doing all of them in one weekend would have been exhausting.

Whew, I expect to do all of this again in two years if the Ubuntu LTS release cycle maintains its pace!