The Real Cost of Self Hosting

How I feel when people tell me to "self-hosting is fun and cool!!!". Image is from a scene in the anime Toradora.
How I feel when people tell me to “self-hosting is fun and cool!!”

Every once in a while I took a look at the various services I self-host and think to myself? Is hosting $X service by myself really worth the cost? What is the cost? Would paying someone to host said service be more cost efficient?

That got me thinking about, what are the costs of self hosting? Let’s talk about it.

Continue reading “The Real Cost of Self Hosting”

Introducing The Art of Sysadmin podcast

I am super excited to announce the launch of a solo podcast project I’ve been working on!

Introducing The Art of Sysadmin, a podcast where I talk about all things sysadmin from a more human perspective rather than a technical one.

The idea for this podcast was one I originally had around the end of 2022, it took me months to actually decide on what I wanted to talk about exactly and some more months to actually sit down and record the episodes.

The first episode is now available here. If you enjoyed it, please subscribe to it in your favourite podcast player and share it with a friend!

Notes from my server upgrades to Ubuntu 22.04 LTS

Almost two years after I wrote my notes from my server upgrades to Ubuntu 20.04 LTS I am here again to write about my upgrades to the current LTS which is 22.04 (Jammy Jellyfish).

As is good practice for such upgrades I waited till the .1 release of the LTS was out to rule out any initial issues from the release. This time around I staggered out and planned out the upgrades over multiple weekends and I left the most difficult servers for last. I also kept notes of all the various issues/quirks I encountered when I did upgrades inside my Obsidian vault.

So let’s get into it.

Continue reading “Notes from my server upgrades to Ubuntu 22.04 LTS”

Notes from my server upgrades to Ubuntu 20.04 LTS

This past weekend I learned that Ubuntu 20.04.1 LTS was out. This meant that I could start upgrading my servers from Ubuntu 18.04 LTS to Ubuntu 20.04 LTS as that is when do-release-upgrade makes the next LTS release available for upgrade.

I started with my web server i.e the server this blog is hosted on. Mostly because I’ve been wanting PHP 7.4 for my WordPress sites and Ubuntu 20.04 would give me that. The upgrade went without issue and after a small configuration change to nginx to make sure it used the new Unix socket for PHP 7.4 FPM, we were all good to go.

From then on, I went ahead and started doing more of the servers, in the order of the servers that were least likely to break with the upgrade.

Breakages and Manual Interventions

Some of my servers failed to upgrade the grub-pc package during the dist-upgrade process. dpkg simply returned a error code. This was fixed by purging grub-pc and installing the grub2 package and then configuring it according to Linode’s documentation on GRUB2. I’m not entirely sure why the grub-pc package failed to upgrade.

All my servers also needed the apt sources list entry for the Icinga repository manually updated to use focal. My Mastodon servers needed repository entries for nodesource manually updated as well.

Ubuntu 20.04 brought with it PostgreSQL 12 and that means manually upgrading the installs and databases on my PostgreSQL 10 servers. This involves dropping the newly created empty 12 ‘main’ cluster and then running pg_upgradecluster on the 10 ‘main’ cluster. After that completed, the 10 ‘main’ cluster can be dropped. I didn’t run into any issues here. I did take a manual backup of database(s) before I started the process and I would recommend everyone do that if possible.

I will however note here that since the process creates a copy of the database into the 12 cluster, you will nearly double your disk usage. This can be a problem if you have large pgsql databases and do not have enough spare disk space on your servers to complete the process. Something to keep in mind if you are planning on doing one of these.

Since PostgreSQL was upgraded to 12, the pgbackrest configuration and stanza needed to be updated. Their documentation goes through the process but I didn’t need to use it as it is fairly straight forward.

Bits, Bobs and Final Thoughts

Overall the process of upgrading my servers to Ubuntu 20.04 was fairly smooth. I didn’t encounter any catastrophic failures or data loss. If I had I could have reverted to the manual snapshots I took of the servers before I started.

I’m glad to finally have PHP 7.4 on my web server and WordPress no longer complaining about having to use PHP 7.2.

Another neat thing to note is that one of my servers was initially provisioned on 14.04 and has over the years been upgraded through LTS Ubuntu releases. So 14.04 -> 16.04 -> 18.04 -> 20.04.

That’s all from me!

TestDisk and your lost partition table

Have you ever tried to resize a ext4 partition using fdisk but end up accidentally wiping your partition table instead?

It can be a panic inducing moment but all is not lost (including your data) .

Turns out that there is a FOSS tool called TestDisk that is designed to be used to recover from such human errors.

From their web page:

TestDisk is powerful free data recovery software! It was primarily designed to help recover lost partitions and/or make non-booting disks bootable again when these symptoms are caused by faulty software: certain types of viruses or human error (such as accidentally deleting a Partition Table). Partition table recovery using TestDisk is really easy.

I can attest to the easy part, I have in the past quite easily recovered a ext4 partition that I accidentally deleted using fdisk.

TestDisk is distributed as static binaries for various operating systems so it is handy to keep a copy of it on any sort of recovery/rescue media that you use.

A protip about fdisk

Do not use fdisk unless you are absolutely certain you know what you are doing with it. While TestDisk can save you if you fuck up, it is better to not take that risk.

Use something safer like parted or my favourite its GUI front-end gparted. In fact, these days I prefer to boot into the gparted LiveCD and perform my partition creations/resizes there.

Booting into a live CD is usually a slower process but it is better to be safe rather than sorry when it comes to things like partitions.