I feel like Im dancing around perhaps the most fundamental piece of my operating system everytime I run and install software. Starting services with systemctl and checking logs with journalctl is the extent of my knowledge.
Do you know of good resources or tutorials for learning how systemd works and how to use it to run software on my desktop and servers? Thanks.
Write a couple of your own toy services as practice. Write a one-shot that fires at a particular time during boot, a normal service that would run a daemon and a mount service that fires after its dependencies are loaded (like, say, a bind mount that sets up a directory under /run/foo after the backing filesystem is mounted - I do this to make fast ext4 storage available in some parts of the VFS tree while using a btrfs filesystem for everything else.) You can also write file watcher services that fire after changes to a file or directory, I use one of those to mirror /boot/ to /.boot/ on another filesystem so it’s captured by my system snapshots.
I’d start by reading the docs so you have some ideas about what services can do, then you’ll find uses that you wouldn’t have thought of before.