PHP and the curse of knowledge

In everything you do a seemingly unrelated lesson will hide. In the case of reinstalling my web server, the task that occupied me for the last few days, I learned something about professionalism, beginners mind, and the curse of knowledge. And that there's always one thing that makes a relatively straightforward process much more complicated. For my web server this was restoring my dad's website's backup.

My dad's website is running on WordPress. In everyday life, this is great. It's easy to use and you can find a lot about it on the internet. If he has any questions about how to do something, he doesn't have to bother me with them. I know next to nothing about WordPress or PHP (the programming language WordPress is written in). I also never used nginx before, the web server that I'm now running. So I wasn't surprised to see that upon testing the website, it only showed an error code pointing to an internal server error.

Even though I knew almost nothing about the software at hand, by now I have at least some experience in debugging my own software (and the systems I'm working with in my job). So, after some (Ok, a lot of) fiddling with several config files to no avail, I started to approach the problem more systematically.

The first step in debugging is to find a minimal example. So, to make sure my server was set up correctly, I wrote the equivalent to a hello world program with PHP and nginx. Turns out, my fiddling with the configs broke the environment and nginx didn't even communicate with the PHP interpreter. After rewriting the config from scratch and adding more complexity piece by piece, the minimal example started working. The website, however, still failed. Digging through the logs I found that the PHP interpreter was crashing when I made a request. Somewhere, something in the website was broken. And slowly but surely I started to reach my wit's end.

Usually, when you want to understand a crash, you look at its stack trace to get an idea of where it's happening. But since the website ran in an environment that was compiled without debug symbols, I didn't have access to the trace. The debug options in WordPress didn't help either. And no matter what I searched for online, I didn't find a solution, or things to try. In absence of any better ideas, I updated my dad's website to the newest version, and installed a clean version besides it. The clean version ran successfully, the updated website didn't. Nothing I did had any effect.

By that point, I had worked longer on recovering his website than on the other steps of resetting my server combined. To vent my frustration, I asked my dad half-jokingly, if he'd be willing to create a new website. To my surprise, he agreed. So instead of further trying to restore his website, I started to extract all posts in a form that would be meaningful to him. If I would have been able to do this without problems, he would be working on a new website by now.

Being a professional isn't about the money you're earning. It's about the attitude with which you approach your work. While I tried to extract the posts (which was harder than I expected), I thought about professionalism and how I would think about somebody I'm hiring to host my website that one day comes up to me and says: "Hey, I had to do some server changes that broke your website. Here are the posts you've written and the images you uploaded, good luck with creating a new one." Now, my dad isn't paying me money. But how you do anything is how you do everything. And this behaviour seemed utterly unprofessional to me.

I took a step back and started to think about it. WordPress is the most used blogging software in the world. It's highly unlikely that I was the first person with this kind of problem. There had to be something about this out there, I just needed a different approach to find it. So instead if trying to nail down the exact crash in PHP, I searched for it in the most obvious way: "wordpress site crashes 502 bad gateway." The first post I found described the exact problem I had and suggested to disable all plugins by renaming the folders where they were stored. And voila, the website worked.

Anyway, that's how I spend the first week of my vacation.

P.S.: Thank you Justus for your help with investigating this issue.