Dreading a future with Copilot

A few days ago, I watched the announcement of Microsoft's Copilot 365. On a technical level, it's impressive. I understand why people are excited about it. I'm dreading where it will lead us.

A part of me believes we hit a point of diminishing returns for software innovation. Yes, I recognize that grouping all things happening in software into one bucket is overly generalizing. And I'm exicted about what happens with Rust and WebAssembly. In many areas, however, the innovation has stopped. Linux was first released in 1991. MySQL exists since 1995. And even if they continue to evolve, at their core, they still resemble their initial versions. But I'm not talking about those. Especially in the context of AI, I fear that we are at a point where the costs started to outweigh the benefits.

Let's look at instant messaging for a moment. Just like with Microsoft's Copilot, one of the core promises of instant messaging was to make us more productive, be reducing pesky waiting time. And sure, it's great that we can use it to reach anyone at any time. But what are the costs of this? You are constantly interrupted, up to a point where turning messaging off is the better productivity tip than using them. They also corroded how we communicate. Not so long ago, people were able to write one letter to meet up across the world. Nowadays, we need ten back-and-forth messages just to meet up at the Café across the street. Does this make the fact we can reach anyone anywhere less impressive? No. Is it worth it? I don't know. Maybe I'm just taking the upsides for granted, while being annoyed at the downsides.

The progress we're seeing in AI is truly mindboggling. Whoever is putting them off, saying that ChatGPT isn't that impressive because it's only data processing and parroting back what it found in the data, doesn't realize how much humans do the exact same thing. And there will be areas, where AI will be tremendously helpful. If we look at chess for a moment, AI already has become a tool to improve at the game. You can review your matches, see the mistakes you made and how you can work on them. But this alone doesn't make you a better chess player. It still takes an incredible amount of effort, studying and grit to improve your playing. Also, you'll never stop blundering, and blundering you Queen will always be painful. So for those, who don't want to investent the time to become better chess players, AI opened another route. They can use it to cheat.

As far as I can seen, cheating in chess isn't a big problem. Even though many people do it, if you play on chess.com, there are algorithm to detect if someone is cheating, and if you look at the profile of a cheater, it's often obvious that they have been cheating. But the rules in chess are simple, and and a bot moves much different than an beginner or intermediate player. This makes the detection relatively easy. Doing the same for generated text, images, or videos will be harder. If someone generates hundreds of scientific papers to support a false claim, how are we going to filter through this? I don't think we are equipped for that, and it's delusional to think that cases like this won't happen. But even without looking at the worst case, most of the use-cases I saw in the Copilot demo annoyed me more than anything.

Much of the software development we're seeing is a technical update based on constraints that no longer exist. When everybody was working in the same office, with the same schedule, having a quick meeting for a status update might have made sense. In a remote setting, meetings are much less useful and can often be replaced, and even improved upon, by writing. And yet, we use Zoom for our standups instead, in a culture shaped by corporate policies that are stuck in the past. And a lot of what I saw in the presentation of Copilot created the same impression. Soon, we'll generate mails from a few bullet points that the recipient will summarize using the same tools. We'll just send auto-generated, corporate noise over our wires, because we'll think it's expected from us. But if the three-bullet-point summary is all that is needed in the first place, should this mail have been written in the first place?

Maybe I'm just not ready for change, and my concerns are a rationalization rather than a well-reasoned position. Maybe we'll figure it out. In any case, a future even more governed by AI than we already are seems unavoidable. Microsoft's marketing material states it's a Copilot, and a human is always in control. But let's be honest. In the way these tools will be used, it is the co-pilot doing everything, so the pilot can take a nap. Soon, he will have forgotten how to fly a plane entirely. And when the plane starts failing and the co-pilot doesn't know what to do, it's already to late.