The 404 Web

Over the years, I’ve changed thousands of URIs. Maybe hundreds of thousands if you count changing everything over from http:// to https://. It always feels kinda wrong to modify URIs. Maybe that comes with a basic understanding that Cool URIs don’t change. But I also understand that, in reality, URIs change constantly. It’s just the nature of the World Wide Web, an endless perpetually evolving flow of digital information, URLs and links included. As stated by the W3C:

There are no reasons at all in theory for people to change URIs, but millions of reasons in practice.

Just thinking out loud here..

I would estimate over half of all URIs will change at some point in their lifespan. Of those, more than half will end up as 404 “Not Found” errors, without any sort of resolving redirect. Considering there are perhaps billions of URIs on the Web, we’re talking about a LOT of 404 errors happening constantly, wasting much time, energy, and resources. Things that we should be conserving.

Over time, it seems the number of solid, non-changing URIs is trending toward zero. For example, the largest Internet players like Apple, Google, and Facebook tend to change their application and documentation URIs I think almost daily. Indeed, it’s becoming increasingly rare to find online resources that aren’t moved constantly or removed completely.

And don’t forget about the millions if not billions of resources that migrated from HTTP to HTTPS (SSL/TLS) over the past ten years. If that qualifies as “changing” URIs, which it does, then most of the Internet is “not cool” according to Tim Berners-Lee, who literally invented the World Wide Web.

Why is this important?

By default, when resources are moved or removed, all links that were pointing to that resource are broken. And without proper redirection, broken resources result in 404 “Not Found” and other errors. Broken pages, broken links in apps, bookmarks, documentation, basically a lot of “broken” everywhere on the Web.

Broken URIs results in wasted time, energy, bandwidth, and other resources. On the server, it’s just wasted memory and processing, dealing with needless redirects and serving requests for nonexistent resources. Bots, scripts, extensions, apps, and bookmarks all sending traffic to absolutely nowhere, just a total waste.

Think about it, do the math taking into consideration there are millions and millions of dead-end requests happening every minute or even every second. You can’t even fathom the amount of collective waste due to constantly changing URIs.

Stability is a virtue

Stability and consistency are virtues worth pursuing. Less wasted time. Less wasted energy. Less confusion. Smarter surfing. Happier users :)

I rejoice when I visit web pages from way back in the day. Like when I visit bookmarks or follow links from past articles and the resources are still online, serving visitors 24/7. That’s just a good feeling. Successful websites are built on a foundation of stability and consistency. Stable URIs make for positive user experience, solid reputation, and quality websites.

When creating online, stability and endurance are as important as first impressions and marketing hype. If you’re online for the long haul, you want to think long-term. Build for tomorrow. Build for the future. And plan your URIs accordingly.

How to build smart URIs

The URI is the “address” of your web pages and resources. You know this from the beginning. When you choose your site’s domain name. People spend countless hours brainstorming and choosing the best possible domain name. It’s gotta “pop”, right? Your domain name is gotta be juuust perfect for whatever it is you’re doing online.

So the key to building smart URIs is to simply apply the same “domain-level” planning to your entire site. The URIs for each page and resource should reflect the same level of care that’s given to your domain name. For example, content management systems (CMSs) by default generate query-string URIs, like:

https://yourdomain.com/?p=321666

Fortunately most CMSs make it easy to change the default URI structure to something better. For example, WordPress provides options to enable permalink structure for your site’s URIs. Permalink URIs look like:

https://yourdomain.com/ai-singularity-coming-soon/

This makes your URIs more informative, user-friendly, readable by humans, and so forth. It also gives you an opportunity to include descriptive keywords in your URIs. Keywords are used by machine algorithms to more accurately describe and place your content. And both of these things — human friendly and keyword descriptive — play an important role in SEO and GEO.

More URI-building tips

Avoid recycled domain names, if possible. Recycled domains previously have been in service and can cause headaches when it comes to traffic monitoring, measuring analytics, dealing with 404 “Not Found” errors, and other annoyances.

For example, one of my “test” slash “demo” sites, Dev-Tricks.com, was purchased around 10 years ago. The name is great but the domain had been used previously for some time. So for the past 10 years, the site’s access and error logs have been filled with an endless parade of 404 responses for past articles which no longer exist. Maybe not a huge deal for the typical weekend administrator. But for anyone who understands the power of log files.. yeah, it can be a real headache.

So before signing on the dotted line and committing to that shiny new domain name, take a moment to check if it’s already been owned. A quick Whois lookup or recycled-domain check is worth the effort.

Fix what’s broken

For the love of sanity, if you absolutely must change your URIs or remove some resources, redirect them or declare them 410 Gone. Or if you prefer old fashioned 404 responses, make them useful. Do NOT leave your visitors stumbling and groping around in the dark broken corners of an ill-planned website. Even you personally couldn’t care less about things like integrity and reputation, take a few moments to redirect your 404 errors for the sake of your visitors. It’s not that difficult and will benefit your site going forward.

420 Gone Forever

There also needs to be a way, some resource or protocol — something — some way for sites to declare a resource “Gone Forever”. Or if you think 410 “Gone” is enough, then someone please find a way to force bots to actually understand and obey that when a server tells you 410 Gone, it means exactly that: the resource is GONE. As in, please stop returning over and over requesting the same 410 resource endlessly proving your programmer’s complete and utter lack of skill and understanding.

Response Code 420 = Gone Forever: Do NOT return to this URI.

A new 420 “Gone Forever” response would be ideal. A way for website developers and admins to completely bury URIs that no longer exist. Scrub them from existence, as it were. So 410 would mean “yes it’s gone, but maybe check again if you want”. But 420 would go further and say, “this resource no longer exists so never check it again.” Unfortunately disobedient bots would ignore both 410 and 420.

410 ’til the end

410 compliance is needed. It’s important. It would save tons of time, energy and other resources that could be spent on useful things. Like all of the legit, non-410 resources. Unfortunately, the server response 410 “Gone” is become impotent. Big search engines like Google and Bing completely ignore the 410 response. Your site can declare a resource 410 Gone all day long and search engines, bots, scripts, and most everything else will continue to request the Gone resources like they were the hottest thing going. It’s… frustrating.

Or maybe 410 compliance is not the solution. Again, just thinking out loud here. Maybe we need to find an alternate way to universally and completely remove and declare missing dead URIs. Because they continue to grow at an exponential rate, wasting exponentially more resources as time goes on. It’s a quietly destructive process that leads to a broken, 404 Web.

About the Author

Jeff Starr = Web Developer. Security Specialist. WordPress Buff.