If you’re a Software Engineer, or adjacent, and are struggling to use AI productively, this is for you. By the end of this, you will have a new tool necessary to partner with AI to write great code. That technique is having a way for the AI to test and or verify whether or not its changes worked properly, completely and end to end.

AI is hugely productive, truly, it is an amazing tool, but like most new tools learning how to use it effectively is the crux of the matter. Just like you can hold a hammer wrong, using the ai incorrectly can lead to less than desirable results. But AI was trained using a lot of context, and with someone or something verifying its work, and we in fact as engineers also have a whole infrastructure around checking if our own work is functioning properly. From Down checkers on the high level, to integration tests on the other end, our infrastructure has already been built to account for common problems between the keyboard and chair. We can use these same techniques and tools in order to make sure AI produces code that works.

The single greatest technique that has given me the greatest gains is giving the AI a way to objectively check if its changes worked properly. This can take many forms, but for webapps, for instance, have the AI spin up an actual browser and check to see if the changes did what they were supposed to. If you do not have a way to get the AI to test its changes you are almost guaranteeing your changes won’t be implemented correctly. The gains from simply giving AI a way to test your changes cannot be understated.

For example the other day I was trying to get Airplay to work on an app I was building, airplay requires https, and though I had https on the live server my local clone didn’t. I was initially just prompting and planning and then pushing it up and checking to see if the server worked properly, the airplay icon was there but clicking on it, and choosing the device didn’t start the actual playback. So I slowed down, instead instructed the ai to set up https locally, and use safari with devices on the local network, to make sure it worked. Only after finally giving Claude this testing environment was I able to actually get the airplay to work, having never ever done airplay myself.

As another example I had a design mockup that was just an image and I needed to implement it with AI. I figured I could just use ImageMagick to compare a screenshot of the implementation, and check how similar it is to the design, telling the AI to make it so it is above 95% similar with only 5% blur applied to the images. With this I was able to 1 shot AI to get it to implement that design for the pages just doing this 1 by 1 for each page in the design. This particular implementation would’ve taken me 3 days, I was able with AI to get it done in 2 hours.

There is a lot of anecdotal evidence out there too that this works well even with other people, There are numerous examples of people using AI to rewrite a codebase , and one of the ways I have seen it done is, they copy all of the tests from the original repo, and then they check and make sure that their new code passes all of the tests. This allows the AI to confirm its own work and gives you an objective meter to how good the code actually works. I have even seen AI researchers use a lot of the same techniques (End of the 2nd paragraph under “Depthfirst’s Security Agent”) ,

Over and over again, I have seen how giving the AI the proper way to test it’s changes is the only way to get it to properly write code. And when you really break down how our job used to go, I would never ever present any work without properly going through and thoroughly checking it end to end, because as familiar as I am with the code, all it takes is 1 character off in the right place to where it won’t work. Why would AI which is essentially just copying usnot need to have at least this, and more?

If you don’t give it a way to confirm its changes, the AI is essentially going in blind and assuming that its changes work properly, most of the time confidently too. Luckily for us it’s actually pretty easy to figure out how to give the AI a testing scenario, simply ask yourself what you would do in order to check whether or not those changes worked properly, and tell the AI to do that. It is also imperative that you check things how the end user will interact with it, If you’re building an API, tell the AI to separately just use curl/wget to make requests, if you’re building a web app use playwright, if you’re building a game for mobile be sure to have a mobile emulator ready and waiting.

LLM’s are so powerful nowadays, and tools so numerous, that all it really takes is that you just tell it how you would test your changes, or what the desired end goal of the changes should be, and there will probably be a tool you can use to make sure it works, from curl, to end to end testing, to great tools like ffmpeg, ImageMagick, Playwright, and more there is probably a way to test your changes, you just have to figure out what the tool is to do so. If you don’t know what tool to use, or think there isn’t one, you can ask the AI for some suggestions on good tools! Or have the AI write the tool it’s self for you! Truly amazing times we live in.

Originally published at https://khalah.substack.com.

One of my favorite tricks for my CLAUDE.md is using a journal to keep track of every change the llm is making to the codebase. And yes I absolutely mean every change. Think about it, we try this with our commit messages, with our documentation, and comments, and yet it takes months for a new engineer to be productive. They are not a perfect solution, even the most well commented repo will tend to miss things, or forget things. However what if we didn’t need to have this limitation anymore, and we could simply document every code change? With the advent of AI, this is now possible.

The CLAUDE.md line that makes an AI_JOURNAL.

```
- CRITICAL: After any changes to the code have been made, be sure to write an entry including date, time, and the intention of the code changes to the file AI_JOURNAL.md. Always add the newest entry to the top or beginning of the file. Always add the list of files that were changed, and what functions or variables were added, removed, or changed in the files.
```

With this one line in my CLAUDE.md I’m able to have a ledger of all of the changes AI is making. I am able to much easier verify where bugs might’ve started, or even what the AI was thinking when it developed a feature. It has been very helpful to me when I need to understand why I have a regression, and also in directing the AI, without having an overly large context being used. When I want to take a feature further with the AI_JOURNAL I tell Claude to look through the journal for hints and reasons as to why something was implemented the way it was.

I have used it in several projects, and especially when it becomes mature, its value cannot be understated. Try it in your own codebases, and instead of explaining to the AI why you built something some way, have the AI refer to the journal. When you review a PR, look at the AI_JOURNAL to get a summary way better than any commit message you’ve ever seen. Happy coding!

Some Example Journals: Very early stage erp, mid stage auto job applier

In the case that they need ways to check their work they are just like humans, and lucky for us we have a lot of the infrastructure already built to check code quality, test coverage, and many other things. We have duplication checks, linters, tests, and even other ai to check if what we built is what we want.

Here are some of my favorite techniques for working with AI:

• Implementing Designs: Use ImageMagick and the compare feature of it to compare screenshots of actual pages to designs.
• Implementing Frontend architecture: Use Playwright to click through the website with a manual test to make sure that whatever feature is being built is actually working properly. I also save a few of these tests for end to end regression testing later.
• Implementing a 2d game: Had the AI write code where it can write out entire sequences of button presses inside of a 2d game repo to make sure that new features don’t break old ones, etc.
• ABR streaming: Implemented a query parameter that showed the bitrate, and allowed for selection, then had the ai choose the bitrate and then check to make sure it was streaming the right one.
• Video Subtitles: Told the AI to load at a specific point and told it what the text should be there if the subtitles are working.
• Implementing duplication checking: I used jscpd, to ensure the code base has less than 2% duplication.

I don’t even ask the AI to do anything unless I can tell it how to check its own work. Nowadays I find that it is wholly unreliable, just like humans, when asking it to do work without giving it a solid check. Think about it if you’re a developer you likely work with a whole team of people who check your work, your product manager, your designer, your superior, and your boss, why would the AI, which still isn’t quite as good as us, need less checks?

I beg you try some of these techniques, and come up with your own! The next time an LLM doesn’t complete your task, ask yourself, did you give it the proper criteria to check its work, and you’ll likely find no. Ask yourself how you would check your own work, and use that as a basis. Shucks nowadays you can even just tell it, come up with your own manual test to check your work. What a time to be alive!

As much as we may feel that the code we write is beautiful, and it may be, the point of code is not to look beautiful, the point of code is to be maintainable, else it gets thrown away and someone else has to redo your efforts. The beauty of the code really is a side effect of, in my opinion, a cognitive bias; we as people tend to believe the things we make are beautiful.

We all think when we’re writing our code, and finally solving some problem that has been plaguing us, that our code is great, but how, often not even 6 months later do we come back to it and think, this is less than perfect? LLM’s writing code have the power to remove us from the trees and allow us to see the forest, scrutinize more, and I think, as a whole, make more functional code. How many of us are a lot more harsh (justifiably so) on LLM code than we would be on our own?

I love writing code not because code is an amazing thing to write, but because of the real world problems you can solve writing code. Sometimes it seems we become so in love with the art of writing code they forget the art isn’t there for arts sake, it’s there to evoke emotion, or in this case the code is there to make someone’s life or work easier. Unlike code though art has little to no intrinsic value, however, even if the code is crap if it solves a problem for people it has value. Losing focus of that ultimate goal is losing focus of why code exists in the first place.

Then there are those that claim AI slop, and I have to say, they’re right; ai does make slop all the time, but I will say, so do humans, and they are made in our image. People who have never built software will probably struggle, but also they will solve problems they have in their lives, and likely problems we have in ours as well. Those unfamiliar with software will make many apps, and crude attempts to validate the possibility of their idea, and they will want to expand, but they will find eventually, their apps unmaintainable and we will profit. The real skill is in figuring out how to translate real world problems into things that can be solved with software.

Any good tool that will be able to create as well as destroy if it is versatile enough. But if we can learn to tame the beast, and control it properly I think we will go very far. I myself have recently built apps in weeks that would’ve taken me 1 year or more before. Just like junior engineers I find llms full of energy, but needing guard rails in order to be the most effective.

With some great and thorough testing, that includes coverage tests, a sensible CLAUDE.md that tell Claude ways to solve problems you’ve seen it encountering, and slowing down making Claude make a plan that you read, you can get some amazing code out of Claude. Things that yes, with time, one could accomplish, but with this i just have to watch it like an overzealous engineer that has a history of taking down production. I should be able to add tests to make that almost impossible.

There has honestly I don’t think ever been a better time to be an engineer, people used to have physical punch cards, write machine code, and read entire manuals to learn how to use a command. And just look at us now, there was a time I couldn’t be on the internet while my mom was on the phone. Now we have code that writes code, better than most humans, we truly live in amazing times.

If I remember correctly, taxi drivers in London didn’t like it when GPS maps with directions came out, and elevator operators didn’t appreciate driver-less elevators. Will we simply be another point in history of people falling to the fear of technology? Probably, for now, but eventually, we’ll see.

Sometimes software engineers forget that code isn’t made for the computer, it is made for the human. As such we tend to write code for machines, instead for future people (including ourselves) that will be manipulating the software. Instead of writing code so that people understand, by being verbose, we write “clever” code so the computer gets .01ms of speed. Most of the time the human time spent adjusting the code will be way more costly than the time that is lost by the speed gain.

Sometimes the code has to be complicated, all codebases I have ever seen, there is always some portion of the code where the “magic” happens. All science, however, when not properly understood might as well be magic. Be sure your code is always understood (well commented). Make it paramount that the “future you”, and anyone else will have no problem following the code.

Just like how the code itself is for the human, so is the resulting software that the code orchestrates. Always remember that all problems you’re trying to solve at their core are the problems of fellow humans, and thus ask yourself; how you can reduce the complexity in their lives while also introducing a new tool (the crux to ux development). Most of the time the answer is to use the tools people are already familiar with, and to mimic patterns they already use. This is why, I suspect, frameworks like React are so popular, it requires minimal specialized knowledge.

Remember that the computer is the machine, and it should be doing the heavy lifting, not the human, so keep your code simple, and more importantly ask yourself and your team, does the code even need to be written?

Why is that though? One would think dealing with such an objective machine some of the objectivity would rub off, and some would say it does, in everything apparently, but the code. I can’t say for sure, but I think this is due to the IKEA effect. The IKEA effect is one in a list of many cognitive biases, and it goes as follows

“The tendency for people to place a disproportionately high value on objects that they partially assembled themselves… regardless of the quality of the end result.”

Anyone that builds software, or has even seen people build software will know the effort that a team, or an individual has to put forth in order to come up with a good product. One would even go so far as to say the effort and mental load are far exceed that of assembling Ikea furniture. To make matters worse there are almost always no instructions, and if they do exist they’re just guidelines or fragments you have to then use to come up with the solution to your problem.

It makes me wonder how many millions are lost on developers feelings, and how many other professions might also suffer from a similar affliction. This could explain a lot of bad rap albums.

Recently I’ve had a front-end app to build and I figured I would use react with react router, I mean hey they’re built to go together, and react is so awesome how could the router not be? Never have more naive words uttered from my oral orifice.

The first gotcha was when you use react router and you have nested routes you have to render this.props.children manually, wtf? I thought the point of a router was to render the pages without so much from me as the react element and the route. This if course has a work around, just don’t use nested routes. But I like my code DRY wet is for the beach, or procreational procedures.

The second gotcha was not every time something renders will it hit the componentWillMount function. It only hits it when the component mounts for the first time. And yea I get it it’s faster to cache the object and not have to initialize it again and rada rada rada. I don’t really care, code should be simple for the developer to understand, and when I have to make another init function that has to be called sometimes when componentDidUpdate is called that adds unnecessary complexity to the code and makes it more confusing, and for a few milliseconds time off? I gander that if that second mount is slow in your app you have other problems to worry about.

So I went and found Navigo, a simple router based on that post on hackernews that goes something along of the lines of “A modern js router in 40 lines.” It’s not 40 lines anymore but it is still sexy, simple and approachable. I love Built a simple wrapper for react apps with it, rendered my within a component, and bam, simple classes now. componentWillMount always gets called when that page is going to load, and no more any need for early returns with this.props.children.

It got me wondering if Netflix will one day pull a YouTube and throw commercials in, first start with 10 second ones you can perhaps skip a few seconds in. They’re slaves to their stockholders, who always want to the revenue go up.

Especially since they have so much original content and it seems like they’re going to keep on this path of making their own shows for the foreseeable future. Why not just throw some extra revenue from advertisers?