If you go back to the first talk ever given on webhooks, it opens on the command-line. Specifically the Unix shell, focusing on one of its defining features: pipes. The idea was that pipes brought a new level of compositionality to programs, and webhooks could bring a new level of compositionality to web apps. Perhaps you could say I was trying to bring the spirit of Unix to the web.

With this last release of Wanix, I’m at it again. This time with the successor to Unix, a little known operating system called Plan 9 from Bell Labs.

Plan 9 has been on my mind for quite a while. In fact, around the time of that first talk on webhooks, the team behind Unix and Plan 9 was being re-assembled to create the Go programming language. I pretty instantly fell in love with the Go worldview, which turns out to be an outgrowth of the Unix and Plan 9 values of simplicity, pragmatism, economy, and ultimately compositionality.

Like Unix, the Plan 9 environment is really made for programmers and system operators. I’ll leave a deeper explanation of what makes Plan 9 great for another post, but I do get into it a bit in the demo video for Wanix:

While I wanted to incorporate Plan 9 ideas into Wanix from the beginning, it wasn’t until we rebuilt it from scratch with that intention that the magic really starts to come through. That’s what this preview release is about.

Wanix is a whole new beast now. It’s no longer a singular computing environment that runs in the browser. It’s now a primitive for building environments in general. The demo shows a shell environment, but this environment is not the point. It’s just a way to bootstrap Wanix so you can use and explore it interactively.

The point of this preview release is to get this primitive out there. I have my uses for Wanix, and I plan to share them with the final 0.3 release, but until then I wanted to let it all percolate. Maybe inspire people to get creative with their own use cases.

Here’s a quick rundown of Wanix features in this release:

Plan 9 inspired design

With the original intention to enable exploring Plan 9 ideas on modern platforms, we’ve ended up with a radically simple architecture around per-process namespaces composed of file service capabilities using similar design patterns to those found in Plan 9.

Single executable toolchain

The wanix executable includes everything needed to produce Wanix environments.

Filesystem is the only API

The Wanix microkernel is now simply a VFS module with several built-in file services exposed via a standard filesystem API. This ends up making the module itself a file service.

Built-in Linux shell

Using the built-in file service primitives, Wanix can bootstrap a Linux-compatible shell based on Busybox. It comes with several helper commands for working with built-in file services.

Tasks and namespaces

The Wanix unit of compute is a task, which is equivalent and compatible with POSIX processes but allows for different execution strategies. Each task has its own “namespace,” which is the customizable filesystem exposed to the task.

Core file services

Wanix includes two singleton file services: one to manage tasks (similar to procfs), and one to manage “capabilities” which are user allocated file services. Built-in capabilities include: tarfs, tmpfs, and loopback.

Web file services

With future non-browser deployments in mind, all web related file services are packaged in a web module, which is also built-in but not considered core. This module includes these work-in-progress file services:

• opfs: For working with the OPFS browser storage API
• dom: For inspecting and manipulating the DOM
• worker: For managing web workers
• pickerfs: Capability wrapping the window.showDirectoryPicker() method (not available yet in Safari and Firefox)
• ws: Capability for working with WebSocket connections
• sw: For configuring the service worker, which is used by the system now to cache all resources needed to run Wanix allowing offline usage, as well as exposing virtual URLs to the root namespace.

Go programmers might also appreciate the filesystem toolkit we’ve been working on since before Wanix. It builds on the fs.FS abstraction in the standard library and gives you DSL-like utilities for defining virtual filesystems like the file services in Wanix. More on that in a dedicated post as well.

So far, the feedback has been really positive. I appreciate everybody taking the time to process it. There’s still lots to do. Wanix is itself its own universe, but it’s just one layer of the Tractor project. After a little vacation I’ll be back to continue work on both fronts. As usual, I’d love help.

Speaking of help, shout-out to Joël Franusic for the help and support. And as usual, big thanks to my GitHub sponsors for making this possible.

Originally published at https://progrium.xyz on May 5, 2025.

In this week’s demo I built a background application using HTML and Apptron. It may seem weird to involve a webview at all for a background app, which is why this was not a use case I had planned, but I was pleasantly surprised to discover how this turned out.

In the last demo, we made an app indicator background utility as a shell script. This week we re-create it using JavaScript in an HTML file. With apptron build we can make it into a standalone executable. This workflow was originally intended for quick webview apps, but because we can let the JavaScript of the page run while the window is hidden, we effectively get a background application.

Apptron wasn’t made just for webview apps. I wanted to expose other native APIs that let you hook into more pre-built UI elements the operating system provides. For now this covers the common ones like app indicators, dialogs, and desktop notifications. In the future I’d like to expose APIs to let you add new items to system menus, control panels, and other places the OS lets developers hook into. These are important to make scriptable so you can easily leverage them for simple workflow hacks. When it’s easy to throw that kind of thing together, you don’t need to find “an app for that,” you can just imagine what would work for you and make it happen.

If this is also a world you want to see, join Apptron early access and help us get this thing out. See you next demo!

-jeff

Originally published at https://dev.to on August 19, 2022.

In this demo I show off the rest of the Apptron CLI, which exposes most of the cross-platform APIs as commands. These commands make great shell scriptable utilities, and you can see how they’ve been designed to facilitate this. By the end, we’ll have created an app indicator (systray) icon and menu that’s created and driven by a shell script.

There is a whole class of power users that doesn’t always want to take the time to “write a program,” let alone learn how. Since the functionality is right there, why not expose it in a way that’s useful for these folks? Sysadmins and devops engineers often spend most of their time at a command prompt, so this feature goes out to all of them.

It turns out having a CLI interface to the APIs also makes it easy to do basic regression testing for the project. It also provides a kind of accessible listing of functionality. And! It helps with our polyglot primitive goal, working as a stopgap before having native library support for a language, since nearly all languages can “shell out”.

As I mentioned in the video, this is a strategy I like to use when making generative primitives: maximize utility by providing multiple ways to access features. Like being language agnostic, this helps meet users where they are, and reach users you might not otherwise. There’s also often a dominant context people think of for some functionality, so showing it in a new or surprising context helps break people out of existing thought patterns and imagine new possibilities.

Let’s see what possibilities open up with the next demo! In the meantime, join early access to explore Apptron on your own.

-jeff

Originally published at https://dev.to on August 5, 2022.

In this demo I use the Apptron build command to create an executable from a single HTML file. No setup, no configuration. If you want to change how the window looks, you can add a meta tag to your HTML file. It’s not only simple, it’s fast.

The way this works is actually similar to the last demo where we built a simple Go application to host our webview app, except Apptron is building it for us. Unlike all other commands of Apptron, this build command does have a dependency, which is the Go compiler. Luckily, this compiler is also a self-contained tool that’s easy to install on any platform.

At some point, I’d like to try embedding the Go compiler with Apptron, but this does introduce some complexity I don’t want to take on just yet. However, if anybody is interested in making this happen, please join early access so you can help us make Apptron great.

Speaking of help, this demo avoids talking about something that’s somewhat necessary on modern platforms: signing. If you’ve ever downloaded an executable on Windows, I’m sure you’ve gotten a giant alert asking if you really want to run it. This is because it’s an unsigned executable, which means it can’t be proven to not be tampered with by a non-author. Platforms are getting stricter and stricter about this.

On Mac, you have to not only sign but notarize your executable before another Mac will be allowed to run it. This is a tricky process that can be done on your own with an Apptron built executable, but I have preliminary tooling to help with the process. The same tooling can also bundle up your executable into a Mac app bundle.

However, I need help getting these last mile features in. So please consider joining early access, not only to try Apptron, but if you have the experience, to help us finish these core features before it debuts as a public open source project.

See you with the next demo!

-jeff

Originally published at https://dev.to on July 21, 2022.

In this demo, I use the Apptron API from Go to build a simple birthday card application defined by HTML and CSS. I also explore the architectural approach of Apptron compared to directly working with platform native APIs.

To expand on the demo, in this post I’ll discuss some of the motivation behind this project. There are a few reasons why I made Apptron.

I’ve always loved the option of using HTML5 in webviews to build cross-platform native app UIs. Native controls definitely have their benefit, but become tricky to do consistently across platforms. Rendering the UI the way a game would with a GPU accelerated render frame is consistent across platforms, but usually means you have to invent your GUI paradigm from the ground up. Using a webview gets you general consistency across platforms, while also getting a well known paradigm for building user interfaces. It also comes with the largest ecosystem of frameworks and pre-built components that can generally be used together, and is constantly improving as the web evolves. It’s not always the best approach, but it’s a nice option to have.

Using a webview for desktop UI is not new, but was not a well known practice until Electron. I’m pretty sure significant chunks of UI in Windows as far back as Windows 95 were actually webviews. Recently, I was surprised to discover some parts of macOS today are actually using webviews. But Electron popularized it by focusing on the cross-platform benefit and riding on the initial success of Node.js, empowering JavaScripters to do more than they ever could before.

Today, Node.js is considered by many, including its author, a bit of a mistake. This is a complex thing to explain, but for me it has a lot to do with the culture of dependencies and complexity in that ecosystem. So if you explicitly don’t want to use JavaScript, or maybe just avoid Node.js and NPM, you don’t have much choice when it comes to easily making webview based apps.

Luckily, we’ve been seeing alternative frameworks for a couple different languages recently. Unfortunately, any language-specific framework is limited to that language. This has always frustrated me as a polyglot programmer. Broad approaches to software design shouldn’t be silo’d off to specific languages, even though obviously you have to pick one for an implementation.

It’s also frustrating to see how complex these full service frameworks become. At their core, they’re just re-exposing platform APIs, but they end up being full-fledged platforms of their own. Keep in mind, it’s not just webviews and windows, there are a bunch of platform APIs for features across all major platforms that you might want to use. Native menus, dialogs, notifications, etc. If you wanted to use one of these in a script or simple program, Electron immediately feels like overkill.

Depending on your language, you may find one-off libraries for these cross-platform native features. A library for webviews, a library for desktop notifications, etc. The problem there is when you want to combine them. Not only are these libraries usually going to be fairly inconsistent in their design, if they even exist, but because of the nature of platform APIs, they often need to set up an event loop and take over the main thread. This makes them difficult if not impossible to use together.

With all this in mind, it seemed the only option was to build a language-agnostic bridge to cross-platform APIs. Apptron uniquely sits somewhere between a full framework like Electron or Tauri, and small one-off libraries you might find for specific APIs. It can be used like a library, a tool, or a micro-framework, but it’s small and self-contained. This is why I describe it as a software primitive or building block.

I hope Apptron is compelling, and I’d like to invite you to try it out in early access. In the meantime, more demos are on the way.

-jeff

Originally published at https://dev.to on July 8, 2022.

Apptron is a software primitive that lets you drive native platform APIs using the technologies you’re already familiar with, enabling cross-platform webview apps and more.

Usable from any language, or even the command line, Apptron gives you webview windows and common platform APIs for your simple scripts, homebrew utilities, or full applications. Building cross-platform programs that leverage native functionality (menus, dialogs, notifications, global shortcuts, etc) has never been more accessible.

These APIs are packed into a single executable that can be used a number of different ways, making Apptron a simple but powerful software primitive.

Zero Config Native Apps from HTML

Turn an HTML file and any other web assets into a self-contained, cross-platform executable in a single command.

Language Agnostic Building Block

Use the Apptron APIs like a library from any language without C extensions or tricky event loops.

Shell Scriptable Native APIs

Use platform APIs from CLI commands to break shell scripting out of the terminal and into the desktop GUI.

These features and more will be showcased in a series of demos coming out over the next few weeks. Given all the different ways it can be used, Apptron is something you really have to see in action. Catch their premieres on Twitch, or subscribe on YouTube so you don’t miss them.

Early Access

In the meantime, we’re opening up Early Access via the Apptron website:

https://tractor.dev/apptron

By joining early access, you’ll be invited to the currently private GitHub repository where you can try Apptron out, submit issues, and help bring Apptron to a stable 1.0 release and become public open source software.

Help Wanted

I’ve been using Apptron in some form with the rest of the Tractor System for over a year now, but to make this a solid standalone primitive I’ll need your help. Please consider joining early access to play around with it and give feedback.

Anybody interested in contributing or helping drive this project should definitely request access and get involved. This is one of many projects I have in the pipeline and I can only do so much.

Thanks

A huge thanks to my sponsors, who I depend on to fund this early R&D work on the Tractor System. As long as it’s sustainable, I look forward to releasing more powerful open source primitives.

Excited to tell you more next week with the first Apptron demo!

-jeff

Originally published at https://dev.to on July 1, 2022.

Creating software is too complicated. Our options are overwhelming, our tooling is too bloated, and things rarely ever “just work”.

8 years ago, the late Joe Armstrong, co-creator of Erlang, gave one of his last talks at the 2014 Strange Loop conference. It was titled “The Mess We’re In” and talked about the increasing over-complexity of software. “Software entropy increases with time,” he said, “we need to reverse entropy.” He said in his conclusion:

“Computing was about controlling complexity, and we have failed miserably.”

An echo of what Djikstra said in the 90s: “It is time to unmask the computing community as a Secret Society for the Creation and Preservation of Artificial Complexity.”

Alan Kay, another computing pioneer, has a great word for this “artificial complexity”: complication.

“The amount of complication [in software today] can be 100s, maybe 1000s of times more than the inherent complexity.”

For 25 years, Kay has been giving talks saying, “The computer revolution hasn’t happened yet!” I believe that this is true, in part, because complication has gotten in the way.

The risks with complexity, complication, and bloat are also only growing. It’s 2022 and we’re still deploying software with logging libraries that let attackers run arbitrary code. We barely understand our own software systems, let alone the tools we use to make them. And yet, underneath this mess is probably the most powerful and constructive tool humanity has ever seen.

Building software today could be as fast and intuitive as we made desktop publishing by the 1990s. Suddenly designers could sit with a client in front of the computer and just execute an idea, making iterations together in real-time. Watching the designer use these intuitive tools, the client might think, “Oh, I could do that”.

To get there with building software we need to reduce the complexity, and especially the complications, of making modern software. It’s not just about user friendliness or making the process more visual. And it’s not about adding more. If anything, it’s about revisiting the idea of doing more with less.

Recently, tools falling under the label of “nocode” have shown the demand for non-programmers to be able to just make what they need, with “no code”. It does sort of hint at what a photoshop of software might be. But every nocode tool is a facade, another layer of complication. They exist for a particular market, offered as a service, with no path to a more general means of making software.

These tools have their place, the same way image macro generators have their place when Photoshop and Canva exist. I think if “nocode” tools could really save us, you’d be able to build a nocode tool with itself. They lack the true generativity of programming.

That said, I am all for anything that can be used as a building block in a software system. Whether it’s a library or a web API or a tool repurposed because you like its workflow. The less you have to build yourself, the better.

My experience has shown me time and again that there is a powerful strategy or guiding principle in what I call “generative building blocks.”

This philosophy applies to more than libraries and web APIs. I design everything as a building block, and everything out of building blocks. This is about more than just modularity, it’s about high-leverage generativity.

I’ve worked on a number of projects and technologies trying to bring out this particular generative building block approach and there are a couple of standout hits that I’ll use as examples.

In 2006 I was working on a hosted source control web app (like GitHub) trying to expose repository hooks. These are shell scripts triggered by your local source control tool that can be used to customize and extend your workflow, almost like registering a callback for an event. Running user code in a web app, especially shell scripts, was not safe or practical. However, imagining if somehow it was, a flood of possibilities hit me. If only web apps could have extension points! Without over formalizing it, I called this “webhooks” and started giving talks about what we could have if we added webhooks to our apps. Throwing a PHP script online was an easy and common activity for web developers then, so functioning as “hook” script with a URL, you just needed to have web apps let you register it and then hit it on certain events.

Webhooks did end up making web apps more extensible and integratable, increasing their value and potential as building blocks. It also allowed for new kinds of API building blocks to emerge like Twilio, where the initial core product was built around webhooks to integrate with telephony and SMS. Webhooks also started creating a demand for what would later manifest as AWS Lambda for event handling, which quickly inspired “serverless” architecture. The concept of functions as a service was something I was pioneering in 2009 because of webhooks.

Later I found myself helping to prototype and conceptualize Docker, which turned software into a kind of standard unit, hence the shipping container metaphor. Although many interpreted it as a lightweight VM to throw whole systems into, I was in love with microservices designed to be run in tiny containers, configured by APIs instead of files, acting more like a little appliance than traditionally managed software. More like a simple building block.

Docker itself became a building block, opening up a whole new generation of platforms and developer tooling. One of my initial use cases for it was as a primitive to create a simple, hackable version of Heroku. Sure enough, using Docker and a few other components I made, I was able to “throw together” a simple Docker-powered Heroku alternative in 90 lines of Bash. This was called Dokku.

Dokku was a bit of a revelation. An expensive, top-tier platform valued in the hundreds of millions of dollars recreated with the right primitives in a weekend as a free Bash script…

I want to have the primitives to create more software like Dokku — extremely simple, hackable alternatives to the world’s best software. The resulting software isn’t really the important part, though. The resulting building blocks are. Why use Figma if you could somewhat easily throw together your ideal figma?

Building simpler alternatives by disaggregating into new primitives expands the adjacent possibility space of what can be thrown together, while also simplifying and commoditizing what it takes to build great software. It actually is a sort of reversing of entropy! Repacking down our abstractions, engineering for generativity.

What I love is how “building blocks” brings to mind a sort of ideal vision we have of building software. Like LEGO, we snap together little components to realize something new. But this metaphor also points out how far that ideal is from reality. LEGO is fun. Building software really isn’t most of the time. And I think trying to express the reality through this metaphor is helpful:

While ALL our software is made of many small bricks or components or modules, most come to us fully assembled with the bricks glued in place and painted over. We can’t change it or even get a sense of how it was composed. Most apps come as an opaque “turn-key solution” to a specific problem because… that’s how you sell things.

Perhaps there’s nothing wrong with that, but we can at least try to find ways to build more software like LEGO kits, where the focus is on good, re-usable, re-combinable building blocks and showcasing what you can do with them. In the long term, I believe this is the way that provides the most value.

A few years ago I realized that I need to build a single cohesive system that lets me express all these ideas, and really show the potential I’ve been talking about. I need to create my own LEGO system for software that would be an umbrella for all my work and let me build the way I want to build, while being at the very least a useful example to others…

So I’m announcing the Tractor System. It’s not one project, but a system of projects building towards a malleable software environment and ecosystem. It is my take on a “Photoshop of software”.

Tractor is a system of building blocks and pragmatic ways of building centered around simplicity and generativity. It’s not a new language, it helps glue languages together. It’s not just an app builder, it’s a systems playground. Much of the tool is made with itself, out of the same primitives you use in it. It’s designed to be as simple and understandable as it can be, top to bottom, inside and out.

The aim is to blur the line between user and developer. The computer is a computer because it can be programmed. The rushed, over-productization of shrink-wrapped software, pandering to enterprise organizations and the computer illiterate, has created so much complication it would seem we NEED big companies to build our software. But we don’t. I’m constantly surprised how simple things really are or can be. With Tractor, eventually we’ll just have operators: people that can use the computer in its full potential to solve problems, create systems, play, learn, work, and have fun.

Some say we live in a post PC era, but the original idea of the PC, the personal computer, is and will always be significant because it puts a magical form of leverage into the hands of individuals for their own use. But instead of this becoming more obvious with time, this generative power has been centralized and sterilized into “pop computing”. Today we consume and are defined by software that benefits engines of capitalism more than engines of equity. You could say, instead of being able to program our own leverage, we are being programmed by those that can.

Luckily, underneath all the apps, social media, and enterprise “solutions”, there is still an imagination compiler. This is what I focus on. This is what I’d love to dust off, polish up, and see where that takes us in the proper hands.

Even if that doesn’t happen, I’d love to just get some better building blocks. Perhaps just find simpler, more enjoyable ways to build software.

As an independent creator, I depend on people like you to help fund this work, and I’d love you to come along and help make the Tractor System a reality by becoming a sponsor.

Onward!
-jeff

Originally published on progrium.com and YouTube on March 15, 2022

Hello everybody!

Since starting work under Progrium Technology Company, I’ve had a single objective: to build a system called Tractor. It’s what I’ve been slowly approaching in my dense 22 year career as an independent programmer. The Tractor System is still hard for me to describe at this stage, but its goal is to make building production-grade personal software systems like building with Legos: fast and fun.

After a difficult year heads-down in 2020, I started posting here in January with the intention of sharing and officially releasing components I’ve been working on that are building up to Tractor. Besides warming up for when I’d eventually talk about Tractor, I wanted people to see how I make independently useful building blocks towards an ideal. That’s a core part of the Tractor philosophy.

Let’s review what I’ve shared so far, and then set up what’s coming in the next 6 months.

The first big release of the year was the macdriver project, which got a significant response on Hacker News. It gives us Objective-C and Apple framework bindings for Go, letting you build simple Apple apps entirely from Go. That wasn’t possible before, so it was a little exciting, but it was early. The native Go APIs included for commonly used Foundation and Cocoa classes were far from complete. They could still be used if you knew what you were doing, but with an ideal of total coverage of Apple frameworks, that wouldn’t be enough. There were also unresolved issues just deciding how to best manage memory and pointers, which I knew would fall on me to figure out and take some time.

Luckily I’ve since gotten a colleague involved in the company, and he’s been helping push macdriver towards a real beta. Part of this was made possible by a project I spun out of macdriver and posted about, though again in a very early state. That project was macschema.

With the macschema toolchain, you can generate API schemas about any Apple framework, class, function, etc based on their documentation and header declarations. This is useful for us in generating framework bindings in macdriver, but it would be useful for any project doing similar work, such as bindings for other languages. This sort of approach will play a big part in how Tractor will integrate with “whatever we want” down the line.

Along the way, I released a few demos for macdriver to make clear the possibilities and provide reference examples. One of them I spun off into a standalone project called topframe, which is also serving as a test bed for how we are doing cross-platform support. More on that in a moment.

Unrelated to macdriver, I released a protocol called qmux with a post explaining why this (but really any) muxing protocol, including and especially QUIC, is such a great primitive for network programming. The project came with two implementations, Go and TypeScript, with more on the way. The post about it was the first explainer article I’d done in a while and people liked it. This was important because the idea behind it was really more valuable than the protocol itself, but regardless we still need implementations of it to exist. To show the idea in action, I built a 130 line version of Ngrok, which turned into a great post walking through how it works.

That brings us to today, half way through 2021. Let’s talk about what’s coming.

More exciting than qmux is what we built on top of it, which is called qtalk. This is my meticulously designed, re-written-several-times network/IPC programming stack. While not that different from just JSON-RPC with pluggable codecs, its two unique features are callbacks and streams. Callbacks means its bi-directional, either side can expose or call methods. Just that alone is something rare in existing RPC stacks, but necessary for callbacks. And while streaming RPC is not new, our streams are full virtual connections, so you can stream more RPC results, arbitrary byte streams, or tunnel something else like a database connection. I’ll share more about the possibilities when it’s released.

So in the second half of 2021, we’ll see qtalk, we’ll see a major update to macdriver, and then with those two parts we can release shelldriver. This is a cross platform API to platform specific resources like windows, dialogs, menus, etc. It’s about native GUI shell integration, it’s not about having every native UI component. In fact, for app UI it really focuses on windows and webviews, a la Electron. Unlike Electron, it can be used from any language that has a qmux and qtalk implementation. The result should be a simple “Electron but as a library” that you can use from Go and other languages.

Once shelldriver is out I will be able to finally start talking about the first real piece of the Tractor system: Tractor Toolkit. I hope to have at least a public demo by the end of the year. The Toolkit is already being shown to friends and sponsors. The work I’ve shared so far is just the tip of the iceberg.

Which reminds me …

I really, really have to thank my sponsors. As things are ramping up, I’m starting to work with more people and this is all self funded. Not only does sponsorship support all this open source work, but sponsors get early access to Tractor Toolkit and see more of what I’m up to.

Also to you, thanks for reading and following along. I’ll be back soon with more releases.

Originally published at https://dev.to on July 5, 2021.

Running a local development server for your app is pretty common, but what if you wanted somebody else to access it? Maybe for a demo, or maybe to debug webhook integrations. If you’ve ever used Ngrok (or perhaps the original localtunnel), you know what I’m talking about.

Today we’re going to make a system that emulates the core functionality of Ngrok, exposing a localhost HTTP server via a public URL.

Just to make sure we’re clear on how this all works, let me explain the problem. Normally when you run a server on your computer it can only be accessed by other computers on the same local network. However, many development servers will bind to 127.0.0.1, or localhost, which means only a client or browser on your computer can access it. Either way, there are times when you might want another party to access it temporarily, but it's simply not addressable from the public Internet.

In the olden days, this could only be achieved if you had access to a server running SSH. You could configure SSH to have publicly addressable ports on that server be “forwarded” to a local port on your computer over an SSH connection. This is called remote port forwarding or sometimes SSH tunneling. Assuming SSH is configured properly, the command to set this up looks like this:

$ ssh -R 8080:localhost:3000 your-ssh-server.com

This would allow somebody to point their browser to your-ssh-server.com:8080 and get your localhost server running on port 3000 as long as this command was running. Around 2009, I noticed how useful this is (especially in the context of webhooks), but was annoyed you had to have an SSH server properly configured for it before you could even do it. Even then, I always found the SSH command difficult to remember how to use for forwarding. I envisioned a command as simple as:

$ localtunnel 3000

So I set about making this happen. My first attempt at building a system with Twisted Python failed because I didn’t quite grok multiplexing. So I ended up wrapping SSH. I wrote a localtunnel client in Ruby that behind the scenes would just run the above SSH command using a server I had set up. The only difference being that I didn't give you a port like 8080 to connect to, I wanted you to get a generated subdomain. Instead of localtunnel.com:8080, you'd get wxyz.localtunnel.com, which was unique to your tunnel as it was intended as a public service.

To achieve that, I just ran a Twisted web server with the SSH server that would figure out which port to use based on the subdomain and proxy to it. This would only work with HTTP because the idea of virtual hosts was actually invented for HTTP. Normally when you get a TCP connection you have no idea what hostname is used.

So I had something like this:

And it worked! It became rather popular. However, due to the slight complexity (Ruby+Python+OpenSSH) and the fact I was too busy to maintain the service, people started cloning it. One jackass even called his clone “localtunnel”.

Another jackass, who was less of a jackass and more my friend and coworker, decided to clone it as a way to learn Go. He called it Ngrok and decided to commercialize it. This bothered me for a number of reasons I won’t get into, but eventually decided it was fine since he really improved on the idea and would at least listen to me if I had anything to say about it.

Anyway, today we’re exploring a toy recreation of this system that we’ll call groktunnel. It removes SSH from the setup and using two libraries lets us build the whole thing in about 130 lines of Go. You’d still need a server to run this on, so it’s not as convenient as a hosted service, but this is more for educational purposes. After all, groktunnel is actually a demo for qmux, one of the libraries I mentioned. Let me explain these libraries.

What qmux does is give us a subset of the SSH protocol to multiplex many connections over a single connection. It was the missing piece of my original Twisted prototype. You can read more about how there aren’t a lot of these protocols (but perhaps with QUIC are also the future of the Internet) in my previous post.

The other library is actually by the author of Ngrok. Called go-vhost, it helps with the problem of virtual hosts. Ideally when a connection comes in to be forwarded down the tunnel we can hand it off wholesale. This way the other end can just start reading off the connection as if received directly. But in order to figure out the hostname used, we have to start reading off the connection up to the Host header. Then we have to hand it off prepended with what was read. What Alan wrote is an abstraction that gives you a virtual listener that lets you accept new connections for a particular virtual host and get the connection as if it hadn’t been read yet. A lot of this is due to the interface based approach of the Go standard library.

This brings us to the main utility function we’ll be writing for this system. A TCP connection established between two peers cannot (easily) be handed off between programs, let alone hosts. We don’t literally “forward” the connection, we sit in the middle and relay the bytes. This is how every proxy, load balancer, gateway, or tunnel works. It sort of welds two sockets together. I call this socket joining. And if you can avoid doing any protocol level stuff, building a proxy just becomes opening a connection to the backend when a connection comes in and then joining them.

Since Go models connections using the interfaces of the io package, we can use io.Copy to simplify reading off bytes from one connection into another until an EOF. A two-way copy like this that closes when finished looks like this:

func join(a io.ReadWriteCloser, b io.ReadWriteCloser) {
    go io.Copy(b, a)
    io.Copy(a, b)
    a.Close()
    b.Close()
}

This minimal approach benefits from setting a deadline on the connections in case one end doesn’t hang up. You can see we copy bytes from a to b in the background, then wait for copying bytes from b to a before making sure both sides are closed once b gets an EOF.

We’ll also define a helper to generate random subdomains. In a production service this is a big deal. You’d want an incredibly large space of possible subdomains, but here we generate 10 random alphanumeric characters using crypto/rand:

func newSubdomain() string {
    b := make([]byte, 10)
    if _, err := rand.Read(b); err != nil {
        panic(err)
    }
    letters := []rune("abcdefghijklmnopqrstuvwxyz1234567890")
    r := make([]rune, 10)
    for i := range r {
        r[i] = letters[int(b[i])*len(letters)/256]
    }
    return string(r) + "."
}

Another quick utility to define that’s common in simple Go programs is a fatal function. This helps simplify all the error checking.

func fatal(err error) {
    if err != nil {
        log.Fatal(err)
    }
}

Now we can get started in our main() which will contain both client and server modes. However, first we'll define some flags useful to both:

var port = flag.String("p", "9999", "server port to use")
var host = flag.String("h", "vcap.me", "server hostname to use")
var addr = flag.String("b", "127.0.0.1", "ip to bind [server only]")
flag.Parse()

The default hostname of vcap.me is a way to make the server easy to run locally. Since localhost doesn't allow subdomains, but any domain can resolve to 127.0.0.1, there are a few public domains that have wildcard subdomains all resolving to 127.0.0.1. If we ran this on a real server, we'd need a domain with wildcard subdomains resolving to the server IP, and we'd probably use port 80 instead of 9999.

Now we’ll start with the client, which is quite simple. It’s activated when there is an argument specifying the local port to expose publicly.

// client usage: groktunnel [-h=<server hostname>] <local port>
if flag.Arg(0) != "" {
    conn, err := net.Dial("tcp", net.JoinHostPort(*host, *port))
    fatal(err)
    client := httputil.NewClientConn(conn, bufio.NewReader(conn))
    req, err := http.NewRequest("GET", "/", nil)
    req.Host = net.JoinHostPort(*host, *port)
    fatal(err)
    client.Write(req)
    resp, _ := client.Read(req)
    fmt.Printf("port %s http available at:\n", flag.Arg(0))
    fmt.Printf("http://%s\n", resp.Header.Get("X-Public-Host"))
    c, _ := client.Hijack()
    sess := session.New(c)
    defer sess.Close()
    for {
        ch, err := sess.Accept()
        fatal(err)
        conn, err := net.Dial("tcp", "127.0.0.1:"+flag.Arg(0))
        fatal(err)
        go join(conn, ch)
    }
}

Here we set up the tunnel over HTTP since we’ll be serving HTTP anyway, and we need a way to tell the client what the generated public host is to display a URL to the user. You can see we get the hostname from a response header.

We’re manually creating an HTTP request with httputil.NewClientConn because it lets us hijack the connection. This simply means it stops processing as HTTP and treats it like a raw TCP connection. Alternatively we could upgrade to WebSocket for this, but this simply works just as well with fewer dependencies.

After hijacking we create a qmux session to start multiplexing virtual connections, or channels, over this connection. This is the tunnel. We don’t open channels, we just sit in a loop waiting for incoming channels and assume they want to be joined with a connection to our localhost server. That’s it.

Now the server. First we set everything up using our flags, including a TCP listener, which we wrap in a virtual host muxer. The meat of our server is in serve() but that runs asynchronously as we listen for vhost muxing errors until shutdown.

// server usage: groktunnel [-h=<hostname>] [-b=<bind ip>]
l, err := net.Listen("tcp", net.JoinHostPort(*addr, *port))
fatal(err)
defer l.Close()
vmux, err := vhost.NewHTTPMuxer(l, 1*time.Second)
fatal(err)

go serve(vmux, *host, *port)

log.Printf("groktunnel server [%s] ready!\n", *host)
for {
    conn, err := vmux.NextError()
    fmt.Println(err)
    if conn != nil {
        conn.Close()
    }
}

Now the heart of this system in serve():

func serve(vmux *vhost.HTTPMuxer, host, port string) {
    ml, err := vmux.Listen(net.JoinHostPort(host, port))
    fatal(err)
    srv := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        publicHost := strings.TrimSuffix(net.JoinHostPort(newSubdomain()+host, port), ":80")
        pl, err := vmux.Listen(publicHost)
        fatal(err)
        w.Header().Add("X-Public-Host", publicHost)
        w.Header().Add("Connection", "close")
        w.WriteHeader(http.StatusOK)
        conn, _, _ := w.(http.Hijacker).Hijack()
        sess := session.New(conn)
        defer sess.Close()
        log.Printf("%s: start session", publicHost)
        go func() {
            for {
                conn, err := pl.Accept()
                if err != nil {
                    log.Println(err)
                    return
                }
                ch, err := sess.Open(context.Background())
                if err != nil {
                    log.Println(err)
                    return
                }
                go join(ch, conn)
            }
        }()
        sess.Wait()
        log.Printf("%s: end session", publicHost)
    })}
    srv.Serve(ml)
}

Here we create a new listener using the vhost muxer passed in to listen for connections on the naked domain ( vcap.me). Then we make an HTTP handler that sets up a new tunnel. First it generates a new hostname and tells our vhost muxer to start listening on it. We'll hold on to that listener.

Then we start writing headers. Remember this is how we tell the client what our generated hostname is. We also use Connection: close to tell the other end we're going to be done and normally hang up, but after we flush the headers with a 200 OK status, we hijack the connection like we know is going to happen on the other end. We make a new qmux session and start a new goroutine.

This goroutine handles connections coming in from our new vhost listener. All this does is sit in a loop accepting new connections, then using our qmux session opens a channel, then joins the new connection with that channel.

Back in our HTTP handler, we call Wait() on our session, which will just block until the connection breaks. This handler has been put into an HTTP server instance set to serve on our original vhost listener for the naked vcap.me domain.

Now we can try it out. You can follow the instructions in the groktunnel README to see it in action.

This little program gives us a client and server that recreates the original localtunnel system. It showcases the use case of tunneling for connection multiplexing, but also how easily you can wire things up in Go using mostly the standard library.

If you wanted to tunnel normal TCP connections, not HTTP connections, this program gets even simpler. However, that’s an exercise for the reader. You can see the full groktunnel source in the demos directory of the qmux project. Hope you were inspired to build something cool!

For more posts like this sent directly to your inbox and to find out what all I’m up to, get on the list at progrium.com. ✌

Originally published at https://dev.to on May 27, 2021.

Towards a socket API with builtin stream muxing

Multiplexing (sometimes called muxing) has a long history of innovation in telecommunications. In 1875, Edison figured out how to transmit four separate telegraph lines over a single wire. In 1910, multiplexing technology was brought to telephony. Within 30 years the Bell System could multiplex 2,400 channels over a single cable. Then in the 1960s we got packet switching, a kind of unlimited multiplexing, which led to computer networking and Internet protocols.

I think of layered protocols and packet switching as “virtualized” multiplexing. Every layer provides a new medium or virtual transport for more protocols to operate, all over the same physical medium. Ethernet allows IPX, IP, and other packets. IP then allows UDP, TCP, and other packets. TCP allows HTTP, SSH, and so on.

The layers nearly end at TCP and UDP because these two protocols provide most of what any programmer would need to make two programs talk. This was more or less set in stone in the 1980s when Unix introduced the socket API, which has been the de facto standard interface for network programming ever since.

This API layer is where you get “application protocols” like FTP, Telnet, DNS, and so many others. TCP, UDP, and the layers below were typically concerned with just getting bytes from one point to another. Application protocols introduced more specific interactions like request-reply, such as with HTTP. Some, like SSH and DNS, became rather multifaceted protocols to serve their domain with many different flows and message types. Rarely would you find an application protocol multiplexed to provide a generic transport for another.

An early exception was SSH. One of the reasons SSH fascinates me is that it was designed for many uses and breaks down into its own set of layers and internal protocols, almost mirroring TLS, TCP, and UDP.

First, SSH establishes an authenticated and encrypted line, similar to TLS. Then you have two very generic ways to communicate over this line. You can send “request” messages that may or may not get a response, similar to use cases of UDP. Or you can establish “channels” that were bidirectional streams just like TCP.

From there, SSH defines how to use these requests and channels as primitives to set up remote shells. But besides that, you could also forward TCP connections, which worked by bridging a TCP socket with an SSH channel, “tunneling” the TCP connection through the SSH connection.

If you’ve ever used Ngrok to open a public endpoint to a localhost server, you may not know it was one of several clones of a tool I made in 2010 called localtunnel. The original localtunnel was just a wrapper around SSH, literally using OpenSSH on the server side.

So SSH was one of the earliest application level protocols designed to further multiplex its transport protocol. The next protocol to multiplex at the application/socket level was HTTP/2. It was introduced in 2015, which is still quite recent, but 20 years after SSH.

HTTP has become a new de facto API, like Unix sockets did. What was originally used to just download files into the browser has been co-opted and repurposed. As people started building more and more complex applications in the browser and with the emergence of web APIs, HTTP evolved into a sort of “do everything” protocol.

In 2012, Google started an effort to design a more efficient HTTP with what they called SPDY, which they were able to test end-to-end in Chrome without people even knowing. After proving its improved performance and cost saving benefits, SPDY basically became HTTP/2. They did a number of things to make it more efficient, but the core idea was multiplexing.

HTTP/2 was more or less HTTP, but over newly introduced multiplexed “streams” similar to normal TCP connections. This makes them roughly the equivalent of SSH “channels”. At least at one point, Ngrok used a subset of HTTP/2 as their muxing protocol the same way localtunnel used SSH.

Both HTTP/2 and SSH do more than multiplexing, and usually aren’t thought of specifically for muxing. Most library implementations don’t expose the lower level multiplexing primitives. SSH at least has clear separation in its specification, where multiplexed channels are a specific and independent layer of the protocol architecture.

After spending so much time with these multiplexing protocols, I noticed something. Most other application protocols also did something similar to multiplexing without realizing.

For example, any request-reply protocol that allows more than one request over a connection is “multiplexing” each request and reply pair. A request and response will often get a shared ID so they can be correlated as other requests and responses can be sent in the meantime. This is the heart of any muxing protocol!

I’ve found implementing RPC or any request-reply over a muxing API not only simplifies the protocol, but opens up the ability to also stream or tunnel other protocols on the same connection. You could “attach” a tunneled database connection with an RPC response.

Working with muxed streams over a single connection gives me a similar feeling to discovering green threads for the first time. Suddenly, streams between the same two points are so cheap and lightweight you can think about using them in totally different ways.

I now believe a socket API with muxing builtin is a better base layer and API for designing application protocols. I was validated when Google pushed multiplexing into QUIC.

If Google could redesign HTTP, why not also TCP+TLS? Their experience with SPDY started them down a path to reduce the overhead of TLS and TCP session establishment even more by creating a more efficient kind of encrypted TCP over UDP. Built into this layer is stream muxing that would not suffer from head of line blocking.

It’s taken nearly 10 years for QUIC to be refined and adopted in the wild and we’re basically there. There’s even a new browser API in the works called WebTransport.

In the meantime, I started designing protocols on the assumption of having a muxing layer directly beneath. Eventually, that would be QUIC, but as a stopgap, what would it be? Remember, there haven’t been any standard TCP muxing protocols.

I considered what Ngrok did with a stripped down HTTP/2, but then I realized… SSH has been around a lot longer. SSH is known to work on nearly every platform and is trusted by basically everybody. Best of all, the SSH muxing layer is simpler and more straightforward than HTTP/2.

So I literally took the muxing chapter of the SSH spec, cut out a couple of SSH specific identifiers, stripped it down to just the channels, and I’ve been using it for the past 5 years. It’s called qmux.

The great thing is that even with QUIC available, qmux can be used over any streaming transport and provide the same foundation for higher level protocols built on muxing. Whether the transport is WebSocket or stdio, as long as it can reliably transport ordered byte streams, qmux drops right in. Protocols built on it will translate easily to QUIC.

I’m only releasing this now because it might not have made sense without all this context or more awareness of QUIC. The banality of a brand new subset of a 25 year old protocol does not escape me, but it’s the shift in perspective that brings potential. In that way it’s similar to webhooks!

If nothing else, I hope people use this story as inspiration to think about using multiplexing to design better, simpler application protocols. Even if for some reason QUIC isn’t the future, you now have qmux!

Originally published at https://dev.to on May 6, 2021.