Ok, short one, but I spent the last day re-living my electrical engineer days and wanted to share in the hope I help others who were navigating the somewhat hard to find / read docs from Brink. It amazes me that in 2020 a device that costs €2–€3k doesn’t come with a ZigBee or Z-Wave module out of the box.

My aim was to be able to fully control the HRU (heat recovery unit — for ventilation but in a way that ensures the air coming into the house is warmed by the air leaving, and so saving heating bills) from Home Assistant.

I have published it all on node-red: https://flows.nodered.org/flow/4e93b2dc8ff6acb5dbbcd48b1df92224

This is the setup I am using to publish events for my Brink Flair 300 heat recovery unit. I have a Raspberry Pi 3, with an RS485>USB, connected to the 2 & 3 pin of the X15 input of the HRU. The flow in Node-Red talks locally to the device via modbus, and then connects to an mqtt server that is used for further connection to Home Assistant, this appears to be the most robust solution.

I used this USB stick: Rpi RS485 USB, but I think any of them will work. You do not need to buy the super expensive ‘digital adapter’ and the app, and with this setup you also do not need the super expensive RF bridge and connectors (though will likely want some physical alternative to this setup to turn it on / off if needed).

It currently provides temperature, set fan speed, actual fan speed, binary sensors for max, medium, normal (same as the 4-way RF switch) and inputs for speed (50–300, +50, -50). I also have another flow connecting this to the output of my air sensor, so when the air quality drops or it detects us cooking it turns the ventilation on!

Huge thanks to https://github.com/sirjackal/brink-modbus as this was essential in making it all work and getting the steer towards Modbus being the way to connect.

The gist below has all of the files referred to in this article: https://gist.github.com/cliftonc/4e93b2dc8ff6acb5dbbcd48b1df92224

Connection to Brink Flair

Use the 2 & 3 connections to the red X15 (note that mine came with a removable red plug already in it — so I just needed to connect it).

To verify it was working, I translated and used the python script from the https://github.com/sirjackal/brink-modbus repository, along with mbpoll.

Amazingly (after reversing the wires as I got it wrong first time around!) it actually worked :)

Then it was a matter of getting it all in node-red, then wiring up to home assistant via mqtt.

Then finally going into Home Assistant itself and creating the sensors (the actual configuration.yaml is in the gist above):

And then last of all the dashboards:

It lives!

I next joined a panel to talk about scalable systems:

And finally was in a second panel talking about organisational change:

Well … we’re now completely, 100%, entirely live with the platform built this way, and have been for a few months, and so we have some good news to report.

The performance and operation of the overall system is great — most individual services are sub 10ms response times, low memory and very reliable (hey — they only do one thing!), the overall pages average 60ms response times even under full load, and all this on significantly less infrastructure then we had previously.

To top it all off using the fully automated deployment pipeline we have done roughly 60 deployments to live since we went fully live with the eCommerce elements two weeks ago. TES previously didn’t do this many releases in an entire year, let alone two weeks. Some of these small bugs, some fully fledged features — but all just one person picking up a ticket, doing the work, pushing to live.

Feel free to find yourself something here if you’re wondering what it is:

https://www.tes.co.uk/teaching-resources/search/

But, like all stories of going live with a complex, multi-faceted system not everything goes right. One of those is that we observed that the composition layer was exhibiting some strange behaviour regarding memory usage after we went live with all 400k of content pages:

Not great at all — we had 7 node processes per server (8 cpus), with 16GB memory per server, and what you see above is the node processes killing themselves with OOM. This is after an earlier round where we had the memory limits within Docker set lower than the Node max heap size, and Docker was killing the processes before we got to this point.

So, we clearly had a memory leak that none of the earlier tests we had been running on any of the individual components (that appeared to not show this behaviour) was highlighting to us.

I embarked on a rapid process of soak testing each of the pieces that fit into Compoxure — the first candidate was the library that gets and then caches the fragments:

https://github.com/tes/reliable-get

A series of commits, including some that removed functionality we weren’t currently using, ensued, along with specific soak tests that were (I thought) designed to find out what is going on with memory growth.

https://github.com/tes/reliable-get/blob/master/benchmark/soak.js

I even used a set of production urls to test against (a lot of them!) to make sure that my test wasn’t too synthetic and so not replicating the live environments. I wrote a little module that dumped memory every second so that I could graph it in Excel …. nada:

So — imagine repeating this process against all the pieces (I won’t bore you but you can see all the commits in Github). Nothing obvious popped out.

The final approach, which myself and Steve Cresswell from Guidesmiths agreed never to share publicly, was to go into one of the live containers, manually install heapdump, restart a worker and then manually trigger heap dumps over the next few hours.

What this delivered however was very enlightening:

1GB of heap consumed by what appears to be the HTML fragments that are being parsed by the parser, and processed to generate the final output.

Looking at retainers, the culprit became clear:

It was the template engine we were using to parse the declarations, Hogan, holding on to sections of the strings that were being processed by htmlparser2, and in so doing forcing V8 to hold on to the entire document.

The specific code that did it is here:

https://github.com/twitter/hogan.js/blob/master/lib/compiler.js#L404

Which basically just caches the template so that on subsequent use it isn’t re-compiled, a very sensible thing to do for ‘normal’ use, but we weren’t using it normally. Once we went live with all of the documents, we were effectively creating a set of micro cached templates for every single document that parsed through it, and then holding the whole document.

But — why does V8 hold on to the entire document if you grab just a small chunk of it? Good question:

Issue 2869 - v8 - Substring of huge string retains huge string in memory - V8 JavaScript Engine - Google Project Hosting

So … our solution was to pull Hogan into the project directly, and remove the caching. It meant nothing was ever holding on to the substr of the document and so V8 could GC it as it should. I’ll package this up into a PR to make it configurable in the next few days.

The result, note the second deployment when we figured it out:

Everyone happy. Key take aways:

1. If you do anything with large documents in node, remember that retaining any substr of it will retain the whole document — be careful.
2. We didn’t actually need this feature of caching the compiled template, and in some respects don’t need the full power of a templating engine to parse the declarations — I should probably have looked deeper under the hood when selecting it and may still replace it with a simpler parser.
3. Load testing to look for memory leaks must be able to replicate real production traffic — not always easy to do.
4. Being able to perform heapdumps in live is invaluable — we’re going to automate it in future.

I’ve been talking at a couple of conferences lately about the approach we are taking at TES Global around declarative service composition (this will explain it).

What I’ve noticed is that people tend to feel a little uncomfortable about putting a proxy like this in front of their application, and lets be honest — I completely understand that fear — this isn’t just a library, this will be between you and your customer and interacting with every piece of HTML your application produces! Dejan outlined some ideas and concerns in a post here.

I won’t talk about how you can try it in small slices (e.g. maybe on just a small part of your application), but rather some of the work I’ve been doing recently to take some of the ideas and concepts from Compoxure and modularise them so that both we and others can use them without having to buy the whole farm.

So, the core of Compoxure has been extracted into two new libraries:

Parxer: https://github.com/tes/parxer

Parxer extracts the core HTML parsing logic out, think of it like a non-streaming version of Trumpet.

Realistically I don’t expect many people find use for this outside of our use case, as there are things with probably ‘better’ APIs out there that if you just want to do HTML transformation, however, if what you want to do is HTML transformation where those transforms involve asynchronous functions (e.g. calls to other services) and you want those to all run in parallel, then this might just be useful for you.

It’s non-streaming for a number of reasons, not the least that I can’t yet get my head around how to do the parsing and interleaving of parallel requests in a streams mindset, so this model is less complex for me to reason about and hence get working. I also find it very fast, despite the fact that it is technically buffering everything vs just sending it as soon as it gets it.

It’s been refactored with a simple plugin architecture, so you can easily add your own handlers, with the defaults that Compoxure uses (url, bundles and test) all built in as core.

var input = “<html><div id=’test’ cx-test=’{{environment:name}}’></div></html>”; 
parxer({ 
  plugins: [ require(‘../Plugins’).Test ], 
  variables: { ‘environment:name’:’test’ }},   
  input, 
  function(err, data) { 
    var $ = cheerio.load(data); 
    $(‘#test’).text() == ‘test’; // true
});

There are a lot more examples in the tests, and of course it is used within Compoxure so you can also look at that source code.

Reliable Get: https://github.com/tes/reliable-get

The second part is the code that interacts with the micro-services themselves, ‘reliable-get’.

Think of this as a wrapper around request that ensures that if the service at the other end isn’t reliable that your application won’t die as well.

var ReliableGet = require('reliable-get');
var config = {
 cache:{
   engine:'redis'
 },
 circuitbreaker:{
   includePath: true
 }
};
var rg = new ReliableGet(config);
rg.get({url:'http://www.google.com', cacheKey:'google', cacheTTL:60000, timeout: 500}, function(err, response) {
 console.log(response.content);
});

So once you have created an instance of reliable get, you can fire off GET requests to as many dodgy services as you like, and this will give you a reasonable amount of protection:

• If your service is higher load than the 3rd party you deal with, this will cache responses from it (completely configurable, plus it will honour cache response headers from the service as well).
• If the service has ever responded with a successful response, and then later dies, reliable-get will serve the last successful (‘stale’) response to ensure your app continues to appear like it is functioning as normal.
• There is a circuit breaker that ensures that if the 3rd party service dies or starts having trouble, a circuit breaker can open that can reduce load. In this model it will continue to serve the ‘stale’ content if it has it. This avoids you hammering a service that might be struggling into the ground while you try to fix it.

Modularity

The other thing that is useful about this process is that it has made the Compoxure code base much smaller and simpler to understand, I’ve found a few bugs in all three projects that are now fixed, overall test coverage is much higher and I think that actually getting others to add features to any one of the three will likely be easier (lets see!).

I think this method actually worked really well in this instance:

1. Write code to get it working.
2. Refactor that code to make it easier to work with, increasing test coverage as you go.
3. Once refactored, extract out modular code into separate libraries. Increase coverage on those libraries.

The danger of starting with lots of modules is that you don’t really understand the requirements for those smaller modules until you compose them together into something more complex.

This way of thinking also has a lot of similarities to the Micro Services — How to Start? — question, e.g. can you only properly identify the right granularity of services if you have previously built something larger that you are now teasing apart?

Cian from NearForm has just put the video from NodeConf EU online:

Micro services are definitely a hot topic at the moment, and many organisations large and small are breaking up their monolithic apps into smaller and smaller components.

Many start out on that journey by breaking out obviously asynchronous tasks — e.g. sending an email — into a separate service that receives messages from other parts of the application via a message bus of some sort.

The instant you do this though, you have just required your developers to now check out two repositories instead of one. Then you break out another service. Three. Then another. Four. Then you get a taste for it. Five. Six. Seven. Fifty. One hundred.

This may sound insane, one hundred repositories? But it’s definitely true — once you get the basics in place that allow you to automatically deploy more than one service it then becomes the default to break out new functionality into new services and on it rolls.

This then creates a challenge though for your development teams. How do they manage to get up and running from a blank start, on their first day, when they have to get hold of over 100 repositories, plus all of the local infrastructure dependencies (MQ, DB, Cache).

There are some fantastic projects that I would highly recommend that can deal with the base level of this — like https://boxen.github.com/ and https://www.vagrantup.com/ — but they tend to focus on everything but the project your working on, e.g. just getting all of the dependencies up and running repeatably and reliably across a number of developer machines.

There are also some great build tools like http://gruntjs.com/ and http://gulpjs.com/ that allow your projects to be built in a common and repeatable way. But by design these tools and their plugin eco-system are very focused on building and managing a single project, not many.

So what is a CTO to do? Well, by very definition an engineer should build be comfortable building their own tools, so that’s what we did.

Bosco

Bosco is a tool to help a team manage all of the projects that get created as a byproduct of a micro service architecture. It doesn’t replace Boxen or Vagrant, it isn’t a build tool that is attempting to replace Grunt or Gulp, but works around them.

Think of the following scenario:

A new developer joins your team. They get a shiny new laptop, maybe you already have boxen on it so all the standard tools are on board.

Their buddy directs them to the team repository on github, which they check out.

They install and run bosco for the first time, prompted to put in the organization’s github name, and an auth key from their account that allows read access to their repositories.

Now, they are ready to go — what can they do with bosco?

Well, broadly, the functionality is split across three main areas:

1. Managing Multiple Projects

These commands are all about making your life a little easier. The ones you use the most:

bosco morning

This is a wrapper for a number of commands (which you can run individually), but is basically the command you run each morning (or across the day), to be caught up on the changes that your team has made across all projects.

It does the following across all repositories: clone, pull, npm install, show activity log.

bosco grep

Allows you to quickly git grep across all the projects.

bosco stash

Stashes any uncommitted changes across all the projects.

I suspect you’re starting to get the drift — there are a set of commands that basically allow you to run the ‘standard’ set of commands you would use within a single project across many projects.

If you don’t want to run the command across all projects, you can filter down by passing an extra parameter:

bosco stash -r name

Where you can specify part of the project name (this is expanded as a regex) and the command will only run against projects that match — this is used in all commands that work across projects.

There are a lot more commands, and you can add your own into your bosco project folder (under /commands).

https://github.com/tes/bosco/tree/master/commands

2. Running and Stopping Multiple Projects

Ok, so you’re a master at using Bosco to make some operations against many projects a little simpler. Now how do you get the project up and running?

First, a small diversion into what you need to do with your services to make Bosco aware of how they start and stop.

package.json and bosco-service.json

If your project is a Node project, then all you need to do is specify a start script in the package.json. Bosco will find this, and when you tell it to run your projects it will use PM2 to start the project via that command.

If you’re project isn’t a Node project (e.g. it may actually just be a Docker container definition for Redis). Then you can specifiy more detailed configuration by placing a bosco-service.json in the root of the project.

Two examples are below:

A Node Project

{
 “tags”:[“review”],
 “service”:{
    “dependsOn”:[“infra-rabbitmq”,”infra-mongodb”,”infra-redis”]
}

This simply adds a tag to the project (this allows you to start a subset based on the tag — e.g. bosco run -t review), as well as define some dependent services that should be started if this one is started.

The start command is in Package.json.

A Docker Service

{
   “service”: {
     “type”:”docker”,
     “name”:”infra-redis”,
     “registry”:”docker-registry.tescloud.com”,
     “username”: “tescloud”,
     “version”: “latest”,
     “alwaysPull”: true,
     “docker”:{
       “HostConfig”: {
       “PortBindings”: {
         “6379/tcp”: [{
          “HostIp”: “0.0.0.0",
          “HostPort”: “6379"
          }]
        }
      }
    }
  }
}

This configuration tells Bosco that this project contains the definition for a Docker container, that when built is published to the registry shown. Bosco will run it after pulling it from the registry (it doesn’t build and run the container from the folder). This is a very light wrapper around the Docker remote API.

Running Projects

Ok, so you’re services are configured — you can now get running.

bosco run

This will run everything.

bosco run -r project

This will run based on a regex match against the project name.

bosco run -t review

This will run based on tags defined in the bosco-service.json files.

bosco run —watch

This will run the project, but in watch mode (using PM2).

Seeing what is running

bosco ps

This will show everything that is running.

Stopping

bosco stop

This will stop everything (the -r and -t tags still work for stop).

3. Managing Static Assets across Multiple Projects

The final piece of the puzzle when dealing with micro-services, especially micro services that render HTML (huh? see this post: compoxure) is that you need a way of managing static assets across projects — css, javascript and images.

Bosco is the natural place to do this, as it already knows all of our projects, and having it take on this extra responsibility wasn’t a big stretch.

bosco-service.json

If your project has static assets, you need to tell Bosco about them.

{
 “build”: {
   “command”: “node_modules/.bin/gulp build”,
   “watch”: {
     “command”: “node_modules/.bin/gulp build —watch”,
     “ready”: “Finished ‘build’”
   }
 },
   “files”: {
     “upload”: {
       “basePath”: “dist/”,
       “js”: [
         “js/moxie.js”,
         “js/plupload.dev.js”,
         “js/tsl-uploader.js”
       ],
       “css”: [
         “css/tsl-uploader.css”
       ],
       “img”: [
         “img/header-bg.png”
       ],
       “swf”: [
         “js/Moxie.swf”
       ]
     },
     “cx”: {
       “html”: [
         “ui/cx-templates/backend.html”
       ]
     }
   }
}

This configuration tells Bosco that the project is built using Gulp (this is optional — if there is no build step you can just point Bosco straight at the files in the project), and that the output of Gulp is a set of files that are defined in the ‘upload’ file group.

You can have as many file group’s as you like, as these are simply collections of assets that are grouped together in the final output (e.g. the JS is concatenated and minified, the CSS is concatenated) into a single file and then deployed together to a CDN.

To make this magic work, there are two key commands.

bosco cdn

This command runs across all of your projects, grabs all of the assets, and starts a server locally on port 7334 (http://localhost:7334) that serves them up. In addition to serving them up, it serves Compoxure fragments that reference the static assets.

This allows you to include the assets from a file group into a given page. Note that you don’t have to use Compoxure, you could do this via another mechanism like ESI, SSI or even Ajax depending on how your app works.

You can also run:

bosco cdn minify

If you want Bosco to minify the assets, exactly as it would if you ran the command to push to a CDN, but locally. This can be useful if you have any issues caused by the minification process.

Finally, once you’re changes are ready you can push the static assets up to a CDN for use in non-local environments.

bosco -e development -b 76 s3push

This will push all of the fragments and assets up to S3, into a path prefixed with ‘development/76'.

You can see a real one here:

https://duqxiy1o2cbw6.cloudfront.net/development/default/index.html

See some of this in action:

https://asciinema.org/a/13142

Summary

Bosco does a lot to help your development flow with Microservices. It isn’t a build or deployment tool — you should use other things for that — but it does aim to make the life of a developer on a day to day basis less complex when dealing with a lot of services.

I’ll be talking more about this at Fullstack in London this week, so do ask questions or let me know of any areas I should flesh out in more detail in future posts.

This year I’m talking about some of our recent work at TES, rebuilding a core part of our technology platform using a micro service architecture.

Why micro services? Well, its an evolution of all the thinking and work I’ve done since I was at ITV 7 or 8 years ago working around trying to bring their disparate IT systems together via the enterprise equivalent of ‘Service Oriented Architecture’. The differences between the two (which is largely semantic, the absence of software companies and management consultants driving a marketing message over substance and {} instead of <>) I can talk about in another post.

A lot of what I did when at MailOnline pushed this even further, but we didn’t really get it right. What I’m talking about in this post is the 3rd generation attempt.

To get a good feel for what a micro service is, and isn’t, I’d suggest you read some of these:

• Martin Fowler — Micro services
• Fred George — Micro services
• Dejan Glozic — Micro services

Compoxure

Compoxure tries to solve the deceptively simple problem: if you break a page up into a more distributed set of underlying services, how do you bring them back together to create a single page.

Break a page a part? What are you talking about? Well, consider this:

This somewhat simple page (which is very important from an SEO perspective), is clearly made up of a number of disparate parts.

1. Site masthead and navigation
2. Resource summary and download
3. Reviews
4. Author panel
5. Recommended / Related resources

Now, the interesting thing about each of these sections is that they each have different data behind them, and they each change at very different rates.

The site masthead needs to be re-used across the site, so it really shouldn’t be stored or managed by the application that renders this page.

The reviews panel is also generic, and can be used across the site — not just on resources. It also changes (from a data and caching perspective) whenever user leaves a review — this is at a different rate to the summary above it.

So, imagine a world where we’ve drunk the micro service kool aid, and so create a set of distinct services — all micro ;) — that do one thing.

1. Masthead service — for a given input URL it will provide you with the masthead and navigation.
2. Resource Summary — for a given resource ID it will give you back the JSON data that represents a resource, or an HTML fragment that represents that data rendered on the page.
3. Resource Reviews — for a given resource ID it will give you back the JSON data that is the reviews, or an HTML fragment that represents the reviews rendered. It also exposes an endpoint for creation of a new review, as well as reporting abuse (e.g. a review that is inappropriate).
4. Author Service — as above but for authors.
5. Recommended Resource Service — for a given resource and user combination, send back a list of resources you think the user would also be interested in.

Great — these can all be built, have their own data sources (kept in sync where necessary by asynchronous messaging in the background — e.g. as a resource is created or changed it drops a message onto an MQ) and be deployed independently — even by different teams.

Now — imagine your the team that has the actual responsibility to build this page. How do you put it together?

The typical option list is as follows:

1. Build a ‘Resource Front End’ application. Have it respond to the original request, and then make a number of service calls for JSON data from each of the services, and then render the entire page.
2. Use Ajax. Build a single page app that calls back to the various services for JSON and renders client side.
3. Use ESI. Use Edge Side Includes — serving a fairly flat response and letting the edge server compose HTML together.
4. Use SSI. Same as above, but closer to home with Server Side Includes.

Now, most organisations go for one of the first two. But both have problems.

If you go with the ‘Front End’ application, you’ve just recreated an application that is in it’s first stages of becoming a monolith, or in the least a single choke point for change.

To make a meaningful change to any part of the page, you’ll need to modify and test not just the service but also the app that brings it together. For any meaningful change to any of the disparate services (e.g. the masthead), you’ll likely want to test it as given you’re rendering the whole page in one go an error in the masthead service will actually cause the entire page not to render. You then move and start implementing code to call the services in parallel, implement circuit breaker patterns, etc. etc.

The second, the Single Page Application approach, is fantastic if you’re behind the firewall or delivering an application vs a content heavy site that depends on SEO. You could go down the render HTML on both client and server React or Airbnb rendr style but this means you have to ‘buy the farm’ and are forever stuck using one of these approaches across your entire estate — as well as NodeJS.

This is not very micro service-ey. Note that if you are behind the firewall, I’d strongly recommend using this approach above all others — its by far the simplest for developers to reason about and understand.

ESI and SSI are interesting — as they point us towards a possible solution, but both have their own challenges when used at scale. What if we could just put something in our HTML markup (e.g. from a CMS) that instructs a layer in our architecture to do the composition for us? But in a way that is simple to understand, decoupled and reliable? It could be its own micro service.

Compoxure is a composition proxy. It’s built as express middleware, so you can build your own application (there is an example in the source) that has all of your configuration, logging etc. vs having to use ours.

It’s deceptively simple:

You put Compoxure in front of a backend that serves HTML that presents the bare bones of the page (as much or as little as makes sense). If you put it in front of your CMS it could actually serve quite a lot of the page.

The HTML contains simple markup that instructs Compoxure to go and get some additional content on the way through (formatted to make it easier to read):

<div 
cx-url="{{server:resource}}/resource-summary/{{param:resourceId}}" cx-statsd-key="resource_summary" 
cx-cache-key="resource:{{param:resourceId}}" 
cx-cache-ttl="5m" 
cx-timeout="2s"></div>

This declaration will get parsed by Compoxure, and do the following if you request /teaching-resources/resource-12345/

• Check the cache at resource:12345 for content. This is currently Redis only, but you could build an adaptor for something else.
• If miss or stale:
• Make a call out to the resource server (defined in config so it can be changed centrally), at /resource-summary/12345.
• It will wrap this call in a circuit breaker, so if it starts to struggle it won’t hammer it into the ground, along with a short timeout.
• If there is any issue, it will simply close up the space (unless configured to display the error — e.g. in development).
• If it is a good response, it will then add it to the cache, and render the response inside the div (or replacing it if you tell it to).
• It will call the logger and statsd functions (which you wire up to your own handlers), so you can keep a very close eye on what it is doing and how it is performing.

What this means is that you can now safely make as many changes as you like to the resource summary service, and provided it continues to render HTML when asked, the risk of any one change breaking the entire page is drastically reduced.

To learn more, check out the github repo — it’s public — and any ideas or contributions are of course very welcome.

https://github.com/TSLEducation/compoxure

If you’re coming to Waterford for Nodeconf I’ll be talking about this and another tool called Bosco — which solves the parallel question — if you have a set of distributed services, how do you manage the static assets like JS and CSS?

See you in Waterford!