Márton Salomváry

Hosting Plausible Analytics on Dokku (2024 Edition)

Thu, 14 Nov 2024 00:00:00 +0000

Plausible is an application that provides web analytics but actually respects your users privacy. Below are the instructions on how to install Plausible on Dokku.

This is a modified version of the work done by Bart Mathijssen which is the modified version of the work done by Kevin Langley Jr.

I made some modifications to the article, changed deployment to using dokku git:from-image instead of a custom Dockerfile and switched to pulling the Docker image from the GitHub Container Registry since images no longer seem to be published to Docker Hub.

Installation

First, you need to SSH into your Dokku Host and create the Plausible app:

dokku apps:create plausible

Next, you need to install the PostgreSQL and Clickhouse plugins, if they aren’t already installed.

dokku plugin:install https://github.com/dokku/dokku-postgres.git postgres
dokku plugin:install https://github.com/dokku/dokku-clickhouse.git clickhouse

Create the instances of the PostgresSQL and Clickhouse databases.

dokku postgres:create plausible-db
dokku clickhouse:create plausible-events-db

Then link the databases to the application (ignore “App image (dokku/plausible:latest) not found” message in the next steps).

dokku config:set plausible CLICKHOUSE_DATABASE_SCHEME=http
dokku postgres:link plausible-db plausible
dokku clickhouse:link plausible-events-db plausible

Next, you’ll slightly adjust the URL that was provided from linking the Clickhouse database and change the name of the environment variable for it as well to match what is expected within the Plausible docker image.

dokku config:set plausible \
  CLICKHOUSE_DATABASE_URL=$(dokku config:get plausible CLICKHOUSE_URL)/plausible_events_db
dokku config:unset plausible CLICKHOUSE_URL

Then you need to generate a secrete key using openssl.

dokku config:set plausible SECRET_KEY_BASE=$(openssl rand -base64 64 | tr -d '\n')

And then, we’ll set the BASE_URL environment variable and set the domain for the application — make sure to use your domain here.

dokku config:set plausible BASE_URL=https://analytics.example.com
dokku domains:set plausible analytics.example.com

Set the required SMTP environment variables to set up transactional emails.

dokku config:set plausible MAILER_EMAIL=admin@example.com \
                           SMTP_HOST_ADDR=mail.example.com \
                           SMTP_HOST_PORT=465 \
                           SMTP_USER_NAME=admin@example.com \
                           SMTP_USER_PWD=password \
                           SMTP_HOST_SSL_ENABLED=true

If you happen to be using Fastmail for outgoing mail, set up a new SMTP-only app password in Settings / Privacy & Security / Connected apps & API tokens / Manage app passwords (access: SMTP).

Then configure Plausible with:

dokku config:set plausible MAILER_EMAIL=sender@yourdomain.com \
                           SMTP_HOST_ADDR=smtp.fastmail.com \
                           SMTP_HOST_PORT=587 \
                           SMTP_USER_NAME=username@yourdomain.com \
                           SMTP_USER_PWD=<the app password> \
                           SMTP_HOST_SSL_ENABLED=false

Deploy Plausible to Dokku directly from the official Docker image.

dokku git:from-image plausible ghcr.io/plausible/community-edition:v2

Override the default (wrong) port mapping:

dokku ports:add plausible http:80:8000
dokku ports:remove plausible http:8000:8000

Initialize the databases:

dokku run plausible /entrypoint.sh db createdb
dokku run plausible /entrypoint.sh db migrate

Finally, you can generate the Let’s Encrypt SSL certificates.

dokku letsencrypt:enable plausible

Now open https://analytics.example.com and set up the first account.

Once the first account is set up, only invited users can set up new accounts. Override this with the DISABLE_REGISTRATION environment variable.

Django: The Good, the Bad and the Ugly

Mon, 04 Dec 2023 00:00:00 +0000

I’ve been working on the largest Django project of my career - which is Shepherd.com - in the last couple of years and have been planning to share my first-hand real-life experience in building non-trivial Django applications for a while. I finally took the plunge to write this up, below follows what I liked and did not like about Django.

The Django web framework can be used for a wide range of products of various complexity, and your mileage may vary on how good fit Django will be. My experience is focused on building a web application with a little bit of JavaScript sprinkled on top. That means not building single-page application (SPA), neither a native mobile app (I do build and optimize for mobile web though), nor a REST API. The applications I have built are monoliths (as opposed to microservices) backed by an SQL database and most of the content is managed using Django’s admin.

If you are building something drastically different from what I described above, my insights and advice here might have limited applicability. Some aspects are heavily opinionated, feel free to disagree.

To give the readers some hint on the scale of the codebase, which I consider to be a medium-sized project (for a small team or a solo developer), here are some code size stats:

Python: 58k lines, 2 MB code (including tests, excluding generated code)
HTML: 8k lines, 400 kB code
JavaScript: 4.5k lines, 90 kB code
(S)CSS: 1.7k lines, 35 kB code

The Good

I have to put forward that I generally like working with Django a lot. What did I particularly like about it? See below.

Python. I absolutely love working with Python. I am proficient in a handful of other programming languages like Java, Scala, Ruby, JavaScript and TypeScript, but for me Python comes close to the top when it comes to speed (both runtime and development cycle), developer friendliness and features (language and standard library).

The ORM. Object-relational mapping, also known as Models in Django. The ORM has a concise and very powerful way of expressing database structure and has powerful data modification and querying methods.

Migrations. Django comes with built-in database migrations, which is a key feature for an ORM. The generated migration files are expressive and easy to read, and do the job without any human intervention most of the time. It is extremely rare that I need to write a migration manually, and when that happens, it’s still an easy job.

Running and managing migrations are also very easy and straightforward using the built-in command line tools.

Management commands. Not that writing command line utilities is rocket science, but Django makes this even easier. I use these extensively for maintenance tasks, debugging and reporting and scheduled jobs.

Stability. Django is 18 years old, and that means stable. It no longer makes crazy changes in architectural direction (perhaps it never did), and it is not being rewritten to a different language. I like that! The release cycle is predictable, version upgrades are smooth. In the recent years, I’ve gone through one major version upgrade and many minor ones, all with reasonable effort.

Upgrading Django itself (excluding third party packages) usually took an hour or less even during a major version upgrade. Surprises were due mostly by not reading release notes carefully enough, or due to some deep customizations to the admin (most of the admin HTML, CSS and JavaScript structure seems to be treated as “internal API” but it is sometimes unavoidable to rely on these).

Contributing. I only have a limited experience with contributing to Django, but that is all very positive. The few issues I ever filed were responded to quickly, and the one changeset I contributed was merged into the mainline within a reasonable timeline. All in all, it seems like a friendly and active community.

The admin. Django comes with an (optional) admin user interface that allows quick CRUD operations on database records with almost no code. The admin is highly customizable, and I have successfully used it to create sophisticated content management systems (CMS) at a reasonable implementation cost and complexity.

Code structure. The framework promotes a simple, flat directory structure called apps or applications within a single project (a “project” is a website, combined from one or more “apps”). The approach may seem confusing at the beginning but it allows “clean code” structure, a folder hierarchy that follows the structure of the problem domain, not implementation details.

Django Debug Toolbar. Not part of Django, it is a third party package. An essential tool in the form of a web GUI that allows inspecting pretty much everything that happened during the rendering of a page. I mainly use it for optimizing SQL queries.

The Bad

There are a few issues that frequently and actively annoy me. I believe that solving these would lead to a significantly better experience for all Django developers.

Python’s dependency management. The horror! The horror! It is not Django to blame, but Python has a number of alternative dependency management tools or disciplines, and I found all of them to be horrible in some way. I settled with Poetry, which is somewhat heavyweight and breaks every once in a while, but does the job most of the time, seems to be the least horrible of all choices. The state of Python dependency management makes Maven or npm look amazing…

Django templates. I have always used Django templates (DTL, the framework’s own template language) for generating HTML, because the default should be fine for everyone, right? It is not the most amazing template language, but does the job just fine, the official documentation recommends sticking with it. (“If you don’t have a pressing reason to choose another backend, you should use the DTL.”)

Unfortunately, the template engine is very slow. Rendering a non-trivial, but also not insanely large or complex HTML page can take tens, sometimes hundreds of milliseconds on Standard-1X Heroku dynos.

Django’s performance optimization docs do actually mention that “heavily fragmented templates” might be slow. Given that many if not most web applications will want to use reusable HTML fragments, using {% include %}, which is documented to be slow, Django templates are not a great choice, except for trivial stuff.

Unfortunately, I realized this too late. Django does ship with an alternate template engine, Jinja2, and migrating an existing code base to a different templating engine is a risky and expensive business.

Lastly, I do not often miss tools from the JavaScript world, but I do miss Prettier, the opinionated code formatter for formatting templates. I use Black for formatting Python and DjHTML does an OK job indenting templates, but it does not come close.

Refactoring models, apps. Clean code means continuous refactoring as the application grows, which sometimes involves splitting Django apps, or moving functionality between them. Although the migration framework supports basic refactoring like renaming fields or changing types, moving models from one app to another is a complicated task, especially when model inheritance is involved.

For example I seem to be forever stuck with the poorly named List model on one project due to some weird dependency loops in the migration history or the inheritance hierarchy. I never managed to figure out why.

The Ugly

I feel a bit “meh” about a few things in Django, that are not that terrible, not deal breakers for most people; quite likely many developers will not even notice or identify them as problems. Still, my life, and maybe the life of a few others would be slightly better if these symptoms did not exist.

Lazy queries. Model property access can and often does result in on-demand database queries also known as the N+1 query problem. This is a Django feature only beneficial for absolute beginners and toy projects. Anything and anyone beyond that do not benefit from feature, and it makes certain page rendering performance issues hard.

As a solution the ORM has tools for eager loading (select_related and prefetch_related) but there is no way to force the usage of them and opt out from the default lazy behavior (although third party solutions exist).

One thing worth noting is that when using async views, lazy fetch on property access fails, so that partially solves disabling lazy fetch, but asynchronous Django has it’s own problems (see below).

Asynchronous support. Django has had asynchronous support since version 3.1. The catch is that it is still work-in-progress, therefore some parts of the framework do have asynchronous support, some do not.

That means one will end up sprinkling code with sync_to_async helpers, which quite quickly gets very ugly.

There is however a bigger catch with async Django, and that is the ORM. Even though the ORM has “asynchronous support” meaning it has async query methods like await Book.object.aget(), the underlying implementation is not truly asynchronous and does not allow concurrent database operations at all. This non-feature is not even documented clearly, in fact I had to learn it the hard way.

To add to the offense, when using an ASGI (asynchronous) server, there is no connection pooling available, which on my projects turned out to be a performance killer and therefore a deal breaker.

The lack of truly asynchronous ORM means it is hard to optimize views that require dozens or queries, no matter whether using sync or async, they will be executed one after another.

Third party Django packages. Django is a popular framework and there are third party packages for everything one can imagine. But most of these are not of “industry grade” quality. Which is OK, this is all free and open source, but one has to be prepared for eventually forking most third party packages in-use, and maintain the fork perpetually.

And even the packages that get semi-regular maintenance often break on new Django versions, sometimes taking weeks to catch up (even with community contributions).

This is of course not specific to Django, all open source communities suffer from lack of resources or interest for maintenance. I do my share of making things better by contributing fixes and filing issues, I wish everyone was doing that.

Notable exception are the packages maintained by the Jazzband community.

Limited alternatives other than ORM or raw SQL. Django’s ORM is very capable and one is able to express rather sophisticated SQL queries using Python syntax. But there are limits to this, and no ORM will likely ever cover all SQL features (standard or vendor specific), and that is fine…

But a few times I hit a wall with a more complex ORM query, which by then is rather complicated to abandon, and sometimes queries are reused elsewhere, and switching to raw SQL means no easy reusing of SQL fragments (except for manipulating strings that contain bits of SQL queries).

Ideally, there should be some low level query builder interface that allows expressing any SQL grammar using Python syntax, but also understand that it’s not something trivial to add to the ORM.

Almost everything is mutable. For those with background in functional programming and immutable data structures, Django might seem like a horrible place. In Python, as well as in Django, almost everything is mutable, object oriented programming and class based inheritance is widely used and relied on.

Not a big deal for me, but I sometimes wish somethink like Python+Django existed that is strictly immutable. I’ve probably spent too many years programming Scala…

Lack of service layer. In most non-trivial web applications one eventually want to contain “business logic” somewhere, ideally following some established patterns.

The missing piece in Django is something between models and views (n.b. what Django calls “views” are called “controllers” elsewhere).

Without further structural decisions, business logic in Django might end up in views, models, managers and querysets. Most often views end up being rather “fat”, containing a lot of business logic.

The perfect architecture can be debated, and is inndeed debated a lot, but I personally believe in “skinny” views and “skinny” models. That means implementing most of the business logic in a layer called “services”. These services are usually plain Python modules named “something_service” containing functions that execute “transaction scripts”.

This is especially useful, or even inevitable for transactions involving multiple models and/or external services (e.g. REST API calls), shared between multiple views.

When the admin app is involved, the service approach becomes messy though. There is no simple way to force the admin to always use whatever business layer we dreamed up, for good reasons (admin is truly just a CRUD GUI on top of database records).

What I ended up doing is overriding the admin’s save_model and save_related methods and call service methods from there, where it was necessary.

No zero-downtime migrations. Not that many ORM-migration tools have that included, but it is worth noting that changes to the database schema can block production applications from accessing the database for the entire duration of the change, which, for large tables, can cause a significant amount of downtime.

Outside of Django are ways around this, for example pt-online-schema-change for MySQL, but it would be nice to have something like this integrated into Django. I think it would fit Django’s role of a “heavyweight ORM”.

No European Django Foundation entity. If you use Django for business, a good way of supporting the project is with donations. As the Django Foundation is registered in the USA, making these donations tax deductible can be hard or impossible in Europe (at least in Germany, where I am tax resident).

Summary

All in all, despite the gripes above, I am very happy with Django and will keep using it on real-life projects. It is rock solid and truly lives up to it’s slogan, which is “The web framework for perfectionists with deadlines”.

I hope some day soon a truly asynchronous ORM becomes a reality. And my next project will use Jinja templates.

Setting up GnuCash with Deutsche Bank Germany

Sun, 05 Mar 2023 00:00:00 +0000

I’ve been recently trying to get my personal finances under control, and part of that was introducing GnuCash to my workflow. GnuCash itself might we worth a separate blog post, but here I’m only documenting how to connect GnuCash with Deutsche Bank in order to automatically import transactions.

Requirements

GnuCash 4.13 or newer. Might work with other versions but I have not tried.

You need to have a bank account at Deutsche Bank Germany. I do not know how much of this applied to Deutsche Bank in other countries.

I think you need to have “HCBI Plus” activated. Log in to online banking and then go to Online SelfServices, Sonstiges, then “TelefonBanking PIN bestellen und HBCI+ Freischaltung” and then make sure “HBCI Plus Status” is activated (“Die HBCI Plus-Nutzung ist aktuell aktiviert.”). If not, you might need to activate it. (For me it was already activated and I can’t recall activating it myself, therefore that might be the default.)

Setting up

Open GnuCash
Open Tools / Online Banking Setup
Click Next, Start AqBanking Setup, click Create User
Select “HCBI backend using AqHBCI”
In the next step, select “Setup a PIN/TAN account” an click Run
In the “Please select the bank” step, enter the following:
- Bank Code: your “Bankleitzahl”, e.g. 10070000
- Bank Name: Deutsche Bank
- Server URL: https://fints.deutsche-bank.de/
- You can also auto-fill the last two using the Select button next to Bank Code.
In the next step, enter your user details:
- User name: this can be whatever you want, for example the account holder’s full name
- User Id and Customer Id: this is your branch number (Filiale, 3 digits) plus your account number (Konto, 7 digits) plus your sub-account number (Unterkonto, 2 digits).
On the certificate Received screen, accept the certificate if you get “The certificate is valid” near the bottom.
When prompted to select a TAN method, select “901 - Mobile-TAN (Version 6)”
When prompted for a PIN, use the 5 digit code you use for signing in to online banking.
At some point you will also be propmted for a TAN which you will receive via SMS text message.
Once finished, close the setup wizard, and on the “Match Online accounts with GnuCash accounts” screen, select which GnuCash account he Deutsche Bank transactions should be synchronized to. If you don’t have one yet, create something like “Assets/Current Assets/Deutsche Bank”.

Testing and using

Once the setup is done, let’s see if it actually works.

Open the GnuCash account for Deutsche Bank.
Go to Actions / Online Actions / Get Balance.
It should show the balance and ask if you want to reconcile.
Then go to Actions / Online Actions / Get Transactions.
Enter some time frame that’s less than the last 90 days.
If you get an error, saying “Error on executing job. Status: rejected (5).”, you need an additional step below.
As the next step, you should see the “Generic import transaction matcher” window, where you will see the to-be-imported transactions and can decide what to do with them.

Fixing ‘Error on executing job’

These instructions are specific to macOS but should mostly be valid with appropriate changes on Linux and Windows:

Open the Terminal app
Run /Applications/Gnucash.app/Contents/MacOS/aqhbci-tool4 listaccounts -v, which should print out something like this (find the right line matching the Deutsche Bank account in question):
```
Account 0: Bank: 10070000 Account Number: xxxxxxxxx  SubAccountId: EUR  Account Type: bank LocalUniqueId: 5
```
Note the number after LocalUniqueId.
Run /Applications/Gnucash.app/Contents/MacOS/aqhbci-tool4 getaccsepa -a LocalUniqueId but substitute LocalUniqueId with your actual id number.
When prompted (“Input:”), enter your online bank PIN.
Try Actions / Online Actions / Get Transactions again, as described above.

Notes

I found the fix to the transaction job in this AqBanking bug report.
Very little documentation is available for HBCI by Deutsche Bank.
GnuCash does not support photoTAN but the bug report has a workaround which I have not tried yet.
I have not yet figured out how to synchronize Deutsche Bank credit card transactions.

Replacing Heroku with Dokku (for Pet Projects)

Mon, 19 Dec 2022 00:00:00 +0000

August 25, 2022 was a sad day in cloud computing, when Heroku announced that they are discontinuing all free plans. Reason being too much time spent on fraud, which I guess translates to folks abusing Heroku for crypto mining.

Anyway, as of writing this, there is already no more free Heroku. Now the question is, should I pay Heroku or someone else from now on, to host my pet projects and experiments? There is definitely more competition in the non-free cloud offerings than in the free ones.

I could use AWS but that’s the opposite of fun, and pet projects should still be fun. I like DigitalOcean since they are better on terms of developer friendliness, but then that’s still not as simple as git push heroku main.

So I decided to have my own “Heroku”, in the form of a self-hosted Dokku server. It’s a tiny PAAS that you can host on a Linux machine and provides a Heroku-like command line interface to interact with.

But you need a server.

My first idea was to install it the NAS that is running 24/7 in my closet but it’s neither powerful enough nor do I want experimental stuff running as root right next to my precious private data.

I decided to use a dirt cheap (4.5 EUR / month) cloud server at Hetzner, not only because it’s cheap but also because pet projects are for trying out new things.

Below is a step-by-step guide to setting up Dokku in a (mostly) automated way.

Setting up the server

I’m a Believer of Configuration as Code, therefore after signing up to Hetzner and obtaining an API token, I did the rest of the setup using Ansible. The instructions below assume that you have a basic understanding how Ansible works.

I’ve put everything in an Ansible playbook named dokku.yml.

- name: Set up a server on Hetzner
  hosts: 127.0.0.1
  connection: local
  tags: server
  module_defaults:
    hcloud_server:
      api_token: "{{ hetzner_api_token }}"
    hcloud_ssh_key:
      api_token: "{{ hetzner_api_token }}"
  tasks:
    - name: Create an ssh_key
      hcloud_ssh_key:
        name: my-laptop
        public_key: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
        state: present
    - name: Create a server
      hcloud_server:
        name: dokku
        server_type: cx11
        image: ubuntu-18.04
        location: fsn1
        ssh_keys:
          - my-laptop
        state: present
      register: hcloud_server_result
    - name: Print server values
      debug:
        msg: "{{ hcloud_server_result }}"
    - name: Add server to inventory
      ansible.builtin.add_host:
        groups: dokku_hetzner
        name: '{{ hcloud_server_result.hcloud_server.ipv4_address }}'
        ansible_user: root

Assuming you’ve set up an Ansible variable named hetzner_api_token holding your API token (preferably using a vault, running the following commands should get a server up and running:

# Install Ansible and dependencies
pip install ansible
pip install hcloud
ansible-galaxy install dokku_bot.ansible_dokku
ansible-galaxy collection install hetzner.hcloud

# Set up the server
ansible-playbook dokku.yml

It is assumed that you want to use the publish ssh key at ~/.ssh/id_rsa.pub for connecting to the server. If that’s not the case, tweak the playbook to your liking.

At this point ssh root@the-address-printed-under-server-values should get you a shell on the server.

Set up a domain

It is recommended (but not mandatory) that you set up a wildcard domain for the Dokku server. I’m using Cloudflare DNS, therefore this section depends on your provider. You can also decide to set up the domain manually.

Add the following to the playbook:

- name: Configure DNS entries for Dokku
  hosts: 127.0.0.1
  connection: local
  tags: dns
  tasks:
    - name: Create an A record for mydomain.com
      cloudflare_dns:
        zone: mydomain.com
        type: A
        record: 'cloud'
        value: '{{ hcloud_server_result.hcloud_server.ipv4_address }}'
        account_email: "{{ cloudflare_account_email }}"
        account_api_token: "{{ cloudflare_api_token }}"
        state: present
    - name: Create an A record for *.mydomain.com
      cloudflare_dns:
        zone: mydomain.com
        type: A
        record: '*.cloud'
        value: '{{ hcloud_server_result.hcloud_server.ipv4_address }}'
        account_email: "{{ cloudflare_account_email }}"
        account_api_token: "{{ cloudflare_api_token }}"
        state: present

You will have to set up Ansible variables for the DNS provider credentials.

Running the playbook again should set up the DNS entries:

ansible-playbook dokku.yml

Verify by running dig mydomain.com or ssh root@mydomain.com. It might take a while until the DNS changes are propagated.

Install Dokku

Because the Dokku folks are amazing, they created an Ansible role for setting it up. Add the following to dokku.yml:

- name: Set up Dokku on the server
  hosts: dokku_hetzner
  tags: dokku
  roles:
    - dokku_bot.ansible_dokku
  vars:
    dokku_hostname: mydomain.com
    dokku_key_file: /root/.ssh/authorized_keys
    dokku_plugins:
      - name: postgres
        url: https://github.com/dokku/dokku-postgres.git
      - name: letsencrypt
        url: https://github.com/dokku/dokku-letsencrypt.git

You can tweak dokku_plugins to your liking. Plugins can also be added later, keep letsencrypt if you want to use free SSL certificates.

Installing Dokku with this command will take a while (~10 minutes):

ansible-playbook dokku.yml

At this point the dokku command should be working on the server:

root@dokku:~# dokku domains:report --global
=====> Global domains information
       Domains global enabled:        true                     
       Domains global vhosts:         mydomain.com

Post-install

A few steps are necessary to enable using free Let’s Encrypt TLS certificates.

dokku config:set --global DOKKU_LETSENCRYPT_EMAIL=me@mydomain.com
dokku letsencrypt:cron-job --add

Use a valid email address :)

This should ideally be done with Ansible as well.

It is not mandatory, but the rest assumes you have the official Dokku client installed on your development machine (wherever you deploy your apps from).

Deploy an app

Now you are ready to deploy your Heroku-compatible apps to your own server using a few simple commands.

If you don’t have any ready, you can play with Heroku’s own example app, like this one:

git clone https://github.com/heroku/python-getting-started.git

Then in the app’s root folder (assuming a Django app using PostgreSQL):

dokku apps:create my-app
dokku postgres:create my-app-database
dokku postgres:link my-app-database my-app
dokku letsencrypt:enable
git push dokku master

Deploy a new version after making a git commit using:

git push dokku master

The app will be available at my-app.mydomain.com.

Note that if you are using the Django example app above, a few more tweaks are needed:

dokku config:set ALLOWED_HOSTS=my-app.mydomain.com SECRET_KEY=something-random
dokku run ./manage.py createsuperuser

Happy pushing, feedback welcome! 👇

Building the Shepherd.com Web Engine

Tue, 20 Apr 2021 00:00:00 +0000

Some time in late 2020 I got approached by Ben Fox, founder and entrepreneur, who needed help building his new venture Shepherd.com - “A better way to discover amazing books”. It is a website of curated book recommendations by authors and famous people organized around topics.

As a book lover, I quickly got excited and agreed to build the website and a content management system (CMS) behind it. The website launched publicly today (20 April 2021), and following is the story of building the software that powers it. Ben runs a diary and newsletter from the founder’s perspective and if you happen to be a book author interested in promoting a book or project, get in touch here.

Landing page of Shepherd.com on the day of the official launch

How it started?

When Ben reached out to me, he already had a clear picture on what he wants and had elaborate design wireframes. The initial requirements were written down in Google Docs which allowed us to quickly come up with a roadmap with estimated costs.

The first plan was quite simple:

Build a headless CMS so that creating website content can be started as soon as possible.
Build a “minimum viable product” (MVP) website and launch as soon as possible.
Elaborate advanced website and CMS features after launch.

Research on book metadata and cover image APIs

At the beginning it seemed like we will want to have an external data source on book metadata and high quality book covers. Even though various options exist, this is a surprisingly hard problem to solve for several reasons:

The data quality wildly varies especially for non-English publications or niche topics.
Book cover quality varies even more, it’s hard to cater for modern designs with high resolution images.
Quality sources can get really expensive.
Licensing and usage rights are usually restrictive and complicated.

Based on Ben’s original shortlist, here is a brief summary of the options we looked at:

Amazon: you need to be an Amazon Associate Program member to use their APIs, which was not an option at this point.
Google Books: easy to use REST API, great search (surprise!), low quality covers only. Free but restricted.
Ingram: awful SOAP API (documentation is 56 pages), high quality metadata with high(ish) resolution covers. Expensive and restrictive.
ISBNdb: simple REST API but the data quality is not impressive. Cheap, restrictive.
OpenLibrary: simple REST API. Free and anyone can contribute data, it’s the Wikipedia of book databases (or the Wild West of it…)
Bowker a not too bad XML REST API, some books have high resolution covers. Not free.

In the end we decided not to use any of these APIs and rely on manual book data entry until later when it becomes clearer what the business demands.

Project management and collaboration

Since it is a two-men team with Ben and me for most of the time, we kept tooling lightweight. Ben writes down the initial requirements for each milestone or major group of features in Google Docs, where we use comments and collaborative editing until clarified and agreed on everything.

We initially used checklists within Google Docs for keeping track of tasks and progress but that quite soon turned out to be not flexible enough due to lack of useable long conversation threads. We now keep track of everything in GitHub Issues which perfectly does the job. I briefly also considered using GitHub project boards but found it a little awkward to use and probably an overkill for this team size. We do use issue milestones though to organize and schedule stories, bugs and other tasks into larger buckets.

Our day-to-day communication is also very lightweight. We do conversations on GitHub Issues when there is one and exchange emails regularly on other topics. We have not seen the need for any sort of real-time, synchronous collaboration so far. I also find the GitHub notification inbox very useful for keeping up with conversations happening on issues.

Software stack

I used the following inputs to decide on what software stack to use:

Ben wanted to get to a working CMS as quickly as possible.
There were no particular user interface or visual design requirements for the CMS part and compromises were acceptable if they speed up delivery.
The CMS was to be used by Ben a very few selected administrators.
There was no user generated content for the immediate roadmap.
The public website consists of mostly static pages with minimal to no interactions other than following links.
High quality books covers were essential parts of the content.

Given these criteria I decided for the Django web framework:

It is a “batteries included” framework that allows focusing on building features at light speed.
It is mature. Having existed for 15 years, I know I won’t run into major issues or lack of documentation and all my possible questions have already been answered on StackOverflow.
Django comes with an automatically generated admin interface which works very well for basic CMS purposes.
It assumes a relational database for which it also automatically generates and manages schema migrations allowing to offload most of the heavy-lifting part of working with data to the database server.
Given the mostly static nature of the public facing website, scaling Django is not a major challenge.

So far no regrets for choosing Django and using the built-in admin for managing the content. There are definitely a few places where I stretched how far one can go customizing the admin and had to go deeper in Django’s source code than I wished for but this has been more an exception than something common so far.

Django Admin – perfectly does the job of a simple CMS

Infrastructure

Given that the system was to be built by me in a solo fashion (collaborating with Ben on the requirements and content building side and a designer responsible for the look of the public website) spending time on anything that’s not closely related to the main goal was out of question. One such area where time and money can saved is setting up and managing infrastructure, also known as DevOps.

For this reason I decided to deploy the application to Heroku. It is not considered to be the “hottest thing” nowadays but it does one thing well: it runs applications with little to no setup or maintenance. The price can go up steeply if your capacity needs grow but that’s a good problem to have. As Heroku did not require me to write my application in any vendor-specific way (except for following the twelve factor principles which is useful on most competing platforms) it’s easy to move elsewhere once the extra effort is financially justified.

Heroku also offers managed PostgreSQL which allows me to not care about database security and backups. It even comes with a 4 days window of point-in-time rollback which drastically improves my quality of sleep.

Uploading, storing, resizing and serving images for book covers and author portraits is an important part of the software but not something worth building ourselves. Heroku does not offer any sort of static hosting itself but they have partnered with Cloudinary which is a CDN and a media management service in one. Setting up Cloudinary with Django integration took me about 30 minutes and we have working uploads and images resized to all responsible design needs.

We use Cloudflare as a CDN, caching and security layer in front of the Django application. Because Heroku offers no static file hosting and Django also recommends solving the problem externally, Cloudflare CDN came handy for serving stylesheets, fonts, images and other static assets. Another selling point for Cloudflare was their cache which we can use for saving a huge amount of Heroku resources by serving a cached version of infrequently changing but still dynamically generated pages.

I am positive that this infrastructure will serve the project for long enough after the public launch so that we can identify growth patterns and make a plan to evolving it (or not) while also building the next round of features.

Continuous delivery

I strongly believe in continuous delivery from day zero of any project. We have a decent amount of automated tests mostly of integration and unit kind, which are ridiculously fast in Python/Django. Most test exercise the whole stack from the HTTP middleware all the way down to an in-memory SQLite database and the entire test suite still completes within a few seconds.

Code quality checks (tests and Pylint rules) alongside of verifying code formatting conformity with Black is done on each commit using GitHub Actions. If the checks succeed, the code is automatically deployed to Heroku.

We do have a staging environment as a separate Heroku app with dedicated database. Just like the production environment, it is also automatically deployed on each commit. We use staging for making sure database migrations apply without issues, to play around with content without messing up the public website and sometimes for previewing features.

Alternatives considered

There are a few alternatives we have or could have considered:

WordPress: we could have taken WordPress off the shelf at one of the many providers but keeping the data structure and workflow heavily customized to our needs was a must from day one which is not something WordPress easily allows.
Django CMS: might be a feasible option for the future but at the beginning a heavyweight CMS did not seem justified.
Ruby on Rails: a reasonable alternative but I’ve been doing more Python work than Ruby recently.
Laravel: if I had any experience with Laravel, it could have also been an option, but I have none.

There are probably plenty of others worth considering, but these were the ones that briefly crossed my mind at the beginning.

The future

There is a long list of features we will keep shipping in the near future. Time will tell how well the infrastructure holds under the load but I don’t expect surprises here. Once we start adding more dynamic or user generated content that does not play nice with heavy caching we might need to reconsider some parts and even move to a cheaper hosting provider from Heroku once we can justify the extra DevOps cost.

If everything goes well, I will write a follow-up post here. (Maybe also when things go bad, but that “should not happen”;)

Have comments or questions? Let’s discuss on Hacker News!

A Lightweight Project Management Workflow

Tue, 13 Oct 2020 00:00:00 +0000

Every time I start working on a new project where the means of coordinating the software development work are not established yet, I propose a lightweight agile workflow distilled from my past experience on a wide variety of projects. This workflow described below is my personal preference and by no means perfect or the only feasible approach. Also nothing here was invented by me.

The workflow is best applied at small teams of 1-10 people working on a single software product. Ideally, the team is composed of one or more developers with roughly similar skillsets, optionally supported by one or more visual/UX designers.

It is also useful to have one person appointed for “running the show”, which means this person is responsible for making sure the agreed upon process is followed and adjustments are made whenever necessary. This role can be taken by the same person for indefinite time but can be periodically rotated as well since it does not require special skills other than knowing this workflow. (In more formal Scrum projects this role is called Scrum master).

The board

The center of this project management approach is the agile board with three swimlanes or columns: ToDo, Doing and Done representing the state of the stories contained within the lanes. The swimlanes contain stories represented by cards or sticky notes. Stories can represent development or design work, new features, improvements, bugs or maintenance tasks. This is also known as Kanban board.

There are no hard constraints on how a story should be phrased and what it should contain but there are a few practices I recommend:

Keep the amount of work per story small. Preferably not more than one person-day per story.
It is OK to add larger stories to ToDo but break it down to smaller ones before the actual work starts.
Be clear when phrasing a story. Use imperative style for new features and improvements (use the words “add”, “create”, “make”, “improve”, etc.). For larger features it might be useful to start with “user story style” optionally followed by detailed requirements.
When adding bugs to ToDo, make sure the distinction between the intended and actual behavior is clear. “Expected the icon to be cornflower blue but got red” is better than “Button is red”.

The rules

This is the team’s daily workflow. Tweak according to your needs.

Planned and committed-to stories are in the ToDo column.
The next story to work on (often but not always the most important one) is at the top of ToDo, the last one at the bottom.
ToDo stories should not be pre-assigned to a person, anyone capable working on the next ToDo should be able to take it.
When someone starts working on a story, it should be moved to Doing.
Ideally nobody should be working on more than one story.
If something gets blocked, better move it back to ToDo.
Once a story is complete, it moved to Done. For most stories “done” means tested and deployed to production.

If the team does daily stand-ups to catch up, it is a great opportunity to ensure the agile board reflects reality. Even if stand-ups are not practiced (very small team or solo developer, cross-timezone collaboration) it is useful to (only) refer to stories that exist on the board when discussing progress.

Besides the daily routine, some teams work in sprints or iterations. In this lightweight workflow it is not necessary to stick to a rigid schedule, however I find reflecting to past progress and planning ahead every once in a while useful.

The tool

The agile board has to manifest itself in some physical or virtual form. Choose the simplest tool possible. If you are a small team all sitting in the same room use sticky notes on a reasonably sized wall surface. If a digital approach is preferred, go for a tool that has the smallest amount of features necessary. Bloated all-inclusive solutions like Jira only result in tears and fist-shaking. I recommend Trello, Pivotal Tracker or when using GitHub or GitLab for source code control they both both offer decent Kanban-like boards. See GitHub’s project management offering and the one of GitLab’s.

Conclusion

Every project needs some structure but keeping the process lightweight for small projects can reduce overhead and friction. If the team or the organizational complexity grows later, there is always to option to “upgrade” the workflow to something more sophisticated.

What is your favorite lean workflow? Can it go even more lightweight than described here? Should it? Let me know what you think!

The History of SoundCleod

Mon, 08 Jun 2020 00:00:00 +0000

This post is about the history of SoundCleod, the open-source desktop app for SoundCloud I started building almost 8 years ago. After 2.0.0 was released recently I thought it might be interesting to share the background story.

The story starts in 2012 with me moving to Berlin and starting a job as an engineer at SoundCloud. When I joined in August, the team was preparing to launch “SoundCloud Next”, a modern single page application that was being written from scratch to replace the old website.

Even though SoundCloud had an official Mac app back then, with the launch of Next that had all the features the company wanted to ship combined with a great user experience, the desktop app was already far behind and got not much later retired. Some folks regretted this decision but most of SoundCloud users were just fine using the website, especially Windows users for whom a desktop app never existed.

Meanwhile, I had the opportunity as a SoundCloud engineer to spend 20% of my time (also known as “hacker time”) on whatever I want as long as it is related to the business the company is doing or a technology the team uses.

And that’s when it clicked in: I want to build a thin desktop wrapper around the SoundCloud web app so that I can use my Mac keyboard’s media keys to control playback and get other desktop-integration benefits like not needing to hunt down which browser tab is playing music.

The Inception

After installing Xcode and investing a few hours into learning a bit of Cocoa and Objective-C, a language I never used before, I made the first commit on Dec 11 2012. Few days later I had a working Mac application.

Before releasing the it to an audience however, I needed a name. Because the project was not meant to be an official app, I wanted a name that’s kind of funny but still recognizable and related to SoundCloud. As a reference to counterfeit brands like “abibas” I renamed the app to SoundCleod and presented it at the next regular internal demo session at SoundCloud.

Image source: reddit.

Colleagues #celebrated my little success, some started using it, some others made small contributions, another one offered help on how to pronounce “SoundCleod”.

Version 0.1 was announced to the public in January 2013 and the source code was published to GitHub under MIT license and with that the journey had started.

Early Years

The public release created a moderate amount of buzz, I had a few people reporting bugs, saying thanks and even had the first contributors send pull requests in early 2013. I kept using my 20% time to add features, fix bugs and keep the small app in good shape. It was a surprising amount of work considering that the app had almost no business logic and a small niche value added on top of SoundCloud’s desktop web experience.

Because any decent desktop software needs easy and frequent updates, version 0.12 with automatic updates was released in November 2013. By that time a continuous delivery pipeline was already in place.

Knowing that SoundCloud had no intention to maintain an official desktop app I kept serving my little niche with a lot of enthusiasm but my excitement about the native Mac app development ecosystem started to fade. I quite soon concluded that I will not want to go in-depth with native Mac or iOS development, which back then mostly meant writing Objective-C (Swift was not yet around).

Going Cross-platform with Electron

Even though SoundCleod’s codebase was not much more than a WebView (Apple’s embeddable WebKit browser component) with a few hundred lines of glue code, I had a few pain points:

Using a weird language I was not familiar with – in a not particularly developer friendly ecosystem.
Not being cross-platform.

Coincidentally, GitHub released Electron 1.0 in 2016, a framework for building cross-platform desktop applications using web technologies like JavaScript, CSS and Node.js.

Happy to ditch native Mac development I rewrote SoundCleod using Electron from scratch and released 1.0 in September 2016 and soon after that a version that was available for Windows as well.

Switching to Electron also meant using its built-in auto updater which required not only setting up and maintaining my own update server but also code signing the application.

Code-signing – as I learned pretty soon – is a complicated matter especially for newcomers not familiar with native application development. Not wanting to pay Apple $100 a year for giving me code signing certificates I decided to use self-signed certificates which are indeed free but the cumulative hassle of dealing with them cost me several times $100, which I only realized much later…

During these years a handful of people said “thanks you” or praised the app publicly, which felt good and encouraging. As many open-source maintainers say it is a lot of work for very little reward and that’s exactly my experience.

The first and until today the only donation the project received in 2018 was a big moment: someone sent me 10 Euros!

Recent Times

Fast forward to 2020 I no longer work for SoundCloud and have been freelancing for a couple of years. Since then I no longer have the free SoundCloud Go+ subscription meaning I rarely use SoundCloud at all. As a freelancer I also no longer have the privilege of spending 20% of my working hours on an open-source project. Every hour I spend on SoundCleod has an opportunity cost – the cost of not working for a client or developing my business in other ways.

I still have bursts of interest in the project. Most recently, thanks to the “bear market” caused by the global COVID-19 pandemic, some of my freelance clients bailed or scaled down ongoing projects giving me plenty of free time.

Besides doing some long overdue maintenance and small bugfixes, I decided to enter Apple’s walled garden and try releasing SoundCleod on the Mac App Store. I knew that Apple notoriously rejects apps for various reasons therefore I did not have high expectations of success. Assuming the attempt is a small effort on my side I decided to go for it as it could have increased my reach and would have allowed to easily charge some small amount for the convenience of the App Store downloads.

The effort turned out to be bigger than I expected. Pushing an Electron app to the App Store is a major hassle but I succeeded submitting it for review, which in turn got rejected within a few hours. Reasons? They did not like the name (no surprise!) and the color of the icon, so I renamed the app to “Cleod for SoundCloud” and changed the logo to green ;)

The App Store compliant name and icon

The branding changes got accepted, however what I did not manage to convince the Review Board about was the functionality. The rejection reason was something like “the app is not more than what you have on your website (sic!)”. I even tried to reason with Slack’s native app as a precedent, which is almost pixel-perfectly the same in Safari. Anyway, appealed, got rejected again and then gave up. I might publish the unedited conversation one day, it’s quite eye-opening…

The Future of SoundCleod

With the recent addition of dark mode I consider SoundCleod to be feature complete. Small features might be added if someone contributes code but I have no plans to add more or anything that might increase the maintenance burden.

Talking of maintenance, although I secretly hoped Electron will save me from this, something still breaks with every major macOS release (to be fair it’s often Apple to blame). To make is worse, things can also break with major releases of Electron making updates a risky business.

With all these in mind I am still planning to keep the project alive for the foreseeable future and if at some point I become too busy to be bothered, someone can fork the project and continue where I left. That’s the beauty of open-source!

Moving a Website to Netlify from GitHub Pages

Wed, 13 May 2020 00:00:00 +0000

The Need

I recently developed a new interest in how my blog and professional portfolio (this very website you are looking at) is performing. The goal was to see some basic numbers like total visitor numbers and a breakdown by pages, maybe referrers as well. Because I removed Google Analytics a while ago to protect the privacy of my visitors (and to not feed the giant with even more tracking data) and also do not think that any client-side tracking can be accurate for a small-traffic website like mine (because of blockers) I looked into server-side traffic analytics options.

The website was hosted on GitHub Pages with a CloudFlare CDN in front of it. GitHub does not offer any sort of analytics, and even if it did, it would not be accurate as most of the requests are served by CloudFlare. CloudFlare does have analytics but it lacks page-level breakdown. I even talked to a CloudFlare sales representative and they confirmed that they only offer more traffic detail in the enterprise package, which costs several thousand dollars a month. Not my budget.

Enter Netlify

A quick ask-around resulted in several recommendations in the direction of Netlify. The service has been on my radar for a while but had no excuse to try it out until today. Netlify Analytics for $9/month/site looks like what I was searching for!

Step One: Add Site to Netlify

Because the source code for my website is in a GitHub repository and uses Jekyll (the static website generator behind GitHub pages) and this setup is supported by Netlify out of the box, the first step was very easy. I signed up with my GitHub account to Netlify and followed the wizard to deploy a website from a repository. After allowing Netlify to access my repos and selecting the one named salomvary.github.com I had the website up and running under https://some-random-name-6a18c6.netlify.app within a few minutes. Looks like we are done!

If you do not use your own custom domain, edit the site settings on Netlify and change some-random-name-6a18c6 to something you like and you can stop reading now.

Step Two: Set up Your Own Domain

If you are using your own domain on GitHub pages like me, the domain needs to be migrated too. The Domain Management area on Netlify nicely explains the options. Unless you want to switch to their own DNS (which is a feasible option) you will have to edit the records at your DNS provider, for which Netlify also offers guidance. In my case the DNS provider is CloudFlare. All I had to do was changing the salomvary.com CNAME record to point to some-random-name-6a18c6.netlify.app. This is slightly different from what Netlify recommended but CloudFlare does CNAME flattening which is kind of equivalent of adding an A record.

Almost immediately after changing the DNS records the website was already served over HTTP by Netlify, and the automatic setup of the free Let’s Encrypt HTTPS certificates did not take more than 10 minutes. One can verify the successful switch-over by looking for the server: Netlify HTTP response header using the browser’s developer tools or the output of the curl -vL http://mydomain.com command.

It is important to note that during these 10 minutes the website was not available due to browsers showing certificate errors this is expected and was acceptable in my case.

Step Three: Fixing GitHub Project Pages

This part only applies if you are switching from a GitHub user or organization site. You can recognize this from the repo being named myusername.github.io (or .com for old sites) or myorgname.github.io. In this case a less known magic is in action which just got broken by moving to Netlify.

The magic is the following. If you do not use your own domain on GitHub Pages, your user/org site is at https://myusername.github.io and your repo pages are at https://myusername.github.io/myproject.

However, if you do use your own domain with GitHub, all project pages will be automagically served under https://mydomain.com/myproject and the .github.io variants will redirect to your domain.

But since switching to Netlify https://mydomain.com/myproject no longer serves the project page and neither does it redirect to anywhere. Netlify simply serves a “404 not found” error page. Even worse, https://myusername.github.io/myproject still redirects to https://mydomain.com/myproject resulting in the same error. The only exception are repos that are neither user nor organization page repos and are configured to use their own custom domain. These will keep working as before.

As long as you are fine with the project pages being served from https://myusername.github.io/myproject with a redirect from https://mydomain.com/myproject the fix is not very complicated: Netlify allows configuring redirects using a file named _redirects placed in the root of the website repo. It should look like the example below, where myusername is your GitHub username and myproject is the GitHub repo name you want the redirect for:

/myproject/* https://myusername.github.io/myproject/:splat
/otherproject/* https://myusername.github.io/otherproject/:splat

When using Jekyll, do not forget to add this to _config.yml:

include:
  - _redirects

If you only have a handful of repos to redirect to, the redirects can be created by hand. If you have many repos, are lazy, or like shaving yaks, you can automate creating _redirects. All you need is jq and curl installed, plus obtaining a GitHub personal access token.

Run this command from a terminal window in the root of the website project:

curl -v -H 'Authorization: token  <your GitHub token>' \
  'https://api.github.com/user/repos?affiliation=owner&per_page=100&page=1' \
  | jq -r '.[]
    | select(.has_pages)
    | "/" + .name + "/* https://yourusername.github.io/" + .name + "/:splat"' \
  >> _redirects

Do not forget to change yourusername to your GitHub username! If you have more than 100 repos on GitHub (check the Link response header for the presence of rel=next) repeat this by changing page=1 to page=2 and so on.

There is one more important thing before celebrating success: you need to tell GitHub to no longer use your custom domain, otherwise myusername.github.io/myproject will keep redirecting to mydomain.com/myproject which we just configured to redirect to myusername.github.io/myproject creating an infinite redirect loop.

This can be fixed by deleting the file named CNAME from the root of the Git repo and pushing the changes to GitHub.

At this point, your website should be serving visitors as it was before.

Step Four: Cleaning Up SEO

There is one remaining problem which might or might not bother you. With this setup, the website content can be available under four different URLs:

The source at https://github.com/username/username.github.com/blob/master/_posts/post.markdown
The GitHub Pages site at https://username.github.io
The Netlify website at https://some-random-name-6a18c6.netlify.app
And https://yourdomain.com

This is OK, as long as you do not care about search engine optimization (SEO), but Googlebot and other crawlers will treat these as duplicate content with appropriate punishments.

Hiding the website source in the GitHub repo is allegedly only possible by not using the master branch unless the repo is made private, which - now with free private GitHub repos - is also an option. This is a solution for #1.

If the repo is not a user or organization repo, GitHub allows turning Pages off from the repo settings page. This can solve #2.

For a user or organization page, there is no option to turn Pages off (which kind of makes sense). Setting the repo to private also does not disable the public website. The only solution is renaming the repo from username.github.io to mydomain.com or whatever you like (from now on, the name does not matter as we do not rely on GitHub Pages’ magic).

Disabling the Netlify “default subdomain” (my-thing.netlify.app) does not seem to be possible, which means web crawlers might also discover it, resulting in duplicate content. Unless there is some robots.txt trickery I am not aware of, or domain-specific redirect rules on Netlify are possible (to be figured out) the only solution is adding canonical URLs to all pages of your website. This, with the help of the Jekyll SEO Tag plugin is not very complicated (and is a good idea anyway). Problem #3 solved.

Conclusion

As we saw the trivial task of moving a static website from one hosting provider can turn into a deep rabbit hole. The good thing is, it did not take as much time as it may seem. In fact writing this blog post has taken much more time than the website migration itself. The other good thing: I did not even have to pay for Netlify, because it’s free, including custom domains.

Well, actually, I did end up paying for Netlify. If you still remember, the original motivation was to gather page-level analytics on the website traffic, so I pulled my credit card and forked out $9 for Netlify Analytics.

Which, to my slight disappointment, is only a tiny bit better than analytics at CloudFlare, offering nothing more than the top x (5?) most visited URLs in the pages section:

(For the curious here is a screenshot of the entire Netlify Analytics page few hours after turning it on.)

It was an interesting rabbit hole anyway :)

Visualizing Data with React and SVG

Tue, 17 Mar 2020 00:00:00 +0000

Although I am a great fan of D3.js when it comes to data visualization and have used it with great success in the past, it might not be the best tool for the job in all cases. In applications where the advanced visualization features of D3 are not needed and React is in use so that efficient mapping of data to DOM is taken care of, the extra learning curve of D3 might not be justified.

This post is a step-by-step introduction to building a simple line chart that visualizes the price of a stock over a couple of days. It is assumed that the reader has experience with React and modern JavaScript, basic knowledge of SVG is useful too. The impatient can find a link to the entire working solution in a Git repo at the end.

To get started, we need some data:

const data = [
  { day: new Date("2020-01-06"), price: 299.799988 },
  { day: new Date("2020-01-07"), price: 298.390015 },
  { day: new Date("2020-01-08"), price: 303.190002 },
  // ...
];

The fundamental first step to visualizing data is establishing a scale that allows mapping the abstract data (time and price in our example) into a visual representation.

For our use case SVG elements offer a convenient way to render our data onto a two-dimensional canvas. The coordinate system of an SVG image looks like this:

Image source: Wikipedia

To get started we will show individual data points - stock prices on a given day - as small circles using a simple React component:

function LineChart({ data, width, height }) {
  // Calculate the extent of the data
  // (not the most efficient way but easily readable)
  const minDay = Math.min(...data.map(({ day }) => day));
  const maxDay = Math.max(...data.map(({ day }) => day));
  const minPrice = Math.min(...data.map(({ price }) => price));
  const maxPrice = Math.max(...data.map(({ price }) => price));

  // Create the horizontal and vertical scale functions
  const x = day =>
     width * ((day - minDay) / (maxDay - minDay));
  const y = price =>
    height - height * ((price - minPrice) / (maxPrice - minPrice));

  return (
    <svg width={width} height={height}>
      {data.map(({ day, price }) => (
        <circle key={day} cx={x(day)} cy={y(price)} r="4" />
      ))}
    </svg>
  );
}

The most interesting part are the scale functions: x(someDate) returns the x coordinate in pixels for a date and y(somePrice) returns the y coordinate in pixels for a price. This means <circle cx={x(date)} cy={y(price)}/> will be rendered to the right place in our two-dimensional space. The resulting SVG looks like this:

To make the chart less empty and allow reading price change trends easier we will connect the circles with a line, turning it into a line chart:

return (
  <svg>
    { /* ... */ }
    <path
      strokeWidth="2"
      fill="none"
      d={line(
        data.map(({ day, price }) => ({
          x: x(day),
          y: y(price)
        }))
      )}
    />
  </svg>
)

To create a <path> that follows a certain course we need to pass in the path coordinates in the d attribute. It is a string of path comands and can be generated using a small helper function:

// Returns a string like M1,2L3,4L5,6
const line = data =>
  "M" + data.map(({ x, y }) => x + "," + y).join("L");

Our chart should look much nicer now:

Because a chart without labels is not a chart (can still be a piece of art but we are doing data visualization here), we will have to make further changes.

But first, in order to prevent our main render function from growing too large, we break it up into a couple of tiny components:

function LineChart({ data, width, height }) {
  // ...

  return (
    <svg width={width} height={height}>
      <Circles data={data} x={x} y={y} />
      <Line data={data} x={x} y={y} />
    </svg>
  );
}

const Circles = ({ data, x, y }) => (
  <g>
    {data.map(({ day, price }) => (
      <circle key={day} cx={x(day)} cy={y(price)} r="4" />
    ))}
  </g>
);

const Line = ({ data, x, y }) => (
  <path
    strokeWidth="2"
    fill="none"
    d={line(
      data.map(({ day, price }) => ({
        x: x(day),
        y: y(price)
      }))
    )}
  />
);

To make room for the labels along the x and y axes we need to slightly shrink the actual line chart and then move it a bit towards the bottom and right. We do this using the translate(x, y) transformation which moves it’s target by x horizontally and by y vertically.

const padding = 40;

function LineChart({
  data,
  width: outerWidth,
  height: outerHeight
}) {

  // Accommodate for the room around the coordinate system
  const width = outerWidth - 2 * padding;
  const height = outerHeight - 2 * padding;

  // Create the horizontal and vertical scale functions
  const x = day => width * ((day - minDay) / (maxDay - minDay));
  const y = price =>
    height - height * ((price - minPrice) / (maxPrice - minPrice));

  // ...

  return (
    <svg width={outerWidth} height={outerHeight}>
      {/* Move down and right to leave room for labels */}
      <g transform={translate(padding, padding)}>
        <Circles data={data} x={x} y={y} />
        <Line data={data} x={x} y={y} />
      </g>
    </svg>
  );
}

// Create SVG translate functions in a readable way
const translate = (x, y) => `translate(${x}, ${y})`;

And now we can add labels along the axes positioned using translate() to the left and bottom side. The x axis is easier to implement:

// ...
return (
  { /* ... */ }
  <XAxis
    data={data}
    x={x}
    transform={translate(padding, outerHeight - padding / 2)}
  />
);

const XAxis = ({ data, x, ...props }) => (
  <g {...props}>
    {data.map(({ day }) => (
      <text key={day} x={x(day)} textAnchor="middle">
        {formatDate(day)}
      </text>
    ))}
  </g>
);

// Return short date like "Jan 10"
const formatDate = day =>
  day.toLocaleDateString("en-US", {
    day: "numeric",
    month: "short",
    timeZone: "UTC"
  });

Because price along the y axis is a continuous value we need to come up with our own discrete values (“ticks”) for placing the labels.

// Calculate ticks for y axis labels
const ticksCount = 5;
const yTicks = [
  ...range(
    // Start from the lowest price
    Math.floor(minPrice),
    // End at the highest price
    Math.ceil(maxPrice),
    // Divide the range into equal parts
    Math.round(
      (Math.ceil(maxPrice) - Math.floor(minPrice)) / ticksCount
    )
  )
];

return (
  { /* ... */ }
  <YAxis
    values={yTicks}
    y={y}
    transform={translate(padding / 2, padding)}
  />
);

const YAxis = ({ values, y, ...props }) => (
  <g {...props}>
    {values.map((price, i) => (
      <text key={i} y={y(price)} textAnchor="middle">
        {price}
      </text>
    ))}
  </g>
);

// Generate numeric values within a range
function* range(min, max, step) {
  for (let i = min; i < max; i += step) {
    yield i;
  }
}

The final result will look like this:

There are may ways how this basic chart can be improved and I have not mentioned testing or usability of the components at all (might do it in a follow-up post). I hope that the basic concepts introduced are a good start for your next data visualization in React!

Once things get more complicated however, I strongly recommend looking at D3. It has amazing documentation, lots of interactive of examples and supports many advanced techniques and optimizations that are hard to get right when writing them from scratch. In fact, the whole approach taken here was inspired with my previous experience with D3.

D3 is also available as small modules which means even if you use React for managing the DOM, you can use part of D3 under the hood. For example the x and y scale functions are available in the standalone d3-scale module, or the <path> commands generator in d3-shape.

You can see the chart component from above running here and the full source can be found in a Git repo.

How to do Remote Meetings Right

Fri, 13 Mar 2020 00:00:00 +0000

As the COVID-19 pandemic makes more-and-more companies decide to go partially or fully remote, it is a very appropriate time to share experience and advice on what can help teams remain productive in this new situation.

I worked remotely for a wide variety of clients in the last couple of years and learned a lot about what the key components of a remote collaboration are. One key ingredient to efficient remote teams is doing meetings right.

Some say remote teams should only communicate asynchronously (via chat, email and documents) but for most, especially for those who only recently switched, setting up conference calls with multiple participants regularly will be inevitable. And it’s harder to do right than most people think.

Probability of "technology works": 50%. Probability of a video call working with video and audio both ways: 50% * 50% * 50% * 50% = 6.25%
— Márton Salomváry (@salomvary) June 6, 2016

There is complicated technology involved in videoconferencing, participants might have different cultural backgrounds, use diverse hardware and software equipment and will dial in from less-than-ideal surroundings.

Therefore it is imperative for everyone to aim for the Perfect Conference Call and hope you will only have minor glitches. Below is my advice on how to do accomplish this.

✋ Only meet if necessary

The best meeting is the meeting that did not happen, keep their numbers low, find other ways to sort things out.

👥 Keep the number of participants low

Inviting more people to a conference call drastically increases the chances of failure. Think twice before inviting someone. Will they really contribute? Do they really benefit? Can they get quickly briefed before or after in writing instead?

If you are invited to a meeting and have doubts whether your participation is useful, decline politely. Don not hang around just because someone invited you.

Also do not invite people as “optional”. Be explicit to avoid spending the first few minutes discussing “whether we should wait for X to join”.

⏱ Be 5 minutes early

Get ready to dial in 5 minutes before the announced start time. This allows you to get comfortable in whatever space you use and test whether the equipment is ready for joining. If you think turning on your laptop at 9:59 for a 10:00 meeting will be fine, you are wrong. You will be late and being late is not only impolite, it wastes everyone else’s time.

If the conferencing tool allows joining early you can even take the opportunity and have bit of small-talk with your colleagues.

🔥 Have a backup plan

If you are a participant, know what alternative ways are available for connecting. Most conferencing solutions allow joining using a good old phone call in case the fancy app refuses to work.

As an organizer of a remote meeting be prepared that a key person will not be able to join due to technical difficulties or they disappear half way through the meeting. If the issue persists for a minute or two, cancel and reschedule for another time. Do not make others wait for someone’s network connection to recover.

👩‍⚖️ Have a moderator

Just like at offline meetings, it is useful to have a person who keeps track of time, reminds people to stick to the agenda, facilitates moving conversations to another channel and reminds people to mute their microphones.

🎧 Use headphones

Even though smartphones and laptops nowadays have decent audio quality and feedback protection for voice calls, using headphones (preferably with built-in microphone) is still the best way of preventing other participants from hearing their own voice back through your connection. Furthermore headphones help you to hear and understand them better and cut you off from distractions.

If you are typing on a laptop while in a call, it is essential to use an external microphone as the internal ones are notorious at picking up and relaying keyboard noise to the other participants.

💻 Know your equipment

There are a gazillion audio and videoconferencing hardware and software, it is not unlikely that you will have to face something unknown in your next meeting (even if you have been working for the same company for a while). Be sure you have the necessary software installed, be that a desktop or mobile app or a browser extension.

Check whether you need to sign in or sign up to join in advance. If you need to use a corporate account expect going through a multi-factor authentication before. This is more likely if you are signing in from a new device or a location you have not used before.

If you need to use a VPN, be sure it is working.

📶 Do not use a Wi-Fi connection

One of the unsolved problems of the 21st century is stable and reliable wireless networking for everyone. If you have cabled network connection available, prefer using that. If you do not, consider buying one, they are cheap and a great investment preventing choppy audio, frozen video or dropped calls.

If you are using your home or office network, it is also useful to have your phone set up to be used as a personal hotspot to have backup if your primary connection goes down.

🔇 Mute your microphone

Whenever you are not talking (especially when for longer periods) turn off your microphone. Nobody wants to hear you breathing or coughing, your phone notifications or fire engines passing in front of your house.

When you want to talk, don’t forget to unmute yourself :)

🤫 Keep your environment quiet

Be sure your environment is as quiet as possible. If you have your family or kids around, ask them to avoid distracting you during meetings. If your home-office has a door, close it. If you have windows open to a busy street, close them.

Prefer not to participate from a café or bar or a busy street. If you really must, try to find a quiet corner.

If you have your camera on, sit in front of a neutral background. People walking behind you does not help. You think your cat walking into the scene is funny? It might be funny once, but it’s more of a distraction for the others.

🏢 Use good quality conference rooms

When some of the participants dial in from a regular office meeting room and some others remotely, keep in mind that people sitting at various distances around a laptop results in a terrible experience for the remote people. They will struggle following the conversation since laptop microphones are designed to pick up the voice of one person sitting directly in front of them, not conversations from a room full of people.

Combined speaker-microphone devices are available for phis purpose, make you employer buy one for each conference room used for remote calls.

Conference rooms often suffer from general poor acoustics, most common is reverb. This can contribute to call quality problems and even though reducing sound reflections does not cost a fortune, it is quite common. If you office suffers from this, prefer smaller rooms where the acoustical problems are usually less apparent.

👔 Be disciplined and polite

Like in other situations, do not talk while someone else it talking, do not cut others off and only interrupt if absolutely necessary. Keep in mind that even if in real life some of this is acceptable, in an on-line situation the slight delay in picture and sound and the drastically reduced meta-communication small “deviances” can be highly disruptive.

👖 Wear pants

You never know when you will accidentally turn on the camera instead of starting screen sharing. It is strongly recommended to wear at least a shirt.

Feedback?

How does your team do amazing (or terrible) conference calls? Do you have something to add or disagree with? Get in touch, see below how!

Does your software development team struggle with remote work? You can hire me for an hour, day, week or more time of consulting.

Memoizing with WeakMaps

Fri, 24 May 2019 00:00:00 +0000

I stumbled upon a performance bottleneck while building a small progressive web app recently. Turns out that Date#toLocaleDateString has terrible performance in Mobile Chrome. This method allows granular control over how a JavaScript Date instance is turned into a human readable string. It is a relatively new feature and in Mobile Chrome it might be as slow as ~100 operations per second.

Does not sound extremely bad but other browsers do several thousand operations in a second. My use case is fetching data containing lots of dates from a server and then formatting them for humans. Certain user interactions result in redrawing the screen and calling toLocaleDateString about 150 times. That results in more than a second delay in Mobile Chrome, definitely not a great user experience.

Time for some optimization! One way would be not using toLocaleDateString and formatting dates “manually” but doing so is more complicated than it sounds. I decided to memoize the formatted strings.

This is how the original function looked like:

function formatDate (date) {
  return date.toLocaleDateString('en-US', {
    month: '2-digit',
    day: '2-digit'
  })
}

Memoizing means, given an input, instead of some expensive computation, return a cached result. We will use WeakMaps to cache the formatted strings.

WeakMap allows storing values by keys where keys are JavaScript objects and the value can be anything. WeakMaps are special because the entries are “removed” whenever the key object is garbage collected. Perfect fit for our caching needs!

The memoized version of the function looks like this:

const dates = new WeakMap()

function formatDateMemoized (date) {
  let formatted = dates.get(date)
  if (!formatted) {
    formatted = date.toLocaleDateString('en-US', {
      month: '2-digit',
      day: '2-digit'
    })
    dates.set(date, formatted)
  }
  return formatted
}

This runs about 24k times per second if the value is already cached. As usual with caching, the performance improvement depends on the rate of cache hits and misses. For my use case this worked fine: the first render after a network response is slow (but talking to the server tends to be slow anyway) and then subsequent rerendering of the same data is very fast. The date instances are garbage collected and the cache is cleaned up when the next server response is parsed.

There is a gotcha though. Even though caching formatted dates works for my use case, this approach is dangerous to generalize.

Firstly, Date in JavaScript is mutable. That means the same date instance might represent 2019-01-05 16:00 at some point and 1970-02-23 10:42 in some other point as the application runs. There is no way to prevent this and if you ever modify the date instances before formatting them, you will end up with hard to debug issues.

Secondly, two different instances of Date might represent exactly the same point in time which means we should use a single cached result for both but we will not because they are cached under different keys.

The only way to prevent bugs resulting from this other than “making sure” Dates are not modified (which relies on humans not making mistakes) is to always use immutable objects as WeakMap keys. For dates Day.js is an option, for other data types Immutable.js is a good start.

A long term solution for my problem would be Chrome fixing the performance issue though :) In case someone wants to repeat it I published the benchmark here.

Old Coding Challenges

Fri, 17 May 2019 00:00:00 +0000

I was browsing through my projects archives and found a couple of old coding challenges I created during the hiring process of past jobs I applied for. I thought it might be fun to share them. Enjoy:)

SoundCloud

In 2012 I applied for a JavaScript Engineer role at SoundCloud. I got offered a job after successfully completing the coding challenge and the on-site interviews in Berlin. I worked for them for 5 years.

Create a web app that allows users to create playlists of ANY tracks on SoundCloud.

Source code: https://github.com/salomvary/sc-playlist

Live: https://salomvary.com/sc-playlist/

Facebook

A Facebook recruiter reached out to me in 2014 about a front-end engineer role. I could not resist and entered the interview process, however, after a successful coding challenge submission I failed the on-site interviews in Menlo Park.

Given a set of events, each with a start time and end time, render the events on a single day calendar.

Source code: https://github.com/salomvary/fb-challenge

Live: https://salomvary.com/fb-challenge/

Axel Springer

This one was for a freelance project in 2019. I successfully passed the interview process but ended up not working for them for other reasons.

[…] App […] which gathers data from two endpoints asynchronously, merges the responses and displays them in any way in the browser.

Source Code: https://github.com/salomvary/as-challenge/

Live: https://salomvary.com/as-challenge/

Bonus Track: Prezi

I applied to a developer position at Prezi.com in 2012. They did not ask for a code challenge but I was invited for a non-binding probation week and starting with a self-introduction using the company’s own presentation product. They offered a job but ended up not working for them because I decided to join SoundCloud instead.

Online presentation on Prezi.com

Type Checking Vanilla JS with JSDoc and TypeScript

Wed, 24 Apr 2019 00:00:00 +0000

Ever wondered about writing plain ~~old~~ modern JavaScript and still enjoying the benefit of strict type checking? Not quite ready to fully migrate existing projects to TypeScript (or Flow or something else)? The good news is: it’s doable!

A less known feature of the TypeScript compiler is it’s ability use type definitions added to JavaScript by using JSDoc comments.

When is this useful?

You are familiar with JSDoc but not ready to learn TypeScript yet.
You are prototyping or targeting modern browsers only.
You want advanced IDE support like code completion and highlighting errors.
You don’t want to use a transpiler, Webpack and friends.
But want to have all the warm feeling of type safety.

Getting started

Enabling TypeScript support on a JavaScript project is straightforward. Add a tsconfig.json file at the project root with the following contents:

{
  "compilerOptions": {
    // Allow checking JavaScript files
    "allowJs": true,
    // I mean, check JavaScript files for real
    "checkJs": true,
    // Do not generate any output files
    "noEmit": true
  }
}

Let’s also add some example code:

greeter.js:

export class Greeter {
  /**
   * @param {string} name
   */
  greet (name) {
    console.log(`Hello ${name}`)
  }
}

main.js:

import { Greeter } from './greeter.js'

const g = new Greeter()
g.greet()

If you open main.js in Visual Studio Code or another editor that has integrated TypeScript support you will see a compile error at g.greet() because we forgot to pass in a required string argument. If you install the TypeScript compiler with npm install --save-dev typescript you can also use the following command to verify your JavaScript code from the command line:

npx tsc

In our case the output will look like this:

$ npx tsc
main.js:4:1 - error TS2554: Expected 1 arguments, but got 0.

4 g.greet()
  ~~~~~~~~~

  greeter.js:5:10
    5   greet (message) {
               ~~~~~~~
    An argument for 'message' was not provided.


Found 1 error.

The compile error can easily be fixed by adding an argument to the function call: g.greet('John').

Using dependencies

Any real project will eventually end up using dependencies from the npm registry. Since all major browsers support native ES6 modules and an increasing number of npm packages are published in this format, it is nowadays possible to develop modularized codebases without any bundling or transpiling tool like Webpack.

It is totally fine to use use a tool, but if you are adventurous like me you can try going all the “vanilla way”. This however needs a bit of further tweaking.

We will use the lodash-es npm module as an example. It is Lodash in the form of many small ES modules.

First install using npm install --save lodash-es. We can import and use the modules directly like this:

import upperFirst from '/node_modules/lodash-es/upperFirst.js'

upperFirst('foo bar')

However the TypeScript compiler does not give us any “protection” here, upperFirst has any type meaning whatever we do, it will compile. We need to install TypeScript definitions separately with npm install --save @types/lodash-es. Once that is done, our tsconfig.json needs to be tweaked a bit:

{
  "compilerOptions": {
    "allowJs": true,
    "checkJs": true,
    "noEmit": true,
    // This resolves types for imports
    // with /node_modules/ prefix
    "baseUrl": ".",
    "paths": {
      "/node_modules/*.js": [
        "node_modules/@types/*"
      ]
    }
  }
}

That’s it. At this point npx tsc should check calls to upperFirst and fail when we make a mistake like passing in an argument of wrong type like upperFirst(42). Visual Studio Code should also pick up the definitions:

Summary

The TypeScript compiler allows type checking code written in plain JavaScript and also leverages type definitions created for third party libraries. This not only allows using the compiler as a “very smart linter” but also improves editor and IDE integration (code completion, highlighting errors). This - combined with the power of native ES6 modules - results in lightweight tooling for projects of any size.

One last thing to keep in mind: not all of TypeScript is supported as JSDoc comments and not all of JSDoc syntax is supported by the compiler. Be sure to check out the documentation and understand the differences.

Introduction to WebAssembly and Emscripten

Mon, 08 Apr 2019 00:00:00 +0000

What is Web Assembly?

WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. ^[source]

Yeah, but what is it really?

Wasm means high level languages like C, C++ or Rust running in the browser nicely interoperating with JavaScript. At near native speed!

Why is that interesting?

You might say JavaScript is already super fast, why do we need something even faster? No matter how fast it is, JavaScript will never beat the performance of a compiled C program in many areas. Think about computation heavy stuff like multimedia or cryptography. Or blockchain!

That on one hand means you can hyper-optimize the heavy part of a JavaScript application using Wasm and keep the rest untouched. On the other hand you can take existing high performant C library off the shelf, compile it to WebAssembly and use it without even knowing how to write C code.

First steps

To start we will need something to compile to Wasm and something to compile with. The C/C++ to Wasm compiler toolkit is called Emscripten and can be downloaded from here. Follow the instructions on the same page to set it up.

Let’s also find something to compile! Because I am working on a multimedia project that uses ffmpeg.js (also compiled with Emscripten) for encoding audio data into AAC format I was wondering how easy would it be to port a standalone AAC encoder to the browser. I will use it as an example for the rest of this post.

Any C/C++ project will do do it as long as it uses the standard ./configure; make; make install toolchain like (probably) 99% of the programs written for UNIX/Linux. If you never used these tools here is a brief explanation.

Compiling the project to Wasm

Once Emscripten is set up we can run the compiler on the C/C++ project with the following commands:

# Not every project has autoconf.sh, FDK AAC does
./autoconf.sh
# Note that we run emconfigure ./configure not just ./configure
emconfigure ./configure
# Depending on the project size, this might take a while:
emmake make

These steps might be slightly different for a given project, it is recommended to consult the project documentation for compilation instructions and then substitute ./configure with emconfigure ./configure and make with emmake make.

If the last command succeeds you should have a compiled LLVM bitcode (sort of an intermediary format before the final Wasm binary) somewhere. Figuring out where it exactly is is a bit tricky as it depends on the project setup. Usually the last lines printed from the emmake make command should give a good hint. In my case, the result is at .libs/aac-enc. You can make sure you are looking at the right format by running file on the output file, like in my case:

$ file .libs/aac-enc
.libs/aac-enc: LLVM IR bitcode

The last step is producing a Wasm binary and a JavaScript wrapper from the LLVM binary. It usually goes like this:

emcc project.bc -o project.js

Where project.bc is the binary produced in the previous step. If it does not have a .bc extension, just rename it, otherwise emcc will complain.

If emcc complains about “undefined symbols” you probably have to include the shared dependencies of the “main” binary on the command line. This step requires familiarity with the build tools and/or the project or some fiddling and intuition. If it’s not obvious what the dependencies are try to find more “LLVM IR bitcode” type of files near the root folder of the project. In the case of the FDK AAC project the complete command is:

emcc .libs/aac-enc.bc .libs/libfdk-aac.2.dylib -o aac-enc.js

This should result in a JavaScript file and a Wasm file ready to be executed from Node.js or a web browser. For example the AAC encoder prints out the usage:

$ node aac-enc.js
aac-enc.js [-r bitrate] [-t aot] [-a afterburner] [-s sbr] [-v vbr] in.wav out.aac
Supported AOTs:
  2	AAC-LC
  5	HE-AAC
  29	HE-AAC v2
  23	AAC-LD
  39	AAC-ELD

If you load the same file in a browser using a <script> tag (or open the generated aac-enc.html) you get the same output logged to the console. Cool but not very useful.

Making use of the compiled code

Encoding WAV files with the compiled Wasm code requires a little more work: we need to pass in files and arguments to the executable.

First we need to recompile the module with the File System API exposed:

emcc .libs/aac-enc.bc .libs/libfdk-aac.2.dylib -o aac-enc.js \
  -s EXTRA_EXPORTED_RUNTIME_METHODS=['FS']

With the recompiled module at hand we have to write a bit of glue code. Interacting with the generated code is possible via the exposed Module object.

Add this before loading aac-enc.js to prevent running the encoder with no arguments initially and to actually expose the module:

var Module = {
  noInitialRun: true
}

Note: the generated JavaScript file is neither a CommonJS module nor an AMD or ES6 one. It’s a global that can optionally be created beforehand for passing in configuration options. It is possible to emit a UMD module instead using the -s MODULARIZE=1 flag of the emcc command or -s EXPORT_ES6=1 for ES6 modules. See more in settings.js.

Next we will have to set up an input file and copy it to the “virtual” file system of the module:

Module.FS.writeFile('input.wav', new Uint8Array(inputBuffer))

The second argument to writeFile is the content of the file in the form of a TypedArray. Obtaining inputBuffer depends on the environment and where the input comes from. Examples:

// If you have a File instance eg. from a form input:
inputBuffer = await (new Response(inputFile).arrayBuffer())

// If you have a URL
inputBuffer = fetch(inputUrl).then((res) => res.arrayBuffer())

// In Node.js with a file path
inputBuffer = fs.readFileSync(inputPath)

Now we can invoke the main method of the compiled application:

Module.callMain(['input.wav', 'output.aac'])

And the last step is to read the output back from the file system:

// Read virtual file into a TypedArray
const output = Module.FS.readFile('output.aac')
// Turn the TypedArray into a File and return it
const outputFile = new File([output], 'output.aac', {type: 'audio/aac'})

The outputFile can then be turned into a blob URL with URL.createObjectURL and downloaded from the browser or directly used in an <audio> element as src. When using Node.js output can be directly passed in to fs.writeFile.

The above code is available as a working example here.

Where to continue from here?

The above simple example is only the beginning. In real life you might want to turn the code above into something reusable, which needs a bit more complicated glue to account for the initialization phase of the Wasm module and to cover error scenarios. I started turning the FDK AAC encoder example into a more elaborate library, the work-in-progress state is available here.

Some to-be-ported codebases might do more than read command line arguments and files, eg. use threads, graphics or audio, which needs special treatment. The Emscripten documentation has an entire chapter on porting.

Even in our arguments-and-files use case, we could do better. The underlying library exposes C functions to encode AAC in a buffered way, without using files directly. This has the advantage or more fine grained control of execution and potentially using less memory but the glue code will be more complicated. It also requires some knowledge of C programming. A good start is the Calling compiled C functions from JavaScript section in the Emscripten docs.

It is also important to note that the Emscripten commands above generate non-optimized code. Make sure you read and follow the Optimization section before using Emscripten and Wasm in production.

Vue.js State Management vs. Clean Architecture

Fri, 11 Jan 2019 00:00:00 +0000

Being independent of frameworks is an important attribute of clean architecture. This post explores the challenges of managing the application state in a way that is less specific to Vue.js.

This is how Ucle Bob explains it in The Clean Architecture:

The architecture does not depend on the existence of some library of feature laden software. This allows you to use such frameworks as tools, rather than having to cram your system into their limited constraints.

This means you should not need to be writing the business logic of your application in a framework prescribed way.

Vue.js basics

Vue.js is a JavaScript framework for building user interfaces for the web. At the heart of the framework lies a reactivity system that maps the component state to rendered DOM.

The example snippet below renders <b>Hello Vue!</b> and if the value of message changes the rendered element gets updated as well:

new Vue({
  data: {
    message: 'Hello Vue!'
  },
  template: '<b>{{ message }}</b>'
})

Reactive state limitations

Any property under data becomes a reactive property when the Vue instance initializes. One might think that this means data can hold arbitrary data structures which can be manipulated any way we like. Unfortunately that is not true, there are limitations:

Object property addition and deletion is not reactive
Array item reassignment is not reactive
ES6 iteration protocols are not supported which means no v-for on custom iterables and no support for Set, Map and friends.

I have not found a single authoritative source explicitly listing which types are supported and which are not as data properties in Vue.js. It seems like the best approach is to stick to using “basic” types (Array, String, Symbol, Number, Object, Boolean and undefined) only and avoid anything else (no custom prototypes, no Date or other language or browser built-in types).

So where is the problem?

The limitations on what types and what methods Vue allows to use might not appear to be a big deal for small and simple use cases. A real life application might eventually reach a point where there appears to be a piece of logic that is independent from the presentation and will benefit from encapsulation - this logic is often called business logic.

The clean architecture approach recommends a business logic implementation to be implemented in a framework independent way but Vue’s constraints mean we can not leverage the full power of modern JavaScript (or whatever language we compile to JavaScript). When implementing data structures or algorithms to be used for representing or manipulating component state, we have to play by the Vue.js rules.

An example

Let’s say we want to implement an application that shows a timeline and allows adding events to it:

<ul id="app">
  <li v-for="entry in timeline">
    {{ entry.year }}: {{ entry.event }}
  </li>
  <li>
    <input type="number" v-model="year">:
    <input type="text" v-model="event">
    <button @click="addEvent">Add event</button>
  </li>
</ul>

<script>
  new Vue({
    el: '#app',
    data: {
      year: 2018,
      event: 'Someone clicked a button',
      timeline: [
        {year: 1946, event: 'ENIAC'},
        {year: 1981, event: 'IBM PC'},
        {year: 2007, event: 'iPhone'}
      ]
    },
    methods: {
      addEvent (event) {
        this.timeline.push({
          year: this.year,
          event: this.event
        })
      }
    }
  })
</script>

This simple example works as expected as we are mutating the state in a Vue.js compliant way.

Extracting the business logic

As the application evolves we might realize that we often have to add events to the timeline, we might want to prevent certain operations on the timeline array or make sure it is always sorted. This will result in extracting a Timeline class, something like this:

function Timeline (initialEntries = []) {
  // Keep the cloned entries array private
  const entries = initialEntries.slice()

  return {
    add (year, event) {
      entries.push({year, event})
      entries.sort((a, b) => a.year - b.year)
    }
  }
}


// Create a timeline with some entries
new Timeline([
  {year: 1946, event: 'ENIAC'},
  {year: 1981, event: 'IBM PC'},
  {year: 2007, event: 'iPhone'}
])

By wrapping the array and keeping it private we can add our own manipulation methods which can ensure the structure is solid.

However if we substitute the plain array with a Timeline in Vue data it will no longer work. Vue has no idea how to iterate over a custom class, we have to convert it to an array:

function Timeline (...) {
  return {
    // ...
    toArray () {
      // Clone the private array
      return entries.slice()
    }
  }
}

An even cleaner option would be implementing a custom iterable but that is not something Vue can directly use.

Calling toArray() from v-for will have the desired results:

<li v-for="entry in timeline.toArray()">

We are less lucky however when it comes to adding entries to the timeline. Even if we replace push with timeline.add this will not result in updating the rendered DOM:

methods: {
  addEvent (event) {
    this.timeline.add(this.year, this.event)
  }
}

Looks like we reached a dead end and in fact we have been told to only use “basic” types and don’t even try to add our own methods.

The way forward

Turns out the workaround is simple: instead of updating an object assigned to a data property in-place, entirely replacing the instance does kick off Vue’s reactive updates:

methods: {
  addEvent (event) {
    // Clone the timeline
    const timeline = new Timeline(this.timeline.toArray())
    // Add an entry to the new timeline
    timeline.add(this.year, this.event)
    // Replace this.timeline with the new instance
    this.timeline = timeline
  }
}

Sharp-eyed readers might notice that at this point we could implement Timeline as an immutable data structure as there will never be a need for in-place updates.

This approach has some downsides though:

It is more code, and more code is more prone to errors and has a higher maintenance cost.
Iterable data structures have to be converted to plain arrays before passing in to Vue. This has a performance cost.
A custom class exposing both immutable and mutable methods can lead to confusion as using the latter will not work with Vue.

Conclusion

Your app might be simple enough or 99% of the business logic lives on the server so that the Vue.js constraints discussed here have no big impact on the architecture.

If you are building something more complicated, my recommendation is the following:

Use Vue.js for building simple and “dumb” components with as little business logic, as possible.
Use Vuex for managing non-trivial application state but keep in mind, Vuex state has the same limitations as Vue.js data.
Keep everything else outside of Vue.js and Vuex and accept some complexity tradeoffs. For example:
- Implement data structures and algorithms in their own classes and modules. Consider using immutable data.
- Separate data access (communication with the server or other storage) in it’s own layer (see the Repository Pattern).

Snapshot Testing Web Audio Code Meetup Presentation

Wed, 05 Dec 2018 00:00:00 +0000

I had the opportunity to do a lightning talk about audio snapshot testing at the last edition of the Berlin Web Audio Meetup.

The example code and a longer explanation are available in this repository.

My First Year as a Freelance Software Engineer

Fri, 23 Nov 2018 00:00:00 +0000

In the summer of 2017 my last permanent job unexpectedly came to an end. I used this opportunity to start doing something different.

Having been an employee for almost two decades I had the idea of becoming a freelancer on my mind for many years. I always found the flexibility of time and space, a wide variety of ever changing projects and my success more depending on my own abilities (as opposed to office politics) appealing in freelancing.

I believe running a business (even if it’s a one-person show) is a useful experience and therefore my career would benefit from becoming independent for a while and offering my services directly at the free market.

Disclaimer: this post is more about self reflection than giving advice to others who want to hop on the freelance train. I do not consider myself an expert in this field, take everything with a grain of salt here and be sure you talk to a qualified expert before making a move.

How did I start?

Getting laid off together with another 140 people from SoundCloud was the best kick start I could have wished for. The layoffs were big news and when I added myself to the Hire a SoundClouder spreadsheet recruiting requests started pouring in. As usual in recruiting, 90% of the inquiries was irrelevant, still by the time I read through hundreds of emails I ended up with a handful of promising leads.

The gardening leave gave me enough time to prepare the paperwork for becoming a freelancer. The first step was to find a tax accountant (Steuerberater). Setting up everything on your own is theoretically possible but I have never heard of anyone in this business doing that.

I briefly considered becoming an e-Resident of Estonia too which supposedly offers no-nonsense bureaucracy for businesses, however being a resident of Germany means I have to deal with the local authorities anyway. Maybe one day for a side-business.

I found a tax advisor who speaks English (back then I was not too confident with my German) and was willing to help. They are quite old fashioned bureaucrats: papers, folders, ring binders everywhere. Germany is a paper-stamp-and-signature based country to an extreme extent, tax advisors are no exception. I convinced them to exchange documents via email although they plain refused to use anything else like DropBox or one of the many cloud based bookkeeping services.

Running a freelance business requires a bank account which is recommended to be a separate one from the one you use for your private needs. As someone who prefers paperless solutions I went for N26 which fulfilled all my expectations. The only paper they used was the envelop they sent the credit card in.

Although this is not a requirement I subscribed for Debitoor, an invoicing and bookkeeping platform. Services like this allow collaboration with your tax accountant but I am currently not using this feature for the above mentioned reasons.

Registering myself with the tax authorities took about a week and from then on I was all set.

The first projects

Thanks to the viral Hire a SoundClouder spreadsheet I had two contracts before even my gardening leave ended. The idea was to work on two projects the same time with 50%-50% time share totaling about 40 hours per week. The motivation behind: more exposure to projects, easy backup if one of them flops, testing out how part time commitment can work.

One quickly turned out to be a flop due to toxic environment and wildly misaligned expectations, the other is still a client I work for. Working on these two contracts taught me that booking myself to 100% of my available time on multiple projects way too much.

Despite closing the deals with the very first clients was quick and low effort, in order to learn more about the market I kept talking to potential clients and recruiters. I probably had a conversation with 15-20 potential leads.

I learned the following: many companies want permanent employees and not freelancers; positioning myself as a generalist might not be the best idea; pricing is insanely hard.

Because I had very little idea how to set a price tag, I used the first leads to experiment in this field. I roughly calculated a baseline hourly rate from my previous year’s salary as an employee and used a -50% to +100% range. I received a wide variety of reactions. They ranged from laughing loud, never answering emails again or to politely negotiating down by 5%.

Self improvements

I believe continuous self improvement in this profession is essential and have been practicing ever since I entered the software engineering career decades ago. I made a few moves over the last year specifically to support my freelance ambitions.

I hired a German teacher and are taking regular lessons. As a result I am no longer scared to join project at German speaking companies. It also gives me more confidence when dealing with legal matter and paperwork.

Also read a couple of “self-help” books on how to be better at running myself as a business:

How To Win Friends And Influence People by Dale Carnegie: large collection of anecdotes on getting along with other humans. Not my favorite but a good start and inspiration on improving people skills. It’s also very American.
The Software Engineer’s Guide to Freelance Consulting by Zack Burt: a kick start into an unknown new world, summarized the most important aspects of freelancing in a compact form. Some parts are US specific.
The Personal MBA by Josh Kaufman (in-progress): another book from the “for the impatient” league, if you never studied business and whatever you learned about economics at high school or university is gone, this book can be used as a refresher. Some concepts are difficult to apply to services like software development.

What worked well

Over the last year I managed to expose myself to working for a wide range of actual and potential clients and this helped me to gain enough experience and confidence to continue. The original plan was to freelance for at least one year and see if it works for me; today the answer is a definite yes.

One important motivation was to be independent of time, space and of fixed working hours. I had doubts in the past whether remote work is for me but a 5 months long contract showed me I can be efficient without showing up in an office every day.

Working less than 40 hours a week is also a big win! I spent most of the extra time on well-being, health and self-improvement. I do not mind the money I did not make while not working at all as long as I can pay all the bills. Although it is hard to measure I think the time I spent working was more focused and the outcome of high quality. The positive feedback I received so far confirms this.

What could I have done better

Contrary to the above I did not organize my time as efficiently as I should have. I did not end up in complete chaos like working at night and sleeping during the day but I wish I was more disciplined with my daily routines. Working remotely on a long project is way more challenging than it sounds.

I could have relied on my senses more when spotting problematic projects and toxic environments. Each disappointment and conflict was preceded by a bad feeling and I still ignored the feeling and went ahead. The lesson? No money is worth causing myself a large amount of pain.

I had big plans for networking and exposing myself “outside” a year ago which quickly faded after the initial excitement, primarily due to laziness. I still believe I should attend meetups or conferences regularly, maybe even present some times, participate in developer communities (especially in-real-life mentoring and helping out people in online forums).

My online portfolio received a quick facelift and managed to add some content but the amount of blog posts was nowhere near what I planned.

Talking of the portfolio, I am still uncertain whether positioning myself as a full-stack software engineer is a good idea (as opposed to a “specialist of something”). Most of the time companies and recruiters are searching for more specific skills (and sometimes too specific for my taste). Even though I could perfectly fit into many of these projects marketing myself as a generalist might turn potential clients away.

How does the future look like?

I will stay on the freelance track, figure out how to position myself better and work on the improvement areas. I will keep you posted!

Tools and services I use

Readers might ask what tools I use to keep the freelance business going. The non-exhaustive list is below:

N26 for banking
Toggl for time tracking and reporting
Tadam app for Pomodoro timer to keep me focused
Debitoor for invoicing and expense tracking
Doxie Go SE scanner to scan and OCR incoming documents
Document shredding bin (Datenentsorgungsbox) from BSR (Berlin’s recycling company) to dispose scanned and archived documents

Things I Learned While Working at SoundCloud

Thu, 26 Apr 2018 00:00:00 +0000

My almost 5 years of employment as a software engineer at SoundCloud came to an abrupt end last summer. Enough time has passed since my last working day so that I can now reflect on my experience and recall the most important takeaways.

Working in a “fast moving environment” taught me how to efficiently work together with people of a wide variety personalities, adapt to different team structures and project management methodologies.
Fast moving environment has downsides too. By the time I established a context with my managers either they moved on or I switched teams and had to start over with someone else.
Being surrounded smart people - especially ones smarter than me - was a great experience. Never lacking inspiration and whenever I got stuck with something there was an expert happy to help out.
I appreciated the postmortem culture, the culture of learning from our own mistakes and not blaming individuals, which was adopted by the entire engineering team.
I learned how there are different ways of providing feedback some are better, some are worse, especially when you disagree. Used these learnings to improve my approach to giving and receiving criticism a lot.
I had no idea how complicated and in many ways problematic the Music Industry is before joining SoundCloud. There I learned a lot, including many things I never wished to know.
SoundCloud was the first place where I worked in a microservice architecture. Learned many lessons on the good and bad ways of implementing this pattern while building a couple of services from scratch and contributing to a lot more.
I learned about the open source model or collective code ownership of shared libraries or services and how lack of clarity and resources can defeat this otherwise great idea.
I got completely sold on continuous delivery and continuous integration as the True Way of shipping software in an agile environment.
As SoundCloud moved towards self-service end-to-end full-stack teams I wore the DevOps hat too. The upside was shipping things even faster, the downside long hours of firefighting at weekend nights (with compensation to be fair).
I also learned the hard way how low quality tooling, slow and flaky builds and convoluted continuous integration setups cause extreme amount of developer frustration, anger and loss of productivity.
I learned the importance of writing high quality commit comments, tried to do my best on improving code documentation too.
I am glad I had the opportunity to write production code in Scala, Java, Ruby, JavaScript, TypeScript, Go, Clojure, SQL, Bash, POSIX shell and maybe others I can no longer recall.
I learned to use singular they as a personal pronoun and not to use “guys” to address a group of people. Had many other lessons about the importance of diversity and inclusion.
No other previous job taught me lessons about dealing with my own unconscious bias.
I also learned the importance of avoiding leading questions while interviewing candidates.
Among many other communication tricks I learned how to email a large group of people using bcc to prevent someone accidentally sending a reply to the entire company.
Black belt masters mentored me in the skill of contributing to any topic on any communication channel using only GIFs and emojis.
I learned who Martin Fowler is and how no serious technical blog post can exist without citing him at least once.
I researched the Probability of Video Call Working in Both Directions Theory during my tenure and established that it is a constant 6.25%.

Real World Experience with ES6 Modules in Browsers

Thu, 25 Jan 2018 00:00:00 +0000

All modern browsers landed native ECMAScript 6 modules support last year. I was excited to try them out on a real project and did not have to wait for too long. I started prototyping a web project for a client a few months ago and decided to use native modules as long as it does not prevent me from getting things done.

ES6 modules in browsers?

This post does not explain browser-native modules in detail. I recommend reading Jake Archibald’s excellent article if you want to know more. Here is an example with the essentials:

index.html:

<script type="module" src="app.js"></script>

app.js:

import lib from './lib.js'

lib.doStuff()

You will need a static web server (most likely you already have one) for development as ES6 modules can not be loaded directly from the file system.
If modules are served from a web server with authentication enabled (eg. HTTP basic) make sure the script tag has the crossorigin="use-credentials" attribute to avoid surprises.

Using external dependencies

As long as you write your own modules and import them with ES6 module syntax, you are lucky. Eventually any non-trivial project will depend on external dependencies. And this is where things will become complicated.

Let’s start with an overview of how people write and publish JavaScript libraries nowadays:

Written in ES6 syntax or older without using ES6 modules
Written in ES6 and using ES6 modules
Written in a compile-to-JavaScript language (like TypeScript or CoffeeScript)

Before publishing to npm or a CDN libraries are compiled into one or more other formats that can (or can not) be consumed by browsers or certain versions of Node.js. The most common publishing formats are:

Published as CommonJS
Published as Universal Module Definition (UMD) or something else that even older browsers can directly use (or with a lightweight loader polyfill)
Published as ES6 modules

Now comes the sad reality. Most of these formats can not be used with the shiny new import './app.js' syntax. Anything that is not in the new module format simply does not work.

Unfortunately even most libraries authored or published in ES6 module format will not work because they target transpilers and rely on the Node.js ecosystem. Why is that a problem? Using bare module paths like import _ from 'lodash' is currently invalid, browsers don’t know what to do with them.

Using UMD modules with import is not possible either. Browsers throw a SyntaxError when attempting to parse a file as an ES6 module if it contains no export definition.

In real life therefore:

You will still need Webpack or something similar to load non-ES6 modules or rely on plain old script tags and global variables. Not pretty but might work until the dependencies are only a handful.
Libraries published in browser-friendly ES6 format can be used directly from a CDN or from a local node_modules folder when served by the web server. In that case you will have to spell out the whole path including node_modules.

Library usage examples:

// Some authors already publish in browser friendly ES6 format.
import Vue from 'https://cdn.jsdelivr.net/npm/vue@2.5.13/dist/vue.esm.browser.js'

// Lodash in ES6 format is published under a different name. When
// using npm, the whole node_modules folder has to be exposed by the
// static web server.
import camelCase from './node_modules/lodash-es/camelCase.js'

// Other libraries are authored in ES6 module format but use bare
// module paths which does not work in browsers. CDNs like Unpkg can
// help by expanding the modules.
import scale from 'https://unpkg.com/d3-scale@1.0.7/index.js?module'

// Sadly there is no module expansion when installed with npm, this
// will NOT work.
import scale from './node_modules/d3-scale/index.js'

See these ES6 module loading examples in your browser.

Loading templates or CSS with import

Quite common practice in transpiler-backed workflows is importing precompiled template or CSS files as modules. Unfortunately native browser imports strictly require the response to be served with JavaScript content type and the body must be parsable as an ES6 module.

There is hope though. One day browsers will implement the System.loader API which would allow adding custom module loaders but the standard is not even ready to be implemented.

Custom loaders with Service Workers

As someone pointed out Service Workers can intercept Fetch API requests. This comes in handy as ES6 import uses fetch behind the scenes.

That means it is possible to turn any fetched resource into a module by turning the response into a string that the browser can parse as ES6 code. For example a template file can be turned into an exported JavaScript string or function.

Unfortunately Service Workers are not yet as widely implemented by browsers as ES6 modules but I was too curious and implemented a rudimentary text file loader. Check out the examples and more specifically the worker itself:

self.addEventListener('fetch', (event) => {
  // Hijack all fetch requests to URLs ending with txt
  if (event.request.url.endsWith('.txt')) {
    event.respondWith(
      fetch(event.request.url)
        .then((response) => response.text())
        .then((body) => {
          // Export the response body as a JavaSript string.
          // The response body has to be sanitized before turning it
          // into JavaScript code.
          // Credits: https://stackoverflow.com/a/22837870
          const newBody =
            `export default "${JSON.stringify(body).slice(1, -1)}"`
          // Replace the original response with an ES6 module
          return new Response(newBody, {
            headers: new Headers({
              'Content-Type': 'application/javascript'
            })
          })
        })
    )
  }
})

This idea can be adopted to load CSS into the page or compile templates on the fly. Beware however, Service Workers is a very new technology.

Fun fact: as of writing you can not write Service Workers as ES6 modules.

Bundling

Any serious project will at some point have to support older browsers that don’t implement native ES6 modules yet. Unless the project only has a very few and small dependencies fetching hundreds of small JavaScript files will become a bottleneck. Bundling is inevitable.

Widely used bundling tools like Webpack or Browserify will have no problem transpiling ES6 modules. Likely you will transpile the code into pre-ES6 modules, therefore <script type="module"> will have to be rewritten to plain old script tags. Check the documentation of your tool to see how that is possible.

I used AssetGraph-builder to bundle the entire project and it did indeed need ugly workarounds as AssetGraph itself does not yet support ES6 modules.

Further gotchas

Since browser-native modules are a very new technology expect development tools to choke on them here-and-there, one such example is Karma I had trouble with.

Another challenge can be supporting old browsers. If you develop using ES6 native modules your raw code will only work in the latest browser versions.

Sharing code in ES modules in Node.js can be tricky too. Native ES6 modules in Node are behind the --experimental-modules flag and require naming modules differently using the .mjs extension. You can use babel-node but (surprise!) “ES6-style module-loading may not function as expected”.

Conclusion

You can start using ES6 modules for experiments, prototyping and small projects today. Anything beyond that will eventually require more tooling. In fact introducing Webpack in development is soon going to happen on my current client project.

My wishlist for the future:

First class ES6 native module support in build tools so that we can still ship optimized and backwards compatible code in production.
Custom loaders for importing templates, CSS and others as modules so that we can use the same approach for modularizing my code everywhere.
Maybe a smart static HTTP/2 server that allows importing large dependency trees (many small modules) efficiently without bundling so that there is no need for insanely complicated development servers. See this interesting topic on ES Discuss.

Switching Back to Firefox

Fri, 22 Dec 2017 00:00:00 +0000

The first web browser I remember using was Netscape Navigator Gold 3.0. I kept using Navigator and its successor Firefox (except for a short affair involving Konqueror while I was using KDE) as my primary web browser both for everyday browsing needs and web development until Google Chrome came out about 9 years ago. Impressed by the speed I switched and did not look back. Until recently…

I was still frequently exposed to Firefox though. As a web developer I always worked on projects that had to support all recent mainstream browsers which includes testing and fixing bugs on Firefox. And I was not impressed. Neither with the visual appeal and ergonomy of the user interface nor the speed and quality (as in lack of bugs and crashes) of the browser engine.

Why bother?

I believe software monocultures are bad and still remember the days when Internet Explorer dominated the web and websites often plain refused to accept me with “This website requires Internet Explorer” or similar messaging.

Feeling increasingly bad about being part of WebKit and Chrome monoculture I tried switching to Firefox earlier this year and failed. It was way too slow, way too buggy and particularly bad for my web development needs.

Then came the Firefox Quantum announcement in November 2017 where Mozilla promised to have made Firefox real fast and and pretty this time. So I decided to give it another try, something I did not regret.

The new Firefox

My first impressions about the user interface were like “wow, this looks way better than it used to!” There are some minor quirks I don’t particularly like, for example the contrast between an active and an inactive window but in general it looks modern (flat design!) and ergonomic.

The download and page rendering speed improvement feels impressive. It still does not feel as fast as Chrome but the difference is small enough not to bother me. I have not had more than one or two crashes over the first month and one of that was caused by a developer tools extension.

Talking of developer tools, this is the only area where Firefox falls short. It has all the features I am used to from Chrome but the whole experience feels sluggish especially when it comes to using the debugger. Sometimes a reloading and stopping on a breakpoint takes 10-20 seconds which is beyond my patience threshold. I also keep bumping into an annoying bug where “quick look” into variables sometimes shows undefined when there is definitely a value. Still, the features of developer tools perfectly meet my needs, the built-in Web Audio debugger is a nice extra I never discovered before.

One more thing. With the Quantum release Firefox deprecated the old extensions api which means from now on extensions have to be written using the new cross-browser Web Extensions standard (sans Safari). All but one of the Firefox extensions I use have been rewritten to the new format. The only thing that drove me mad is the Send to Kindle extension which Amazon did not bother to upgrade. I really do not see how a near trillion dollars worth company did not have the time and resources to fix their official extension…

Firefox mobile

Based on the success on desktop I installed Firefox on my iPhone too. Not that it makes a big difference on terms of browser engine, rather for going full-on with the Firefox journey.

No big complaints here, it does the job. The “readability mode” is slightly inferior to the one of Safari and I never know certain features are under the top “…” menu or the bottom hamburger one.

Firefox mobile actually comes in two separate editions the second being Firefox Focus (branded as Firefox Klar in German speaking countries) for enhanced privacy and ad blocking but I have not had time so far to form an opinion about it.

My future with Firefox

Looks like Firefox will remain my primary browser on all platforms for the foreseeable future. I have high hopes that Firefox will remain on the market, hopefully even gets stronger and Mozilla will not mess it up.

How to Continuously Deploy a Front-end Project Using GitLab and Surge

Fri, 15 Dec 2017 00:00:00 +0000

I recently built a web project from scratch for a client who had no existing infrastructure for web development. As a believer in continuous integration I decided to go with automated deployment from the very beginning.

After a quick research on off the shelf solutions and came up with the following combination:

Host the source code on GitLab
Build and deploy using GitLab’s continuous integration pipelines
Host the web application on Surge

The advantage of this setup is the lack of the need for setting up your own server software while both cloud solutions have free plans to start your project with. Moreover GitLab’s free plan includes private repositories.

This blog post will guide you through the first steps of setting up GitLab and Surge for your own web project.

Requirements

It is assumed you know the basics of Git, have a GitLab account set up and configured with your ssh key and have the following software installed:

Recent Node.js
Git command line

Setting up the project

If you don’t have one yet, use your favorite text editor or IDE to create a web project in an empty folder with an index.html at the top level. You can also copy the html, js and css files from the example project I created for this post.

Publishing the project to GitLab

First you will need a local Git repository. Set one up using the commands below:

git init
git add index.html main.css main.html
git commit -m 'First commit'

Next create a new project on GitLab and publish the local repository with these commands (copy the origin URL from GitLab):

git remote add origin git@gitlab.com:some-username/my-project.git
git push -u origin --all

Adding static web hosting

We will use Surge for quick and free static web hosting. It comes with a command line tool that can be installed using npm. Run these commands in the root of the project:

npm init -y # If you don't have a package.json yet
npm install --save-dev surge
node_modules/.bin/surge .

The first time it’s used Surge prompts for an email and password. Choose a subdomain you like or let Surge generate a random one. The output should look like this:

$ node_modules/.bin/surge .

Surge - surge.sh

          email: example@example.com
          token: *****************
   project path: .
           size: 1 files, 57 bytes
         domain: my-awesome-project.surge.sh
         upload: [====================] 100%, eta: 0.0s
propagate on CDN: [====================] 100%
           plan: Free
          users: example@example.com
     IP Address: 45.55.110.124

Success! Project is published and running at my-awesome-project.surge.sh

Open http://my-awesome-project.surge.sh in your browser to see the published web project.

Important: Surge prompts for a subdomain every time you run it, if you want to stick with one, save it with the project:

echo my-awesome-project.surge.sh > CNAME

This is the right moment to make a Git commit:

echo node_modules >> .gitignore # Avoid commiting Node.js dependencies
git add CNAME package.json package-lock.json .gitignore
git commit -m 'Added Surge static hosting'

Setting up continuous deployment

GitLab’s continuous integration pipeline can be configured with .gitlab-ci.yml placed in the root folder of the project. Create one with the following content:

image: node

before_script:
  - npm install

deploy:
  stage: deploy
  script:
    - node_modules/.bin/surge .

Before pushing .gitlab-ci.yml to GitLab the pipeline has to be configured with Surge access credentials so that it can publish to the domain you configured in the previous step. First obtain a token from Surge:

$ node_modules/.bin/surge token

    Surge - surge.sh

              email: me@example.com
              token: *****************
              token: <COPY THE TOKEN FROM HERE>

Next go to GitLab -> Your project -> Settings -> CI / CD -> Secret variables

Add a key named SURGE_LOGIN and set the value to the email you use with Surge
Add a key named SURGE_TOKEN and paste the token into the value from the output of surge token

And now commit and push .gitlab-ci.yml to GitLab. In a short while you should see the pipeline running under GitLab -> Your project -> CI / CD -> Pipelines. If it fails click on the red failed pipeline and the failed job to see the full terminal output of the deploy job for troubleshooting.

From now on every change you push to GitLab will automatically be deployed to Surge.

Bundling the project

No serious front-end project can get away without a bundling tool nowadays. To keep things simple I used AssetGraph-builder. This tool has the advantage of taking care of bundling JavaScript, CSS and HTML files by using an HTML file as an entry point with zero configuration - perfect for a newly created simple web application.

Setting up AssetGraph-builder is not more complicated than running these two commands:

npm install --save-dev assetgraph-builder
node_modules/.bin/buildProduction --outroot=dist index.html

Check the contents of dist folder to see the bundled project.

To do the bundling automatically on GitLab, tweak the contents of .gitlab-ci.yml:

image: node

before_script:
  - npm install

deploy:
  stage: deploy
  script:
    - node_modules/.bin/buildProduction --outroot=dist index.html
    - cp CNAME dist/
    - node_modules/.bin/surge dist

Commit and push all changes to GitLab to run the pipeline. Don’t forget to add dist folder to .gitignore. After a short while the bundled project should be republished to Surge.

Next steps

It should be easy to adopt this approach to your favorite front-end toolchain. Using Yarn instead of npm or bundling with Webpack or Browserify is not much different from the tools used here.

A decent software project has automated tests. Fortunately we already have a pipeline set up with the potential of running our tests. Read the GitLab CI documentation on how to set up advanced pipelines with multiple jobs and stages.

The examples above are published to a GitLab project including public access to the pipeline.

Alternatives

If your project is actually a static web site without the need for a sophisticated bundling process or automated tests a static site generator and/or GitLab Pages or GitHub Pages would do the job.

There are also alternatives for continuous integration, Travis CI for example is free for public projects.

Getting Started with MapReduce the TDD Way

Tue, 17 Oct 2017 00:00:00 +0000

Writing and running the first lines of MapReduce code for Hadoop is an involved process that often turns beginners away. I will use the test driven development approach to give an introduction to writing MapReduce jobs and unit and integration tests without even installing a Hadoop server.

Requirements

I will use Scala and sbt for the code examples below because I like conciseness of the language but the approach should be easy to adopt to Java or any other JVM language.

To run the code examples the only thing you will have to install is sbt. I used version 1.0.2. Use your favorite terminal application to launch sbt commands.

You will also need a text editor or an IDE with built-in sbt support (I recommend IntelliJ IDEA with Scala plugin). The examples will assume you are using sbt from the command line, feel free to use your IDE’s equivalent of the sbt commands.

Although they do not have to be installed separately, this tutorial uses Scala 2.12.3 and Hadoop 2.8.1.

Introduction

Big data is a hot topic nowadays and Apache Hadoop is one of the popular software in use for storing and processing large data sets. There are many tutorials on how to get started with MapReduce - Hadoop’s underlying programming model - but not many of them serve the needs of the impatient developer. Moreover there is almost no time spent on testing a MapReduce job and Google search results often suggest using MRUnit, a now defunct and unsupported tool.

The official MapReduce Tutorial and many other learning material use the “word count problem” as an example - this tutorial will follow the tradition. The goal is to count the occurrence of the words in a bunch of text files.

While I was an engineer SoundCloud I had to learn Hadoop the hard way. Working with seasoned big data engineers meant I had access to a well maintained cluster and there was always someone to help out solving complicated problems but material or good advice on making your first steps did not exist. This experience inspired me to write this post.

The basics of MapReduce

To give you a basic understanding of what Hadoop and MapReduce are, these are the basics:

Hadoop manages and stores data in sets of key value pairs both of which can be treated as certain types or simply as binary data.
Mapper is a function that takes a key - value pair and returns zero or more key - value pairs.
Reducer is a function that takes one key and many values (outputs from the mapper) and returns zero or more key - value pairs.
Input and output key and value types might be different for both the mapper and the reducer but there are a few important rules:
- The input types of the mapper must match the key - value type of the input data.
- The input types of the reducer must match the output types of the mapper.

Keep in mind there is a lot more beyond this explanation. This post will not discuss how Hadoop stores the data in a distributed manner neither how the mapper and reducer functions are executed. If you want to know more I recommend starting with the official Hadoop documentation and the MapReduce Tutorial.

Setting up a project

To start a new sbt project create an empty folder with a file named build.sbt in it having the following contents:

name := "hadoop-mapreduce-tutorial"

version := "1.0"

scalaVersion := "2.12.3"

libraryDependencies ++= Seq(
  "org.apache.hadoop" % "hadoop-client" % "2.8.1",
  "org.scalatest" %% "scalatest" % "3.0.1" % Test,
  "org.mockito" % "mockito-core" % "2.8.47" % Test
)

This will add the Hadoop Java library and the test frameworks as sbt-managed dependencies.

As you might have noticed I use ScalaTest test framework and the Mockito library for managing test doubles. Any other framework should do it, feel free to adopt the code to your favorite one.

Running sbt test from the terminal should be successful and print something like “No tests were executed”.

Writing a mapper

To repeat the basics, mapper is a function that takes a key - value pair and returns zero or more key - value pairs.

In our case the input values are all the lines in the text files, the mapper will split these into words (tokenize). The output key will be the word itself and the value always 1 meaning “this word occurred once”. The input keys are ignored in this mapper.

Let’s create an incomplete implementation in src/main/scala/TokenizerMapper.scala:

import org.apache.hadoop.io._
import org.apache.hadoop.mapreduce._

class TokenizerMapper
  extends Mapper[LongWritable, Text, Text, IntWritable] {

  override def map(key: LongWritable,
                   value: Text,
                   context: Mapper[LongWritable, Text, Text, IntWritable]#Context) {
    // TODO: implement the mapper
  }
}

Add a unit test for the mapper

A simple unit test would look like this in src/test/TokenizerMapperSpec.scala:

import org.apache.hadoop.io._
import org.mockito.Mockito._
import org.scalatest._
import org.scalatest.mockito.MockitoSugar

class TokenizerMapperSpec extends FlatSpec with MockitoSugar {

  "map" should "output the words split on spaces" in {
    val mapper = new TokenizerMapper
    val context = mock[mapper.Context]

    mapper.map(
      key = null,
      value = new Text("foo bar foo"),
      context
    )

    verify(context, times(2)).write(new Text("foo"), new IntWritable(1))
    verify(context).write(new Text("bar"), new IntWritable(1))
  }

}

This is what happens here:

The test subject will be an instance of the mapper.
The MapReduce framework provides a Context instance in production. In unit tests we will pass in a test double instead.
Invoke the mapper by calling map() with some test input and the context.
Ensure there were calls to context.write() with the given arguments (the words and 1) using verify().

Run sbt test to see the test failing.

Implement the mapper

And now we are ready to implement a real mapper, let’s get back to TokenizerMapper. Change the contents to the following:

import org.apache.hadoop.io._
import org.apache.hadoop.mapreduce._

class TokenizerMapper
  extends Mapper[LongWritable, Text, Text, IntWritable] {

  override def map(key: LongWritable,
                   value: Text,
                   context: Mapper[LongWritable, Text, Text, IntWritable]#Context) {

    val words = value.toString.split("\\s+")

    words.foreach { word =>
      val key = new Text(word)
      val value = new IntWritable(1)
      context.write(key, value)
    }

  }
}

This is a rather naive implementation but it should be fine for out exercise. Verify the tests are passing by running sbt test.

Writing a reducer

Reducer is a function that takes one key and many values (outputs from the mapper) and returns zero or more key - value pairs.

The word count reducer will take a word as an input key and the value set will be as many ones (value 1) as many times the word occurred. The reducer has to count the ones to output the total occurrences for the given word.

A minimal reducer looks like this in src/main/scala/IntSumReducer.scala:

class IntSumReducer extends Reducer[Text, IntWritable, Text, IntWritable] {
  private val result = new IntWritable

  override def reduce(key: Text,
                      values: Iterable[IntWritable],
                      context: Reducer[Text, IntWritable, Text, IntWritable]#Context) {
    // TODO: implement the reducer
  }
}

Add a unit test for the reducer

Writing a unit test for the reducer is very similar to how we did it for the mapper. Add these lines to src/test/IntSumReducerSpec.scala:

import org.apache.hadoop.io.{IntWritable, Text}
import org.mockito.Mockito._
import org.scalatest._
import org.scalatest.mockito.MockitoSugar

import scala.collection.JavaConverters._

class IntSumReducerSpec extends FlatSpec with Matchers with MockitoSugar {
  "reduce" should "sum the values" in {
    val reducer = new IntSumReducer
    val context = mock[reducer.Context]

    reducer.reduce(
      key = new Text("one"),
      values = Seq(new IntWritable(1), new IntWritable(1)).asJava,
      context
    )

    verify(context).write(new Text("one"), new IntWritable(2))
  }
}

Explanation:

The test subject will be an instance of the reducer.
The MapReduce framework provides a Context instance in production. In unit tests we will pass in a test double instead.
Invoke the reducer by calling reduce() with some test input and the context.
Ensure there were calls to context.write() with the given arguments (the words and 1) using verify().

Run sbt test to make sure everything compiles and see the newly added test failing.

Implement the reducer

At this point we are ready to implement an IntSumReducer that passes the tests:

import java.lang.Iterable
import org.apache.hadoop.io._
import org.apache.hadoop.mapreduce._
import scala.collection.JavaConverters._

class IntSumReducer
  extends Reducer[Text, IntWritable, Text, IntWritable] {

  override def reduce(key: Text,
                      values: Iterable[IntWritable],
                      context: Reducer[Text, IntWritable, Text, IntWritable]#Context) {

    val sum = values
      .asScala
      .map(_.get())
      .sum

    val value = new IntWritable(sum)
    context.write(key, value)
  }
}

Run sbt test to see all tests passing.

Running the MapReduce job

It is not a widely advertised feature but the Hadoop Java library allows executing MapReduce jobs locally without any Hadoop server running. This is a useful step towards learning how to come up with something that can later be deployed to a production environment and more importantly for our topic: it allows easy integration testing.

Running MapReduce jobs with or without a Hadoop cluster requires some glue code which acts both as an entry point (the main method of the runnable application) and the wiring that connects inputs with the mapper and reducer and dumps the output somewhere.

Let’s use Scala’s built-in App class to create a simple executable in src/main/scala/WordCount.scala:

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.hadoop.io.{IntWritable, Text}
import org.apache.hadoop.mapreduce.Job
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

object WordCount extends App {
  val conf = new Configuration
  val job = Job.getInstance(conf, "Word Count")
  job.setJarByClass(classOf[TokenizerMapper])

  FileInputFormat.addInputPath(job, new Path("input"))

  job.setMapperClass(classOf[TokenizerMapper])
  job.setReducerClass(classOf[IntSumReducer])

  job.setOutputKeyClass(classOf[Text])
  job.setOutputValueClass(classOf[IntWritable])
  FileOutputFormat.setOutputPath(job, new Path("output"))

  job.waitForCompletion(true)
}

What happens here?

A Job instance is created
The job is configured with input as the input path (relative to the current working directory)
The mapper and the reducer are added to the job
Output is set up to a folder named output with key and value types specified
The job is started and the application will not exit until the job completes or fails

There is a lot more to learn about job setup, one important thing to know is the lack of compile time “compatibility checks”. If the input/output types do not match somewhere, the job will fail with an exception at some stage of running.

Running the job locally

Some test input and a logger configuration is missing before we can run the job locally. First create a few text files in a folder named input under the project root.

Add input/file1 with the following contents:

Hello World Bye World

Add input/file2 with the following contents:

Hello Hadoop Goodbye Hadoop

To enable verbose logging create src/main/resources/log4j.properties with the contents below. This helps tracking job progress and allows debugging issues.

log4j.rootLogger=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%nV

At this point you are ready to run the application with sbt 'runMain WordCount'. After a few seconds the job should exit with 0 status and write a text file named output/part-r-00000. If you check the contents of the file, it should look like this:

Bye     1
Goodbye 1
Hadoop  2
Hello   2
World   2

Note: the entire output folder has to be deleted before running the job again, otherwise it will fail.

Adding an integration test

The above setup is almost good enough for a basic integration test with the exception that it is not automated. Automating is fairly easy, just create src/test/scala/WordCountIntegrationSpec.scala with the contents below.

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path}
import org.scalatest._

import scala.io.Source

class WordCountIntegrationSpec
  extends FlatSpec
    with Matchers
    with BeforeAndAfterEach {

  "WordCount" should "write out word counts to output folder" in {
    WordCount.main(Array())

    val output = Source.fromFile("output/part-r-00000").mkString
    output should equal(
      """|Bye	1
         |Goodbye	1
         |Hadoop	2
         |Hello	2
         |World	2
         |""".stripMargin)
  }

  override def afterEach() = {
    val fs = FileSystem.get(new Configuration())
    fs.delete(new Path("output"), true)
  }

}

Explanation:

The main method is invoked “manually” in the same process as the test runner.
Once the job finished, the output file is read to a string and compared against the expected contents.
The test deletes the outputs after each run.

You should see the integration test passing when running sbt 'testOnly WordCountIntegrationSpec'.

Summary

From here on you should be able to extend these ideas to build and test more complicated real life MapReduce jobs. As a next step I recommend going in-depth with the official MapReduce Tutorial and continue from there to more detailed sections of the Hadoop documentation.

Packaging a MapReduce job into a jar file for running it on a real Hadoop cluster is beyond the scope of this post. If you are using Scala and sbt I recommend checking out sbt-assembly.

You can also find heavily commented code examples from above in this GitHub repo.

Delivering Applications with Electron Presentation

Mon, 30 Jan 2017 00:00:00 +0000

There are many steps and obstacles between a working Electron application and people happily downloading and using it on various platforms. I recently rebuilt SoundCleod, the unofficial SoundCloud desktop player from Objective-C into an Electron app.

I had the opportunity to give a talk at the Berlin.JS meetup about integrating with the framework and shipping an installable application to end users. Download the slides for the presentation.

Level Up On Shell Scripting

Wed, 15 Oct 2014 00:00:00 +0000

I recently decided to replace myself with a bunch of small shell scripts in as many tedious daily routine tasks as I can. But knowing how rusty my scripting skills were I decided to refresh my knowledge a little bit.

Here is advice, resources and tools I found useful on this journey:

Try writing portable scripts.
Forget Bash, understand what POSIX shell is.
Watch the Shell Haters Handbook video.
Use this POSIX Reference.
Dash as /bin/sh is a useful collection of portability challenges.
Install ShellCheck (brew install shellcheck when using HomeBrew).
Add ShellCheck to your editor (automagically works when using Vim+Syntastic).
Write tests to verify your scripts. Roundup is a simple and useful test runner, check this page out for a more thorough overview of the options.

And what’s next once we mastered shell scripting? Maybe learn AWK…

D3.js on SoundCloud Stats Presentation

Sun, 30 Mar 2014 00:00:00 +0000

During my long stay in San Francisco I took the chance to dive deep into the local tech communities. I recommend everyone who spends at least a couple of weeks there to check out meetup.com and visit a couple of local groups - it will be an interesting experience.

What I would also strongly recommend is presenting something at one of those meetups - which is what I did there. I had the opportunity to give a talk at the Bay Area d3 User Group on how we rebuilt the new stats pages of SoundCloud using D3.

Download the slides or watch the video.

Introduction to AngularJS

Sat, 05 Oct 2013 00:00:00 +0000

A while ago I gave a talk at budapest.js JavaScript Meetup about AngularJS. The slides of my presentation are available for download here: Introducing AngularJS.

Thanks everyone for coming!

JSHint Everywhere

Sat, 24 Nov 2012 00:00:00 +0000

What is JSHint? It is JSLint tamed for humans.

I always wanted to have JSHint properly integrated into my development flow but never took the plunge. Until now. Prerequisites: nodejs installed in your PATH.

Install JSHint CLI

The command line interface to JSHint can be installed via npm:

npm install -g jshint

And you will be able to do this in the shell:

# JSHint can verify recursively every js file in a folder:
$ jshint .

# … or a single file
$ jshint assets/app.js

# to ignore others' problems:
$ jshint assets/!(vendor|require.js)
assets/app.js: line 23, col 1, 'console' is not defined.

1 error

The jshint command searches the current folder and its parents for a .jshintrc file and falls back to ~/.jshintrc. This enables project-level jshint options. A gitignore-style .jshintignore can be provided to exclude certain files or folders from syntax check.

Note: to have the !() syntax you might need to enable the extglob feature in bash (see bash manpage).

Add JSHint to VIM

Install syntastic. Done.

Syntastic will automatically pick the appropriate syntax checker for most languages. If you happen to have both JSHint and JSLint installed, syntastic has to be configured to ue the former. Add this to vimrc:

let g:syntastic_javascript_checker = 'jshint'

Add JSHint as a compiler to VIM

Useful for JSHinting whole projects with large number of JavaScript files - possibly with many errors.

Install jshint cli (see above) and create ~/.vim/compiler/jshint.vim with the following contents:

CompilerSet errorformat=%f:\ line\ %l\\,\ col\ %c\\,\ %m
CompilerSet makeprg=jshint

Then in VIM comand line:

:compiler jshint

"Validate current folder recursively:
:make .

"Validate current file:
:make %

"Validate selected files:
jshint assets/!(vendor\|require.js)

You might need to add this line to .vimrc to enable advanced patters in bash:

set shell=/bin/bash\ -O\ extglob

Make collects errors output the quickfix window (:copen). You might want to use the location list window instead. In this case use :lmake and :lopen. Read more about quickfix/location list window: :help quickfix.txt.

Credit goes to Jonathan Jacobs for this trick.

Add JSHint to Rake build pipeline

It might be useful to automate running JSHint on the codebase before deployment, from CI or as part of the test phase. Using rake it is as simple as adding this to Rakefile (replace assets with the folder to be JSHinted):

task :jshint do
  system('jshint', 'assets') || abort('JSHint check failed')
end

The downside of this approach is that it requires JSHint cli to be installed. If that’s not an option, there is a gem called jshintrb which a provides ruby wrapper around JSHint and a Rake task (JavaScript runtime is still required).

Install:

gem install jshintrb

Or add this line to Gemfile:

gem "jshintrb", "~> 0.2.1"

Add jshint task to Rakefile:

require 'jshintrb/jshinttask'

Jshintrb::JshintTask.new :jshint do |t|
  t.pattern = 'assets/**/*.js'
  t.exclude_pattern = 'assets/{vendor/**/*,require}.js'
  t.options = JSON.parse(IO.read('.jshintrc'))
end

Cons of JshintTask: does not use .jshintignore, neither the same magic for config lookup as JSHint cli does.

How to Center Horizontally a Position Fixed Element with CSS?

Tue, 24 Apr 2012 00:00:00 +0000

The common approach is width:100px; left:50%; margin-left:-50px which is simple but will only work with fixed width. Here is how to do it in a more flexible way.

The CSS:

.container { 
	/* fixed position a zero-height full width container */
	position: fixed;
	top: 0;
	left: 0;
	right: 0;
	height: 0;
	/* center all inline content */
	text-align: center;
}

.container > div {
	/* make the block inline */
	display: inline-block;
	/* reset container's center alignment */
	text-align: left;
}

…and HTML:

<div class="container">
	<div>I will always be at the top center.</div>
</div>

Tested on recent Chrome, Firefox, iOS 5, Android 4, IE9. Should work in every browser that supports inline-block. Try it on the demo page.

Modular JavaScript, AMD and RequireJS

Mon, 12 Mar 2012 00:00:00 +0000

I had the opportunity today to give a talk at the budapest.js JavaScript Meetup. Here are the slides of my presentation on Modular JavaScript, AMD and RequireJS.

Thanks everyone for coming!

IFRAMEs Automagically Fit Their Content on iOS

Fri, 16 Dec 2011 00:00:00 +0000

The result of a recent struggle with IFRAMEs on iOS Safari is an interesting discovery: IFRAMEs can expand to fit their (content documents) content without any JavaScript. Even cross-domain! There are two important notes:

Adding scrolling="no" attribute to IFRAME disables this behavior.
It only works with dynamically changed content: the IFRAME will not fit the initial content.

Example code:

<iframe width="100" height="100" src="content.html"></iframe>

content.html:

<div>
	<button 
		type="button"
		onclick="this.parentNode.style.height='300px'">
		resize
	</button>
</div>

Click the button and see the magic! Try example code online from your iOS device.

Tested on iOS4 and iOS5, does not work in any other browser I tried. Could not find any relevant documentation on this behavior except a StackOverflow post.

Getting Started with GitHub Pages and Jekyll

Thu, 03 Nov 2011 00:00:00 +0000

GitHub Pages are the perfect way for geeks to create personal or project websites, blogs. GitHub uses Jekyll, a static site generator.

Requirements

Install Jekyll if you want to preview the generated pages locally. Ruby and Gem are the only prerequisites.

gem install -v 0.11.0 jekyll

Optional: install Pygments if you want to test code syntax highlight locally. See the Jekyll Install page for details. It is recommended to use the same Jekyll version as GitHub if you want to push it.

Create the Basic Layout of Your Site

 .
 ├── _layouts
 │   └── default.html
 ├── _posts
 │   └── 2011-11-11-hello-world.markdown
 └── index.markdown

default.html will hold the HTML code that wraps your content, e.g:

<!doctype html>
<html>
	<head>
		<title>Getting Started with GitHub Pages and Jekyll</title>
	</head>
	<body>
		{{ content}}
	</body>
</html>

Fire up your favorite text editor, open _posts/2011-11-11-hello-world.markdown and create your first post in your favorite syntax (mine is Markdown). The post file name must start with a properly formatted date.

---
title: My First Post
layout: default
---

## Hello world!

Now you can run Jekyll locally:

jekyll --server --auto

Open http://localhost:4000/2011/11/11/hello-world.html and you should see the miracle.

It is useful to have an index.html to list your posts, let’s create index.markdown:

---
title: My First Jekyll Blog
layout: default
---

{% for post in site.posts %}
- [{{ post.title }}]({{ post.url }})
{% endfor %}

Open http://localhost:4000/ and voilà! Now you can add posts, improve the layout and implement some nice design. The following pages are useful for learning more about the tools used:

Jekyll README
Jekyll’s YAML Front Matter is the page metadata between the --- lines.
Jekyll’s Template Data are the variables available in the pages.
Learn to use Liquid, the ‘macro’ language of the pages, and Jekyll’s Liquid extensions.
Learn Markdown or whatever markup syntax avaliable to Jekyll you prefer.

Push it to GitHub

Once you are happy with the newly created site, you can publish it to GitHub. Two options are available:

User/organization pages: myname.github.com repository will be available at http://myname.github.com
Project pages: myproject repository’s gh-pages branch will be available at http://myname.github.com/myproject

More instructions can be found at the GitHub Pages Introduction.

Tips and Tricks

Use `baseurl`

If you are creating project pages, every url has to be prefixed with the project name, e.g:

<a href="/myproject{{ post.url }}">{{ post.title }}</a>

It can even be made configurable:

<a href="{{ site.baseurl }}{{ post.url }}">{{post.title }}</a>

Create _config.yml with the following content:

baseurl: /test

Override this with Jekyll’s --base-url [url] command line switch locally if you like so (e.g. to serve from the root).

Use HTML5 in Pages or Layouts

Maruku understands HTML5 tags. Add this to your _config.yml:

markdown: maruku