r0b's random ramblings

Notes on restoring my iPod Classic

2025-11-19T19:00:00Z

Notes on restoring my iPod Classic (4th Generation)

I recently got into iPods again. I wanted to have an mp3 player again, be a bit less reliant on streaming services and add some intentionality to my music listening. I knew my parents had at least one old iPod laying around their house so I got them to dig one out! What they found was an iPod Photo, a “classic” iPod a.k.a the 4th generation.

My new old iPod Classic!.

After browsing the excellent iFixit guides, I found they have an entire series on this specific iPod so I got to work ordering up some parts and restoring the device. After a fair few hurdles, it’s all working and I’m listening to it right now as I’m writing this. What follows are some of my notes and key points from the process, in the hopes it might help someone else. I won’t go into specifics, I fix it has you covered there.

What I modified

I wanted to keep this as “pure” an iPod experience as possible, I avoided flashing Rockbox. Instead opting to keep the original OS, often referred to as Pixo.

The iPod wouldn’t boot or stay charged so I set about changing the battery and it seemed relatively easy to swap the HDD for an SD card so I swapped that too. The device went from a 20GB HDD to a 64GB SD card! I’m also hoping that the SD will be more battery efficient. It’s a slightly convoluted 50 Pin to CF Adapter into a CF card to SD card adapter and finally an SD card to go in that.

For the 50 pin adapter, I missed the iFixit step to trim down one of the nodules to make it fit and it wasn’t obvious that not all the pins are needed, it’s ok for it to overlap on one side. I also bent over a dual jumper header so it fit better inside the reassembled iPod case.

I spent a fair while exploring the BIOS-like debug menu to see what was going on with the iPod. I didn’t really learn much here. It was useful to understand how it works.

You hold down select + menu to reboot the iPod, then while booting hold down prev track + select and you’ll get a glitchy animation into the boot menu. The most useful tool I’ve found here is to look at the battery stats, you can test most bits of the hardware here too. There are lots of commands for testing the hard drive, but as I swapped that for an SD card they weren’t useful to me and didn’t work.

Charging the device

I followed this iFixit guide and got my battery from eBay. The main issue I’ve had after replacing the battery and HDD was getting the device to charge. From looking around online, a lot of people claim the iPod doesn’t charge very well over USB and people recommend using a Firewire connection instead. In 2025 firewire doesn’t really seem like a good option!

People also claim the JST connector pins (the one used to plug the battery into the PCB) gets a bit loose and some recommend bending the pins a bit to get a better connection. For me, it helped to arrange the battery wire so it was feeding from “the back”, so the cable loops around the left and comes down into the socket, where “down” is where the 30-pin socket is. Also ensuring a tight bend to keep things together.

The iPod seems fussy about what will charge it. Mine won’t charge on a PC, even though data works, and it won’t charge on many USB power adapters. Definitely try a few combinations before abandoning the iPod/cable. Mine works best from an Anker multi-USB power hub that I use to power my Raspberry Pis, of all things.

Replacing the HDD

I followed this iFixit guide and got this Compact Flash adapter and this 50 pin adapter from Amazon. It was weird that not all the pins are used, it needs to “overhang” on the left side. If you trace the lines on the circuit you can see that those pins aren’t connected to anything anyway. There is a surprisingly bright LED when the card is being read, but I guess you never really see that. I also bent/folded the jumper so it fit inside the iPod nicely. I do wonder if it affects the battery.

Because the SD card setup is so much smaller, I stuck a bit of anti-static foam to the underside of the 50-pin adapter. This presses the adapters up against the case a bit and stops anything from rattling around. I had some lying around, it was probably 5mm thick and quite compressible.

I think the iPod can handle macOS and Windows formatting of the SD cad, I went with fat 32 because I thought it would be most compatible. I ended up using windows to sync the iPod, more on that below, so fat 32 worked well for that.

Synchronising the device

I started out using macOS and was pretty surprised to see a neat iPod icon in Finder! The wonder stopped there though, I couldn’t get the integration to work very well. It often struggled to recognise the device or would get into a stuck state where the only way to fix it was to reboot my entire mac. Not ideal.

Once I tried synchronising with Windows and iTunes I never went back. Opening iTunes and installing the “iPod Support” was very simple and I haven’t run into many issues.

The other benefit of using Windows for me was that I’m still in the progress of ripping my old CDs. When I did this on Music on macOS, it very quickly started messing up my streaming albums and it broke a few things on my iPhone. Using windows was a lot cleaner and didn’t interrupt my normal iPhone music streaming through Apple Music. I’m still to decide where to put my developing music collection, it’s currently a big folder in iCloud.

There are a few options when you start synchronising an iPod, and I had no idea what they mean:

convert bitrate
It turns out the iPod will store whatever you throw at it, be it AAC, MP3 or Apple Lossless. You can use this option to transcode audio to AAC at a bitrate at the time of synchronising. I’ve had weird issues where this option would lose album artworks, it might be worth transcoding yourself with something like ffmpeg.

I also started making a script to transcode an entire directory (recursively) of music files into a new directory formatted to AAC 256kbps while preserving metadata & artwork and removing Album Artists (which don’t work on iPods)

manually manage this iPod
This forgoes iTune’s synchronisation UI so you can just drag and drop music onto the device. This is very useful. You can also inspect the tracks on the iPod within iTunes so you can see if the artwork or metadata is set correctly. You don’t even need to have the music in iTunes to do this.

Another note on synchronisation is that iPods, at least mine, don’t support “album artist” which for me makes the Artists menu pretty useless. A track has an Artist and Album Artist field which you can use to set who’s in the song but also how the tracks should be grouped by separately. For example most tracks in ten days by Fred Again… feature different artists, so for each track it includes every artist in the “Artist” field but each track just has Fred Again… in the “Album Artist” field. But unfortunately the iPod doesn’t do this grouping so I ended up overwriting all the Artists fields.

two other options
I can’t remember the other to options, I don’t think they were as important.

Battery Life

The battery controller on the iPod tries to work out the maximum and minimum charge that can be stored in the battery. It does this when it is charged to the maximum and drained to the minimum. I think it then uses these values to inform the battery level it displays.

When you first put a new battery in, I’d advise continuously charging it up even after it claims it is full. Then use it until it is completely empty and won’t play any more. That way it will get to its true “full” value and “empty” value and the battery indicator will work properly.

I started ripping my music with Apple Lossless (ALAC). I’m not sure if that was a good idea or not yet. It seems that is not best for the iPod and it uses significantly more power to play these tracks. Reddit recommends storing tracks as AAC at 256kbps, I’m still testing this theory but it makes sense to me.

Audio books

If you want to use audio books, it turns out they are just a regular m4a file but renamed m4b (b for book I guess). I had a CD and wanted to rip each track as a chapter so I made this tool: m4b Editor. You drop in your audio files and it’ll generate you a single m4b preserving the metadata and setting each track as a chapter.

It all runs completely in the browser, there’s no server here! It runs ffmpeg using WASM which is pretty cool. It seems significantly slower than running the CLI, but hey you only need to do this a few times.

Apple USB SuperDrive

If you want to use the SuperDrive with Windows, you’ll need to install the driver for it. You can get them as part of the Bootcamp support package. Download that and install AppleODDInstaller64 and it should work and show up in iTunes. It's here-ish:

bootcamp{version}\BootCamp\Drivers\Apple\AppleODDInstaller64.exe

Useful links

Here are some miscellaneous resources that were also useful during this whole process.

iPod Photo iFixit guides
The iFixit tools I have
Reddit post on iPod music quality
iFlash — seems like a good idea but I didn’t try them
iPod won’t charge post
iPod gen 4 service guide
iPod gen 4 spec
potential C library for talking to iPods
iTunes artwork finder
Apple SuperDrive on Windows

Thanks for reading, I hope this was useful! Drop me a line on Mastodon if you have any questions.

Using Explicit Resource Management with TypeScript and Postgres

2025-01-29T12:30:00Z

Overview

Explicit Resource Management is a newish JavaScript feature that lets you add tear-down functionality to function scopes. This means if your code sets something up, you can have it automatically clean things when that code completes.

It's available in Node.js 20.4.0 +, Deno 1.38+ and TypeScript 5.2+, so you should be able to use it on newer projects.

It looks something like this:

using file = openFile('notes.txt')

Then openFile implements this API to close the file when it is not used anymore.

My Code

So I had some code, I wanted to see how the footprint changed. Roughly my code:

import postgres from 'postgres'
import { getPostgresMigrator } from 'gruber'

export async function runMigrations(direction: string) {
  const sql = postgres('postgres://...')

  const migrator = getPostgresMigrator({ sql, directory })

  try {
    if (direction === 'up') await migrator.up()
    else if (direction === 'down') await migrator.down()
    else throw new Error('Unknown direction <up|down>')
  } finally {
    await sql.end()
  }
}

And it turned into this:

export async function runMigrations(direction: string) {
  await using sql = getPostgresClient('postgres://...')

  const migrator = getPostgresMigrator({ sql, directory })

  if (direction === 'up') await migrator.up()
  else if (direction === 'down') await migrator.down()
  else throw new Error('Unknown direction <up|down>')
}

Which needed this magic:

function getPostgresClient(url) {
  const sql = postgres(url)

  Object.assign(sql, {
    [Symbol.asyncDispose]: async () => {
      await sql.end()
    },
  })

  return sql
}

Notes

It takes an extra level of nesting out of the code
I wish TypeScript let you assign symbols to things without complaining
I could remove the use of return to simplify the code
Synchronous and async are different, one uses Symbol.dispose and the other Symbol.asyncDispose and requires the await using keywords.
It feels like this could be very powerful, especially when all your favourite libraries start using it.

Async iteration for JavaScript EventTargets using ReadableStreams

2024-06-28T17:00:00Z

Working on an internal app platform at work I found a way to iterate a set of events that get fired on an EventTarget. I did a cursory web search and only found some old StackOverflow posts so promptly dived in to try and implement it myself.

An EventTarget is what a lot of JavaScript primitives are based on. It lets you listen to event on DOM elements, or WebSockets or EventSources. You'll probably be familiar with the call sign:

something.addEventListener('click', (event) => {
  console.log(event)
})

I wanted my code to look more like:

for await (const event of somethingElse) {
  console.log(event)
}

I'm sure I've seen something like this before, but I cannot remember where that was. I've been slowly getting my head around the Web Streams APIs for the Hub project and I remembered that a ReadableStream also implements Symbol.asyncIterable.

That symbol allows you to use for await with things, so ReadableStream opens the door to use it in a custom way. A ReadableStream is a definition of an asynchronous stream of data that can be piped somewhere and there is a notion of a buffer/queue of that data so that it doesn't use up all your memory. So we can do this:

const stream = new ReadableStream({
  start(controller) {
    something.addEventListener('click', (event) => {
      controller.enqueue(event)
    })
  },
})

for await (const event of stream) {
  console.log(event)
}

I thought this was pretty neat, for EventTarget you could event declare a method that takes a set of event names and creates the generator for you:

function stream(eventNames) {
  return new ReadableStream({
    start(controller) {
      for (const name of eventNames) {
        something.addEventListener('click', (event) => {
          controller.enqueue(event)
        })
      }
    },
  })
}

for await (const event of stream(['click', 'mouseover', '...'])) {
  if (event.type === 'click') {
    console.log('click', event)
  }
}

That is a bit over-the-top but you can see it can start to create a friendlier interface over those events while keeping that flow of code. In this case the code will keep listening forever and never exit the for loop.

It isn't quite as simple as above, it should also clean-up after itself and remove those event listeners:

function stream(eventNames) {
  // Store the listeners so they can later be removed
  let listeners = {}

  return new ReadableStream({
    start(controller) {
      // Generate a listener for each event and hook it up
      for (const name of eventNames) {
        listeners[name] = (event) => controller.enqueue(event)
        something.addEventListener('click', (event) => listeners[name])
      }
    },
    cancel() {
      // Remove each of the listeners from the event target
      for (const [name, callback] of Object.entries(listeners)) {
        something.removeEventListener(name, callback)
      }
    },
  })
}

let countdown = 5
for await (const event of stream(['click', 'mouseover', '...'])) {
  if (event.type === 'click') {
    console.log('click')
  }

  // Break out of the loop after 5 events have been listened to
  countdown--
  if (countdown <= 0) break
}

console.log('stop')

This time it stores the event listeners when it starts the ReadableStream so they can be removed when the stream is cancelled. That cancel method is called when the stream is no longer being used, in this case it is when we break out of the for await loop.

The only thing I'm not sure about this approach is how the internal queuing/buffering works. I think it has an upper limit of things it will buffer and will start to drop things if that limit is reached. I think that is configurable but I will have to read up more on ReadableStreams.

Learning a bit of Für Elise

2024-06-09T17:00:00Z

I recently tooted that I was learning Für Elise on the piano. I've also been doing recordings and sharing them with my Mum, how cute!

I thought it was kind of interesting to hear the difference in progress, this is probably only interesting to me. But here it is.

I was also curious about how I could incorporate audio on my blog.

First recording

Recording number one

I'm clearly still learning the notes and finger work here
I think I'd just been able to make it all the way through

Second recording

Recording number two

I'm getting more confident, but still missing a few notes
I'm still struggling with the second phrase, so much stretching fingers!
I added a little bit on the end playing with the bass notes

Third recording

Recording number three

This is where I'm at at time of writing
I've started using the sustain pedal for the base bits
Its all flowing a lot better
I've also learnt I can plug my phone into my Piano's USB-C and get a perfect recording
I can't believe I messed up that last note 😔

Well that's that, hope you found it interesting! 🤷

Get to da Terminal

2024-02-23T10:50:00Z

I wanted a single shortcut to open a new Terminal so first I had to pick one. I had been using CMD+OPT+C in VSCode for a while to bring up the terminal pane and while dabbling with Nova I set up the same shortcut to open a new local terminal too. This had worked well for a while, but it was actually overriding the macOS default keyboard shortcut to show the colour picker, so it wouldn’t work as a global shortcut.

I spent a hot minute scanning my keyboard to work out what to use until I landed on the §/± key. I’m on a UK ISO macOS keyboard, so that is the button to the left of my 1 and below my ESC. To my knowledge CMD+§ isn’t a well-known shortcut so that’s what I’ve adopted.

First I moved my VSCode + Nova keyboard shortcuts to that new combination and VSCode will sync it between both of my Macs too. Nova I just had to configure it in both.

For a global keyboard shortcut I created a Shortcut in the Shortcuts app. This just runs and opens Terminal.app. You might be able to do something clever so that it gets the “current folder” if available and pass it to the Terminal to open at that location, but I couldn’t work out how to do that. Finally, I assigned that a keyboard shortcut to the Shortcut under the little “info” tab.

With application-specific shortcuts, they take precedence of the global ones so if I use it in Nova or VSCode then it opens a terminal in-app and if I use it in another app it opens the macOS terminal. Perfect.

Misc

I’ve been trying Prompt 3 for a bit and currently have the Shortcut opening that while I trial it.
With my Keycron keyboards, I tried remapping the § button with Via to actually just trigger the old CMD+ALT+C keyboard shortcut. That did work for a while but was annoying in laptop mode and where the shortcut wasn’t available.
I’m not sure if Terminal.app (or Prompt.app) support opening them at a certain folder, at least through Shortcuts. That would be really cool if it could capture my location in Finder and open the terminal there instead.
After setting up the keyboard shortcut in Shortcuts.app, it doesn't seem to take effect in apps that are already open. e.g. If Mail was already open when the shortcut is set, it won't work until Mail has restart.

Isomorphic JavaScript, Web Standards and Documentation Driven Design

2024-01-08T22:00:00Z

I was posting on Mastodon about a documentation driven design process recently and thought I'd do a post to share the thing I was thinking about at the time and dig into the process and design a bit.

Web Standards

I've been thinking about server-side JavaScript a bit recently and especially the move towards web-standards on the backend. My first experience was when Deno came along with a really nice "serve" API. It is literally just a function that takes a fetch Request and you return a Response.

I really like this API for two reasons:

It's minimalistic, there is no res to mutate, no next to call or ctx to worry about.
It's based on a feature-rich web standard; parse JSON, stream bytes, read headers. It's all there.

Generally, that got me thinking about other standards but also the promise of them. Web standards don't come and go every night. They're well thought out and don't really get deprecated.

Background

At work I work on lots of small to medium Node.js projects often by myself and each project is somewhat similar but I've often tweaked something or tried out something new. I love that I can do this and get to explore lots of things. But, I have lots of similar but not quite the same code which does make it difficult to go back to projects.

This time I took some time to think about what my ideal common ground for a project would be and how a common ground could help compose projects together in the future. If I have added magic link auth to one project, how can I reuse that in another project, for instance.

Documentation driven design

This all lead me to writing documentation for my ideal library through documentation-driven-design. There might be a more formal name for it, I'm not sure. The idea is to design a thing from the perspective of someone first using and learning about it.

It really gets you to think about the APIs and contracts you're creating. How there could be modules and how those modules could work together. Then you also need to think about how you explain those concepts which really forces you understand what is important and what isn't.

It is also freeing. There is no burden of any code being written. If you want to change an API to be cleaner or simpler, you can. Straight away. You don't have to worry about how that effects your codebase, how tests will need to change or things be reorganised.

You do still need to ground yourself and remember what is possible. That's why I think it works best when you have a strong idea of what you want to make and a rough of idea of how it will work internally. I think if you went in completely blue sky, you would design something that wouldn't be possible to make.

Gruber

So, the thing I was thinking about was a hypothetical server-side JavaScript library. Funnily enough, there is some interesting documentation to look at. Check it out at robb-j/gruber.

I'd love to know what you think.

→ Rob

Eleventy WebC in a nutshell

2023-12-02T00:00:00Z

I've been learning about WebC recently and I'm pretty sure I'm converted. So I thought I'd put together a set of notes from my experience of learning it. To do this, let's create a little project together. From scratch.

Prerequisites

Before starting, I'm going to assume:

You are happy to write some JavaScript
You're familiar Eleventy
You maybe understand custom elements (a.k.a web components)

mkdir 11ty-webc-nutshell
cd 11ty-webc-nutshell

npm init -y
npm install @11ty/eleventy @11ty/eleventy-plugin-webc

That'll create us a project folder, set it up to use NPM and install our dependencies.

Configuration

Now we need to configure Eleventy to use WebC. This is pretty simple, we'll add our eleventy config file. I'm going to use eleventy.config.cjs, but there are several of other names you can use.

My current thinking is this is more important than a "dot file" like .gitignore and I'd like to be specific this is CommonJS.

const eleventyWebc = require('@11ty/eleventy-plugin-webc')

module.exports = function (eleventyConfig) {
  eleventyConfig.addPlugin(eleventyWebc)
}

Pages

Pretty simple so far. Now lets create our first page. This uses the new .webc extension (hint: it's just HTML, so you can tell your editor about that to make it prettier).

index.webc

---
layout: html.webc
---

<main>
  <h1>Hello, there</h1>

  <my-first-component></my-first-component>
</main>

This all looks familiar so far, if you've used Eleventy before that is. It starts of with Front Matter which is meta-information to tell Eleventy things. Here we say we'd like to use another file as the layout. So this page will be rendered inside the html.webc which we'll create a bit later.

After the Front Matter, we create our page. It has a nice big title and it uses a custom element! WebC is based on web standards, it hooks in to custom elements and starts to do it's magic. So let's create our first WebC component!

Components

By default, WebC will automatically pick up any .webc file you put into the _components folder. Let's create _components/my-first-component.web:

You can configure where Eleventy/WebC looks for these components or you can use the default location if you want.

<p class="call">General Kenobi</p>

Thats what a WebC component is, it's just HTML. There are lots of other clever things you can do, but at it's purest it can just be HTML. Eleventy will replace <my-first-component> in the page with our <p> element for you. If you just want a way of reusing HTML code, WebC is a fine way to do that.

So to be a bit more interesting, let's give our component some style.

_components/my-first-component.web

<p class="call">General Kenobi</p>
<style>
  .call {
    color: rebeccapurple;
    text-decoration: underline;
  }
  .call::after {
    content: '!';
  }
  .response {
    color: blue;
    text-decoration: underline;
  }
</style>

Here we've made the text look cooler with some CSS. By default, WebC will see that we've used a <style> tag and will take it out of our template to be bundled up. We'll tell it where to put it later in our layout.

This style is global so we're using a class to make it more specific. You can alternatively add webc:scoped to the <style> tag to auto-generate a name for you and make it so the style only applies to things inside the WebC file.

By adding CSS (or JavaScript) it changes how WebC behaves a little. Without it, WebC will take the HTML content and straight replace it anywhere it is referenced. This is known as a html-only component. With scripts or styles, the component remains a web-component so the <my-first-component> remains in your HTML and it's contents becomes everything else in your in your .webc file.

You can configure how the content is inserted with a <slot> tag, more info

So let's add some JavaScript too, add this script tag to the existing file, as long as it is at the top-level of the HTML document in your component file.

_components/my-first-component.web

<!--
  previous example code here...
-->
<script>
  class MyFirstComponent extends HTMLElement {
    connectedCallback() {
      setTimeout(() => {
        this.appendMessage('This will make a fine addition to my collection')
      }, 2_000)
    }
    appendMessage(message) {
      const p = document.createElement('p')
      p.textContent = message
      p.classList.add('message')
      this.appendChild(p)
    }
  }
  window.customElements.define('my-first-component', MyFirstComponent)
</script>

Because WebC is not in html-only mode, it keeps the <my-first-component> element in your rendered HTML and you can attach code to that. We create our client-side custom element, MyFirstComponent by subclassing HTMLElement. Then we use define on the custom elements registry to get it hooked up. When someone visits the page, the browser attach our code onto the HTML element and call connectedCallback to let us know.

As a demonstration, we add another styled message after waiting an extra 2 seconds.

Layouts

Ok so what happens to these scripts and styles, and how do they fit into a nice HTML document? It's completely up to you! We'll use a layout to add the styles and scripts into our rendered HTML, create _includes/layout.webc:

<!doctype html>
<html>
  <head>
    <title>My first WebC</title>
    <meta charset="utf8" />
    <meta name="viewport" content="width=device-width" />
    <style @raw="getBundle('css')" webc:keep></style>
  </head>
  <body>
    <template @html="this.content" webc:nokeep></template>
    <footer>My first WebC 2023</footer>
    <script type="module" @raw="getBundle('js')" webc:keep></script>
  </body>
</html>

There's are a few bits going on here, let's break it down.

As mentioned above, one of the powers of WebC is that it can bundle up your JavaScript and CSS for you. WebC takes out those <script> and <style> tags from your .webc files and we have to get them into to our HTML document. We do this with some familiar <script> and <style> elements! These ones though have special attributes to the files generated by WebC and insert them into the document. Other brief notes:

The scripts and styles are per-page so only the resources you need are on each page.
As WebC removes <script> and <style> elements, here we need to tell it not to by adding the webc:keep attribute.
Any webc: attributes are removed from the HTML you publish.
The value of @raw can actually be anything from Eleventy, e.g. things from the Front Matter or Data Cascade. In this case it's a shortcode provided by eleventy-plugin-bundle (which WebC uses under the hood).

The second interesting bit is that we're using a @html attribute on a <template> element. This takes the HTML generated from processing our page and puts it inside the <template> element. There is a special webc:nokeep attribute on there too. This tells WebC that we aren't interested in the template itself and to get rid of it, replacing it with whatever our page has rendered.

Round up

To summerise, we:

Set up a fresh Eleventy project and added the WebC plugin
Created a page using WebC as the templating language
Added the WebC component that the page uses
Made the component more interesting with styles and scripts
Laid out the whole page using a WebC layout

If you want to skip to the end, who doesn't, see all the code in one place at examples/eleventy-webc.

Current questions

A few things I've been thinking and don't currently have answers for:

Can a webc:setup access "props"?
How can you modify the attributes on non-html-component?
I keep getting a Uncaught SyntaxError: Identifier '...' has already been declared & have to restart the 11ty.

Next steps

My team has a custom-elements based design system, partly using every-layout, and it has an existing Eleventy plugin to generate styles and scripts. I wonder how I could migrate or provide a way to do this via WebC instead. There are currently some pretty big regexes I'd rather not have to maintain!

Here are some places you could go next:

The WebC reference is the obvious place to look for more information and inspiration. You can do stuff like if statements or for loops or provide slots to describe how WebC will embed content in your component.
Props and Attributes are really powerful and let you process Eleventy data into your templated content.
11ty/eleventy-plugin-bundle is a good place to look if you're interested in how the script and style bundling works. A big snoop through the code helped me work out what was going on.
MDN has a good guide on getting started with custom elements if you want to learn more about those.
I haven't dived into it yet, but webc:setup looks really interesting
I'm working on a "year in review" type thing for a coffee club I'm in and it's using WebC, robb-j/year-in-rebrew if you're interested.

Found this useful or spotted a mistake? Let me know on Mastodon!.

Creating a HTTP proxy with Deno

2023-08-12T00:00:00Z

I've been playing around with Deno quite a bit recently. I'm really liking all of the integrated tooling and it's web-standards based approach.

I wanted to create a little proxy so that I could access our Grafana dashboard on a Raspberry Pi-based display in our office. The problem is that Grafana requires authentication, but there was no way to configure the kiosk to provide it.

So I set about to create a proxy server that could inject the authentication needed to access the dashboard. I ran the proxy on the Raspberry Pi and pointed the kiosk to the proxy rather that grafana directly. Now it easily boots up and shows our dashboard.

The code

proxy.ts

Deno.serve({ port: 8080 }, async (request) => {
  const { pathname, search } = new URL(request.url)
  const url = new URL('.' + pathname, 'https://example.com')
  url.search = search

  const headers = new Headers(request.headers)
  headers.set('Host', url.hostname)
  headers.set('Authorization', Deno.env.get('PROXY_AUTHORIZATION'))

  return fetch(url, {
    method: request.method,
    headers,
    body: request.body,
    redirect: 'manual',
  })
})

That's all you need. I'll break it down and talk about some of the nice Deno things I found along the way.

First, we're using the new Deno.serve API which I really like. It's nicely based on web-standards and uses the same Request and Response objects that the fetch API uses. This is really useful later.

The API is pretty minimal too. You say what port you want to run on and provide a callback to process the request and return a response. One gripe is the request.url is not a URL object but just a string representation of the URL.

const { pathname, search } = new URL(request.url)
const url = new URL('.' + pathname, 'https://example.com')
url.search = search

This bit grabs the pathname from the request being made, e.g. /some/path, and combines it with the target URL to decide where to proxy to. There is a little hack here, because we know the pathname on the URL will always start with a /, we can prefix it with a . to make it a relative pathname. So if the target URL also has a pathname in it, they will be combined together.

It also copies the URL search parameters across to the new URL too, so they are preserved.

const headers = new Headers(request.headers)
headers.set('Host', url.hostname)
headers.set('Authorization', Deno.env.get('PROXY_AUTHORIZATION'))

One key part of the proxy is that it needs to force the Host header to be set to the host being proxied to. If the server at the other end of the proxy uses a reverse proxy, this needs to be set so it knows how to handle the request. If we didn't do this, the host would be set to localhost and that would confuse the target server.

The other bit here is that it injects the Authorization header, which is what we wanted to do all along. It grabs the value from an environment variable and puts it onto the headers, ready to be part of the proxy request.

return fetch(url, {
  method: request.method,
  headers,
  body: request.body,
  redirect: 'manual',
})

The final bit is to return a new Response (or promise of one) to the Deno.serve handler. Because the API uses exactly the same objects as the fetch API we can do this directly without any custom processing needed. Here it returns a promise for a request that shares the same HTTP method, uses the customised headers, streams the request body and tweaks the redirection logic.

That's it, all you need to do is run it with a deno command:

deno run --allow-net proxy.ts

I hope you found this interesting. I've been quite liking Deno recently, I think it's worth a try. I've also been playing around more in-depth with a little proxy server for some work and personal infrastructure, goldiprox, if you're interested to see some more Deno proxy stuff. There is also a more fleshed out example in examples/proxy.ts to check out.

Spotted a mistake or have some feedback, let me know on Mastodon!.

Host an ics calendar feed with Eleventy

2023-05-21T00:00:00Z

I recently wanted to host an ical feed for some events that I was organising. For the last MozFest, we added a feature to let you subscribe to the events in your MySchedule in your own calendar. People quite liked this feature and I thought it could work nicely in an Eleventy website.

Under the hood a calendar feed is a ical file that is hosted on the web. In a calendar client, you subscribe to that URL and it will periodically fetch that file, parse out the events in there and show them in your calendar.

You could manually create or template that ical file, but that would be pretty fiddly and rife for mistakes, so let's use the ical-generator package instead.

Set up

Let's create a fresh Eleventy project to see how this all works. First we'll need an NPM to install Eleventy and ical-generator.

mkdir eleventy-ical
cd eleventy-ical

npm init -y
npm install @11ty/eleventy ical-generator

All of the code is at examples/eleventy-ical if you want to jump ahead and see it all in one place.

Events collection

In this set up, each event will be a page in an Eleventy collection. In Eleventy you can group pages using tags to later query for them in other pages. We can also tell Eleventy not to render these pages by setting permalink: false in their front-matter. We can do both of these easily by setting it using a directory data file which means there is less to configure in each event page.

So let's create the events/events.json in a new events directory like below:

{
  "tags": ["events"],
  "permalink": false
}

Let's put an event in our new collection. You can name these whatever you want, for my calendar I've numbered them and padded the start so they're nice and alphabetical in my IDE. Each event has a name and location along with the start and end dates. Then the page body can be used for the even description. Here's events/001.md:

---
title: Meeting 001
start: 2023-05-02T15:00:00Z
end: 2023-05-02T16:00:00Z
location: User Study Space
---

This is gonna be the best event ever!

Calendar metadata

To generate a feed, its useful to define some of the metadata so it can also be used within other Eleventy templates. So let's create _data/calendar.json:

{
  "title": "My calendar",
  "description": "Events about all the cool things",
  "organisation": "Rob's calendars",
  "url": "https://feed.r0b.io"
}

This isn't necessary just to generate the feed but it is useful to share these values with other Eleventy templates, like pages that showcase the feed.

Eleventy template

Now we have an event and the metadata, we can finally generate our feed. To do this, we'll use an Eleventy class-based template. This means that we can dynamically generate a page with the exact contents we want.

Here's the feed.11ty.js to create:

const {
  ICalCalendar,
  ICalAlarmType,
  ICalCalendarMethod,
} = require('ical-generator')

module.exports = class FeedTemplate {
  // Setup Eleventy data for this template,
  // namely set the name of the file to be generated
  data() {
    return {
      permalink: 'feed.ics',
    }
  }

  // The render method is called
  render({ calendar, collections }) {
    // Generate a calendar object based on the calendar configuration
    // plus information provided by eleventy
    const cal = new ICalCalendar({
      name: calendar.title,
      description: calendar.description,
      prodId: {
        company: calendar.organisation,
        product: 'Eleventy',
      },
      url: calendar.url + this.page.url,
      method: ICalCalendarMethod.PUBLISH,
    })

    // Loop through of each of our events using the collection
    for (const page of collections.events) {
      // Create a calendar event from each page
      const event = cal.createEvent({
        id: `${calendar.url}/${page.fileSlug}`,
        start: page.data.start,
        end: page.data.end,
        summary: page.data.title,
        description: page.template.frontMatter.content,
        location: page.data.location,
      })

      // Add an alert to the event
      const alarm = new Date(page.data.start)
      alarm.setMinutes(alarm.getMinutes() - 15)
      event.createAlarm({
        type: ICalAlarmType.display,
        trigger: alarm,
      })
    }

    // Generate the ical file and return it for Eleventy
    return cal.toString()
  }
}

There a few bits going on here. The file is exporting a class which is our Eleventy template. Eleventy will create an instance of this class for us and call the methods when it needs to.

Eleventy will first call data() on our class to generate the data for the page, similar to the front-matter in a markdown file. We use this to set the permalink to tell Eleventy what we want our file to be called.

Next Eleventy will call the render(data) method. This takes all the data Eleventy has generated from the cascade and passes it as a parameter for use in JavaScript. Here we destructure the calendar field, generated from the data above, and collections. It's also useful to remember that you have access to the this inside these methods which have extra information, like the page's URL.

Inside the render, it first creates a calendar object and sets up the metadata for the feed.

After setting up the calendar it loops through each page in the events collection. For each page it creates a corresponding event object and adds it to the calendar. One important point about events is the id attribute, this should be unique and persistent so calendar clients know whether to create a new event or update an existing one. To keep this unique it combines the URL of the feed itself and the identifier of the event.

I added a bit of logic here to create an alert on the event 15 minutes before the event starts. This doesn't have to be hardcoded and you could define this on more of a per-page basis if you like.

Finally, the render function generates the ics and returns it for Eleventy to create a file using the cal.toString() method.

Timezones

You can set the timezone on the calendar object or on the events themselves, but it isn't as simple as that. An ical feed needs a VTimezone object which you need a generator for. This can be provided by a library like @touch4it/ical-timezones. There are some good docs in the ical-generator readme for more information.

For my use-case I didn't need this. I used a little Node.js script to generate the pages which creates the date objects for me and automatically writes them with the correct offset in UTC so they're at the right time. By default all dates in the calendar are assumed to be in UTC.

Generating the feed

Now that everything is set up, you should have a directory structure like this:

.
├── _data
│   └── calendar.json
├── events
│   ├── 001.md
│   └── events.json
├── feed.11ty.js
├── node_modules
│   ├── ...
├── package-lock.json
└── package.json

We can run Eleventy and it will generate our ical feed for us! This will create _site/feed.ics which contains the content of our feed, ready to host on the internet.

npx eleventy

Next steps

timezones — As mentioned above, I didn't really get into timezones, so things could definitely be improved here. The ical-generator docs are the best place to start for this.

create a site — You might want to host a little static website alongside your feed. It could simply link to the feed URL with some subscribe instructions or it could pull in event information in a fun dynamic way.

host the site — The website needs to be hosted for people to subscribe to your events!

better event descriptions — This set up dumps the markdown content into the body of the events but markdown won't work in calendar clients.

per-event alerts — You could define an alerts section in an event's front-matter and use that to set different alerts on different events.

generator script — My feed was Tuesday-based so I created a script to create an event on the next occurring Tuesday and fill in the front-matter for me. You could do something similar or even take arguments for how you want to create the events. If you were feeling fancy you could hook up something like decap-cms to edit the events using a web UI.

Hit me up on Mastodon if you liked this, have feedback or just want to know more!

Embed JSDoc comments in an Eleventy website

2023-05-03T00:00:00Z

You can use JSDoc to automatically generate a documentation website to fully describe the contents of your API. This works by adding special comments around your JavaScript or TypeScript code that describe the code in more detail. For JavaScript you can even describe the types to better help the people using your API.

An example JSDoc comment

/**
  It erm ... adds two numbers together
  
  @param {number} a The first number
  @param {number} b The number to add to the first number
  @returns {number} The sum of the two numbers
*/
function add(a, b) {}

JSDoc can then generate a website to showcase your API for you. Personally, I've found these generated sites hard to navigate and difficult to get to the information I need. You can create your own template but that requires learning the ins and outs of JSDoc and how it's templating works.

For a recent project, I was creating a site to demonstrate and document a design system. I wanted to reference the JSDoc I'd already written in the code, rather than duplicate it. I found the ts-morph package on GitHub which makes it easier to parse TypeScript's Abstract Syntax Tree (AST). My ideal integration was to only pull out the JSDoc comments and embed them as markdown on the site.

AST is a data structure that represents the code in a file, rather than the raw text string. It allows it to be queried and modified in-place. AST Explorer is a good tool to play around and inspect the trees generated from code files.

There are a couple of benefits to this general approach:

These doc comments only need to be written once. The same comment can be seen in an IDE code completions and on the documentation website. Those comments only need to be updated in one place, so can't get out of sync.
The comments are close to the code that they document. So it's easier to update them when the code the document changes. This feels in the vein of Locality, from The Unicorn Project's Five Ideals.
You have complete control of how your documentation site looks and feels. It's important to properly think through documentation. I've sometimes tried "Documentation driven development" where the docs are written before any code to get a feel for how something should work from a consumer's perspective, rather than jumping into the technical implementation.

How it works

Ok you're sold, or still interested to learn more? To show how it works, we'll create a fresh Eleventy website, along with an "API" to document and hook up. In a terminal, let's scaffold the project and fetch NPM dependencies:

mkdir eleventy-jsdoc
cd eleventy-jsdoc

npm init -y
npm install @11ty/eleventy typescript ts-morph @11ty/eleventy-plugin-syntaxhighlight markdown-it slugify

All of the code is at examples/eleventy-jsdoc if you want to jump ahead see it all in one place.

Let's make our library, lib.js, this is the API we're creating and want to document. We're going to pull these JSDoc comments through into our website:

/**
  It ermm ... adds two numbers together
  
  ```js
  import { add } from "my-api"
  
  const answer = add(40, 2);
  ```
  
  @param {number} a The first number to add
  @param {number} b The number to add to the first number
  @returns {number} The sum of the two arguments
*/
export function add(a, b) {
  return a + b
}

/**
  Greet the nice person by their name
  
  ```js
  import { greet } from "my-api"
  
  const message = greet('Geoff Testington')
  ```
  
  @param {string} name The name of the person to greet
  @returns {string} The boring greeting message
 */
export function hello(name = 'General Kenobi') {
  return `Hello there, ${name}`
}

Now add an Eleventy configuration file, eleventy.config.js, to add some custom logic:

const markdownIt = require('markdown-it')
const syntaxHighlight = require('@11ty/eleventy-plugin-syntaxhighlight')
const slugify = require('slugify')

// Create our own markdown-it instance to be used in a few places
const md = markdownIt({ html: true })
md.disable('code')

// A snippet to generate some HTML for a given API export
const apiDoc = (item) => `
<div class="apiDoc">
<h3>${item.name}</h3>
${md.render(item.content)}
</div>
`

module.exports = function (eleventyConfig) {
  eleventyConfig.addPlugin(syntaxHighlight)
  eleventyConfig.setLibrary('md', md)

  // Manually watch out API so that
  eleventyConfig.addWatchTarget('./lib.js')

  // NOTE: There is a bit of a hack here in that Eleventy mutates
  // this instance so it magically gets syntax-highlighting applied.
  // This is also exploited in the `apiDoc` shortcode below.
  eleventyConfig.addFilter('md', (content) => md.render(content))

  // A shortcode to render the JSDoc comment for an export from a given entry-point
  eleventyConfig.addShortcode('apiDoc', (api, entrypoint, name) => {
    const item = api?.[entrypoint]?.[name]
    if (!item) {
      throw new Error(`Unknown API item '${name}' in '${entrypoint}'`)
    }
    return apiDoc(item)
  })

  // A filter to generate a URL-friendly slug for a text string
  eleventyConfig.addFilter('slug', (text) => slugify(text))

  // A utility to pretty-print JSON
  eleventyConfig.addFilter('json', (value) => JSON.stringify(value, null, 2))

  // Tell Eleventy to use Nunjucks
  return { markdownTemplateEngine: 'njk', htmlTemplateEngine: 'njk' }
}

Next let's create a HTML base layout for our site, _includes/html.njk, which our pages can use:

<!DOCTYPE html>
<html>
  <head>
    <title>My Fancy API</title>
    <meta charset="utf8" />
    <meta name="viewport" content="width=device-width" />
    <!-- Some base styles so raw HTML doesn't look gross -->
    <link
      rel="stylesheet"
      href="https://cdn.jsdelivr.net/npm/water.css@2/out/water.css"
    />
    <!-- A light-theme for our code -->
    <link
      rel="stylesheet"
      href="https://cdn.jsdelivr.net/npm/prismjs@1.29.0/themes/prism.min.css"
      media="(prefers-color-scheme: light)"
    />
    <!-- A dark-theme for our code -->
    <link
      rel="stylesheet"
      href="https://cdn.jsdelivr.net/npm/prismjs@1.29.0/themes/prism-tomorrow.min.css"
      media="(prefers-color-scheme: dark)"
    />
    <style>
      /** Something to make our code snippets POP */
      .apiDoc { padding-left: 1em; border-left: 5px solid var(--text-bright); }
    </style>
  </head>
  <body>
    {{ content | safe }}
  </body>
</html>

And now, we can use that layout to add our first page, index.md, a basic homepage to link to our other pages:

---
layout: html.njk
---

# My Fancy API

This is the site that will tell you all about the API and how to use it.

- [Guide](/guide)
- [Docs](/docs)

Now for the first proper page, guide.md. This is a page to showcase specific bits of the API. This uses the apiDoc shortcode registered in the Eleventy config to directly embed our JSDoc comments! The shortcode needs the api data object passed to it, more on that later, along with the entry-point and the named export you want to embed.

---
layout: html.njk
---

[Home](/)

# Getting started

> This page demonstrates pulling specific api exports using the shortcode

This is a detailed guide to using the API, these methods might be useful:

{% apiDoc api, 'lib.js', 'add' %}

Once you've got the hang of that, you can try the `hello` method:

{% apiDoc api, 'lib.js', 'hello' %}

To show another use of the integration, we can make a page that enumerates the whole API to document everything, docs.md. This shows how you can do whatever you like with that api data object, if you pull in extra information from ts-morph you can access that here. For this example, it loops through each entry-point and their corresponding named exports to dump them all out into the page.

There is also a little "Debug" section so you can see what was put onto that api object.

---
layout: html.njk
---

<section>

<p><a href="/">Home</a></p>

<h1>API Docs</h1>

<p>This is all the things the API does, in alphabetical order </p>

<blockquote>This page shows how to enumerate each entry point in the API and render each export from them</blockquote>

<details>
<summary>Debug</summary>
<pre>{{ api | json }}</pre>
</details>
</section>

{% for entrypoint, items in api %}

<section>
  <h2 id="{{ entrypoint | slugify }}">{{ entrypoint }}</h2>
  
  {% for name, item in items  %}
  {% if item.content %}
  
  <div class="apiDoc">
    <h3 id="{{ name | slugify }}">{{ name }}</h3>
    
    {{ item.content | md | safe }}
  </div>
  
  {% endif %}
  {% endfor %}
</section>

{% endfor %}

That is the basic site setup, now we need to start linking it up with TypeScript with ts-morph. For that we'll need to add a TypeScript config file, tsconfig.json:

{
  "compilerOptions": {
    "allowJs": true
  }
}

To get this api object we've seen in the templates, we'll use an Eleventy global data file, _data/api.js. It runs ones to generate a data object with JavaScript. This is the brunt of the integration, there is quite a bit going on here and the TypeScript AST is quite complex.

It loads the predefined entry-points up and parses their AST nodes into memory. With those nodes we go through and find the code which is exported, i.e. code that uses JavaScript's export modifier, and then collect the JSDoc comments from them. This version is setup here to ignore any exports marked with @internal, but you could change that check for anything you like.

const path = require('path')
const { Project, Symbol } = require('ts-morph')

const jsDocText = /\/\*\*([\s\S]+)\*\//
const jsDocTag = /^[ \t]*?@.*$/gm

// Which files should be imported and inspected
const entrypoints = ['lib.js']

function api() {
  const project = new Project({ tsConfigFilePath: 'tsconfig.json' })

  // This is where our processed AST is going to get stored and return
  const output = {}

  // Loop through each of our entry-points
  for (const entry of project.getSourceFiles(entrypoints)) {
    // Create a friendly name for the entry-point (by removing the absolute path)
    const entryName = path.relative(process.cwd(), entry.getFilePath())

    // Start compiling the entry-point
    output[entryName] = {}

    // Loop through each symbol that is exported from the file
    // NOTE: these may be export symbols in the entry-point
    // and not the actual code definitions with useful information on them
    for (let symbol of entry.getExportSymbols()) {
      // Skip any symbol marked with @internal
      if (symbol.getJsDocTags().some((t) => t.getName() === 'internal'))
        continue

      // If it is an alias, make sure to get what is aliased instead
      if (symbol.isAlias()) symbol = symbol.getAliasedSymbolOrThrow()

      // Put the export into the entry-point
      // You could do more processing here to capture more information if you like
      output[entryName][symbol.getEscapedName()] = {
        entryPoint: entryName,
        name: symbol.getEscapedName(),
        content: joinDocComments(symbol),
        tags: symbol.getJsDocTags(),
      }
    }
  }

  return output
}

/**
  Compose all doc comments on a Symbol together into one markdown string.
  It gets the text out of a JSDoc comment,
  then strips out the annotations and joins them all as markdown paragraphs
  
  @param {Symbol} symbol
*/
function joinDocComments(symbol) {
  const sections = []

  // Each symbol might have one or more declarations,
  // each of which might have zero or more JSDoc comment regions
  for (const declaration of symbol.getDeclarations()) {
    for (const range of declaration.getLeadingCommentRanges()) {
      const match = jsDocText.exec(range.getText())
      if (!match) continue
      sections.push(match[1].replaceAll(jsDocTag, ''))
    }
  }

  // Join all sections together with two newlines to make sure they are
  // seperate paragraphs in markdown
  return sections.join('\n\n')
}

module.exports = api()

You should now have a directory structure something like this:

.
├── _data
│   └── api.js
├── _includes
│   └── html.njk
├── docs.njk
├── eleventy.config.js
├── guide.md
├── index.md
├── lib.js
├── package-lock.json
├── package.json
└── tsconfig.json

With all that setup, we can build and serve our site with npx eleventy --serve and open it up in the browser 🥳. It should look like the pictures below:

The first page is a "guide", which shows how to embed specific bits of the API using the apiDoc shortcode.

The guide page with some crafted notes and JSDoc snippets embedded in-between.

The second page is a catch-all "docs" page that dumps the entire API grouped by entry-point.

The docs page listing out each entry-point and each named export in them.

To recap,

There is an API we want to document in lib.js
That file has JSDoc comments in, documenting the code in-place
From _data/api.js, we parse the AST of the code and the JSDoc comments to make it available in Eleventy
That data is available in templates globally as api
There is an apiDoc shortcode to quickly render a named export from a specific entry-point
It all gets built into a nice website with syntax highlighting from prism

Next steps

My use case was just getting those JSDoc comments out of the code and into the website, but in exploring that there are more things I think you could do.

better embedding — I left the apiDoc shortcode quite brief on purpose, you might want a url-slug id in there or to use a different heading tag.

Actually use TypeScript — The example lib.js is just JavaScript, you can of course use it with TypeScript too. I was just trying to keep things simple here.

More TypeScript integration — There is a load more information in the TypeScript AST that isn't being used, I tried getting it to generate code signatures before but didn't get very far. There are also some cool things that could be done with the "tags" in JSDoc comments, e.g. @param, maybe they could be processed more and put into HTML tables or something.

A library — with some more iteration, and some interest, there could be a nice library/eleventy-plugin here to make this a lot easier in the future, it's quite a lot of code in this post 🙄.

Work out the "hack" in eleventy.config.js — I'm not sure how properly get the syntax highlighting to work without relying on the mutation of my md instance in there.

Configure the Watch Target — there is only one file in the API here but with multiple you'll want to pass a glob pattern to eleventyConfig.addWatchTarget to make it reload when any of your source code changes.

Props to Zach for prompting me to do this.

Hit me up on Mastodon if you liked this, have feedback or just want to know more!

Yoath released

2023-03-08T00:00:00Z

A macOS status bar menu showing available OATH accounts ready to generate codes for

If you’ve got a YubiKey and you want to quickly get OATH codes from your macOS status bar it’s the app for you! Check out Yoath on the App Store.

Oh and it’s open source, because I wouldn’t trust an app that talks to my #YubiKey that wasn’t!

Using URLPattern to add "edit on GitHub" support to my blog

2023-02-18T00:00:00Z

While wanting to make a quick change to a blog post, I wondered how hard it could be to add a simple "edit on GitHub" shortcut to my site. On github.com while in a repository, you can press "." which opens the same repo on github.dev which is a lightweight vscode web app that lets you edit and commit changes back. I wanted to do something similar with my blog, but I only need to edit one post at a time.

Here is the whole thing in action, opening on GitHub with Arc's Peak

I started off adding an event listener to know when the "." key is pressed:

window.addEventListener('keyup', (event) => {
  if (event.key !== '.') return

  // Code goes here ...
})

Next I needed to know if the visitor (me) is on a blog post, luckily I remembered that URLPattern is a thing! Its relatively new thing (in 2023) and there isn't much support for it. As this feature is just for myself and I know what browser I'm running on that isn't an issue, but it should still be implemented with progressive enhancement. So the feature is ignored if the API isn't available:

// If URL pattern doesn't exist, go no further
if (!('URLPattern' in window)) return

Now to use the API, you create a URLPattern object and use it like a regex to match URLs. There is a string-based and object-based ways of defining a URLPattern, I went for object-based. As I only wanted to match the pathname part, with object-based other attributes (like the host or protocol) are treated as wildcards so will not be considered in the matching.

const pattern = new URLPattern({
  pathname: '/post/:slug/',
})

This pattern will match when any URL is passed to it that follows that pathname convention AND you can use it to get the "slug" part out too, which is useful soon.

const result = pattern.exec(location.href)
if (!result) return

I used the pattern like this, testing the current location of the visitor to see if it matches the pattern. I'm a fan of early-exiting to avoid loads of nesting, so the callback stops here if the current URL does not match.

location.href = `https://github.com/robb-j/r0b-blog/edit/main/content/post/${result.pathname.groups.slug}.md`

The final step is to redirect the user to the "edit" page on GitHub, which is templated with the "slug" extracted from the URLPattern. You can use the pattern-matching in any of the components of a URL, here it grabs the slug from the matched pathname and uses it to create the edit on GitHub URL.

I wasn't sure if it should open a new tab or redirect the current page. Because of Arc's new Peek I decided to keep it on the same page and it all works very nicely.

I hope this was interesting, let me know what you think on Mastodon!

Trying to make a vanilla web app

2023-01-28T00:00:00Z

At work we have a Coffee Club. It goes back a while but we cooperatively bought a coffee machine and have been trying to work out the best way to organise coffee buying ever since.

A brief history of the club

In our old office we were split on two floors and upstairs had an "official" lab-sanctioned and paid machine and on our floor we had naught. One Black Friday a few of us clubbed together to buy a bean-to-cup machine and a bunch of coffee beans.

We had this brilliant idea that we could ask people who wanted to join to buy 3 bags of beans to be in the club. This solved our need for beans running out and we had more people to talk to on coffee breaks. Yes, we just made a Ponzi scheme.

When we, very quickly, realised that Ponzi schemes don’t work, we set about making a fair system to decide who should be the next person to buy coffee beans. We hooked up a Raspberry Pi and RFID sensor to self-report cups drank to a Google sheet and a Google form to register purchases.

The machine now knew who had bought the least amount of beans for the number of cups they had drank and sent them a nice email to tell them about it.

Last year we rebuilt the system to be a native piece of software that ran entirely on-device and this included adding a nice 7" touchscreen. One of my goals was to see how feasible it was to build a web app without any frameworks. (Cue the rest of this post)

The home screen of the coffee club, above our bean-to-cup machine.

How it works

The app is made up of a few pages:

The home page which shows general stats, daily messages and collective usage information.
A scan page which registers a cup drank and shows personal stats, navigated to when you scan your card.
A product page to log a coffee purchase and set the weight if it is a new product, triggered by scanning a barcode when on the scan page.
The register page which is shown when a new RFID card is scanned. You can associate with a person on our group website or be anonymous.

The register somewhat-cheekily pulls down the search.json from our group website to populate it's list, we need to make this a little more formal.

Each page is a HTML document, CSS stylesheet and a bit of JavaScript. The structure of the page is all in the HTML file and dynamic elements are put in <template> tags to be populated when needed.

One of the things that SPAs (Single Page Applications) break is natural browser navigation. With coffee club it had the static html files and some query parameters, so normal page navigation could occur.

The system defaults to the home page (index.html) then when someone scans an RFID card it goes to /scan.html?card=abcdef123456, all good. You can’t do any fancy url-parameters like /scan/abcdef123456/ which I was hung up on for a while but it doesn’t really matter does it, no one sees this! The client side javascript can pick up the URLSearchParams easily by creating a URL from location.href.

const url = new URL(location.href)

Data binding

The first thing I missed was data-binding which SPA frameworks rely heavily upon. I toyed with lightweight ones like alpine or petite-vue but held firm and ended up making my own minimalistic version. It looks like this:

const state = reactive({ profile: null })

bind(state, ".totalCups", (state, elem) => {
  elem.innerText = state.profile?.cups ?? "~"
}

Whenever state.profile changes it will call the callback with the latest state and the element that matches the query selector. It also calls the callback straight away to render the initial state.

Internally it uses Proxy objects, and it was quite a fun exercise to learn how they work and play around with them.

Kiosk display affordances

Something that was different to design for was that this device was to run in a kiosk-like mode and I put in extra effort to think through the UI components we used so that the app didn’t get stuck anywhere.

The main idea was to make sure it always got back to the home page. This meant each page needed a countdown to either perform an action or cancel. I ended up with buttons with countdowns on them to try to show their relation. I say try because from watching people use the app, they still tend to press the button even when the countdown indicates that it will happen anyway.

HTML templates

In HTML you can have <template> tags which are not shown to viewers and let you start to do things programatically. I used these in the app for dynamic bits of the page which I wanted to control when they were shown without navigating to a new page. For example if you'd scanned your card but didn't want to register a cup so pressed "no coffee", it would stay on the same page but swap out the main element with something different. I created a wrapper to grab a template and fill it in:

<template id="noCoffee">
  <span id="name">…</span>
  <button id="done">Finish</button>
</template>

// Create a template hydrator
const noCoffee = hydrateTemplate('#noCoffee', (elem) => {
  const name = elem.querySelector('#name')
  name.innerText = 'Geoff Testington'

  const done = elem.querySelector('#done')
  done.addEventListener('click', () => { /* ... */ }))
})

// Add it to the DOM
rootElement.appendChild(noCoffee())

These templates worked pretty well, definitely a bit more verbose than their Vue or React counterparts but pretty serviceable. Being more-specific to implement did make me re-think what needed to be dynamic and what could be built into the page itself which was interesting. In retrospect this is a bit of a precursor to "islands architecture" that tools like 11ty/is-land and fresh are proposing.

The only dependency

Eventually I did concede and add a client-side dependency. It was for charts because they are just a hard problem and there has been lots of good effort gone into solving that already. Until now everything could be made to work with the browser but charts are a different beast.

I went for chart.js to show some nice usage graphs on the home screen, it’s nice to see the average cups for each day of the week compared to the current week. Why are Tuesdays our most popular day?

Event-driven issues

Another oddity about this interface is that it is not driven by cursor/pointer interactions. As mentioned above it’s designed to work with countdowns so you don’t have to touch the screen very often. The other part is the RFID reader and barcode scanner (used to register purchases of beans).

Because it is not a Single Page App, it’s a bit harder to handle those events and difficult to handle across actual page navigations. The sensors work by emulating a keyboard under-the-hood so you need to keep state to know what’s been typed and that can get lost if the page navigates away.

For the current version the app has a common "Scanner" class that can be easily created to listen and process the raw events then emit specific events like “card scanned” or “barcode detected”. I think this could be better handled with SharedWorkers in the future but we haven’t got around to trying them yet.

Honourable mentions

A few things in modern web dev that make things easier:

The fetch API
URL constructors (especially that second base parameter)
URLSearchParams
ESM
Intl.NumberFormatter

And a mention to Dan Jackson, who's the main brain behind this system, designed our human-friendly csv-based filesystem and made it easy to talk to the various input devices. Also EveryLayout for taking layout out of the equation.

Conclusions

I think it’s safe to say you can make a web app without an SPA framework like React or Vue. You definitely loose a bit of developer experience, but all the tools are there for you to try it yourself. The main loss I found was data-binding and I don’t think most people would be interested in creating their own on top of Proxy objects.

This specific app is a bit of an outlier in the whole SPA debate as it’s running on very precise hardware and software. In this case it might have actually made sense to use something like Vue.js to speed up development and avoid those event hang-ups. We have considered this a few times! But in writing this I think it has been useful to explore the space and question the norm.

Thanks for reading, please let me know what you think!

Creating drag interactions with setPointerCapture in JavaScript

2023-01-07T00:00:00Z

Pointer events are the web's answer to mouse vs touch interactions, so you can add one event and it will be triggered consistently whether you are touching on mobile or clicking on a computer. I recently played around with some fun cards on my personal website.

The project cards on my website. Click to flip over the card or click & drag to move them about.

The main issue with implementing a drag like this is that as soon as the cursor exits the target element, it stops receiving the events it needs. This is where setPointerCapture comes in.

How it works

To get this working you listen for pointer events or set the .onpointer* methods to your callback. I've always been one to prefer the event listeners over setting callbacks but recently have come to realise that if you only have one exclusive interaction with an element, it makes sense to use the methods. It's easier to handle (pun intended) and it's simpler to manage in more dynamic environments (you can't re-add the same listener by accident).

So, to create a drag effect, let's add some HTML to manipulate:

<div class="projectBoard">
  <article class="projectCard"></article>
</div>

And give it a rough style:

/* Create a space to move the card about in */
.projectBoard {
  min-height: 960px;
  min-width: 1048px;
  position: relative;
}
/* Give the card a size and something to see */
.projectCard {
  position: absolute;
  width: 320px;
  height: 180px;

  background: royalblue;
  border-radius: 1em;
}

Now let's add the drag with JavaScript:

// Start by iterating through all the cards that we want to add the interaction too
for (const card of document.querySelectorAll('.projectCard')) {
  // Keep track of where interactions started, remember that this is scoped for each card
  let startPosition = null

  // Listen for "pointerdown" events, when a touch/click starts on that element
  card.onpointerdown = (event) => {
    // Ignore this event if it is not a left-click or touch
    // or if clicking an anchor tag on the back of the card
    // https://developer.mozilla.org/en-US/docs/Web/API/Pointer_events#determining_button_states
    if (event.button !== 0) return
    if (event.target instanceof HTMLAnchorElement) return

    // This stops normal browser' drag behaviour
    // Note: It can cause trouble if you have an anchor on the card
    event.preventDefault()

    // This is the key bit, it binds this `PointerEvent` to the element until you tell it to stop
    // So if the pointer leaves the element, it still receives the relevant events.
    card.setPointerCapture(event.pointerId)

    // Lets remember where the card started to track if the pointer moved or not
    startPosition = [event.screenX, event.screenY]

    // Only add the "move" event handler once the interaction has started
    // Using the "on" method means we can easily remove it later
    card.onpointermove = (event) => {
      // In this setup the cards are positioned absolutely in a container
      // so setting the top/left will move them about in their container
      // It adds the "movement" from the event to it's existing offset in that direction
      card.style.left = `${card.offsetLeft + event.movementX}px`
      card.style.top = `${card.offsetTop + event.movementY}px`
    }
  }

  // Listen for "pointerup" events, when a touch/click that started on the element and was captured ended
  card.onpointerup = (event) => {
    let dx = event.screenX - startPosition[0]
    let dy = event.screenY - startPosition[1]

    // If the pointer didn't move at all since it started, treat it as a click
    // I used css elsewhere to do a flip animation based on the `data-side="front"` vs back property.
    // Ideally it would be a vector distance algorithm (A^2 + B^2 = C^2 stuff)
    if (dx === 0 && dy === 0) {
      card.dataset.side = card.dataset.side === 'front' ? 'back' : 'front'
    }

    // Remove the "move" event now
    card.onpointermove = null

    // Important! remember to release the pointer from the element
    card.releasePointerCapture(event.pointerId)
  }
}

Thats an extra verbose implementation, not too many steps!?

First we add the onpointerdown callback to listen for pointer events starting. In that callback it captures the pointer which binds all events to that element, even if they aren't pointing to it. Then it adds a onpointermove callback to update the relative position of the card when the pointer moves.

Lastly, it sets the onpointerup callback to handle when the pointer event finishes. It releases the captured pointer and removes the onpointermove callback. If the pointer didn't move since it started, it treats it as a click and flips over the card.

You can find the source for my homepage's version here, if you want to dig a little deeper.

Bonus: card flip animation

I adapted this w3schools tutorial to create a nice flip effect for my cards. Instead of on-hover though, I wanted to be able to toggle the cards with a click. To store that state I turned to HTML data attributes which can be used in CSS styling. You could also use a HTML aria attribute, as long as the meaning matches up with the interaction you're creating.

The card flip animation like a ... card flip. Very skeuomorphic

The code above will toggle the data-side attribute on the card between front or back when you click on it. It needs more markup that the above example:

<!-- The card is now a wrapper that sets up the 3d position -->
<article class="projectCard" data-side="front">
  <!-- There is an inner element that is responsible for animating rotation on the Y axis -->
  <div class="projectCard-inner">
    <!-- The front-side of the card, an image in this case -->
    <div class="projectCard-front">
      <img src="https://example.com/image.png" width="320" height="180" />
    </div>

    <!-- The back-side of the card, more infromation about the project -->
    <div class="projectCard-back">
      <p>
        My fancy project — it was really cool and goes over at least one line
      </p>
      <p><a href="https://example.com">More info</a></p>
    </div>
  </div>
</article>

This effect needs a few elements to take place, the wrapper (projectCard), an inner element and the front and back elements.

Let's style them:

.projectCard {
  position: absolute;
  width: 320px;
  height: 180px;
  background-color: transparent;
  perspective: 1000px;
  box-shadow: 0 3px 1px rgba(0, 0, 0, 0.05);
}

The wrapper element sets the perspective attribute which controls how far the object is away from the user when using 3d transforms. A smaller number puts it closer to the user, 1000px looked nice for these size cards. If the number is less than the width of the card, then the animation will clip through the "camera" (where the viewer is) and look glitchy.

.projectCard-inner {
  position: relative;
  width: 100%;
  height: 100%;
  transition: transform 500ms ease;
  transform-style: preserve-3d;
  text-align: center;
}

The inner element sets up the animation by setting transition and transform-style. The transition will animate the transform property on the element whenever it changes.

The transform-style statement makes child elements exist in their own 3d space which lets the back be rotated 180 degrees away from the front in the 3d world.

/* Show the card's backside when that data property is set */
.projectCard[data-side='back'] .projectCard-inner {
  transform: rotateY(180deg);
}

When the inner element has the attribute data-side="back" it rotates itself (and it's children) 180 degrees on the y axis. These transforms composite, so when the parent is rotated 180 degrees, the back is rotated 180 degrees + 180 degrees which is a full rotation so it is facing forwards again.

Because there is also a transition setup, it animates! These dataset attributes can easily be set from JavaScript using the dataset field, shown earlier.

/* Style the front back back of the card the same way */
.projectCard-front,
.projectCard-back {
  aspect-ratio: 16 / 9;
  position: absolute;
  width: 100%;
  height: 100%;
  backface-visibility: hidden;
  border-radius: 6px;
  overflow: hidden;
}

The front & back elements share styles so they are exactly the same and don't get misaligned.

.projectCard-back {
  /* Flip the back of the card (relatively to the whole card) */
  transform: rotateY(180deg);

  /* Style the text and surround with a border */
  background-color: white;
  color: black;
  border: 5px solid black;

  /* Center child elements */
  display: flex;
  flex-direction: column;
  justify-content: center;
  align-items: center;
  gap: rem;
}

The back element has the final piece of the puzzle, it is rotated away from the visitor. Because both front and back have backface-visibility: hidden; set, you cannot see the reverse of the elements and they don't bleed through each other.

Wrapping up

That's my card dragging and a cheeky flip animation too, you can see at the bottom of my homepage and I've also attached a demo into the blog too!

If you have any questions or feedback, reach out to me on Mastodon.

Deploying ESP32 with SPIFFS using github actions

2022-11-25T00:00:00Z

Generating and flashing firmware onto an ESP32 can be a bit difficult. I've been using one for a new project and here is what I've learned. This post is in two parts, the pipeline I ended up creating and some key things I've learnt that I didn't think was obvious.

Background

This is my first time developing firmware for a chip like the ESP32 so my approach may be a little different from firmware veterans. My goal is to automate as much of the process as possible and have the entire pipeline codified for reproducible builds that run automatically.

I've not got on well with Arduino IDE and the recent v2 seems to have broken some plugins too. I also didn't like the global nature of library dependencies and lack of a manifest to define them. I opted instead for a command line based approach that could run on my computer or on GitHub Actions.

The pipeline

The pipeline I eventually setup looked like this:

A new version is pushed to GitHub, like v1.2.3
It compiles the Arduino firmware
Generate static files that are bundled into a SPIFFS partition
Create a static website that can be used to flash the device or download assets
Deploy the static website to GitHub pages

1. Create a release

A release is triggered using the npm version command locally, which bumps the version in package.json, commits the change as x.y.x and tags the commit as vx.y.z.

2. Setup GitHub actions

When a tag with a v prefix is pushed to GitHub it runs an action to build and publish the release. It first sets up the environment, with a recursive git checkout. thirdy-party libraries are added as git submodules so they can be kept track of and upgraded dependably. It then installs and sets up arduino-cli and node.js and installs NPM dependencies. arduino-cli is then used to install published libraries and add the ESP32 package.

3. Generate the firmware

The ESP32 I'm using is a captive portal that serves a web app, so next it builds the web-app using parcel. This packages everything up nicely, minifies the code and makes it compatible with older browsers. Then it uses ESP32's mkspiff tool to wrap all those files up into a spiffs.bin partition.

4. Generate flashing app

Next it creates the flash tool which is a website that uses esp-web-tools to create an interface to flash the ESP32 from a browser. Sadly only Chrome is supported at this time, it needs the Web Serial API. This is a little html file which loads the tool and has the firmware binaries adjacent to it for the tool to load them in. It's all joined together with a manifest file, which specifies the firmware partitions and where to put each binary.

5. Deploy flash tool

With the flash tool built, a GitHub action takes those html, js, css, json and binaries and deploys them to GitHub pages, so you can access the tool or directly download the firmware and manifest.json.

Example GitHub workflow

workflow.yml

name: Release

on:
  push:
    tags: [v*]

jobs:
  build:
    name: Compile web flash
    runs-on: ubuntu-latest

    permissions:
      pages: write
      id-token: write
      contents: read

    environment:
      name: github-pages
      url: '${{ steps.deployment.outputs.page_url }}'

    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          submodules: recursive

      - uses: actions/setup-node@v2
        with:
          node-version: '16'

      - name: Install Arduino CLI
        uses: arduino/setup-arduino-cli@v1

      - name: Setup Pages
        uses: actions/configure-pages@v2

      - name: Install dependencies
        run: npm install --no-audit

      - name: Build the app
        run: npm run -w app build

      - name: Setup arduino
        run: ./bin/esp32.sh

      - name: Compile firmware
        run: ./bin/esp32.sh

      - name: Build the flash tool
        run: npm run -w web_flash build
        env:
          NODE_ENV: production

      - name: Upload pages artifacts
        uses: actions/upload-pages-artifact@v1
        with:
          path: web_flash/dist

      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v1

Hurdles

What is a sketch

An Arduino sketch is a folder with a same-named file inside it with a .ino extension. So for MyProject/MyProject.ino, the sketch is the folder MyProject.

The Arduino docs also reference a "sketchbook" which I think is just a folder that has multiple Arduino sketches in it, but it is a different thing.

IDE setup

My reccomendation for an IDE is Visual Studio Code with these extensions:

C++ tools to get intellisense for Arduino code
YAML to validate GitHub workflow or other YAML files
EditorConfig to make sure all your indentations match up
Prettier to auto-format html/css/ts/js/yaml/md files

To get C++ tools working with Arduino, create a .vscode/c_cpp_properties.json and modify the default configuration to include:

{
  "configurations": [
    {
      ...
      "name": "ESP32",
      "includePath": [
        "${workspaceFolder}/arduino/**",
        "~/Library/Arduino15/packages/esp32/hardware/esp32/**",
        "~/Documents/Arduino/libraries/**"
      ],
      "defines": ["ESP32=1"],
      ...
    }
  ]
}

This will set the required hash-defines for ESP32 development and tell the extension where to load the Arduino libraries. It's not perfect and not all imports works but it gets most of the way.

Setting and configuring arduino-cli

The first step in automation was getting the build done from the command line. arduino-cli can be quickly installed and you can create an ardunio-cli.yaml file which will automatically pass parameters for you. This file has to be in the same directory as you use the cli and you can use arduino-cli config dump to quickly see what is configured.

ardunio-cli.yaml

board_manager:
  additional_urls:
    - https://dl.espressif.com/dl/package_esp32_index.json

The configuration is useful for ESP32 development because you can set board_manager.additional_urls which is needed to install the ESP32 packages. You can also install the libraries you need to. I ended up having bin/setup.sh to do this one-time config so it can easily be ran locally and on GitHub actions. It sort-of serves as a definition of the dependencies of the Arduino code.

bin/setup.sh

#!/usr/bin/env sh

arduino-cli core update-index
arduino-cli core install esp32:esp32@1.0.6
arduino-cli lib install ArduinoJson@6.19.4 AnotherPackage@x.y.x

There is a sketch.yaml which lets you specify libraries but it didn't work with my custom libraries setup.

Custom libraries as submodules

The ESP32 library I was using has some third-party dependencies so I wanted to codify those too I didn't want them installed globally, I wanted them dependably in one place at a specific version.

The best way I found to codify these dependencies was to add them as git submodules to the project, so a specific version or commit can be pinned within the parent git repository. It does add an extra step to git submodule init during setup and git submodule update needs to be run every so often.

I hoped Arduino IDE would automatically load them if they were in lib, libraries or the src folder which it claims to check and compile, but I couldn't get this working. It needed an extra argument to the arduino-cli compile command instead. You can pass a --libraries option which is a custom folder of libraries, which worked out well because all the submodules were in a folder together.

3/11/22 I've not found a way to get this working with the Arduino IDE yet...

Arduino partitions

I had to learn how flashing an ESP worked. You have a set amount of flash storage on the device and several binaries that need placing at specific places in that storage. These are called partitions and one of the partitions tells the device where the other partitions are.

When you compile with the IDE or arduino-cli it creates two binaries app.bin and app.partitions.bin, the first is your compiled firmware, the second is the partitions you're going to use.

You can choose different partitions by passing --build-property build.partitions=scheme, where scheme is the partition scheme you want to use. More on these below. You might want to have more space for your app, or a larger spiffs section for example.

The partitions table must be flashed at 0x8000 and be 0xC00 bytes long, where the other binaries go depends on which partition scheme you use. By default arduino-cli uses the ... "default" schema.

If you want to use a custom bootloader, it needs to be flashed at 0x1000.

The ESP32 package uses csv files to define the partitions and you can find them in: $ARDUINO_DIR/packages/esp32/hardware/esp32/1.0.6/tools/partitions/. Below is the contents of default.csv in that directory.

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x140000,
app1,     app,  ota_1,   0x150000,0x140000,
spiffs,   data, spiffs,  0x290000,0x170000,

Here we can find that our app should be at 0x10000 and be 0x140000 bytes long and our spiffs should be at 0x290000 and 0x170000 bytes long. These numbers become very useful as we create SPIFFS and flash them below.

Creating a SPIFFS partition

SPIFFS is an in-memory filesystem the ESP32 can use to store extra files alongside the app. For my app it's where web assets (html, js, css and images) go that the ESP serves as a captive portal.

There is a plugin for Arduino IDE to generate a SPIFFS partition. It takes the contents of the data/ folder inside your sketch, creates a partition for you and flashes it to the ESP. I couldn't get this working and the manual process didn't fit with my automation goals.

To generate one from the CLI you need need to know a few things. First, where the ESP32 Arduino package gets installed, this is $HOME/Library/Arduino15/packages/esp32 on my mac. You can use arduino-cli config dump to see where it looks, the path is under directories.data

Next you need to find the tools directory in there and find the mkspiffs binary. There are a few hard-coded versions in there so it makes sense to have them as script variables.

You can get the Arduino directory from the CLI with a little help from jq.

bin/build.sh

#!/usr/bin/env sh

ESP_VERSION=${ESP_VERSION:-1.0.6}
SPIFFS_VERSION=${SPIFFS_VERSION:-0.2.3}
ARDUINO_DIR=`arduino-cli config dump --format json | jq -r .directories.data`

# Compile firmware
# → My sketch is named `arduino`, not very original
# → Surprisingly you need --export-binaries otherwise it does nothing
# → My custom libraries are submodules in "arduino/libraries"
arduino-cli compile --verbose \
  --fqbn esp32:esp32:esp32 \
  --libraries arduino/libraries \
  --export-binaries \
  arduino

# Generate the spiff
# → "arduino/data" is where my web assets are
# → I chose to put the binary alongside the binaries arduino-cli built
# → Lots of magic numbers here, more below
${ARDUINO_DIR}/packages/esp32/tools/mkspiffs/${SPIFFS_VERSION}/mkspiffs \
  -c arduino/data \
  -b 4096 \
  -p 256 \
  -s 0x170000 \
  arduino/build/esp32.esp32.esp32/arduino.ino.spiffs.bin

There are three very-specific numbers here and they have to be exactly right. So it took a while to find out exactly what these numbers should be. I found the block-size, -b , and page size, -p in the official docs.

The size parameter was a little harder to track down. The size is important, it needs to be the exact size of the spiffs partition you want to mount, otherwise it won't work. More info on this in Arduino partitions above.

Flashing from the CLI

The final hurdle was getting a consistent flash from the CLI, to quickly develop the firmware and try out different fixes. This took the form of another bash script:

bin/flash.sh

#!/usr/bin/env sh

ESP_VERSION=${ESP_VERSION:-1.0.6}
SPIFFS_VERSION=${SPIFFS_VERSION:-0.2.3}
ESPTOOL_VERSION=${ESPTOOL_VERSION:-3.0.0}
ARDUINO_DIR=`arduino-cli config dump --format json | jq -r .directories.data`

BOOTLOADER="${ARDUINO_DIR}/packages/esp32/hardware/esp32/${ESP_VERSION}/tools/sdk/bin/bootloader_dio_80m.bin"

# You can flash the app, bootloader and partions with arduino-cli,
# but I preferred a single command to do it all
# arduino-cli upload -p "/dev/cu.usb..." --fqbn esp32:esp32:esp32 arduino

$ARDUINO_DIR/packages/esp32/tools/esptool_py/${ESPTOOL_VERSION}/esptool \
  --chip esp32 \
  --before default_reset \
  --after hard_reset \
  write_flash -z \
    0x1000 $BOOTLOADER \
    0x8000 arduino/build/esp32.esp32.esp32/arduino.ino.partitions.bin \
    0x10000 arduino/build/esp32.esp32.esp32/arduino.ino.bin \
    0x290000 arduino/build/esp32.esp32.esp32/arduino.ino.spiffs.bin

This is a bit of a monster of hard-coded configuration and more specific numbers are needed again! --chip is simple, I'm building for an ESP32, its not the "board name" (esp32:esp32) though. --before and --after run their operations at their respective times, so it does a regular reset before flashing then a hard reset after flashing.

There is also a new BOOTLOADER variable, which is the path to where the ESP32 package has the bootloader I wanted to use. This lets you flash the bootloader onto the ESP too. It makes sure the latest versio of the bootloader is installed and that its compatible with the compilled firmware.

The main chunk of this command is information from the partitions file, namely the address' of where to put the app and spiffs. The bootloader and partitions go at hardcoded locations for the ESP as previously discussed.

A nicety of esptool over arduino-cli upload is that you don't need to specify a USB device, it picks it for you.

Where is esptool and mkspiffs

When you install the ESP32 package with arduino-cli it also installs esptool and mkspiffs which are handy CLI tools for ESP32 development.

You can use this little jq trick to get your ARDUINO_DIR and then find the binaries, as long as you know what versions it installed.

I only found the versions through trial and error.

SPIFFS_VERSION=${SPIFFS_VERSION:-0.2.3}
ESPTOOL_VERSION=${ESPTOOL_VERSION:-3.0.0}
ARDUINO_DIR=`arduino-cli config dump --format json | jq -r .directories.data`

# esptool location
${ARDUINO_DIR}/packages/esp32/tools/esptool_py/${ESPTOOL_VERSION}/esptool

# mkspiffs location
${ARDUINO_DIR}/packages/esp32/tools/mkspiffs/${SPIFFS_VERSION}/mkspiffs

esptool docs →

Misc

One of the commands needed the python to be on my $PATH, but that is no longer the case on macOS. So I had to create a symlink to the python3 binary. On an M1 mac with Homebrew it looked like:

sudo ln -s /opt/homebrew/bin/python3/opt/homebrew/bin/python

It seems the esp32 package for Arduino CLI has specific versions of esptool and mkspiffs bundled with it, but it still makes sense to keep those as variables so the script can be updated at a later date.

Bonus: monitor serial from the CLI

If you're using the Serial within your firmware code you can use the arduino-cli to get the serial output right in your own terminal. You'll need to know your baud rate, which is the number you pass to Serial.begin(...).

bin/monitor.sh

#!/usr/bin/env sh
set -e

# This is a little hack,
# If you call the command without an argument, it suggests one that is probably an ESP32
if [ -z "$1" ]
then
  echo "Usage:\n  $0 $(echo /dev/cu.usbserial-*)"
  exit 1;
fi

# Put your own chip and baud rate here
arduino-cli monitor -b esp32:esp32:esp32 -p "$1" -c baudrate=115200

Next steps

Generate a GitHub release and attach the firmware binaries directly to it.
Explore merging all binaries into one and see if there are any benefits

ESM Node.js TypeScript with subpath exports

2022-11-05T00:00:00Z

ES Modules is the future for JavaScript. TypeScript support has been lagging behind, but it is finally catching up. Here's what you need to use them with TypeScript:

src/index.ts

export default function main() {
  console.log('This is the main export')
}

src/another-module.ts

export default function alternativeMain() {
  console.log('This is an alternative export')
}

tsconfig.json

{
  "compilerOptions": {
    "outDir": "dist",
    "moduleResolution": "Node16",
    "module": "ES2020"
  },
  "src": ["src"]
}

package.json

{
  "name": "@robb_j/my-module",
  "version": "1.2.3",
  "type": "module",
  "scripts": {
    "build": "tsc"
  },
  "files": ["dist/*"],
  "exports": {
    ".": {
      "types": "./dist/main.d.ts",
      "import": "./dist/main.js"
    },
    "./*.js": {
      "types": "./dist/*.d.ts",
      "import": "./dist/*.js"
    }
  },
  "devDependencies": {
    "typescript": "^4.8.4"
  }
}

type: module and exports are the key parts here. The type field tells Node.js that this module will use ESM, the obvious first step.

exports is a newer field and is used to tell Node.js how to import this module. It lets you customise which files to load depending on the module-type and TypeScript uses it to tell it how to load associated types.

The "." export is the files that are imported when no subpath is provided, in this case it will load the main.js file. The "./*.js" export is used when a subpath is used that ends with "js", that same wildcard is used to find the file too. More info →. If you had commonjs files too, you can link them up here too.

You need to use .js when importing files from within TypeScript (even when importing another TypeScript file), so it makes sense to use the .js here too. It helps to locate the type definitions too.

Running the example

# cd to the project

# Install dependencies
npm install

# Build the code to see what happens
npm run build

Full example

To see all the code, see the examples folder.

Bonus: testing locally

To test a module locally, you can use local relative dependencies. This works better than the npm link method, which I've always struggled with.

In another folder:

package.json

{
  "private": true,
  "type": "module"
}

Install the module relatively:

npm i ../relative/path/to/module

Then you can test it and you should have types and IDE help too.

consumer.js

import main from '@robb_j/my-module'
import alternativeMain from '@robb_j/my-module/another-module.js'

Getting started with kube-prometheus-stack

2022-02-02T00:00:00Z

I recently added monitoring and alerting to my Kubernetes stack to discover and respond to failures faster. It took me a lot longer I than expected to get my head around kube-prometheus-stack and get it doing what I wanted it to, so here's what I found out.

Background

I've spent the last year building and running infrastructure to host virtual conferences (Climate:Red, MozFest '21 & '22 & Planet:Red). Running these large deployments by myself has led to a lot of automation. I start off automating testing and integration, then building and releasing containers and then deployment, configuration and scaling of those containers.

I'd previously used kube-prometheus-stack to see what was going on inside my Kubernetes clusters, but always knew there was a lot more to it than the fancy Grafana graphs. For MozFest 2022 I decided it was time to look into this properly.

The conference infrastructure relies on several background tasks, running as Kubernetes CronJob resources, which can fail very quietly. If one of the jobs failed, the schedule might become stale or new users might not be able to sign in. I needed something to quickly surface those background errors and report them.

Setup

kube-promethus-stack is pretty easy to get installed, DigitalOcean even offers a one-click install after deploying a Kubernetes cluster! That internally uses Helm to deploy prometheus-community/kube-prometheus-stack. So you can:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack

This deploys prometheus-operator/kube-prometheus, which is an operator that uses CRDs to deploy and configure instances of Prometheus and Alertmanager. It uses them to deploy a standard instance of Prometheus and Alertmanager and adds a tonne of rules for monitoring standard Kubernetes resources.

One confusion point was that internally kube-prometheus-stack is deployed using Jsonnet templates and the helm offering is community-operated, so there are some differences between both sets of documentation.

How it works

Prometheus is a time series database which periodically scrapes different servers to fetch metrics. It offers a query language, PromQL, to query those metrics over time and a rule system that triggers alerts based on queries. Alerts have a name and key-value labels of information about them, like the namespace the resource was in or name of the related container.

The stack comes setup to monitor the resources within Kubernetes using kube-state-metrics and has lots of rules already setup to monitor things like Pods crashing.

Alertmanager takes Prometheus' alerts and lets you group and filter them based on the associated labels. It provides routing to send different alerts to different receivers, like an email or Slack message.

Configuring Alertmanager

The default Prometheus instance is already configured to collect metrics on Kubernetes resources and alert on failures. I was interested in KubeJobFailed and KubePodCrashLooping rules to detect failed jobs or bad deployments. Alertmanager needs to be configured with a receiver to send them somewhere. prometheus-operator provides the AlertmanagerConfig custom resource to do just that. The default Alertmanager instance automatically regenerates it's internal configuration whenever any AlertmanagerConfig is changed in any namespace. So you can create a custom-alertmanagerconfig.yml and kubectl apply -f -it:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: custom-config
  namespace: production
spec:
  route:
    groupBy: ['job']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: 'sendgrid-smtp'
    matchers:
      - key: severity
        value: warning

  receivers:
    - name: 'sendgrid-smtp'
      emailConfigs:
        - to: notify@example.com
          from: noreply@example.com
          smarthost: smtp.sendgrid.net:587
          authUsername: apikey
          authPassword:
            name: sendgrid-alerts
            key: SENDGRID_API_KEY

It's worth pointing out AlertmanagerConfig is a v1alpha1 CRD so it will most likely change in the future.

This configuration is a camel-case version of regular Alertmanager config and the operator merges them together into one configuration. This creates a route that groups jobs by their job label and sends them to the sendgrid-smtp receiver. You can use matchers to filter the alerts the route receives based on their labels.

There is a bit of hidden logic in how prometheus-operator merges these configurations back together. It will ignore any namespace matchers you set and instead match the namespace the AlertmanagerConfig itself is in. In this case it would be namespace=production, so make sure to put the config in the same namespace you want alerts from.

Receivers configure how to actually send alerts, in this example via smtp. This one will email alerts to notify@example.com via SendGrid's smtp server. It references a secret, which needs to be in the same namespace as the AlertmanagerConfig, to specify the password. So you don't have to put credentials in your CRD.

Debugging

Detecting errors

First, you want to make sure the operator is accepting accepting your AlertmanagerConfig. It will log any errors from the config or let you know that it successfully reloaded. You can watch those logs:

kubectl logs -f -n kube-prometheus-stack deploy/kube-prometheus-stack-operator

Inspecting generated config

If the configuration is loading correctly, you can inspect the generated secret that it is stored in:

kubectl get secret \
  -n kube-prometheus-stack \
  -o=jsonpath="{.data.alertmanager\.yaml}" \
  alertmanager-kube-prometheus-stack-alertmanager-generated \
    | base64 -d

Configuration options

If you're unsure about what you can put into an AlertmanagerConfig, doc.crds.dev is a great website that describes CRDs based on their schema.

doc.crds.dev/github.com/prometheus-operator/prometheus-operator →

Inspecting prometheus & alertmanager

You can port-forward Prometheus and Alertmanager to see what is going on in a browser. Looking at Prometheus is useful to explore the default alerts and their generated labels and also inspect the raw metrics. Looking at Alertmanager lets you see the grouping that is applied and the alerts that have been triggered.

# Inspect prometheus
kubectl -n kube-prometheus-stack port-forward svc/prometheus-operated 9090:9090
open http://localhost:9090

# Inspect alertmanager
kubectl -n kube-prometheus-stack port-forward svc/alertmanager-operated 9093:9093
open http://localhost:9093

Testing

To make sure alerting is setup, create an obviously bad workload and see if it triggers an alert. e.g. this deployment.yml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bad-app
  namespace: production
spec:
  selector:
    matchLabels:
      app.kubernetes.io/app: bad-app
  template:
    metadata:
      labels:
        app.kubernetes.io/app: bad-app
    spec:
      containers:
        - name: bad-app
          image: busybox
          command: ['exit', '1']

After you kubectl apply -f it, you should first see a "Pending" alert in Prometheus After the for duration has passed, it should fire the alert. Once fired, you will see the new alert in Alertmanager and receive it in whatever receiver(s) you have setup.

Next steps

Choosing rules is the next logical step. This setup will alert on all of kube-prometheus-stack's default alerts, which are a good first step. But if that's too much info, you need to find the alerts that are useful to your specific stack and setup matchers. Looking through prometheus's default alerts should be a good first step, i.e. localhost:9090/alerts.

Meta alerting is useful to ensure confidence that monitoring and alerting is all going as expected. Prometheus constantly fires a Watchdog alert so you could use a dead-man's-switch type service that alerts you when the alert stops firing.

Different receivers let you send different notifications to different places, perhaps based on severity or the team responsible. You could have seperate alerts for frontend or backend teams for example.

Setup persistence, the default kube-prometheus-stack doesn't persist any metrics beyond a reboot. For alerting purposes this is fine but if you need to keep those metrics around you'll want to look into configuring some PersistentVolumeClaims.

Custom metrics is a very interesting next step. You'll need to tell your Prometheus instance to start scraping your services or pods directly using a ServiceMonitor or PodMonitor and define custom /metrics endpoints on your containers that expose prometheus metrics specific to your stack. For MozFest there is a 'site-visitors' widget and it could be interesting to track that over time and have a custom alert when it is over a certain value.

Resources

These were helpful links I used while getting this setup:

prometheus-operator.dev provides a relatively good overview of whats going on, but does miss out on some of the nuances that go on behind the scenes. The design page is a very good starting point explaining what each CRD is for. The user-guides section in their GitHub repo has some good guides for getting started and configuration.
doc.crds.dev was really helpful debugging what needs to go inside the AlertmanagerConfig resource and the descriptions helped to describe what different fields do and how they operate together.
Alertmanager Configuration is useful to know what should and shouldn't be going into those configs and debugging what the generated configurations do. The visual editor was kind of useful, but not very.
The Artifact Hub page has some useful information on it too.

Conclusion

These were the steps I ended up with to get notifications when things start going wrong inside my Kubernetes cluster. It helpfully surfaces things that are going wrong and lets you know about them. If you found this helpful or have questions, let me know on Twitter!

Creating custom JavaScript errors

2021-09-12T00:00:00Z

Creating custom errors in JavaScript can be very useful. If you want to handle those errors in a catch block you can use the instanceof operator to check for that specific error. And with TypeScript, you can then safely use the fields and methods on your custom error.

Another benefit of custom errors is that you can provide your own constructor which lets you pass more information to the error. Which is information you can use in an error handler.

Custom JavaScript errors are almost as simple as creating an Error subclass, but there are two things you can do to make them even more useful.

Set the name

The first thing to enhance custom errors is to set this.name inside the constructor. This means your error name will appear in stack traces rather than the generic Error:.

Re-capture the stack trace

An unfortunate side-effect of creating a custom error is that your custom constructor will appear in the stack trace. This is because your super call is technically creating the error and the JavaScript VM records that in the generated stack trace. This isn't useful and makes stack traces messier.

Removing this makes it easier to get to the code that's thrown the error. You can do this with the Error.captureStackTrace static method.

An example

To demonstrate this, let's create a custom error that captures extra information for a http API, so we can throw a desired http status and an error identifier.

class ApiError extends Error {
  constructor(status, apiCode) {
    super(`There was an error with your API request: "${apiCode}"`)
    this.name = 'ApiError'
    this.status = status
    this.apiCode = apiCode
    Error.captureStackTrace(this, ApiError)
  }
}

That's all you need, you could use it like this:

throw new ApiError(404, 'notFound')

// or

throw new ApiError(400, 'login.emailNotProvided')

and this will produce a stack trace like this:

/◦◦◦/custom-errors/main.js:4
  throw new ApiError(404, "notFound");
  ^

ApiError: There was an error with your API request: "notFound"
    at runApp (/◦◦◦/custom-errors/main.js:4:9)
    at main (/◦◦◦/custom-errors/main.js:12:3)
    at Object.<anonymous> (/◦◦◦/custom-errors/main.js:15:1)
    ◦◦◦ {
  status: 404,
  apiCode: 'notFound'
}

See examples/custom-errors for the exact source code.

The output shows that it has namespaced the error with ApiError: which is from us setting this.name. The first line of the stack trace is not ApiError's constructor but runApp which was the method which threw the error. Because of this, it now shows you the exact line of code that threw the error in the excerpt.

You can also see that status and apiCode have been stored on the error.

Catching errors

To complete this post, here's an example of catching an ApiError and using the extra fields.

const express = require('express')
const app = express()

// From above
class ApiError extends Error {
  /* ... */
}

// A route which results in an error
app.get('/', (req, res, next) => {
  next(new ApiError(404, 'notFound'))
})

// An Express error handler, more info at:
// http://expressjs.com/en/guide/error-handling.html
app.use((error, req, res, next) => {
  console.error('Handled error', error)

  if (error instanceof ApiError) {
    res.status(error.status).send({ apiCode: error.apiCode })
  } else {
    res.status(500).send({ apiCode: 'unknownError' })
  }
})

app.listen(3000)

This shows throwing an ApiError in an ExpressJs context and handling it with an error middleware. It checks for an ApiError with the instanceof operator, and then safely uses the fields on it to generate a http response. The logic also nicely allows a generic http 500 error to be sent when unknown errors are thrown.

Bonus: static methods

When using custom errors another pattern I've used is to add static methods to easily create common errors. The does bring back the stack trace issue, so another Error.captureStackTrace is needed.

To demonstrate this, lets add some common errors to ApiError:

class ApiError extends Error {
  static notFound() {
    return new ApiError(404, 'general.notFound').trimStack()
  }
  static unauthorized() {
    return new ApiError(401, 'general.unauthorized').trimStack()
  }

  // Same as above
  constructor(/* ... */) {
    /* ... */
  }

  trimStack() {
    Error.captureStackTrace(this, ApiError)
    return this
  }
}

I found the utility function trimStack helps keeps these static methods easier to read and understand.

Now you can quickly create common errors with throw ApiError.notFound(), which is easier to read and hopefully leads to less mistakes. For the full source code see See examples/custom-errors.

Creating a Nova Extension with TypeScript

2021-07-25T00:00:00Z

I've been quite getting into Nova recently, a macOS-only IDE developed by Panic. It's a "Mac-assed" editor and I feel it fits a lot more naturally into my flow.

I've gone far enough into making a few extensions for Nova to make it work for me more. Here I want to share my experiences in doing that, specifically around setting up a project with TypeScript. Writing an Extension with TypeScript requires a few different steps to the recommended setup.

Prerequisites

Familiarity with TypeScript
Understanding of terminals, i.e. the Mac's default Terminal.app
Knowledge of NPM and using packages

Any time I use A → B → C notation, it means to navigate through macOS menus

Setup

First, create a folder to put your Extension in, this will be referred to as the root folder going forwards.

# Create a folder to put the project in
mkdir nova-example
cd nova-example

# Quickly setup NPM
npm init -y

# Install development dependencies
npm install --save-dev esbuild typescript @types/nova-editor-node

# If you have the Nova CLI installed...
nova .

Now open this folder in Nova and create a Nova Extension inside of it. The reason for this nesting is to keep TypeScript source-files and development tooling outside of the Extension, then compile JavaScript files into it.

A Nova Extension is essentially a folder with a .novaextension file extension.

In Nova, select Extensions → Create New Extension... Then choose the type of extension you want to make, for this tutorial I will choose Blank. Make sure to put the new extension inside the folder we created above. Remember to close the new window Nova opened after creating an extension, because we want to have the root folder open in Nova.

If you don't see Extensions in the Nova menu, make sure you have enabled development mode in Nova → Preferences → General → Extension Development.

Configure TypeScript

Create tsconfig.json in the root folder:

{
  "$schema": "https://json.schemastore.org/tsconfig",
  "display": "Nova",

  "compilerOptions": {
    "target": "es6",
    "module": "ES2020",
    "moduleResolution": "node",
    "newLine": "lf",
    "strict": true,
    "noEmitOnError": true,
    "skipLibCheck": true
  },
  "include": ["src"]
}

A few points to note:

We're targeting ES6 which lets us use newer JavaScript features and still support older macs. It should support macs from 2016 onwards.
noEmit is set to true because we're using esbuild to bundle things, not TypeScript's compiler, tsc.
The $schema section helps JSON Extension to validate and suggest completions
typescriptlang.org/tsconfig has great info about possible options

For improved linting:

Add these to your compilerOptions

    "noImplicitReturns": true,
    "noFallthroughCasesInSwitch": true,
    "forceConsistentCasingInFileNames": true,

Create a build task

Next, create a build Task which will take TypeScript, bundle any imported files together and output JavaScript into the Extension folder. Go to Project → Project Settings... then create a Task called Development (or whatever you want). Then in the Build section, enter the script below. Make sure to update --outfile with whatever your Extension is called.

#!/usr/bin/env sh

# Ensure the build fails if TypeScript fails
set -e

# Lint TypeScript source code
npx tsc --noEmit --pretty

# Bundle into JavaScript
npx esbuild \
  --bundle \
  --format=cjs \
  --outfile=Example.novaextension/Scripts/main.dist.js \
  src/Scripts/main.ts

You could put this in a script like bin/build.sh if you want to run it outside Nova, then link the script in Project Settings... using the path option.

Now you can compile TypeScript with a cmd+B or by pressing the build button. The output file is called main.dist.js so you can add it to your gitignore with a *.dist.js.

Write some TypeScript

Finally we can write some TypeScript, let's create a script which will be the Extension entry point. If you have the TypeScript Extension enabled, you should start seeing the auto suggestions and linting while typing this out.

src/Scripts/main.ts

export function activate() {
  const message = new NotificationRequest('hi')
  message.title = 'Hello, world!'
  message.body = 'Lorem ipsum sil dor amet'
  nova.notifications.add(message)
}

export function deactivate() {
  // ...
}

Configure the Extension

To link everything up, configure your Extension's extension.json, to ensure it uses our main.dist.js. For the purpose of this tutorial, set activationEvents so your code always gets run. In production, you'd want to configure that to only activate the Extension when it is needed, so users aren't running it unnecessarily.

More information about activationEvents

Example.novaextension/extension.json:

{
  "main": "main.dist.js",
  "activationEvents": ["*"]
}

Run the Extension

Run a build with cmd+B and it should generate your code into Example.novaextension/Scripts/main.dist.js, providing there are no TypeScript errors.

Run the Extension locally with Extensions > Activate Project as Extension. This will activate this Extension for any active Nova windows and will automatically reload it when you rebuild (or if any other files inside your Extension folder change).

When activated, you should see a notification in the top right of Nova 🎉 it's all working!

Notes

If you change the strings and re-build, you should see the notification again with the new message.
Esbuild will bundle any node_modules that you import, but you should be careful about what those modules do. The JavaScript that Nova run's isn't the same as a Node.js environment and that might break some packages. You only have Web APIs and the nova global.
Make sure to update your activationEvents before publishing.

If you found this useful, Tweet me and let me know!

Bundle JavaScript with Eleventy and esbuild

2021-06-27T00:00:00Z

Static site generators are great... as I have previously mentioned, here is how to bundle JavaScript into your Eleventy site too.

When you're making a website with a static site generator, you have already chosen to not to create a Single Page App (SPA). But you might still want to add JavaScript to add dynamic features.

You can use an Eleventy JavaScript class template to bundle up multiple JavaScript files into a single backwards-compatible file using esbuild. Esbuild is a newer JavaScript bundler written in Go and it's very performant compared to tools like WebPack and Parcel.

The only downside to esbuild I've found is that it only goes back to es6, so no IE support...

Setup

Let's start with an empty project to see all that’s required.

# Create a project folder
mkdir esbuild-eleventy
cd esbuild-eleventy

# Create an npm project
npm init -y

# Install Eleventy and esbuild
npm install --save-dev @11ty/eleventy esbuild

Create some JavaScript

Next, create src/js/app.js using some modern JavaScript features, this is the file we want to bundle.

async function main() {
  await new Promise((resolve) => setTimeout(1000, resolve))
  console.log('done')
}

main()

Then create an Eleventy JavaScript class template scripts.11ty.js, this is in charge of bundling JavaScript for the website.

const esbuild = require('esbuild')
const { NODE_ENV = 'production' } = process.env

const isProduction = NODE_ENV === 'production'

module.exports = class {
  data() {
    return {
      permalink: false,
      eleventyExcludeFromCollections: true,
    }
  }

  async render() {
    await esbuild.build({
      entryPoints: ['src/js/app.js'],
      bundle: true,
      minify: isProduction,
      outdir: '_site/js',
      sourcemap: !isProduction,
      target: isProduction ? 'es6' : 'esnext',
    })
  }
}

A Note on JSX

If you want to use JSX, like in Using jsx WITHOUT React, you can add these parameters to the esbuild.build call:

{
  jsxFactory: 'createElement',
  jsxFragment: "'fragment'",
}

Similar to compiling Sass, this is an Eleventy class-based JavaScript template, that exports a class which Eleventy will instantiate. Eleventy will call the data method first, which is the same as the front-matter of a markdown file. We use the permalink to tell Eleventy not to create a file for this template, as esbuild does this for us. Eleventy then calls the render method which we use to call esbuild.

The esbuild JavaScript API handles outputting files and we tell it to put them in the same place the Eleventy does. A benefit of this is that the Eleventy template can be used to bundle multiple JavaScript entry points in one go, by passing entryPoints.

See esbuild's API docs for more information about the options you can pass here, like configuring source-maps or minifying code.

Configure Eleventy

Finally, configure Eleventy to use our template by creating .eleventy.js:

module.exports = function eleventyConfig(eleventyConfig) {
  eleventyConfig.addWatchTarget('./src/js/')

  return {
    templateFormats: ['md', '11ty.js'],
  }
}

This does two things. First, it tells Eleventy to watch for file changes in the src/js folder. So in development, when you edit a JavaScript file Eleventy will rebuild and reload the development server. Second it tells Eleventy to look for our .11ty.js templates and load them.

Build the site

This site doesn't have any markdown, but it will still generate our bundled JavaScript.

npx eleventy

You should see the JavaScript generated into the _site folder now, all done! 🎉

Compile Sass with eleventy

2021-05-25T00:00:00Z

Static site generators are great for generating HTML from markdown files. Eleventy (11ty) is a simple JavaScript based generator, named after the 11 templating languages it supports.

Static HTML is great, but websites need CSS too! My go to css preprocessor is Sass, so I wanted to find a way to compile Sass directly in 11ty projects. After a bit of experimentation, the best way I found to do this was with 11ty class templates. Using 11ty means there is only one process to worry about and it auto reload on sass changes.

Setup

Let's start with an empty project to show what’s required:

# Create a project folder
mkdir sass-eleventy
cd sass-eleventy

# Create an empty npm project
npm init -y

# Install 11ty and sass
npm install —save-dev @11ty/eleventy sass

Create some sass

Next, create src/sass/theme.scss:

$primary: #096e4f;

h1 {
  color: $primary;
}

Then back at the root of the project create theme.11ty.js:

const path = require('path')
const Sass = require('sass')

module.exports = class SassTemplate {
  data() {
    return { permalink: '/styles.css' }
  }

  render(data) {
    return Sass.renderSync({
      file: path.join(__dirname, './src/sass/theme.scss'),
      includePaths: [path.join(__dirname, './node_modules')],
      outputStyle: 'compressed',
    }).css
  }
}

This is an 11ty class-based JavaScript template, 11ty will create an instance of our class and call the data method. This provides the front matter for the template, similar to the top bit of a markdown file. We use this to set the permalink value to tell 11ty we want to create a file called styles.css.

11ty then calls render with the front matter data. This method returns what we want to put in the file. Here we use the Sass library to render the input file and return the css to put into the file. Pass includePaths here if you want to import sass from npm dependencies, like bulma.

Configure 11ty

Finally, configure 11ty to use our sass template with an .eleventy.js:

module.exports = function eleventyConfig(eleventyConfig) {
  eleventyConfig.addWatchTarget('./sass/')

  return {
    templateFormats: ['md', '11ty.js'],
  }
}

This does two things. First, it tells 11ty to watch for file changes in the src/sass folder. So in development, when you edit a Sass file 11ty will know to rebuild and reload the development server. Second it tells 11ty to look for our 11ty.js files, so the sass template gets loaded.

Build the site

This site doesn't have any markdown, but it will still generate our css.

npx eleventy

You should see the style.css file generated into the _site folder now, all done! 🎉

Future improvements

This setup could be improved, but its a good starting point and fine for lots of setups.

Things to improve:

Generate sass source maps, this might need Sass to generated in the data method and using the pagination api to create multiple files.
It might be improved by using asynchronous data / render methods, rather than using Sass.renderSync

Quick Array to Map Conversion in JavaScript

2021-02-28T19:50:00Z

Quick Array to Map Conversion in JavaScript

TLDR

const people = [
  { id: 'geoff', name: 'Geoff' },
  { id: 'helen', name: 'Helen' },
  { id: 'tim', name: 'Tim' },
]

const peopleMap = new Map(people.map((item) => [item.id, item]))

console.log(peopleMap.get('tim'))
// { id: 'tim', name: 'Tim' }

Converting an array to a map is a useful tool to turn an O(n) operation into O(1). If you're often looking up a value in an array that isn't changing, you can optimise it like this. Instead of looping through the entire array to find a value, you can look up an item by some key.

It's worth noting that this is not useful if you only have to do a single lookup of the array, you should just loop through it in that case. If you are often looking up values in the same array, that is when this is beneficial.

How does it work

JavaScript's Map has been around for a while and is designed for situations where an object has dynamic keys. The constructor takes an entries value, which may seem a little strange at first. This entries value is an array of key-value pairs which it can use to initially fill in the map.

const entries = [
  ['geoff', { id: 'geoff', name: 'Geoff' }],
  ['helen', { id: 'helen', name: 'Helen' }],
  ['tim', { id: 'tim', name: 'Tim' }],
]

Each item is an array where the first subitem is the key and the second is a value. When passed to the Map constructor, the key will be the index to lookup by and the value is the thing to lookup.

To quickly create these entries, we can use Array's map method which converts each item of an array into something else, based on a function you provide. You can use this to easily convert an array of objects into an array of entries.

const people = [
  { id: 'geoff', name: 'Geoff' },
  { id: 'helen', name: 'Helen' },
  { id: 'tim', name: 'Tim' },
]
const entries = people.map((object) => [object.id, object.name])

The arrow function passed to map takes an object and converts it into an entry where the key is the object's id and the value is the object itself.

Putting it all back together again:

//
// These are the objects we want to create a map of
//
const people = [
  { id: 'geoff', name: 'Geoff' },
  { id: 'helen', name: 'Helen' },
  { id: 'tim', name: 'Tim' }
]

//
// This creates a map from our array of people
//
const peopleMap = new Map(people.map(item => [item.id, item])

//
// Now we can find Tim
//
console.log(peopleMap.get('tim'))
// outputs: { id: 'tim', name: 'Tim' }

Now you have a map which you can quickly and efficiently lookup from. I've found this especially useful in Vue computed properties. When you have an array of models that you want to lookup values from, you can setup a computed property that generates a map. When the array changes it will automatically recompile the map for you based on the new array.

Running Node.Js as a systemd service

2020-11-16T09:58:50Z

systemd is a software suite that provides an array of system components for Linux operating systems.

For us, it lets you run services, start them on boot then check and manage the status of them. This is useful for making sure a node.js script is running whenever a raspberry pi is running. To run through this I'll be installing a node.js app called blinkit.

1 - Installing Node.js

Assuming you're starting from scratch, say from Raspbian lite, you'll need to install node first.

VERSION=v10.20.1-linux-armv6l

# Create and enter a temporary directory
TMP=`mktemp -d`
cd $TMP

# In the temp dir, get the node binaries and extract them
curl -sLO https://nodejs.org/dist/latest-v10.x/node-$VERSION.tar.gz
tar -xzf node-$VERSION.tar.gz
mv node-$VERSION /usr/src/node

# Link the binaries onto the $PATH
ln -s /usr/src/node/bin/node /usr/bin/node
ln -s /usr/src/node/bin/npm /usr/bin/npm
ln -s /usr/src/node/bin/npx /usr/bin/npx

# Clean the temporary directory
rm -r $TMP

Feel free to change the version of course, I used this older one as it supports armv6 on the Pi Zero I was installing on.

2 - Setup a user

We'll need a system user that the app will run as, we'll keep it simple and call them node. Then we'll give them sudo and gpio access (this is a special group on Raspberry Pis) and create a folder for the app to be in.

# Create the unix user
adduser node

# Add them to groups
usermod -aG sudo node
usermod -aG gpio node

# Create a node-owned folder to put the app in
# You call this whatever you like, probably the name of your application
mkdir -p /usr/src/blinkit
chown -R node:node /usr/src/blinkit

3 - Checkout the the app's repo

Now you need to get the code that you want to run, and put it in that new folder.

# Become our new node user
su node

# Clone the repo
git clone git@github.com:robb-j/blinkit.git /usr/src/blinkit
cd /usr/src/blinkit

# Install the app's production dependencies
# There could be trouble here if any of your dependencies require extra binaries, like python
# You might need to debug you packages and see what they require a bit
npm install --production

4 - Create a systemctl service

To run with systemd we need a file which tells systemd how to run and manage our service. For blinkit I put this file inside the repo which is pulled down, so it is available at /usr/src/blinkit/blinkit.service

blinkit.service

[Unit]
Description=blinkit
Documentation=https://github.com/robb-j/blinkit/
After=network.target

[Service]
Type=simple
User=node
ExecStart=/usr/bin/node /usr/src/blinkit/src/cli.js serve
WorkingDirectory=/usr/src/blinkit
Restart=on-failure

[Install]
WantedBy=multi-user.target

This service file is what systemd uses to know our application exists and what to do with it. You should change the Description, Documentation, ExecStart and WorkingDirectory statements to match your application.

Next we need to tell systemd about our new service. We will symlink the service file into the systemd directory, then restart the daemon and enable and start the new service.

# Your service name will be what you set `Description` to above
SERVICE=blinkit

# Link the service file into place
sudo ln -s /usr/src/blinkit/blinkit.service /lib/systemd/system/blinkit.service

# Reload the daemon so it knows about the new file
sudo systemctl daemon-reload

# Enable our new service
sudo systemctl enable $SERVICE

# Start the service
sudo systemctl start $SERVICE

Now the new service should be running and doing whatever it does. You can check its output with:

# -f follows the logs in real time
# -u reverses the order so its newest first
journalctl -fu $SERVICE

If you restart your host, the service should now start up during the boot. Congratulations! 🎉

Useful links:

My first generator function

2020-07-25T00:00:00Z

function* myFirstGenerator() {}

I've always strayed away from generator functions, they are foregin, different and generally confusing. From debugging babel-ified code I've understood that async code gets transpilled into generator code, but I never understood what it meant or how they worked. I found my first use of one and was pretty pleased with myself, so I'm going to share what happened.

For context, I was building something to make a git repo into a headless cms. It uses netlify-cms to directly edit a git repo then use node.js to serve that content as API. The bit I was working on was a command that clones the repo, reads and validates the files, then puts them into redis to be served by http.

So why did I use a generator? The problem I was facing was when reading in files in specific folders and validating them against a structure. But, I only wanted to write to redis if all the files are valid. You can write a nice method to read in a folder using a glob, validate each file's frontmatter against a structure and write to redis. But to seperate out the logic made the code more complex and much more verbose.

I also used superstruct version 0.10.x for the first time and it's really good

Say we have this method, which uses a glob pattern (e.g. src/pages/*.md) to match local files, read in the markdown frontmatter, validate it against the struct and return a tuple of validation errors and parsed records.

async function readAndParse<T>(
  pattern: string,
  struct: Struct<T>
): [Error[], T[]]

It would be very easy to add extra arguments so that function could also save the resulting records somewhere, but it couldn't wait until other calls of the function have also validated.

I wondered if I could define what I want it to do as data, then run it more manually:

const contentToParse = [
  { pattern: 'pages/*.md', key: 'pages', struct: PageStruct },
  { pattern: 'events/*.md', key: 'events', struct: EventStruct },
  { pattern: 'tags/*.md', key: 'tags', struct: TagStruct },
  { pattern: 'posts/*.md', key: 'posts', struct: PostStruct },
]

This is when I wondered about generators, was this what they're useful for? I had a quick browse through the MDN docs, then came up with this:

// Create a generator for each content type
const generators = contentToParse.map(async function* (job) {
  // In the first run, read and validate files that match the pattern
  const [errors, records] = await readAndParse(job.pattern, job.struct)

  // End the first run by yielding the errors
  yield errors

  // In the second run, save the validated records to our redis cache
  await redis.set(job.key, JSON.stringify(records))
})

With a generator function you get an object which you can programatically tell to run up to the next yield statement. This is good for my use case as I can create a generator for each content type and run each one upto but before saving the records. From there you can check for errors and then decide to proceed or not.

As code, it looks like this:

// You can asynchronously run each generator in parallel
const firstRun = await Promise.all(generators.map((gen) => gen.next()))

// Then grab all the errors out and flatten into a single array
const allErrors = firstRun.map((result) => result.value).flat()

Now I could exit early if there are any errors, or if there aren't any continue on:

// Run the generators again, which will continue on after the `yield` above
await Promise.all(generators.map((gen) => gen.next()))

And it works! I'm sure generators are much more powerful than my little use case, but I nerdily enjoyed my first encounter with them.

Automating developer operations for Node.js

2020-04-10T00:00:00Z

Dev agility is all about getting your code into production as fast as possible and turning the time you invested into product value. The faster you can securely and reliably get your code into the hands of your users, the better.

One part of this is packaging up code into containers, ready to be deployed on servers. Containers are great at defining the exact environment to run code in and automating the creation of them means you can get code live faster.

Automating this process means less time manually running tests and building containers. To do this you can hook into the git flow you're already using.

The workflow is as follows:

structure commits to describe changes to the application
semantically version the app based on those commits
[not covered] automatically build container images based on those versions

There are a couple of npm packages to help with this.

yorkie lets you define scripts that run on when you interact with git
@commitlint/cli is a utility for linting commit messages
standard-version analyses commits to automatically version your app and generate changelogs

Semantic versioning

This workflow uses semantic versioning to denote the changes to the application. From this you can see the relation between different versions and work out what changes are safe to be deployed.

There are three parts to a semantic version major.minor.patch. If the application is a major change, e.g. 1.2.3 to 2.0.0, you know there is some extra work needed to migrate a deployment. If there is a minor or patch change, e.g. going from 2.0.0 to 2.0.1 or 2.1.0 you know it's safe to deploy the new version as it (should) only contain backwards compatable changes.

More about Semantic versioning →

Conventional commits

Conventional commits are a standard format for commit messages that describe the type of changes made. It also forces commits towards atomic single-concern commits, i.e. only working on one bug/change/feature at a time.

For example:

fix: stop user from dropping users table is a backwards-compatible fix and generates a semver patch
feat: add new fancy buttons is a backwards-compatible enhancement and triggers a semver minor version
BREAKING CHANGE: redesign login form is a breaking change and triggers a major semver change

You can also supply a commit body and/or footer to add more info to your commit. Here's a full example:

BREAKING CHANGE: redesign login form

We reviewed user feedback and moved all the buttons around
to make the login flow as easy as possible

resolves #3

I find the footer works nicely with github's fix/resolves syntax

More info on conventional commits →

Standard version

The final piece of the puzzle is standard version, it is a command that parses git commits to work out whats changed since the last version. Because the commits are "conventional" it can work out what the next semantic version of the app should be and it can accurately generate a changelog from commit messages.

For example, if you'd commited three fix:-es and a one feat: it would increase the semver's minor part. It'll then go away and generate a changelog with those commit messages and run the npm version command to increament that minor version. It creates a commit for those changes and tags it with the new version.

When you push up a tag you then have a pipeline of your choice to build a container image.

The process

# Commit your atomic change
git add somefile.txt
git commit -m "feat: add that really cool thing you wanted"

# Generate a new version
npm run release

Or a more complex example:

git checkout -b feature-branch

# work on a feature ...

git commit -m "fix: add check for divide by zero"

# some sort of code review ...

git checkout master
git merge feature-branch
npm run release
git push --follow-tags origin master

# ci magic to build image ...

Most of these can be done inside your favourite IDE

Project setup

Start by adding these dependencies

npm install --save-dev \
  yorkie \
  @commitlint/cli \
  @commitlint/config-conventional \
  standard-version

Then add this to your package.json:

"commitlint": {
  "extends": ["@commitlint/config-conventional"]
},
"gitHooks": {
  "commit-msg": "commitlint -e $HUSKY_GIT_PARAMS"
}

First it sets up commit lint to ensure your commits meet the conventional commits standard. This means you can't commit unless you

Second it sets up a git hook to run commitlint on commit-msg, which means your commit will fail if it doens't meet the standard.

Next add this script to your package.json.

"release": "standard-version"

Now you can perform a release with npm run. Like this:

npm run release

Thats it.

Write commit messages that are conventional commits
Use npm run release to version releases and generate changelogs
Next setup a CI/CD to generate a docker image based on git tags

Tales from the bashrc: d1

2020-03-28T00:00:00Z

alias d1='docker container run -it --rm'

Sometimes you just want to pop into a container to see whats in there, run an arbitrary command or maybe avoid something entering your bash history. I made it nice and short and pass a couple of params, from the --help page:

-i, --interactive    Keep STDIN open even if not attached
-t, --tty            Allocate a pseudo-TTY

The first gotcha of docker container run is the it doesn't start interactively, any input you enter after starting a container doesn't get passed to it. -i and -t solves this by attaching STDIN of your terminal to the container.

--rm                 Automatically remove the container when it exits

The second bit is --rm, this one is more of a preemtive housecleaning. If you've been running containers for a while you'll realise how quickly they build up. If you use --rm it automatically removes the container after it finishes, which is perfect for these little ephemeral containers.

Having this means there are less stopped containers on your system and also less unused images too. Images will stick around while they have a container attached to them.

For more maintenannce you can always do a docker system prune which picks up these things too.

Examples

Here are a couple of things I regularly use this for:

# Start a node.js repl using node.js version 12 in alpine
d1 node:12-alpine

# Start (b)ash in nginx to inspect the default config / filesystem
d1 nginx:1-alpine ash

# Run an unsecure local mongo database
d1 -p 27017:27017 mongo:3

# Quickly serve a directory using nginx, this could be another alias ;)
d1 -p 80:80 -v `pwd`:/usr/share/nginx/html nginx:1-alpine

That's d1 – a simple alias which I actually use every day.

alias d1='docker container run -it --rm'

Useful Raspberry Pi WiFi commands

2020-03-21T00:00:00Z

I keep needing these commands when doing pi things, so I thought I'd put them here in one place:

# Edit the wifi config file
# e.g. to add a new router
sudo nano /etc/wpa_supplicant/wpa_supplicant.conf

# Reconfigure the wifi on the pi, reloading the config
sudo wpa_cli -i wlan0 reconfigure

# Check the status of the wifi connection
sudo wpa_cli -i wlan0 status

Spoofing a Raspberry Pi's mac address

2020-02-22T00:00:00Z

It's surprisingly simple to make a raspberry pi appear as a different mac address on a network.

Why you want to spoof it isn't important, all you need to do is modify /boot/cmdline.txt by appending:

smsc95xx.macaddr=aa:bb:cc:dd:ee:ff

Then reboot your pi (sudo reboot) and it will force it to appear as that mac address.

Tales from the bashrc: bashrc

2020-02-13T00:00:00Z

alias bashrc='nano ~/.bashrc && source ~/.bashrc'

I'm that lazy, but I'm quite happy with this. It simply edits my bashrc and automatically reloads the config into the current terminal. I think I just found it faffy to edit the right file on mac, only to try the thing I changed and realised I forgot to source the file too.

It also works for zshrc too, although it does take a bit longer to re-source a zshrc (maybe because I'm using oh-my-zsh).

alias zshrc='nano ~/.zshrc && source ~/.zshrc'

Just be careful editing this alias iteself. I got into a bad loop when trying this where it would constantly keep opening nano every time I closed it.

Connecting a Raspberry pi to 802.1x WiFi

2020-02-03T00:00:00Z

First you'll want to connect your pi either via ethernet, an actual keyboard & monitor or ssh-over-usb if you're fancy. All the wifi config on a pi is setup in /etc/wpa_supplicant/wpa_supplicant.conf.

First though, you need to generate a hash of your password, so your password isn't stored in plaintext on the very-hackable pi.

# ssh you@yourpi.local
read -s -p "Password: " pass && echo -n $pass | iconv -t utf16le | openssl md4 | sed 's/(stdin)= //'

# then type your password and hit enter and you'll get:
# Password: abcdef123abcdef123abcdef123

# From now on $PASSWORD will be this hash
PASSWORD=abcdef123abcdef123abcdef123

# clear the $pass variable if you're paranoid ;)
unset pass

Now you can start editing your wpa_supplicant.conf file:

sudo nano /etc/wpa_supplicant/wpa_supplicant.conf

First make sure it contains these values at the top, setting the country code to wherever the pi is. In newer pis it needs this to be set for wifi to work

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

Then add a network block for the connection you want to conenct to. Where $SSID is the name of the router, $USERNAME is your username for the wifi and $PASSWORD is the hash from above. Be careful with the quotes!

network = {
	ssid="$SSID"
	proto=RSN
	key_mgmt=WPA-EAP
	pairwise=CCMP
	auth_alg=OPEN
	eap=PEAP
	phase2="auth=MSCHAPV2"
	identity="$USERNAME"
	password=hash:$PASSWORD
}

These work for the 802.1x wifi I'm connecting to

Now you can use the wpa_cli to restart the pi's network interface.

# Tell it to reconfigure itself with the changes from wpa_supplicant.conf
sudo wpa_cli -i wlan0 reconfigure

# You can check on what its doing with this:
sudo wpa_cli -i wlan0 status

# You can also check for an ip with
ifconfig wlan0

# And you can check its working with this
nslookup duck.com

Now your pi's on the internet! 🥳

Credit to Dan Jackson for the required arguments and password hash generator.

Tales from the bashrc: npr

2020-01-27T00:00:00Z

alias npr='npm run -s --'

I added a new alias to my .bashrc today. I was fed up with npm's run command, it does two things that were annoying me.

Needlessly noisy

The first thing is that it's noisy. It spews out lots of information that I've never found useful and distracts from the real error(s).

For instance, running npm run lint in this repo spews out all this:

> @robb_j/r0b-blog@1.0.0 lint /Users/rob/dev/r0b/blog
> eslint

sh: eslint: command not found
npm ERR! code ELIFECYCLE
npm ERR! syscall spawn
npm ERR! file sh
npm ERR! errno ENOENT
npm ERR! @robb_j/r0b-blog@1.0.0 lint: `eslint`
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the @robb_j/r0b-blog@1.0.0 lint script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/rob/.npm/_logs/2020-01-14T20_35_05_835Z-debug.log

Whereas npm run -s lint goes straight to the error:

sh: eslint: command not found

Option stealing

The second annoying thing is that it steals dash-dash options, which gets infuriating when you're making a command-line tool.

If you ran npm run cli --help, the --help doesn't make it to your application, it instead shows the help for npm run. This is fixed by adding -- after run.

The alias

So I created npr, which combines these two things into a new smaller command. It's not going to change the world, but it makes my life a tiny bit easier.

alias npr='npm run -s --'

A minimal kiosk mode for a Raspberry Pi

2020-01-20T00:00:00Z

When I was making my Smart mirror I made a webapp to run on the Raspberry Pi hidden behind the frame. From all the tutorials I saw, they reccomended using a full instillation of the Raspbian desktop, then installing chrome, then hacking away at it. I didn't like this.

I felt this was a waste as all I wanted was to run chrome. I didn't need a window manager, or any of the countless pre-installed apps that come with raspbian. I wanted a minimal footprint on the pi, with hopefully only chrome. This means less to manage and a smaller security footprint.

Then I found this blog post, which promised to do all of that.

Steps

Start with Raspbian Lite (not the desktop version). Then install the packages required to run chromium and set the pi to boot straight into to the console.

sudo apt-get update -qq

sudo apt-get install --no-install-recommends xserver-xorg-video-all \
  xserver-xorg-input-all xserver-xorg-core xinit x11-xserver-utils \
  chromium-browser unclutter

# Go to: Boot Options > Console Autologin
sudo raspi-config

19/10/25 — On Debian 13+ chromium-browser seems to have been renamed to just chromium

Next edit /home/pi/.bash_profile to automatically start the gui. There's a check for the bash context first, so you don't accidentally start chromium whenever you ssh in.

if [ -z $DISPLAY ] && [ $(tty) = /dev/tty1 ]
then
  startx
fi

The last bit is to setup /home/pi/.xinitrc to run chromium whenever you run startx. Here's the full list of chromium arguments.

#!/usr/bin/env sh
xset -dpms
xset s off
xset s noblank

unclutter &
chromium-browser https://yourfancywebsite.com \
  --window-size=1920,1080 \
  --window-position=0,0 \
  --start-fullscreen \
  --kiosk \
  --incognito \
  --noerrdialogs \
  --disable-translate \
  --no-first-run \
  --fast \
  --fast-start \
  --disable-infobars \
  --disable-features=TranslateUI \
  --disk-cache-dir=/dev/null \
  --overscroll-history-navigation=0 \
  --disable-pinch

It disables the cursor and screensaver. Then runs chromium with *all* of the flags. Set https://yourfancywebsite.com to the website which you want to display. And set --window-size to the size of your display (it's horizontal first and vertical after the comma).

You may also want to uncomment disable_overscan=1 in /boot/config.txt so that the pi boots up using the full display.

Now whenever the pi boots up it'll go into the console then on into chromium. If you want to exit you can hit Alt+F4, then enter startx to start up the browser again.

Using jsx WITHOUT React

2020-01-12T00:00:00Z

From my experience, jsx has always been synonomous with React. But jsx can in fact be used without it. Jsx lets you write xml structures in javascript and apply your own meaning to them. You can see Facebook's specification here.

Jsx is not meant to be implemented as part of the Ecmascript specification, but rather used by preprocessors (like babel) to transform jsx code to native javascript code. Much like typescript is transpilled into javascript.

What jsx does

Jsx is passed to babel and it converts your xml into javascript function calls. You pass a pragma option to babel which tells it how to process your xml. For instance if you set pragma to createElement and run this script through it:

let myDocument = <p id="name">Hello, world</p>

Babel would convert your code into:

let myDocument = createElement('p', { id: 'name' }, 'Hello, world')

Which has nicely transformed our code into something a browser (or node.js) will understand. We still have to craft our createElement though. In React this is a method which creates components, but we can make it do whatever we want.

Pragma method signature

A pragma method takes two parameters and a rest argument, for example:

createElement(tagNameOrComponent, attributes, ...children)

tagNameOrComponent is either a string tagName of a native element (like a <p> or <div>) or a variable to something in the scope of your jsx. Maybe you want to pass a function and call it inside you pragma method.

attributes is an object of key-value pairs that you passed to the xml element.

...children is a rest argument of any xml children that are inside the parent tag. They have already been processed by your pragma method.

An example

So now to use this in a real (trivial) example. First setup a node project and install some dependencies.

# Create a project folder
mkdir jsx-without-react
cd jsx-without-react

# Create an npm project
npm init -y

# Install babel
npm install @babel/cli @babel/core @babel/plugin-transform-react-jsx

Next we'll need a .babelrc which will tell babel how to process our jsx.

{
  "plugins": [
    ["@babel/plugin-transform-react-jsx", { "pragma": "createElement" }]
  ],
  "comments": false
}

And then add an index.jsx file

function createElement(tag, attributes, ...children) {
  console.log({ tag, attributes, children })
}

let myDocument = <p>Hello, world</p>

Now you can run this command to generate your javascript.

npx babel index.jsx -d dist

Which will create a new dist/index.js file with the transpilled code.

An example with dom elements

For a real world example, lets use jsx to create dom elements and render them. With the same setup as before, lets change our index.jsx to:

// A jsx pragma method to create html dom elements (more info below)
function createElement(tagName, attrs = {}, ...children) {
  const elem = Object.assign(document.createElement(tagName), attrs)
  for (const child of children) {
    if (Array.isArray(child)) elem.append(...child)
    else elem.append(child)
  }
  return elem
}

// Setup some data
const name = 'Geoff'
const friends = ['Sarah', 'James', 'Hercule']

// Create some dom elements
const app = (
  <div className="app">
    <h1 className="title"> Hello, world! </h1>
    <p> Welcome back, {name} </p>
    <p>
      <strong>Your friends are:</strong>
    </p>
    <ul>
      {friends.map((name) => (
        <li>{name}</li>
      ))}
    </ul>
  </div>
)

// Render our dom elements
window.document.getElementById('app').replaceWith(app)

You can add // eslint-disable-next-line no-unused-vars to ignore annoying eslint errors

And to test our code, add an index.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <meta http-equiv="X-UA-Compatible" content="ie=edge" />
    <title>jsx WITHOUT react</title>
    <link
      rel="stylesheet"
      href="https://cdn.jsdelivr.net/gh/kognise/water.css@latest/dist/light.min.css"
    />
  </head>
  <body>
    <div id="app"></div>
    <script src="dist/index.js"></script>
  </body>
</html>

Then run the transpiller again and open your index.html in a browser. I snuck in water.css to make raw html prettier.

npx babel index.jsx -d dist

My createElement function

Here is my createElement from above, it does a couple of things.

function createElement(tagName, attrs = {}, ...children) {
  const elem = Object.assign(document.createElement(tagName), attrs)
  for (const child of children) {
    if (Array.isArray(child)) elem.append(...child)
    else elem.append(child)
  }
  return elem
}

First it creates a dom element with document.createElement and assigns the jsx attributes onto it with Object.assign. This lets you set most element properties like id or Element#className.

Next it loops through the child elements, which have already been generated into dom elements with this function. With each element it adds them as a child. I chose Window#append because will it create text nodes for you too. It checks for arrays children too so it can add any nested children, like the friend's mapping above:

<ul>
  {friends.map((name) => (
    <li>{name}</li>
  ))}
</ul>

pragmaFrag

There is another babel parameter which lets you handle jsx short fragments. You can pass pragmaFrag in your .babelrc to tells babel how to handle fragments.

If you setup a .babelrc like this:

{
  "plugins": [
    [
      "@babel/plugin-transform-react-jsx",
      { "pragma": "createElement", "pragmaFrag": "'fragment'" }
    ]
  ]
}

Then you can:

function createElement(tagName, attrs = {}, ...children) {
  if (tagName === 'fragment') return children
  // Same as above ...
}

const elements = (
  <>
    <p>Hello,</p>
    <p>world!</p>
  </>
)

It will now pass whatever you set pragmaFrag to your pragma method, which you can handle in any way you like. In this case 'fragment' will get passed to createElement which the method handles by returning the child elements. This can be useful if you don't want to create too many extra dom elements.

That's what I know about using jsx without react. I've found it useful for tidying up small web apps which only need dom access.

You can find the example at robb-j/r0b-blog

r0b's random ramblings

Notes on restoring my iPod Classic

Notes on restoring my iPod Classic (4th Generation)

What I modified

The boot menu

Charging the device

Replacing the HDD

Synchronising the device

Battery Life

Audio books

Apple USB SuperDrive

Useful links

Using Explicit Resource Management with TypeScript and Postgres

Overview

My Code

Notes

Async iteration for JavaScript EventTargets using ReadableStreams

Learning a bit of Für Elise

First recording

Second recording

Third recording

Get to da Terminal

Isomorphic JavaScript, Web Standards and Documentation Driven Design

Web Standards

Background

Documentation driven design

Gruber

Eleventy WebC in a nutshell

Prerequisites

Configuration

Pages

Components

Layouts

Round up

Current questions

Next steps

Creating a HTTP proxy with Deno

The code

Host an ics calendar feed with Eleventy

Set up

Events collection

Calendar metadata

Eleventy template

Timezones

Generating the feed

Next steps

Embed JSDoc comments in an Eleventy website

How it works

Next steps

Yoath released

Using URLPattern to add "edit on GitHub" support to my blog

Trying to make a vanilla web app

How it works

Page-based navigation

Data binding

Kiosk display affordances

HTML templates

The only dependency

Event-driven issues

Honourable mentions

Conclusions

Creating drag interactions with setPointerCapture in JavaScript

How it works

Bonus: card flip animation

Wrapping up

Deploying ESP32 with SPIFFS using github actions

Background

The pipeline

Hurdles

What is a sketch

IDE setup

Setting and configuring arduino-cli

Custom libraries as submodules

Arduino partitions

Creating a SPIFFS partition

Flashing from the CLI

Where is esptool and mkspiffs

Misc

Bonus: monitor serial from the CLI

Next steps