Building a Chord Chart App for Someone With No Free Hands

VIP is a small app I built for a major touring artist's VIP sessions: the intimate pre-show sets where he plays a handful of songs for a room of fans. The whole brief fits in one sentence from the README: an iPad on a stand, at arm's length, with a guitar in both hands, so nothing important needs a finger.

Someone says or types a song title, the chord chart comes up in big type on a dark screen, and it scrolls itself at a rate set once per song. That's the product. Most of the work went into the parts you don't see.

Designing for someone with no free hands

A touchscreen is a bad input device for a guitarist mid-song. So I treated touch as the fallback and built three other ways in:

• Voice. "Scroll", "stop", "faster", "slower", "next", "up a key", "play" plus a title.
• A wave over the front camera to turn the page.
• Keyboard keys, which quietly covers Bluetooth page-turner pedals, because those just send arrow and page keys.

All three feed a single event, so the performance screen doesn't care where a command came from:

window.dispatchEvent(new CustomEvent("vip:command", { detail: { cmd: "page" } }))

The voice side had two details worth writing down. First, Safari ends speech recognition after every pause, which in practice means after every command. The fix is blunt: when it ends, and we still want it on, start it again.

rec.onend = () => { if (this.on) { try { this.rec.start() } catch (e) {} } }

Second, in performance mode I act on interim results, not just final ones. Waiting for the recogniser to finalise "stop" costs a noticeable beat, and when the chart is running away from you, a beat matters. Acting on interim text means the same word can fire twice as the transcript firms up, so there's a 900ms debounce. The stop words are checked first, so if a phrase contains both "stop" and "go", stopping wins.

When hands are busy, the cheapest command to get wrong is the one that stops things. Everything else can wait for a second attempt.

Forgiving fuzzy matching

Voice recognition in a loud room mishears constantly. A title comes back with a letter dropped, a word pluralised, a number spelled out, an apostrophe gone. A search box that demands an exact match is useless here.

The matcher normalises first: lowercase, strip apostrophes, turn & into "and", and rewrite a couple of spelled-out numbers into digits, because the recogniser says "sixty nine" where the title has digits. Then it scores in tiers, from most to least confident:

1. Exact match (1000)
2. Title starts with the query (800, minus title length, so shorter titles win)
3. Title contains the query (620, minus where it starts)
4. Word-by-word matching, where each spoken word can be a prefix of a title word or within a small edit distance of one
5. Whole-phrase near miss, allowing edits up to 28% of the length
6. The phrase sitting near-miss somewhere inside a long title
7. Album name contains it (40)

The per-word tolerance scales with word length, which is the bit I'd reuse elsewhere:

const max = w.length >= 6 ? 2 : (w.length >= 4 ? 1 : 0)

Short words must match exactly or as a prefix. A four-letter word gets one typo. Six letters or more gets two. Without that, every three-letter word matches every other three-letter word and the results turn to soup. So "brokn promise" still finds a title like "Broken Promises", while a short word like "on" never fuzzily matches "of", "or" and "in".

The edit distance is bounded: it tracks the best value in each row and bails as soon as that exceeds the limit. It's run against every title on every keystroke, so not finishing a hopeless comparison is the whole optimisation.

Spoken requests also get their lead-in stripped ("hey", "play", "pull up", "let's do", "the song") before any of this runs, because nobody says just the title.

When I built the native app I ported this to Swift line for line, with the same weights, so the web and the iPad rank songs identically. If two clients search the same list, they should agree on what comes first. Otherwise someone learns one app's quirks and gets surprised by the other.

The wave gesture

I expected this to need something clever. It didn't. It's frame differencing on a tiny image:

• Grab the front camera at 320x240, draw each frame into a 48x36 canvas.
• Sample only the top 45% of that frame.
• Convert to greyscale, compare with the previous frame, average the absolute difference per pixel.
• If it crosses a threshold (26), turn the page, then ignore everything for 1.6 seconds.

The top-band restriction is the important decision, and the comment in the code explains it better than I can here: his hand passes above the iPad, while the guitar and the strumming arm sit lower in the frame. Sample the whole image and every strum turns the page. Sample the top and only a deliberate wave does.

The cooldown does the rest. A single wave produces several frames of motion, and without the cooldown one wave would flip three pages. There's also a small meter on screen showing motion against the threshold, which makes tuning in a new room a matter of watching a bar instead of guessing.

The right constraint on the input often beats a smarter model of it.

Sync, then no sync

VIP started life reading its catalogue from another app of mine, a song-request tool for the same tour where fans pick what they want to hear. The idea was that songs, keys and charts would live there and VIP would pull them down.

Within the first afternoon I had to answer the obvious question: what happens when a chart gets fixed on the iPad and the next sync arrives? I split the fields in two. Catalogue facts (title, album, year) always came from the source. Authored fields (chart, tab, key, BPM, capo) filled in from the source, but once edited locally they were left alone:

next if !blank_here && song.chart_edited_at.present? && !@force

chart_edited_at was stamped by a before_save whenever any authored field changed, and the sync set a flag so its own writes didn't count as edits. Matching fell back from remote ID to title, so songs seeded locally got adopted instead of duplicated. Every run logged how many songs it had skipped to protect local edits.

It was a reasonable design, and it lasted about five hours. That evening I deleted the whole thing: the sync service, the rake tasks, the sync-run table and the tracking columns. The charts were always going to be authored in VIP, by the people using VIP. The other app didn't really own them, it just had them first. Once that was clear, all the conflict logic was protecting against a situation I could remove instead.

The best sync conflict rule is not having two sources of truth. VIP got a proper song editor and a setlist builder (buttons to reorder rather than drag-and-drop, so it works with a finger on an iPad) and became standalone.

Where it landed

The performance screen then moved to a native SwiftUI app, and Rails became a password-protected admin that serves the catalogue as one JSON payload. No paging, no partial sync. The iPad writes that payload to a local file and reads everything from it. A failed sync leaves the old file alone, and a snapshot ships in the app bundle so first launch is never blank. Venue wifi is not something I want a show to depend on.

Scrolling there uses a CADisplayLink nudging a real UIScrollView by fractions of a point each frame, because at slow speeds you want the chart to drift, not step. The fuzzy matcher and chord parser came across. Voice and the wave gesture are the web version's for now: the native app has the camera and microphone permission strings in place, but that code hasn't been ported yet.