Rebuilding RFSPager: Laravel to Go, and a POCSAG Decoder of Our Own

RFSPager decodes the POCSAG pager network the NSW Rural Fire Service and Fire and Rescue NSW still run on, and turns it into live incident feeds and push notifications for volunteers and the wider community. It is a community project, not an official dispatch or life-safety system, and it has grown to a bit over ten thousand users across a web app and iOS and Android.

For a few years it ran on Laravel, with the pages themselves arriving from a chain of rtl_fm piped into multimon-ng, scraped by PagerMon, and POSTed into the API. Over a few days this month I replaced the whole thing: the backend is now a single Go binary, and the receivers run a POCSAG decoder we wrote ourselves. This is the story of both halves, because they turned out to be the same story, which is that the old pipeline had too many moving parts and not enough control over any of them.

The old stack

The backend was Laravel 10 with the full buffet: Lighthouse for GraphQL, Horizon on Redis for queues, Pusher for websockets, Nova for admin, Telescope and Pulse for observability, Meilisearch for search, Sanctum for tokens. On the radio side, each Raspberry Pi ran one Docker container per dongle per frequency. Inside each container, rtl_fm demodulated FM audio into multimon-ng -a POCSAG512, and PagerMon's reader.js scraped multimon's stdout and POSTed the text to the API.

Count the moving parts in the path of a single page: a tuner, rtl_fm, multimon-ng, a Node scraper, a PagerMon server, a webhook plugin, and only then Laravel. Seven things, each of which could and did fail independently, and the receiver at the front of it was started by hand in a root tmux session that did not survive a reboot.

It mostly worked. "Mostly" is the problem with a system people rely on during a fire.

Part one: Laravel to Go

The dedupe bug that ate 41% of the table

The symptom was duplicate pages. The same message would land in the feed two or three times. When I went looking, the deduplication logic had a window that was supposed to treat identical text to the same pager address within five minutes as one message. It was built on Carbon, and Carbon's DateTime is mutable. The essence of it:

// The ±5 minute window, roughly as it was:
$start = $timestamp->subMinutes(5);   // mutates $timestamp in place
$end   = $timestamp->addMinutes(5);   // mutates it again, back to ~now

// $start, $end and $timestamp are all the SAME object.
// The "ten minute window" is now a single second.

Both variables pointed at the same object, so after the second call the window had collapsed to the instant the message arrived. Only exact same-second repeats were ever caught. In the week I measured, 41% of the rows in the table were exact duplicates. Two and a bit million rows, nearly a million of them junk.

The fix in PHP is a one-liner (copy() the instance, or use CarbonImmutable), and I did apply it in place to keep the bleeding stopped. But a check-then-insert across several redundant receivers also has a race that no amount of immutability fixes, and that is the kind of thing you want the database to arbitrate, not four PHP workers hoping for the best.

A feed query that read the whole table

The main feed is "most recent pages, newest first, excluding soft-deleted rows and rows whose capcode we do not recognise". Nothing in the schema indexed that shape. So Postgres read every row in the table, threw away the million-and-a-half that did not match the filter, and top-N sorted whatever survived. Page one took 2,144 milliseconds, and every scroll paid it again, because the GraphQL layer paginated with a connection type that also ran a COUNT on each page.

Search was worse. It was ILIKE '%term%', which cannot use a normal index, so a search for a term that matched nothing walked the entire feed at about 3.7 seconds.

Realtime that was not

Pusher was configured to broadcast new messages. The prod failed_jobs table told the real story, thousands of rows of:

Pusher error: cURL error 6: Could not resolve host: wss.rfspager.app

A realtime notification system whose realtime had been quietly broken for a long time, failing several times an hour, re-queuing, failing again. Next to it, a hazard-reduction importer that had been returning 403 every five minutes since roughly September of the previous year, because RFS had moved that feed behind a CMS login and nobody had noticed the cron quietly dying. The queue was full of noise, so real failures hid in it.

This is the part that actually decided it for me. I was keeping the thing alive by SSHing into the box and editing PHP in the release directory, patches that a redeploy would have silently reverted. When your operational model is "edit prod by hand and hope nobody deploys", the framework is no longer helping you.

The Go rewrite

The replacement is one statically-linked Go binary. chi for routing, sqlx straight onto the same Postgres database Laravel was using, and that is nearly the whole dependency list. The pieces that used to be separate services became features of the one process:

• Realtime is Postgres, not Pusher. An AFTER INSERT trigger on messages fires NOTIFY rfs_message. The server holds one dedicated listener connection, and fans that notification out to the SSE endpoints the apps subscribe to. Horizon, Redis and Pusher all disappear into one LISTEN/NOTIFY mechanism and a SELECT ... FOR UPDATE SKIP LOCKED worker.
• REST, not GraphQL. The four-hundred-line schema is gone. The mobile and admin apps talk to typed REST endpoints.
• Auth reuses the existing hashes. Login does a bcrypt.CompareHashAndPassword against the same users.password column Laravel wrote, so every existing account kept working with no reset. Tokens are HS256 JWTs with a token_version counter I can bump to revoke.
• Notifications and in-app purchases are native. APNs and FCM directly, and Apple StoreKit2 receipts validated locally by verifying the JWS signature chain up to a pinned Apple root certificate, so there is no third-party billing SDK and no per-check call to Apple.

Migrations are embedded in the binary with go:embed and run on boot, each in its own transaction, serialised across processes with a Postgres advisory lock and recorded in a schema_migrations table. Deploying is: cross-compile for linux/arm64 with CGO_ENABLED=0, copy one file, restart one systemd unit.

Running two backends on one database

The risky part is that I did not want a big-bang cutover. I wanted Go and Laravel pointed at the same physical database for a while, with Laravel still able to serve the old app versions until everyone updated.

That constrains what the Go migrations are allowed to do. The rule I held to: Go only ever adds columns and tables, never renames or drops anything Laravel reads. Where the shapes genuinely conflicted, a trigger bridged them. Go normalised pager agencies into their own table with a foreign key, but Nova still read and wrote the old free-text agency column on capcodes. So a BEFORE INSERT OR UPDATE trigger keeps the two in sync in both directions: set the foreign key and it backfills the text, set the text and it upserts the agencies table and fills the key. Laravel never knew anything had changed underneath it.

Pagination got the classic trick on the way past: a keyset cursor ordered on (created_at DESC, id DESC), fetching LIMIT n + 1 and using the presence of that extra row to decide has_more. No COUNT, stable while live rows are prepended by SSE, and the same cost whether you are on page one or ten thousand pages deep into two million rows.

And the dedupe moved into the database where it belonged:

// Keyed on the pager address, released on commit.
pg_advisory_xact_lock(hashtextextended(address, 0))

Take the lock on the address, re-check for duplicates inside the transaction, then insert. The check is two-pass: exact MD5 of the text within five minutes, plus a match on the first thirty characters within thirty seconds to catch retransmissions whose tail got corrupted. Even with a correct window, three redundant receivers hearing the same page still produced about 6.7% duplicates before this lock went in. That is a different number and a different bug from the 41% above, and worth keeping them straight: one was a collapsed window, this one is a genuine check-then-act race.

The cutover

Because the Go ingress endpoint deliberately speaks the same contract the old PagerMon webhook did (an apikey header, and it accepts both JSON and form-encoded bodies), switching the live page feed from Laravel to Go was a Caddy config change, one line repointing a hostname. Not a single third-party receiver had to be touched.

Two traps made cutover interesting. The first: every historical row had a "has this been fanned out to devices" flag that was null, because that column was new. If I had simply started Go against the live database it would have cheerfully tried to push all two million historical pages to every phone at once. So the cutover ran a deliberate UPDATE messages SET fanned_out_at = NOW() WHERE fanned_out_at IS NULL first, a four-minute write across the whole table to mark history as already delivered, and only then started the server.

The second was quieter: prod had no database backups at all. Laravel had never run any. So before I trusted the Go deployment I upgraded Postgres from 14 to 17 (a logical upgrade, five minutes and forty-seven seconds of downtime, and remember to re-ANALYZE because a copy upgrade loses the planner statistics), and stood up a nightly pg_dump that is age-encrypted and pushed to Cloudflare R2, with a decrypt-and-checksum round trip to prove the backup is restorable. It costs about sixty US cents a month. It is the single most valuable thing the migration produced, and it has nothing to do with Go.

The headline numbers, once the indexes existed and the query shapes were right:

Part two: multimon-ng to our own POCSAG decoder

The backend was the half I expected to do. The decoder was the half that turned out to matter more.

Why multimon-ng had to go

The complaint that started it, more or less verbatim from my own notes at the time:

We are seeing a tonne of decode issues. Our decoder is simply not up to scratch and needs to be fixed. Other stations running RTL-SDR and POCSAG on Linux don't have issues with text like we have.

multimon-ng is a fine tool, but it is a general-purpose decoder reading an audio stream, and its failure mode under a fade is to give up: read its POCSAG source and it abandons a message after two consecutive codewords it cannot correct. On a VHF paging signal with a receiver in a suburban house, two bad codewords in a row is a Tuesday. You lose the tail of the message, or you lose the message, and the audio-pipe design gives you no way to know a drop happened or to recover alignment after it.

The emblem of all this was one capcode: the Fire and Rescue beacon that transmits once a minute, every minute, forever. It was so reliably mangled that someone had long ago aliased it in the database as "Rubish" and set it to be ignored. In the week before I touched any of this, one receiver logged that beacon 2,264 times and decoded it cleanly twice.

The receiver, end to end

So the receivers now run a single Go binary that does the whole chain itself, from raw radio to decoded page:

RTL-SDR IQ @ 1.92 MS/s
  -> NCO mixer            shift each channel to baseband (±187.5 kHz)
  -> FIR low-pass + decimate ÷50   -> 38.4 kHz
  -> FM discriminator     arg( x[n] · conj(x[n-1]) )
  -> matched-filter bit slicer      512 / 1200 / 2400 baud, in parallel
  -> POCSAG framer + BCH(31,21) error correction
  -> authenticated WebSocket to the server

One wideband capture at 1.92 MS/s, with the dongle tuned to the midpoint between the two paging frequencies and each channel mixed down from its offset in software. That alone fixes a structural limit of the old setup: one dongle now covers both RFS and Fire and Rescue, where before it was one dongle, one container, one frequency.

The mixer is a numerically-controlled oscillator, a complex phasor rotated per sample and renormalised once per block, rather than calling cos/sin for every one of nearly two million samples a second. The FIR computes only the outputs it is going to keep after decimation. The FM demodulator is the textbook quadrature discriminator: the argument of each sample times the conjugate of the last. None of this is novel. The point is that it is ours, it runs block-at-a-time, and we can see inside it.

POCSAG, and actually correcting errors

POCSAG itself is a tidy little format once you have clean bits. A 576-bit preamble of alternating ones and zeros, then a 32-bit sync word, then batches of sixteen 32-bit codewords, each codeword a 21-bit payload protected by a BCH(31,21) code and a parity bit.

const (
    SyncWord     = 0x7CD215D8
    SyncInverse  = 0x832DEA27
    IdleCodeword = 0x7A89C197
    codewordsPerBatch = 16
)

The decoder does several things multimon-ng does not, and each of them came from watching real noise:

• It corrects errors instead of giving up. The BCH(31,21) code (generator x^10 + x^9 + x^8 + x^6 + x^5 + x^3 + 1, i.e. 0x769) can correct up to two bit errors in each codeword. I precompute the syndrome-to-error-pattern table for every single and double bit error at startup, so correction is a table lookup. Two bad codewords in a row is not the end of a message, it is two codewords to repair.
• It confirms sync twice before believing it. A batch's pages are staged and only committed once the next 32 bits also line up as a sync word. The probability of two false syncs in a row is around six in ten quintillion, which is a number I am comfortable trusting instead of a single lucky match.
• It refuses to hallucinate. Two guards exist specifically to stop the error correction from doing damage. Anything within a few bits of the idle codeword is treated as padding before BCH can "correct" it into a phantom address. And a candidate address arriving in the middle of a message has to decode with zero errors to be believed, so a noisy message codeword cannot be error-corrected into a fake capcode and split one page into two.
• It understands gaps. The SDR layer reports exactly how many samples it dropped on a USB overflow. When that happens, the decoder finalises the part of the message the BCH already verified and re-hunts for sync, rather than decoding across the hole. Without this, roughly half of the garbage codewords a gap produces will pass BCH by chance, which is precisely how you get real pages with junk tails and phantom pages on random capcodes.

A content filter at the end makes the post-or-drop decision on the text itself, not on the flags (noise flips the flags): alphabetic messages, at least six characters, and if it is longer than a dozen characters it has to contain a space, because real dispatch text has word boundaries and BCH-passed noise does not.

The decoder that was too slow

Here is the honest part, because the first version of this Go decoder was worse than multimon-ng in one specific way, and it took me a while to see it.

The decode logic was right. The DSP was naive: it did the mixing and filtering one sample at a time, with the trig inside the loop. On my laptop it was fine. On a Raspberry Pi 4 it ran at 1.22 to 1.27 times realtime at full clock, which means it could not keep up with the radio. The SDR layer, correctly, drops IQ buffers rather than stalling the USB pipe when the consumer falls behind, so it was shedding samples constantly: about 16 to 18% of the stream gone, logged as an iq buffer overflow warning around 12,500 times a day. Every dropped buffer inside a burst shifted the codeword alignment, about half of those misaligned codewords passed BCH anyway, and that is where the garbled tails and the phantom pages were coming from. Not from multimon-ng. From us.

The fix was to rewrite the hot path to work a block at a time: the phasor mixer with its once-per-block renormalisation, the FIR computing only decimated outputs, the matched filter on the slicer. Same maths, different granularity. The result on the same Pi:

Fifteen times the headroom, and the drops went to zero. All of the "bad decoder" symptoms were a performance bug wearing a correctness bug's clothes.

The Rubish beacon

Which brings it back to that once-a-minute beacon, the one nobody could decode and so had named "Rubish" and switched off. It is the perfect test signal: known structure, known content, exactly one per minute, so "did we decode it intact" is a yes or no you can count.

Before: two clean decodes out of ten thousand transmissions in a week. After the block-DSP rewrite: 43 out of 43, every single minute. It has since held at 96 to 100% across both sites for weeks. A page that was literally labelled rubbish and ignored is now a free health check that emits 1,440 known-good decodes a day, and if it ever stops, I know a receiver is sick before any human one does.

What one binary bought us

The scoreboard, end to end. The backend went from PHP-FPM plus Horizon plus Redis plus Pusher plus Nova plus Telescope plus Pulse plus Meilisearch, to one Go binary and one web container. The feed got about a hundred and fifty times faster. Realtime went from a broken third-party websocket service to a Postgres NOTIFY. Duplicates went from two in five rows to none. The database went from zero backups to encrypted, verified, offsite nightlies. And the receiver went from a four-process audio pipe babysat in tmux to a single static binary under systemd that survives reboots and a dongle yanked out of its socket, correcting bit errors the old pipe could only drop.

The theme, if there is one, is control. Every piece of the old system was something I was orchestrating rather than owning: a queue worker's retry policy, a websocket vendor's DNS, a general-purpose decoder's decision to give up after two bad codewords. Replacing them with code I can read, in one process I can profile, is not about Go being fast or Laravel being slow. It is that when a volunteer is relying on a page arriving, I would rather the reason it did or did not arrive be a line I can point at.

The decoder, the receiver and the server are open source, under the AGPL, if you would like to read the parts I only described here. And the beacon is still transmitting, once a minute, forever. These days we hear all of it.