The Gophercon EU Berlin 2025 talks are now available on youtube, which means it is time for me to rewatch them and ponder again about the things I learned, the things that made me laugh, and the things that reignited my love for Go.

This was my second in-person Gophercon, the last one being Gophercon EU 2022, also in Berlin, and I think I enjoyed this one more. I get the feeling that the organizers get better at this every year, with small touches that increase inclusivity and make a more diverse group of people feel welcome. Not to mention some cute ways to spread the word of the Gopher.

At this one, there was a little booth where you could assemble a little gopher out of stickers, put it all on a postcard, write down a message and address, and they would mail the postcards anywhere in the world.

The quality of the talks themselves always vary based on the speakers but this year was special for me. You see, I got to see Dave Cheney in person and got to tell him in a slightly high-pitched fanboy voice about how much his writings inspired me to pick up Go. I did have to then look Matt Ryer in the eye and tell him (he was standing right next to Dave) that I enjoyed his writing as well, just not as much as Dave’s. Matt, being the gracious human being that he is, told me that he felt the same way. I think he was the second-biggest fan of Dave’s at that conference.

Levity aside, there were some quality talks given by funny and passionate people and some sad reminders of what is currently wrong with the world (a speaker unable to attend due to the current geopolitical situation).

I won’t go through all the talks, you can do so yourself at GopherCon Europe: Berlin 2025. I will go through a few specific ones that I enjoyed more than the rest.

Talks

The titles are also links to the youtube videos.

The Things I Find Myself Repeating About Go by Dave Cheney

I’ve always been a big fan of Uncle Bob’s Clean Code school of programming. He used Java in his examples but the ideas he put forward are applicable to any programming langauge.

Although I quite enjoy deep technical talks (and there were plenty of those), I love it when someone with long experience in the field talks about the basics of writing good code in a way that spans languages and technology stacks. Especially when they do it with as much of a mix of gravitas and humor as Dave brings to the stage.

That was what Dave’s talk was all about: how to write easy-to-read, maintainable code. I didn’t agree with all that he said (is it a sign of maturity, being able to disagree with your heroes?) but I appreciated all of it.

Some takeaways:

• Good naming is worth more than a sea of comments (though comments are never a bad idea). Sometimes even the lack of a full name can be good naming based on the context.
• Reduce conditionals to reduce the load on our fleshy thinking blobs.
• Keep to the left. Left-aligned code reads faster and easier. If you can get rid of an else statement by a little code-rearrangement, it’s often (not always) worth doing it.
• main.run! A pattern I had already picked up from Matt Ryer’s blog post. If you can make your entry-point accept environment-agnostic parameters, you can ship your code to various environments much more easily, not to mention making it a lot more testable. Stdout becomes an io.Writer, os.Args becomes a []string, context.Background becomes just another context.Context and probably more things you can think of.
• Make the default case the true case. If your users don’t want to give you their data, call the conditional variable DenyTelemetry instead of having a AllowTelemetry and then setting it to false. As Dave says “Bad things happen when you accidentally ingest data for a user who has denied telemetry.”, and this applies to all kinds of scenarios.
• Helpers, write lots and lots of helpers. Code in a helper is named and it is testable.

There were many more gems in his talk and I won’t go over them all or in much detail because I want you to go to youtube and see his talk. It’s a pleasure I dare not deny you.

Am I fanboying too hard right now?

Panel with the Go Team by Damien Neil, Jonathan Amsterdam, Michael Pratt, Michael Stapelberg

A classic Q&A session with some of the Go Team, featuring Michael Stapleberg (Protobuf), Damian Neil (Security), Michael Pratt (Runtime/Performance), and Jonathan Amsterdam (Static Analysis/MCP SDK).

Questions were submitted beforehand, and again, I won’t go into all of them, but just some I find interesting. The bold part of each sentence is also a timestamped link to that part of the video.

• Support immutable datatypes: the team (and especially Michael Pratt) agree that it would be a wonderful feature to have but they don’t know how they would do it. I don’t know about you but I’ve often held up people like the Go Team on a sort of intellectual pedestal and it’s nice to see that they too are just people trying to solve challenging problems and sometimes the answer is: “We’re not sure how to do this so we’re leaving the issue be for now.”

• Data Science/ML Libraries: the team sees Go’s role in all this quite similar to my own: Go is a language that helps productionize what’s already built in, say, Python. They are committed to implementing important AI specs (such as MCP SDKs) but apart from that they don’t see great value in directly involving Go in ML training.

• Bare Metal Go: not something they’re actively considering, it seems, though they are interested in a new proposal submitted by people from the Tamago project.

• Most Useless Feature of Go (lang and stdlib): This mostly centered around things they would remove from the standard library, mainly libraries that are badly designed or badly designed APIs. Can’t say I haven’t had thoughts like that about my own code every few weeks.

• Error Handling 😬: One of the biggest points of contention within the Go ecosystem, the Go team’s answer was (and I’m paraphrasing): “we don’t know how to make it better in a way that makes the majority happy, we’ve given up for now and will focus on more impactful matters”. They’ve blogged about this recently as well. Can’t blame them, to be honest. I’m sure a huge number of people find Go’s error handling to be a PITA. I’m not one of them. For me, it’s a mild annoyance. There are also many for whom it doesn’t matter and may even be preferable. And to change something so fundamental would take a concerted effort from the Go team. And without a clear majority benefit I can see why they would rather spend that effort somewhere else.

• Non-nil Function Arguments: I’ve seen nil-pointer dereferncing called The Billion Dollar Mistake. Maybe I’ve lived for too little a time to appreciate this, and perhaps my elders have horror stories that would curl my toes, but I’ve never really had a problem with nil (null) pointers. I can count on the fingers of my hands the number of times in my 10+ year career I’ve had a nil pointer reference error. Perhaps others’ mileage has varied. Claude tells me (you can do more thorough research of your own if you wish) that the total valuation of the tech industry in 2009 (when Tony Hoare gave that talk) was around 11-12 Trillion USD. Which means Tony Hoare’s billion-dollar mistake caused a 0.0000909091 % or so of damage. I don’t think it’s anything to lose sleep over. Write me and convince me I’m wrong if you’re so inclined. I would love a different perspective on this.

  The Go team’s take on this was similar to the immutability problem: it would be nice to have, but currently they don’t see an elegant way to solve it. By this I suppose they mean the guts of the Go compiler. Syntax and DevX-wise I think Rust’s Option type is the GOAT.

  For some added hilarity, do sympathize with Christopher Null, especially if you ever wrote code that made the life of people like him more difficult than it needed to be.

• Performance of Garbage Collector: The hardware landscape keeps changing. One day you’re running on a dual-core machine, and then suddenly you’re on a 256-core one. I can imagine scaling garbage collection in such an environment is a unique challenge. Michael Knyszek is the person making sure this scaling happens and he has a proposal for something called the Green Tea 🍵 Garbage Collector.

There was a lot more they talked about. This was just a little taste of a few things I personally found interesting. Do check out the full talk.

Refactoring Go in Large Codebases by Brittany Ellich

Brittany started off by reminding us all that as software engineers there are certain ways of thinking we relegate ourselves to. In the context of updating existing software we look at our old messy piece of code and think to ourselves: “I can do better if I rewrite this.” It’s the kind of trap we all fall into as succinctly summarized in Brittany’s talk (each line links to the part of the talk where she talks about it):

• It will be faster to just rewrite it
• We will move faster with clean code
• The current technology stack is holding us back

Watch the full talk to learn what advice she gives on how to go about refactoring (instead of rewriting) software.

Go Protobuf by Michael Stapelberg

Michael talked about the what, why and how’s of Protobuf, especially as it relates to Go and then moves on to talk about the Opaque API.

The Opaque API has been around since the end of 2024 but isn’t something I have given much thought until this talk reminded me that I really should be using getters instead of direct field access. Call it my failing at avoiding Hyrum’s Law. It would have made much of my code cleaner.

Fortunately for me, smarter people than I (such as Michael) are about to force my hand. At some point in the future, field access will be replaced by a xxx_hidden_ prefix (wonder who came up with that one 🤔) and I will have to update all my code to use accessor methods.

Luckily for me, the smart people are also considerate people, and they’ve created a tool (open2opaque) that rewrites your existing code to conform to the Opaque API.

The Opaque API, apart from providing a cleaner API to work with and promoting safer coding practices, also allow for memory optimizations which provide significant speedups by reducing allocation counts. I encourage you to both read the blog post and watch the talk for yourself.

Michael not only mentions the open2opaque tool but also talks about how it was written.

Michael also mentions protoscope: Cool little tool (and representational format) for inspecting the protobuf wire format.

Faster Go Maps With Swiss Tables by Michael Pratt

Fascinating deep talk on how Go maps work in general, the challenges involved in an operation such as increasing a map’s size, and how Swiss Tables come in and make everyone’s lives faster.

I won’t talk about it much, myself, I cannot keep all the details in my head no matter how many times I go over them, but I find them fascinating, nonetheless and I think you should watch the talk.

For the Love of Go: How to Contribute Really by Arati Rana

The beauty of this talk lies in the humility and courage of its presenter. Arati Rana took her first steps into contributing to Go, struggled once and again, and kept at it. She shares the story of adding T.Output to go, a process that was filled with a lot of twisting paths.

Y: Recursion The Hard Way by Eleanor McHugh

I left this one for last. Eleanor McHugh is probably the funniest person I’ve come across in a while who doesn’t do comedy for a living. Her talk softened my brain, then took the little melty bits and rearranged them in interesting ways.

I think I’ll need to watch it at least 15 more times to make any sense of it but that’s okay because watching her give the talk is one of the most pleasant experiences in my life, so far.

I won’t try to explain the talk. My fingers might start melting as well. In short, it is about doing recursion without a function calling itself.

Wait, that doesn’t sound right. Or does it? 🤔

Other Things

Between sessions, beautiful artworks made by talented gophers were displayed on the screen.

Lunch was served up by Fraulein Kimchi and Papa Africa Kitchen, simple hearty meals that gets one through a day of talks.

I had the fufu with peanut sauce from Papa Africa.

Breakfast (and snackies thoughout the day) was served by I don’t know who but I was a big fan of these long round pastries filled with spinach. I wish I had taken a picture. I think I ate more of those than actual lunch or the much more healthier option of fruit. There was a lot of fruit also on offer.

I stayed in a housing complex of sorts with lots of small rooms arranged around a central courtyard. You have to pass through multiple doors to get to where you need to go, so I slept peacefully knowing any would-be murderers were probably guests staying there. I would rather be killed by hotel cutlery than a shank from a street hobo.

The room I was in (and most rooms there, I can imagine) was functional. That’s the only way I can really describe it. It wasn’t sparse, it wasn’t messy or dirty, it just felt extremely functional. The furniture was old and well-used, there was only as much furniture as one could need, everything was quite minimal. It felt…minimal. I am at a loss for words to describe it because I still have mixed feelings about it.

On the one hand it does not meet my expectations of even a basic hotel room. It lacked pizazz! At the same time it was cleaner and more well-maintained than almost any hotel I’ve been to. It confounds my senses even as I think back on it. And I like it. And I would go back 11/10.

On my way back to the airport, I came across something…disturbing.

At the airport, the German version of the TSA kept my Oathbringer keychain. This was the last picture I managed to keep of it.

Before I ordered myself some RTSP compatible cameras, I needed a quick way to get an RTP stream to test out the general idea. The only cameras I have on hand are my iPhone, my MacBook and an old IP camera from which I’ve so far failed to extract an RTP stream.

I decided to see what I could do with my MacBook camera and after some trial and error landed on using mediamtx.

From it’s Github README:

MediaMTX is a ready-to-use and zero-dependency real-time media server and media proxy that allows to publish, read, proxy, record and playback video and audio streams. It has been conceived as a “media router” that routes media streams from one end to the other.

I used mediamtx in conjunction with good old ffmpeg to get the RTP stream from the Mac camera. ffmpeg takes the camera input, does some encoding voodoo and then sends it to mediamtx which serves it as an RTP stream.

This is the script I ended up with after a lot of help from Claude (and with added comments for clarity):

#!/bin/bash

# Exit immediately if any command fails
set -e

# RTSP Stream Manager
# Get camera device from first argument, default to device 0 if not provided
DEVICE=${1:-0}
# Variables to store process IDs for cleanup
MEDIAMTX_PID=""
FFMPEG_PID=""

# Cleanup function to gracefully stop both services
cleanup() {
    echo "Stopping services..."
    # Kill FFmpeg process if it's running
    if [ ! -z "$FFMPEG_PID" ]; then
        kill -9 $FFMPEG_PID 2>/dev/null
        echo "FFmpeg stopped"
    fi
    # Kill MediaMTX process if it's running
    if [ ! -z "$MEDIAMTX_PID" ]; then
        kill -9 $MEDIAMTX_PID 2>/dev/null
        echo "MediaMTX stopped"
    fi
    exit 0
}

# Set up signal traps to run cleanup on script interruption or exit
trap cleanup SIGINT SIGTERM EXIT

# Start MediaMTX streaming server in background
echo "Starting MediaMTX server..."
mediamtx mediamtx.yml &
# Store the process ID for cleanup
MEDIAMTX_PID=$!

# Give MediaMTX time to initialize before starting FFmpeg
sleep 2

# Display connection information to user
echo "Starting camera stream from device $DEVICE..."
echo "View RTSP: rtsp://localhost:8554/live"
echo "View Web: http://localhost:8889/live"
echo "Press Ctrl+C to stop both services"

# Start FFmpeg to capture camera and stream to MediaMTX
ffmpeg -f avfoundation \           # Use macOS AVFoundation for camera access
       -video_size 1280x720 \      # Set capture resolution
       -framerate 30 \             # Set capture frame rate
       -i "$DEVICE" \              # Input device (camera number)
       -c:v libx264 \              # Use x264 video codec
       -preset ultrafast \         # Use fastest encoding preset (lower CPU usage)
       -tune zerolatency \         # Optimize for low latency streaming
       -pix_fmt yuv420p \          # Set pixel format for compatibility
       -f flv \                    # Output format (FLV for RTMP)
       "rtmp://localhost:1935/live" &  # Send to MediaMTX via RTMP
# Store FFmpeg process ID for cleanup
FFMPEG_PID=$!

# Wait for FFmpeg process to complete or be terminated
wait $FFMPEG_PID

Note: I also learned about the trap command in bash which allows you to run a cleanup function when the script is interrupted or exits. I read further on it at The Bash Trap Command on Linux Journal by Mitch Frazier.

mediamtx.yml is an extremely barebones file:

# Minimal MediaMTX config
paths:
  all: {}

I needed to have something, for without it mediamtx would complain about missing authentication configuration. This file doesn’t have any authentication configuration, so I’m guessing mediamtx sees it and is like: “I have an explicit config file, but it doesn’t have any authentication, so I won’t require any”. Good enough, I suppose.

On running, the script gives us both an RTSP stream and something we can see on a web page. I am not quite sure what that’s doing. Though by looking at the network requests, I see there’s a Content-Type application/sdp response, which suggests to me WebRTC since an SDP (Session Description Protocol) is typically used in WebRTC. Pretty neat!

I can view the RTSP stream in VLC by going to File -> Open Network and entering rtsp://localhost:8554/live.

Hope you had fun reading!

Here’s a recent problem I faced. I had several enums in protobuf files that I wanted to save as database columns. But instead of their integer representation, I wanted to store their string representations.

Let’s say this is the enum I wanted to store.

package userama;

enum UserType {
	UNSPECIFIED = 0;
	SUPER = 1;
	COMMON = 2;
}

The generated Golang code for our enum looks like this.

type UserType int32  
  
const (  
    UserType_UNSPECIFIED UserType = 0  
    UserType_SUPER       UserType = 1  
    UserType_COMMON      UserType = 2  
)  
  
// Enum value maps for UserType.  
var (  
    UserType_name = map[int32]string{  
       0: "UNSPECIFIED",  
       1: "SUPER",  
       2: "COMMON",  
    }  
    UserType_value = map[string]int32{  
       "UNSPECIFIED": 0,  
       "SUPER":       1,  
       "COMMON":      2,  
    }  
)

...more stuff

Note the two exported maps, <EnumName>_name and <EnumName>_value. These allow us to convert between the string and int32 representations of our enum values.

Let’s say this is the struct representing the database row.

type User struct {
	ID    string           `pg:"id"`
	Name  string           `pg:"name"`
	Email string           `pg:"email"`
	Type  userama.UserType `pg:"type"`
}

Since the underlying type of the generated enum code in Golang are int32s, storing this struct naively in the database (using a library that uses the corresponding struct tags to map the fields to column names) would result in the type column storing an integer value.

For a value:

user := User{
	Id: "dwedwedwdwd",
	Name: "Shaunion",
	Email: "Shaunion@Onion.on",
	Type: userama.UserType_SUPER
}

We get the table:

But those aren’t very human readable if, for some reason, we need to inspect your database columns directly. We want to store the string representation of the enum in the database and then cast it to the typed int32 when reading it back.

How do we do this?

SQL Scanner + Valuer Interfaces

We can implement the Scanner and Valuer interfaces, using a wrapper that takes advantages of the exported maps.

type UserTypeWrapper struct {
	userama.UserType
}

// Value implements the SQL Valuer interface, allowing us
// to map a Golang value into a postgres column.
func (u *UserTypeWrapper) Value() (sql.Value, error) {

	// Since the underlying type is just an int32, you could technically
	// assign any value, whether predefined or not, to the field. It's
	// good to have a check.
	if stringVal, present := userama.UserType_name[u.UserType]; present {
		return stringVal, nil
	}

	return nil, fmt.Errorf("%d is not a valid value", u.UserType)
}

// Scan implements the SQL Scanner interface, allowing us to
// map a value from a postgres column to a variable.
func (u *UserTypeWrapper) Scan(src any) error {

	asString, cast := src.(string)
	if !cast {
		return fmt.Errorf("%v is not a valid string", src)
	}

	int32Val, present := userama.UserType_value[asString]
	if !present {
		return fmt.Errorf("%s is not a valid value", asString)
	}

	u.UserType = int32Val
	return nil
}

Note: We could have also used u.UserType.String() to get the string version of the enum, the only problem with that being the lack of any validation. If we were to assign the value 56 to u.UserType, String() wouldn’t complain. It would fall back to printing 56, which is a very confusing error to try to debug.

The only problem with this approach is that we will have to do this for every single enum type. We will have to only do this once for each new enum type you introduce, but introducing a new enum type now comes with extra work.

We can do better by creating a protobuf plugin to handle this for us.

Parts of a Protobuf Plugin

When generating go code using the protoc tool like so.

protoc --go_out=. --go_opt=paths=source_relative

We’re making use of at least two separate binaries (that I am aware of): protoc and protoc-gen-go.

Protoc

protoc is the protobuf compiler. It is installed on my machine via homebrew (there are also other ways to install it).

➜  ~ which protoc
/opt/homebrew/bin/protoc

It is the entry point for protobuf compilation (transpilation?). It parses and validates incoming protobuf files, resolving any dependencies along the way, ensuring type safety and any other goodness for which we prefer strict data schemas.

It’s output is not Golang code, but something called a Code Generation Request (itself represented in protobuf wire format). This is then piped to the Stdin of whatever code generator plugin we want to use to generate our final output.

That’s what the --go_out and --go_opt flags do. They indicate to the protoc binary that we want to generate Golang code and store it at a certain path (and with some options).

As you might have guessed, protoc looks for a binary in our path named protoc-gen-go or more generally for a given flag --<name>_out it will look for a binary protoc-gen-<name>.

Protoc Go Plugin

protoc-gen-go is the official plugin for generating Golang code from a protobuf file. It takes as its input a Code Generation Request via Stdin and then writes out the corresponding Golang code to its Stdout. This is then read back by the protoc compiler and stored in the correct paths.

If we want to pass options to the plugin we can do so via the --go_opt flag.

Building our own Plugin

Let’s build a plugin that augments the Golang code generated by protoc-gen-go and adds the methods that would make it satisfy the Scanner and Valuer interfaces.

We start off by defining the basic structure.

import (
	...
	"google.golang.org/protobuf/compiler/protogen"
	...
)

func main() { 
	if err := run(context.Background(), os.Args[1:], os.Stdout); err != nil { 
		os.Exit(1) // Only exit point in the program.
	} 
} 

func run(ctx context.Context, args []string, stdout io.Writer) error { 
	// Handle version command.
	if len(args) > 0 && args[0] == "version" { 
		fmt.Fprintln(stdout, version) 
		return nil 
	} 
	// Main plugin logic.
	protogen.Options{}.Run(func(gen *protogen.Plugin) error { 
		// Process proto files here.
		return nil 
	}) 
	
	return nil 
}

Handling the version command isn’t strictly necessary but is a nice quality of life addition to our plugin.

After some basic boiler plate our plugin code will start executing by invoking protogen.Optipns{}.Run. It takes a callback which provides us with the parsed (and validated) protobuf file. gen.Files is where each of the files that were fed to the compiler are available.

Since we are targeting enums, we need to loop over the input files and gather up the enums.

	...
	protogen.Options{}.Run(func(gen *protogen.Plugin) error { 
	
		for _, f := range gen.Files { 
			if !f.Generate { 
				continue // Skip files not marked for generation, this includes Google's well-known types as well as any imports or dependencies which the compiler would have pulled in for type-checking but which we aren't inrerested in.
			} 
			
			// TODO: process the file.
		}

		return nil 
	}) 
	...

Protobuf enums can appear in two places: top-level enums, and nested enums.

package userama;

enum UserType {
	UNSPECIFIED = 0;
	SUPER = 1;
	COMMON = 2;
}

message User {
	enum Type {
		UNSPECIFIED = 0;
		SUPER = 1;
		COMMON = 2;
	}
}

Both are valid ways of defining enums and which one you pick would depend on your usecase. If an enum is hyper-specific to a message type, you might define it as part of that message type itself. In this case you would refer to it as User.Type instead of UserType. For enums that might be shared by many messages it makes more sense to create a package level (or top level) enum.

We need to account for both cases in our code.

// TemplateData holds the data structure to be used by Golang's text/template package.
type TemplateData struct {
    PackageName string  
    Enums       []EnumData  
    Version     string  
}  
  
type EnumData struct {  
    Name        string
}

Next we collect our enums.

...

for _, f := range gen.Files { 

	data := TemplateData{  
	    PackageName: string(file.GoPackageName),  
	    Enums:       make([]EnumData, 0),  
	    Version:     version // It's a good idea to include the version of your tool that generated this code.
	}

	if !f.Generate { 
		continue
	} 
	
	// Top-level enums.
	for _, enum := range f.Enums { 
		data.Enums = append(data.Enums, EnumData{ 
			Name: enum.GoIdent.GoName, // Go type name.
		}) 
	} 
	
	// Nested enums from messages.
	for _, message := range f.Messages { 
		data.Enums = append(data.Enums, collectNestedEnums(message)...) 
	}
}
...

Since messages can contain other messages, we need to recursively check them and collect enums.

func collectNestedEnums(message *protogen.Message) []EnumData { 
	var enums []EnumData // Direct enums in this message.
	
	for _, enum := range message.Enums { 
		enums = append(enums, EnumData{Name: enum.GoIdent.GoName}) 
	} 
	
	// Recurse into nested messages.
	for _, nestedMessage := range message.Messages { 
		enums = append(enums, collectNestedEnums(nestedMessage)...) 
	} 
	
	return enums
}

Once collected, we will use text/template to generate our enum code. This is the template we will use.

var enumScannerTemplate = template.Must(template.New("enum_scanner").Parse(`
// Code generated by protoc-gen-enum-wrappers {{.Version}}. DO NOT EDIT.
package {{.PackageName}}

import (
    "database/sql/driver"
)

{{range .Enums}}
// Value implements driver.Valuer for database storage.
func (e {{.Name}}) Value() (driver.Value, error) {
    if stringVal, present := {{.Name}}_name[e]; present {
		return stringVal, nil
	}

	return nil, fmt.Errorf("%d is not a valid value", e)
}

// Scan implements sql.Scanner for database retrieval.
func (e *{{.Name}}) Scan(src any) error {

	// No point in trying to scan a nil value.
	if src == nil {
		return nil
	}

    asString, cast := src.(string)
	if !cast {
		return fmt.Errorf("%v is not a valid string", src)
	}

	int32Val, present := {{.Name}}_value[asString]
	if !present {
		return fmt.Errorf("%s is not a valid value", asString)
	}

	*e = int32Val
	return nil
}
{{end}}
`))

{{range .Enums}} helps us loop over our collected enums, and then it’s a matter of inserting the enum names in the right places.

The final part consists of executing our template and generating a file to store the results.

filename := file.GeneratedFilenamePrefix + "_enum_scanners.pb.go" 
g := gen.NewGeneratedFile(filename, file.GoImportPath)

The GeneratedFilenamePrefix comes from the original .proto filename. For user.proto, this creates user_enum_scanners.pb.go. The .pb.go suffix follows protobuf conventions.

var buf bytes.Buffer 
if err := enumScannerTemplate.Execute(&buf, data); err != nil { 
	...
} 

g.P(buf.String())

g.P() writes lines to the generated file.

And with that we’ve written our plugin to implement valuer/scanner on enums. The google.golang.org/protobuf/compiler/protogen package takes care of the rest, converting our output to a CodeGeneratorResponse, that is then read back by protoc.

Going back to our example:

When we first found out we were having our baby boy, I was overjoyed. The terrifying reality of parenthood did not set in until I held my child in my arms for the first time.

Those first few days were the hardest. The first few weeks were all but a blur, and the first few months had us sleep deprived and somewhat cranky. All normal things, I am told by many a parent.

It was not until recently that I have become comfortable enough looking after my child and family that I am able to let my brain wander into its more hobbyish moods. And when it did, it decided to start solving some real world problems, though I leave it up to you to decide if the returns are worth the trouble.

We have been using one of the many baby tracking apps on the market to keep track of when we feed our baby, when he sleeps, how often he poops, and other beautiful bodily functions. It’s a simple app in its engineering, but exquisite in its interface and UX.

It maintains a “baby” as an entity for which it records various activities: food, bottle, sleep, etc. Some activities like “bottle” or “diaper” which are points in time have a time field for when they occurred. Activities like “sleep” have a start and end time. Activities can also have groups (such as “fruit” or “vegetables” for the “food” activity), and all activities can have a “notes” field. I and my partner’s user accounts are added to the baby as “caregivers.” When we update an activity on our phones, it syncs to both devices with detaials on who added the record.

It’s a system that has worked well for us and one we’ve used for months now. Recently, I wanted it to work even better (or I got lazier?).

We have Google nest devices scattered around the house. We use them to play music, play white noise for when baby sleeps, control smart lights, learn about the weather, all the usual smart home stuff. So I asked myself: wouldn’t it be nice if it could answer questions like “when did baby last have a bottle?”, “how long was baby’s last nap?”, “how much did baby drink last week?”.

And thus started my journey.

Syncamajiggy

Since the baby tracking app did not have an API I could access, step one was to figure out how to sync that data to a server I controlled. I don’t have any mobile dev experience, but with Claude’s help I figured out how to get a rooted Android emulator running on my Mac.

➜  ~ sdkmanager "system-images;android-30;google_apis;arm64-v8a"

I had downloaded an APK of the tracker app from https://www.apkmirror.com/ and based on its metadata, Claude informed me this would be a good system image. The command above downloads it to my system.

➜  ~ avdmanager create avd -n rooted_emulator -k "system-images;android-30;google_apis;arm64-v8a" -d pixel_4

This command creates an Android Virtual Device(AVD) based on a Pixel 4 device definition and the system image I had downloaded earlier. I named it rooted_emulator.

Finally…

➜  ~ emulator -avd rooted_emulator -writable-system -no-snapshot -selinux permissive

This command launches the virtual device in an emulator making the system partition writable (-writable-system, we’ll soon see why that’s needed), foregoes saving any state (-no-snapshort) and does not block security policy violations (-selinux permissive).

When it came to installing the APK into the emulator, I had my first surprise. My last brush with anything Android was back in 2011 when life was much simpler. You got a slightly shady APK from somewhere and installed it on your rooted device and enjoyed some fleeting pleasures not readily available on the Play Store.

These days APKs can come in multiple parts! Apparently Google Play can use Android App Bundles to bundle together parts specific to the device its being downloaded to. The split nature of the APK could account for resources for varying screen sizes and densities, languages, and even dynamic features! Very cool stuff.

I installed the APK using…

➜  ~ adb install-multiple base.apk split_config.*.apk

Once installed I could launch the app, log into my account and view my data. Now it’s time to figure out how this app is talking to the backend.

I used the free version of HTTP Toolkit a brilliant piece of software that automates the process of intercepting HTTP calls.

It supports many applications and for Android in particular it has a plethora of options.

For me, it was as easy as choosing Android Device via ADB (which becomes ungrayed when it detects an active device via adb). The program requires a rooted device with a writable system. This is because in order to successfully decrypt HTTPS traffic it needs to inject an SSL certificate as a system level certificate.

Let me explain.

Android phones (devices?) support configuring a proxy server. While that allows us to intercept the traffic from the apps, it’s not much use since pretty much all traffic these days is encrypted via SSL certificates.

HTTP Toolkit solves this by injecting its own certificate into the system certificate store of the device. It has to be on the system level since most apps don’t support the user certificate store. And for this we need a rooted, writable system.

A rooted emulator with a writable system partition solves this. So with that…

… we have inception interception!

It took me a while to figure out which requests were from the baby tracker app. HTTP Toolkit intercepts all traffic from the device and I don’t know if there’s a way to filter down to a single app.

With trial and error I found the requests relevant to my quest. After some false starts I discovered the tracker app was talking to a Firebase Cloud Storage system. The crucial request is https://firestore.googleapis.com/google.firestore.v1.Firestore/Listen which subscribes to updates from collections (which are analogous to SQL tables). This was what kept the app in sync with event updates and what I had to replicate on the server side.

Using the cloud.google.com/go/firestore Golang module, and a generous amount of guidance from Claude, I was able to build a simple backend server that keeps listening to the specific collection that records baby’s daily events and syncs them to a locally hosted postgres server.

The next step was to hook this up to a voice assistant.

Dialog what?

Trying to figure out how do this with a Google Nest device was frustrating. Their DialogFlow thing seems to be deprecated (or not well supported?), their Conversational Agents thing does not seem to work with Nest devices, and navigating their myriad of products and technical jargon was giving me a headache.

It’s a problem I bashed my skull against for a while. I even considered somehow sending the commands from the Nest to my HomeAssistant instance and using HA’s voice tech. But that just felt convoluted. I kept having this feeling that this shouldn’t be this hard.

And then I remembered why I was having this feeling. I had once built a very simple Skill for an old Amazon Echo device. Although I remembered nothing about the process, I remembered it being relatively painless. I also remembered that the Amazon Echo ecosystem supports a Skills marketplace, something the Google Home ecosystem doesn’t seem to. At least not without pain.

It led me to unearthing my old Amazon Echo (1st or 2nd gen, I believe). Building a basic skill that sent a request to my server and replied with a nicely formatted string took less than 2 hours. The ease of use compared to Google’s nightmare was so stark that I nearly wept.

After that, there was no looking back.

Be Intentional

Alexa skills are built around the concept of “intents” - essentially things users want to accomplish. When we create a skill, we define these intents and provide sample utterances that might trigger them. The model reminded me of pattern matching in a language like Rust or Elixir.

The lifecycle of an Alexa request looks something like this:

1. User says: “Alexa, ask hey baby when was baby’s last bottle?”.
2. Alexa identifies “hey baby” as my skill’s invocation name.
3. It matches “when was baby’s last bottle” to my LastActionIntent with the slot actionType set to “bottle” (more on Slots below).
4. My backend receives this intent and slot data as a JSON payload.
5. My code queries the database and returns a response.
6. Alexa speaks the response to the user.

Since I am not planning on publishing this Skill to the Alexa marketplace (or whatever it’s called, I can’t say: “Alexa, when was baby’s last bottle?”. That’s something that can be achieved with [Name-free interactions])(https://developer.amazon.com/en-US/docs/alexa/custom-skills/understand-name-free-interaction-for-custom-skills.html), which are only available for published skills. Name-free interactions trigger Alexa to look for the skill that best matches the utterance and use that to process the request.

Looking at the JSON I’ve defined in my interaction model (which is just a fancy way of saying “the configuration of my skill”), you can see I’ve focused on a few key use cases.

For example, one of the most common questions we have is about the last time our baby had a bottle:

{
    "name": "LastActionIntent",
    "slots": [
        {
            "name": "actionType",
            "type": "actionType"
        }
    ],
    "samples": [
        "last {actionType}",
        "when did baby last have {actionType}",
        "when was baby's last {actionType}",
        "when was the last {actionType}"
    ]
}

This intent captures various ways we might ask about the last occurrence of an action. The actionType slot can be filled with values like “bottle”, “diaper”, “nap”, etc. I’ve defined these in a custom slot type:

Slots in Alexa skills function like variables in programming - they’re placeholders for dynamic values that users provide when they speak. We can define our own slot types (like actionType above) with a specific set of valid values and synonyms, similar to enums in programming. Amazon also provides built-in slot types like AMAZON.DATE, AMAZON.NUMBER, and AMAZON.TIME that handle common data formats and natural language variations without us having to implement them ourselves.

{
    "name": "actionType",
    "values": [
        {
            "name": {
                "value": "bottle",
                "synonyms": [
                    "drink",
                    "formula"
                ]
            }
        }
    ]
}

Beyond just knowing when things happened, we also wanted to track quantities like how much formula was consumed:

{
    "name": "LastBottleVolumeIntent",
    "slots": [],
    "samples": [
        "how many milliliters was baby's last bottle",
        "what was the volume of baby's last bottle",
        "how much was baby's last bottle",
        "last bottle volume",
        "how big was the last bottle"
    ]
}

And we can get quite specific with stats over time periods:

{
    "name": "BottleStatsIntent",
    "slots": [
        {
            "name": "bottleMetric",
            "type": "bottleMetricType"
        },
        {
            "name": "datePeriod",
            "type": "AMAZON.DATE"
        }
    ],
    "samples": [
        "how {bottleMetric} did baby drink {datePeriod}",
        "how {bottleMetric} did baby have in bottles {datePeriod}",
        "how {bottleMetric} formula did baby have {datePeriod}",
        "how {bottleMetric} did baby drink",
        "how {bottleMetric} from bottles"
    ]
}

The bottleMetricType slot configuration.

{  
    "name": "bottleMetricType",  
    "values": [  
        {  
            "name": {  
                "value": "bottles",  
                "synonyms": [  
                    "many bottles",  
                    "number of bottles",  
                    "times"  
                ]  
            }  
        },  
        {  
            "name": {  
                "value": "ml",  
                "synonyms": [  
                    "milliliters",  
                    "volume",  
                    "much",  
                    "quantity"  
                ]  
            }  
        },  
        {  
            "name": {  
                "value": "total",  
                "synonyms": [  
                    "all"  
                ]  
            }  
        }  
    ]  
}

I found myself being quite thorough with the sample utterances. Unlike traditional UI design where you know exactly what controls are available to users, voice interfaces are much more open-ended. I don’t always know how I am going to phrase my question when I get around to asking.

On the backend side, handling these intents is straightforward. I built a small Go server that receives requests from Alexa, maps the intents to database queries, and returns formatted responses. The handler for the LastActionIntent looks something like this:

func handleLastActionIntent(ctx context.Context, request alexa.Request) (alexa.Response, error) {
	actionType, err := request.GetSlotValue("actionType")
	if err != nil {
		return nil, errors.Wrap(err, "failed to get action type")
	}

	// Query the database for the last entry of the given action type
	entry, err := db.GetLastEntry(ctx, actionType)
	if err != nil {
		return nil, errors.Wrap(err, "failed to get last entry")
	}

	// Format the response
	timeAgo := humanize.Time(entry.Timestamp)
	text := fmt.Sprintf("Baby's last %s was %s", actionType, timeAgo)
	
	return alexa.NewSimpleResponse(
		fmt.Sprintf("Last %s", actionType),
		text,
	), nil
}

When the request comes in, I extract the slot value, query my database, and format a human-friendly response like “Baby’s last bottle was 2 hours ago.” The Alexa SDK handles converting this text response to speech.

You might have noticed that slot values can have synonyms. In my code I map these to their canonical value before converting them into a query. This is made easy because for any slot synonym the Alexa request includes the canonical value.

Alexa’s built-in slot types impressed me. The AMAZON.DATE type, for example, handles natural language date expressions like “yesterday” or “last week” without me having to implement date parsing myself. There are similar built-ins for numbers, durations, and other common data types.

Testing the skill was also fairly painless. The Alexa Developer Console has a simulator that lets you type or speak test commands and see exactly how your skill would respond, including the full JSON of the request and response.

Trust but Verify

Since this backend exposes the data about my dear baby boy, security wasn’t optional. Thankfully, Amazon provides mechanisms to verify that requests are genuinely coming from Alexa and not someone particularly interested in how many diapers I’m going through.

Each request sent to the skill’s endpoint includes two crucial security elements:

1. A URL pointing to an Amazon certificate.
2. A cryptographic signature of the request body.

The verification process is more complex than I initially expected.

First, we need to validate the certificate URL itself (found in the SignatureCertChainUrl header). Amazon requires it to follow a specific format - it must be HTTPS, hosted on Amazon’s S3 servers, and follow a particular path structure. This prevents attackers from pointing to their own certificates.

// Simplified example of URL validation
if !strings.HasPrefix(parsedURL.Path, "/echo.api/") {
    return errors.New("certificate URL path does not start with /echo.api/")
}

Next comes timestamp validation. Each Alexa request includes a timestamp, and we need to verify it’s recent (within 150 seconds). This prevents replay attacks where someone could capture a valid request and resend it later.

The real cryptographic magic happens in the signature verification. We download Amazon’s certificate, verify it’s valid and specifically issued for Alexa services, and then use it to check the signature against the request body.

For performance reasons, we also cache the certificate. The certificate doesn’t change frequently, so there’s no need to download it for every request.

Finally, we verify that the skill ID in the request matches the specific skill’s ID, ensuring that only that skill can access the backend endpoint.

// Verify the application ID
if requestAppID != s.config.AlexaSkillID {
    return errors.Errorf("Expected skill ID: %s, got: %s", 
                        s.config.AlexaSkillID, requestAppID)
}

And with that…

OneOf types, aka Tagged Unions are data structures that are used to hold one of a finite list of distinct types. A variable of a tagged union type can hold a value of one of several types defined for that tagged union type. This might be easier to understand with an example via pseudocode.

type Media = Movie | Show | Short

var favorite1: Media = Movie{}
var favorite2: Media = Show{}
var favorite3: Media = Short{}

var favorites: [Media] = [favorite1, favorite2, favorite3]

Variables favorite1, favorite2, and favorite3 all have the data type Media but have values Movie, Show and Short, which works because the Media data type has been defined as a Tagged Union of all 3 other data types. Consequently, the favorites variable, which is defined as an array of Media values, can have values of the 3 defined data types in it.

Crucially, the values all take up the same space in memory and only one can be set at any given time. My knowledge here becomes fuzzy, but typically, the memory allocated for a OneOf corresponds to the largest possible field within it, with additional space for internal bookkeeping.

Protobuf has OneOf as its implementation of the Tagged Union data type. A message field can be marked as oneof with the variants specified.

message Event {
	oneof media {
		Movie movie = 1;
		Show show = 2;
		Short short = 3;
	}
	int64 timestamp = 4;
	optional bool had_fun = 5;
}

message Movie {
	string title = 1;
	string director = 2;
}

message Show {
	string title = 1;
	string runner = 2;
}

message Short {
	string title = 1;
	string shorty = 2; // I couldn't think of something unique, here.
}

Protobuf allows assigning only one of the 3 fields out of movie, show and short, and setting one automatically clears any of the others set, ensuring that memory for only one of the types gets consumed.

Protobuf’s descriptor.proto is a protobuf file that describes the structure of protobuf files. It hurts my head a little to think about it.

Let’s inspect the DescriptorProto message, which is the schema for a protobuf Message.

// Describes a message type.
message DescriptorProto {
  optional string name = 1;

  repeated FieldDescriptorProto field = 2;
  ...
  ...
  repeated OneofDescriptorProto oneof_decl = 8;
  ...
}

Let’s look at the relevant parts of FieldDescriptorProto and OneOfDescriptorProto and then we’ll talk about them.

// Describes a field within a message.
message FieldDescriptorProto {
	...
	// If set, gives the index of a oneof in the containing type's oneof_decl
	// list.  This field is a member of that oneof.
	optional int32 oneof_index = 9;
	...
	// If true, this is a proto3 "optional". When a proto3 field is optional, it
	// tracks presence regardless of field type.
	//
	// When proto3_optional is true, this field must belong to a oneof to signal
	// to old proto3 clients that presence is tracked for this field. This oneof
	// is known as a "synthetic" oneof, and this field must be its sole member
	// (each proto3 optional field gets its own synthetic oneof). Synthetic oneofs
	// exist in the descriptor only, and do not generate any API. Synthetic oneofs
	// must be ordered after all "real" oneofs.
	//
	// For message fields, proto3_optional doesn't create any semantic change,
	// since non-repeated message fields always track presence. However it still
	// indicates the semantic detail of whether the user wrote "optional" or not.
	// This can be useful for round-tripping the .proto file. For consistency we
	// give message fields a synthetic oneof also, even though it is not required
	// to track presence. This is especially important because the parser can't
	// tell if a field is a message or an enum, so it must always create a
	// synthetic oneof.
	//
	// Proto2 optional fields do not set this flag, because they already indicate
	// optional with `LABEL_OPTIONAL`.
	optional bool proto3_optional = 17;
}

// Describes a oneof.
message OneofDescriptorProto {
  optional string name = 1;
  optional OneofOptions options = 2;
}

I pasted the whole documentation for the proto3_optional field since it has heavy implications on how OneOfs are implemented. OneOfDescriptorProto is relatively simple and for us only the name field matters.

To fully grasp how to detect and make use of OneOf fields, we must understand both how OneOfs are represented and how optional fields are represented.

Remember that optional bool had_fun field? That wasn’t there just for flavor but to demonstrate how optional fields need to be taken into account when making use of OneOfs.

Looking at DescriptorProto we see a field repeated OneofDescriptorProto oneof_decl = 8;. This field is what lists all OneOf fields in the message. Using our very first example of message Event, oneof_decl would contain 1 value:

DescriptorProto{
	Name: "Event",
	OneofDecl: []OneofDescriptorProto{
		{
			Name: "media",
		},
		{
			Name: "_had_fun" // Keep an eye on this one. We'll get back to it.
		}
	},
}

Wait, what’s that _had_fun doing in there? We didn’t define that as an OneOf! Keep reading.

The fields of the message would have info on each field:

DescriptorProto{
	Name: "Event",
	Field: []FieldDescriptorProto{
		{
			Name: "movie",
			OneOfIndex: 0,
		},
		{
			Name: "show",
			OneOfIndex: 0,
		},
		{
			Name: "short",
			OneOfIndex: 0,
		},
		{
			Name: "timestamp",
			OneOfIndex: nil,
		},
		{
			Name: "had_fun",
			OneOfIndex: 1,
		},
	},
}

Of interest to is the value of the OneOfIndex property. The description of the property tells us how it works.

If set, gives the index of an OneOf in the containing type’s list. This field is a member of that OneOf.

That seems straightforward enough. In the Event message, we have two OneOfs listed (even though we did not list the second one, ourselves). And each of the fields tells us whether it is part of one of the OneOfs (indicating by index which OneOf it belongs to) or not (if the value is nil).

As expected, movie, show, and short have the OneOfIndex of 0, part of the OneOf located at the 0th index of the OneofDecl list, media.

But why is had_fun is also a part of an OneOf?

Turns out, because of some backwards compatibility shenanigans, optional fields are also represented as OneOfs, with the two options being (I suppose) either the concrete type or nothing.

So marking had_fun with the optional label causes the descriptor to generate a synthetic OneOf. A OneOf that’s not a real OneOf.

When parsing this schema for our own purposes we have to be aware of this synthetic OneOf. We detect real OneOfs by checking if the OneOfIndex is not nil and the Proto3Optional is false. If both conditions are met, we can be sure that the field is part of a real OneOf.

All this was discovered over a frustrating afternoon of reading through documentation and experimenting. It was annoying but also rewarding. Hope you enjoyed the read.

Their page looks like this.

Each day there are two episodes, one for the 13h news and another for the 19h news. Clicking through, we can see that each episode has chapters within it with timestamps and titles. It’s a list of these titles with links to that chapter (for quick viewing) that my girlfriend was interested in gathering, without having to manually click on each episode, each day, every day.

I started off by exploring the html source of the landing page to find content already present. There’s no need for complicated scraping if I can just grab the relevant html section, perhaps something between <ul> or <ol> tags.

No such luck. The initial document that loads is pretty barebones.

<!DOCTYPE html>
<html lang="nl">

	<head>
		<meta charset="UTF-8">
		<title>VRT MAX</title>
		<script>var s = document.createElement('script'); s.src = '/vrtnu-resources/loaders/scriptLoader.js'; s.async = false; document.head.appendChild(s);</script>
	</head>
	<body>
	
		<div id="root"></div><noscript>U moet JavaScript inschakelen om deze app te gebruiken.</noscript>
		<script>document.body.onload = () => loadBodyScripts();</script>
	</body>
</html>

I have elided much from the head but the body is as you see it: sparse. It looks as if all the actual content is loaded via Javascript. Not the best design, if you ask me. I’m not sure there is enough interactivity on a page like this to justify denying non-Javascript enabled users access to the page.

Okay, so we can’t do anything without running Javascript. So let’s do something to run Javascript.

I originally wanted to program this in Go since that is my daily driver. But after exploring libraries, I couldn’t find anything that beats puppeteer for the sheer number of features available, not to mention ease of use, wealth of documentation, and example projects.

I started off by inspecting the page after all the Javascript had run and found my content.

After playing around a bit I decided on this CSS selector: #main ul. Short and gets the job done. Time to extract the list.

async function extractEpisodeList(url) {

	try {
		// Launch the browser
		const browser = await puppeteer.launch();
		
		// Create a new page
		const page = await browser.newPage();
		
		// Navigate to the specified URL and wait for the network to be idle
		await page.goto(url, { waitUntil: 'networkidle0' });
		
		// Wait for any client-side rendering to complete
		await new Promise(resolve => setTimeout(resolve, 2000));
		
		// Get the fully rendered HTML content
		const content = await page.content();
		
		// Extract the list items and their links
		const listItems = await page.evaluate(() => {
		
			const list = document.querySelector('#main ul');
			
			if (!list) {
				return [];
			}
		
			return Array.from(list.querySelectorAll('li'))
				.map(li => {
			
					const a = li.querySelector('a');
	
					return a ? {				
						href: a.getAttribute('href'),
						content: a.getAttribute('aria-label') || a.textContent.trim()
					} : null;
		
				})
				.filter(item => item !== null);
		});
		
		// Close the browser
		await browser.close();
		return listItems;
		
	} catch (error) {
		throw error;
	}
}

The code should be pretty self-explanatory. I had to fiddle a bit to find the right query selectors to extract the link and title after I got the list. It’s important we use waitUntil when loading the page to ensure most network activity is complete. networkidle0 waits for at least 500 ms of no new network connections.

The end result is a list of episodes.

[
	{
		"href": "...",
		"content": "..."
	}
]

If you spend some time on that page you’ll notice the episode list is a sort of carousel within which new episodes get loaded as you scroll horizontally to the right. By default only 13 episodes or so are listed. But this is fine for us since we will check this page every day and we only care about new episodes.

Before we start grabbing the chapters for each episode, let’s talk about a very small optimization. We could scrape the entire list of episodes every day, but given that we have only 2 new episodes each day, we could be re-scraping 11 episodes we have already scraped, every single day. Let’s perform a very small optimization step.

We will maintain a list of episodes we have already processed, and each day process only new episodes. This is achieved by this function (which itself calls the function above to grab the episode list).

async function updateEpisodeList() {

	const url = 'https://www.vrt.be/vrtmax/a-z/vrt-nws-journaal/';
	
	try {
		const renderedContent = await extractEpisodeList(url);
		
		// Read the local file "episode-list.json"
		let localEpisodes;
		
		try {
		
			const rawData = await fs.readFile(episodeListFilePath, 'utf8');
			localEpisodes = JSON.parse(rawData);
		} catch (error) {
		
			console.error('Error reading or parsing local file:', error);
			return;
		}
		
		// Compare extracted episodes with local episodes
		const extractedSet = new Set(renderedContent.map(item => item.href));
		const localSet = new Set(localEpisodes.map(item => item.href));
		
		const newEpisodes = renderedContent.filter(item => !localSet.has(item.href));
		
		const removedEpisodes = localEpisodes.filter(item => !extractedSet.has(item.href));
		
		// Print the diff
		console.log('Diff between extracted and local episodes:');
		
		if (newEpisodes.length > 0) {
		
			console.log('\nNew episodes:');
			
			newEpisodes.forEach(episode => {
				console.log(`- ${episode.content} (${episode.href})`);
			});
		}
		
		if (removedEpisodes.length > 0) {
		
			console.log('\nRemoved episodes:');
			
			removedEpisodes.forEach(episode => {
				console.log(`- ${episode.content} (${episode.href})`);
			});
		}
		
		if (newEpisodes.length === 0 && removedEpisodes.length === 0) {
			console.log('No differences found.');
		}
		
		return {
			newEpisodes: newEpisodes,
			removedEpisodes: removedEpisodes,
			localEpisodes: localEpisodes
		};
	} catch (error) {
		throw error
	}
}

Once again, most of the code should be self-explanatory. We read our existing episodes list from a file, compare against the scraped list and generate lists of new episodes (and some extra stuff but those aren’t relevant).

Armed with new episodes to scrape, we need to figure how to extract the chapters of the episodes, mainly their title and the timestamp at which they start. And if we can generate a link that will take you to the exact part of the video, even better.

For this I started inspecting the episode page itself. As with the main page, the episode page is sparse and uses JS to load everything, but looking at the XHR calls, I noticed something.

Is that a Graphql endpoint mine eyes spy?! I definitely know how to use one of those.

This made the whole process much simpler. I inspected the Graphql response, found the part with the episode chapters and the timstamps and bundled them up in a message I could send to a Slack channel. I won’t show that code here as it’s fairly mundane.

Afterwards I had the thought that perhaps the main page also loaded its episode list via a Graphql call and I did not need to scrape anything. But I already had so no point thinking about it further.

At this point I had to decide where to run it. This wasn’t planned to be a live service, just a script that ran every day so something like Fly.io wasn’t necessary. I remembered a tech meetup I had attended earlier in the year at Theo Technologies and the talk on Zero Budget Solution Engineering Toolbox by Daniel Dallos. He had talked about using Github actions as a way to run jobs on a schedule, while at the same time storing small amounts of data in the Github repository as a JSON file. I highly recommend watching the talk as he mentions a lot of useful tricks and tools.

So I whipped up a Github a workflow yaml file.

name: Scrape  
  
on:  
  workflow_dispatch:  # this allows us to manually run this workflow
  schedule:  
    # Runs at 12:00 UTC every day  
    - cron: '0 12 * * *'  
  
jobs:  
  run-script:  
    runs-on: ubuntu-latest  
  
    steps:  
      # Step 1: Check out the repository  
      - name: Checkout repository  
        uses: actions/checkout@v3  
  
      # Step 2: Set up Node.js (adjust version as needed)  
      - name: Set up Node.js  
        uses: actions/setup-node@v3  
        with:  
          node-version: '20'  
  
      # Step 3: Cache npm dependencies  
      - name: Cache dependencies  
        uses: actions/cache@v3  
        with:  
          path: ~/.npm  
          key: ${{ runner.OS }}-node-${{ hashFiles('**/package-lock.json') }}  
          restore-keys: |  
            ${{ runner.OS }}-node-  
  
      # Step 4: Install dependencies (if you have a package.json)  
      - name: Install dependencies  
        run: npm ci  
  
      # Step 5: Run your JavaScript script  
      - name: Run the JavaScript script  
        run: node scrape.js  
  
      # Step 6: Commit and push changes back to the repository  
      - name: Commit and push changes  
        run: |  
          git config --global user.name 'github-actions[bot]'  
          git config --global user.email 'github-actions[bot]@users.noreply.github.com'  
          git add .  
          git diff --quiet && git diff --staged --quiet || (git commit -m "Update results from scheduled script" && git push)  
        env:  
          # Ensure you have write access to the repository via a GitHub token  
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

There is some stuff in there about loading NodeJS, cacheing dependencies and whatnot, but the meat of the code is in Steps 5 and 6 where we run our scraping and then commit the results back to the repository.

And with that I had a tool that would let me know every day about the episodes of the VRT NWS Journaal.

Hope you enjoyed the read.

Failed to compute set of methods to expose - Symbol not found

This cryptic error has reared its ugly head a few times in the past when working with grpc in Golang. I can’t remember how I’ve solved it before. Probably with some combination of stackoverflow answers and random github gists with advice that I have now forgotten. When it happened most recently, I decided to try to actually understand what was going on under the hood.

Note: For the purposes of this post I am using Golang v1.22.3, libprotoc 3.21.12, google.golang.org/grpc v1.63.2, and google.golang.org/protobuf v1.34.1.

I had gone a long time without encountering this error, only stumbling upon on it again while using the protovalidate library for validating protobuf messages. Used in an interceptor it provides a way to validate input messages on gRPCs.

According to the docs, I should use the validate.proto files by importing them into my project and referencing them in my primary proto files.

syntax = "proto3";

import "buf/validate/validate.proto";

message User {
  // User's name, must be at least 1 character long.
  string name = 1 [(buf.validate.field).string.min_len = 1];
}

I am not a fan of having to add files under arbitrary directory structures so I did not store my validate.proto (and expression.proto and some others that are dependencies of those two) under the buf directory. I stored them under the validate directory. This is what things looked like.

├── services.proto
└── validate
    ├── expression.proto
    ├── priv
    │   └── private.proto
    └── validate.proto

services.proto being my primary proto file of interest. It imported validate.proto like so:

import "validate/validate.proto";

My generation script does not generate any code from validate.proto, since we’re using the schema for its options only, that then get baked into the output for my services.proto file.

For performing the validations on these baked, generated options, I’m using github.com/bufbuild/protovalidate-go. Code example from its github repo:

package main

import (
	"fmt"
	"time"
	
	pb "github.com/path/to/generated/protos"
	"github.com/bufbuild/protovalidate-go"
	"google.golang.org/protobuf/types/known/timestamppb"
)

func main() {
	msg := &pb.Transaction{
		Id:           1234,
		Price:        "$5.67",
		PurchaseDate: timestamppb.New(time.Now()),
		DeliveryDate: timestamppb.New(time.Now().Add(time.Hour)),
	}

	v, err := protovalidate.New()
	if err != nil {
		fmt.Println("failed to initialize validator:", err)
	}

	if err = v.Validate(msg); err != nil {
		fmt.Println("validation failed:", err)
	} else {
		fmt.Println("validation succeeded")
	}
}

Compiling the protobuf into Golang works: the gRPC server registers the RPC server defined in the schema and my program runs. If I call any of the RPCs, it works fine.

It’s when I try to use the reflection server is when we run into trouble. I’ve built a sample project to demonstrate this over at https://github.com/BadgerBadgerBadgerBadger/protoreflect-error-test/tree/main/error-state.

Our proto folder looks like this:

.
├── dependencies
│   └── secondary
│       └── secondary.proto
├── gen.sh
└── primary.proto

Our gen.sh:

set -e;  
  
# Get the directory where the script is located  
scriptDir="$( cd -- "$( dirname -- "${BASH_SOURCE[0]:-$0}"; )" &> /dev/null && pwd 2> /dev/null; )";  
echo "script directory: $scriptDir";  

# ensure output directory exists
outDir="$scriptDir/lang_go";  
mkdir -p $outDir/primary;  

# build the secondary proto
protoc -I="$scriptDir/dependencies" \  
    --go_out="$outDir" --go_opt=paths=source_relative \  
    --go-grpc_out="$outDir" --go-grpc_opt=paths=source_relative \  
    secondary/secondary.proto  

# build the primary proto
protoc -I="$scriptDir" \  
    --go_out="$outDir/primary" --go_opt=paths=source_relative \  
    --go-grpc_out="$outDir/primary" --go-grpc_opt=paths=source_relative \  
    primary.proto

Our primary.proto and secondary.proto files:

syntax = "proto3";  
  
option go_package = "github.com/BadgerBadgerBadgerBadger/protoreflect-error-test/proto/lang_go/primary";  
package protoreflect.error.test.primary;  
  
import "dependencies/secondary/secondary.proto";  
  
service Main {  
  rpc GetPrimary (PrimaryRequest) returns (PrimaryResponse) {}  
}  
  
message PrimaryRequest {  
  secondary.SecondaryMessage primary_request = 1;  
}  
  
message PrimaryResponse {  
  secondary.SecondaryMessage primary_response = 1;  
}

syntax = "proto3";  
  
option go_package = "github.com/BadgerBadgerBadgerBadger/protoreflect-error-test/proto/lang_go/secondary";  
  
package protoreflect.error.test.secondary;  
  
message SecondaryMessage {  
  string value = 1;  
}

I won’t paste the go code here that runs the server since it’s of less interest to us but you can find that here.

When we run this, and then try to use grpcui to introspect the server, we get:

➜  ~ grpcui -plaintext localhost:8000
Failed to compute set of methods to expose: Symbol not found: protoreflect.error.test.primary.Main
caused by: File not found: dependencies/secondary/secondary.proto

This is the error that gave me a significant amount of headache before I fully understood not only what causes it but also what causes it in my specific situation.

I stumbled across this github issue. If you’re smart (which I am not) you will immediately understand the problem and figure out a solution. It took me a bit longer since I also like to understand how things work.

Since dependencies/secondary/secondary.proto is the culprit let’s look at two things:

• How are we importing that dependency?
• How are we building that dependency?

Looking at our primary.proto file, we are importing it as:

import "dependencies/secondary/secondary.proto";

And while the reflection server is trying to find it by that path, it can’t seem to.

Looking at our gen.sh script, we see:

protoc -I="$scriptDir/dependencies" \  
    --go_out="$outDir" --go_opt=paths=source_relative \  
    --go-grpc_out="$outDir" --go-grpc_opt=paths=source_relative \  
    secondary/secondary.proto

We are building our dependency with the path secondary/secondary.proto.

“But Badger”, you might ask, “Why does it matter what path we use to reference our dependency while building?” And I might answer: “I didn’t think it would, but apparently it does!”

As jhump says in his comment to that github issue I linked:

this is a problem in the Go protobuf runtime regarding how file descriptors that are compiled into your binary are “linked”.

They are linked purely by name. What that means is that the name (and relative path) used to compile a proto with protoc must exactly match how all other files will import it.

Which essentially means that the name you use while building a certain dependency gets encoded in the Golang code of the generated file as the path for that dependency. To me this feels silly, but without jumping into google.golang.org/protobuf’s code and understanding why they built it this way, it would be similarly silly to pass judgement.

While jhump’s example pointed to this file I could not find a similar example in my own code due to the differences in library and tool versions.

I looked at the Golang code generated and followed this code path:

• https://github.com/BadgerBadgerBadgerBadger/protoreflect-error-test/blob/main/error-state/proto/lang_go/secondary/secondary.pb.go#L145
• https://github.com/protocolbuffers/protobuf-go/blob/v1.34.1/internal/filetype/build.go#L121
• https://github.com/protocolbuffers/protobuf-go/blob/v1.34.1/internal/filetype/build.go#L138
• https://github.com/protocolbuffers/protobuf-go/blob/v1.34.1/internal/filedesc/build.go#L91
• https://github.com/protocolbuffers/protobuf-go/blob/v1.34.1/internal/filedesc/build.go#L112
• https://github.com/protocolbuffers/protobuf-go/blob/v1.34.1/reflect/protoregistry/registry.go#L114

I’m sure there’s an easier way to do this, but I did it by stepping through the code using Goland’s debugging tools.

func (r *Files) RegisterFile(file protoreflect.FileDescriptor) error is the function responsible for registering a protobuf file to the Global Registry

Debugging our way to line 125 we see that the path has been encoded as secondary/secondary.proto, which is the path that we used when building the file.

![[shows-path-of-compiled-protobuf-file.png]] It looks to be that the path used to build the file is the one that gets embedded as the path to the file. And the gRPC reflection server tries to look up the file using that path.

Since the file does not actually reside at that path, the reflection fails.

The solution is easy enough: use the same path while building the dependency as used to reference the dependency.

The corrected code is at https://github.com/BadgerBadgerBadgerBadger/protoreflect-error-test/tree/main/working-state

I hope you enjoyed the post and if you know why things were implemented this way, let me know.

Aside: The internet tells me that the C shell was the first to introduce the history command. It would not be hyperbolic to say that without history my task of recreating my setup would be significantly more difficult. I used it extensively to figure out what I ran when I needed to do this all those months ago. I should probably containerise or chefise or ansibilise or something.

I live in a country that uses DSMR meters and have one in my home. My girlfriend wanted a closer look at our electricity consumption, so I decided to hook up a raspberry Pi to the meter’s P1 port and send those readings to my home lab server using the Pi as a relay.

I bought a RJ12 –> USB cable off of Amazon and tried to read from my Pi’s USB port by using first cu and then pyserial (which is a much friendlier interface).

# Using cu
cu -l /dev/ttyUSB0 -s 115200 --parity=none

# Using pyserial
python3 -m serial.tools.miniterm /dev/ttyUSB0 115200 --xonxoff

Initially neither tool gave me any readings and I nearly tore my luscious hair out before finally realising that it was a bad cable I was using. I returned it and bought a more respectable cable from robbshop.nl.

And that gave me results!

I had a few different ways of turning the meter readings into useful insights:

1. Run HomeAssistant on my Pi with direct access to the port: Not an option since my HA instance was already running on my home lab server. The Pi was meant to be a relay only.
2. DSMR to MQTT: I could run something that would read the DSMR telegrams and send them to an MQTT broker to which I could subscribe my HA integration. Definitely an option.
3. Serial to Network Proxy: The option I ended up going with since it felt the simplest to me. I would run a program that would expose the DSMR telegrams over a TCP connection.

The program I landed on for doing this is ser2net With a little help from ChatGPT I figured out this ser2net config. Comments describe what the various fields do.

# defines an alias confver, I'm not entirely sure if this does anything
define: &confver 1.0

# begins a new connection definition and assigns it an alias `con00`.
connection: &con00
  # specifies how incoming network connections will be accepted, in this case the Telnet protocol with RFC 2217 support (I no clue what that means), and the connection will be over TCP and listen on port 2013c (this part I understand)
  accepter: telnet(rfc2217),tcp,2013
  # specifies how the serial device will be connected, includes path to the serial device file and port settings, and it is to be configured in local mode (not sure what local mode means)
  connector: serialdev,/dev/ttyUSB0,115200n81,local

I paired this with HA’s DSMR Slimme Meter integration with it set to listen on the network, pointed at my Pi on port 2013.

And we have lift-off!

Note: Solar readings are coming in via a different integrations and configured together with the DSMR readings using HomeAssistant’s Energy Dashboard.

Clarification: This is literally about my laundry. I am not being metaphorical. I am not talking about my feelings or my secrets. I am talking about my clothes.

Intro

Our washing machine is a Samsung something-something that sends me a notification on the SmartThings app when it completes a wash cycle. This is useful since we almost always need to follow up a washing cycle with a drying one and maybe even another load of laundry. The machine isn’t a combined-washer-dryer so someone has to move the wet clothes from our smart washer to our dumb dryer.

The fact that our dryer is “dumb” is what led me to my automation journey. If it was Wifi-enabled and also sending out notifications as it completed a cycle, I wouldn’t be writing this post.

Having been spoiled by knowing exactly when my washing has completed (so I know when to go move things to the dryer) I decided to do something similar for the dryer. This is especially useful, for example, when the clothes need an extra drying cycle or when there are more wet clothes that are waiting in the washer. In these times it is useful to know if the dryer has finished a cycle.

And since it is not a smart device, I have to add the brains, somehow.

Plan of Attack

This is how I decided to approach the problem:

• I have a Shelly smart plug that I would put between Drew (the dryer is called Drew) and the wall.
• The plug would transmit power readings to my Home Assistant setup.
• I would do something to detect when the power reading goes down from some high number to some very low number (I’m assuming it never goes to 0 since the little LED panel needs power).
• I will notify myself of this somehow.

It’s the something and somehow parts I was unsure of. My Home Assistant setup is as basic as it gets. I only started my home automation journey a month or two ago and was unfamiliar with the possibilities. However, this felt like something that should be within the realm of HASS’s capabilities.

Discovering Automations

The first thing I discovered as I dug further into Home Assistant was Automations. In Home Assistant you create automations via the Automations system. You can turn on a light when a motion sensor senses someone in the room; you can turn on the vent when humidity reaches a certain level; you can make all the lights turn pink and romantic music play over the speakers when it is February 14th and your beloved comes home from work. The possibilities are endless.

The Automation Setup

Figuring out the Automation setup I wanted was straightforward. There are 3 key components to the Home Assistant Automation system.

1. Triggers: The thing that kicks off an Automation. These can be of many types ranging from a number on a sensor changing, to an incoming MQTT payload.
2. Conditions: An optional extra check to ensure that when the Automation triggers everything is in the state you want it to be. You can check the value of sensors, check the on/off status of devices, or even check if the sun is up.
3. Actions: This where we do something in our Automation. This is where you will want to set up that electric rooster crowing when the sun comes up (if you want to build your own weird alarm clock for some reason), or send a message on a webhook when the kids are using too much electricity.

Thinking over my needs I decided on the following for my Trigger/Condition/Action setup:

1. Trigger: I want the Automation to start when Drew’s power draw falls below 10 W (as reported by Sheila, the smart plug), and has remained under 10 W for more than 5 minutes.
2. Condition: I want to ensure this Automation only fires if we have consumed more than a 100 Watts of power sometime in the last 5 minutes.
3. Action: Inform me somehow that this has happened.

While the Trigger and Condition were straightforward, the Action was a bit more nebulous. At first, I wasn’t entirely sure how to notify myself. I looked into Home Assistant’s Restful Notifications but could not make it work for the life of me. Then I decided to check if I could send an email. I found Home Assistant’s SMTP Notifications and decided to give it a go. Again, I could not make it work. It asked me to enter my Gmail password in the Home Assistant configuration file, but I was not comfortable doing that. I decided to look for another way.

I finally stumbled upon the Home Assistant iOS App which I installed on my phone and pointed to my Home Assistant server. This allowed me to send notifications to my phone (my phone became available as a target for notifications).

The full Automation setup looks like this:

alias: Is it dry yet?
description: Notifies when Drew (the dryer) has finished a drying cycle.
trigger:
  - id: Sheila Power Below 10 W
    platform: numeric_state
    entity_id:
      - sensor.sheila_power
    below: 10
    for:
      minutes: 5
condition:
  - condition: numeric_state
    entity_id: sensor.sheila_power_5_minutes_ago
    above: 100
action:
  - service: notify.mobile_app_sharpie
    data:
      message: Drew has finished a cycle. You should unload him.
      title: Dryer Cycle Finished
  - service: notify.mobile_app_buttercups_android
    data:
      message: Drew has finished a cycle. You should unload him.
      title: Dryer cycle finished.
  - service: media_player.play_media
    target:
      entity_id: media_player.thuis
    data:
      announce: true
      media_content_type: mp3
      media_content_id: https://<static-host>/drew-finished-drying.mp3
mode: single

notify.mobile_app_sharpie and notify.mobile_app_buttercups_android are the entities exposed by the Home Assistant iOS and Android apps on my phone and my girlfriend’s phone respectively. media_player.thuis is the entity exposed by my Google Nest speaker group. Not being simply content to send a notification to my phone, I also wanted to play a sound on my Google Nest speaker group. The sound is a mp3 file that I host on a static host.

sensor.sheila_power is the sensor entity exposed by my smart plug and what lets me monitor the power draw of the dryer. sensor.sheila_power_5_minutes_ago is a sensor that I created using the SQL Integration. I created a custom sensor to get the power draw 5 minutes ago.

select *
from states
where metadata_id = (select states_meta.metadata_id
                     from states_meta
                     where states_meta.entity_id = 'sensor.sheila_power')
  and datetime(last_updated_ts, 'unixepoch') < datetime('now', '-5 minutes')
order by last_updated_ts desc
limit 1

In combination this ensures that this Automation only triggers when our power draw falls below 10 W for at least 5 minutes, and we have consumed more than 100 W of power just before. This gives me a good indication that the dryer has finished a cycle and I should go unload it.

Exposing to the World

I should mention that my Home Assistant server was not available to the internet at that point. All integrations were over local Wi-Fi with firewall rules to prevent any external access. I did not have to expose my Home Assistant server to the internet to get notifications on my phone since they are both part of a Tailnet. This meant that as long as I had the Tailscale App on my phone and Tailscale VPN enabled, I could access my Home Assistant server from anywhere in the world.

And this would have been good enough for me, but my girlfriend also needs notifications on her phone and asking her to install Tailscale and turn it on/off before she could access the notifications was a bit much.

So I decided to expose my Home Assistant server to the internet.

My first attempt was using Let’s Encrypt and DuckDNS, following a guide (or three) similar to this. I won’t go into all the details, but I will say that it was a pain. Ultimately, after struggling for several days, I gave up. I gave up and decided to go back to my girlfriend, head bowed, and give her the bad news.

But then I stumbled upon Tailscale Funnels. Since I was using Tailscale anyway, with a single additional invocation, I could have a secure, encrypted, tunnel to my Home Assistant server.

➜  ~ tailscale funnel --https=8443 --bg 8123

No extra DNS required since I can use my Tailnet’s name and no extra certificates required since Tailscale handles all of that for me.

The only caveat to this approach is that Tailscale does not yet support domain name handling. So I can’t assign a nice domain name to my Home Assistant server. Furthermore, I only have 3 ports available to me. This means that if I were to have other services on that machine that I want to expose to the world, I can at most have 2 more, and they will have to be referenced by their port numbers.

This is something I will have to come back to later, but for now both me and my girlfriend have access to our Home Assistant server from anywhere in the world and are happy ducks.

Of course, I limited her account to only have access to the notifications and some views. I don’t want her messing with my automations and I don’t want a third party poking around my smart home setup.

Whenever I have a long-running process on my home-lab I start it in a screen session. I check back on it occasionally to see if it has completed. The same could be true of something I’m running on my local machine, though, in that case, it is in a live terminal session. The latter is usually during the course of my day-job and is somewhat of an easier problem to solve since I am at my machine for the duration of the process.

For my local machine (which is a MacBook), I use the say command extensively. It’s trivial to add a say "Done" at the end of a command chain to aurally inform me of a task’s completion. But that wasn’t quite spicy enough for me, so I came up with something more interesting.

➜  ~ which sdd
sdd () {
	if [ $# -eq 0 ]
	then
		title="Done"
		text="Done"
	else
		title=$1
		shift
		text="$@"
	fi
	osascript -e "display notification \"$text\" with title \"$title\"" && afplay ~/Documents/star-wars-b1-battle-droid_kampfdroide-roger-roger-sound.mp3
}

Behold the sdd function I add to the end of every script and command chain. I can invoke it with a title and an optional message, or invoke it with nothing and let it default to “Done” and “Done”. When invoked, I will be shown a notification on my screen along with an audible “Roger! Roger!”

Disclaimer
I have no idea if this is required. I find copyright laws almost maliciously confusing. Anyway, just to cover my backside:

I do not own any rights to the “Roger, Roger” voice line, which is associated with the Star Wars franchise created by George Lucas and owned by Lucasfilm Ltd. and The Walt Disney Company. This phrase is used here under the principles of fair use for educational or informational purposes only.

This is all fine and dandy for my local machine, but what about my home-lab? I don’t want to keep a live SSH session running all the time, nor do I want to have to check back on it at odd intervals. Nor am I at a terminal all the time!

The solution to that comes from my Home Assistant setup (look out for another post that talks about my Home Assistant and dryer smartification journey).

I have created an Automation that listens to a webhook and sends a notification to my phone whenever that webhook fires.

alias: Notify Sharpie
description: ""
trigger:
  - platform: webhook
    allowed_methods:
      - POST
    local_only: false
    webhook_id: "<webhook-id>"
condition: []
action:
  - service: notify.mobile_app_sharpie
    data_template:
      title: Notification
      message: "{{ trigger.json.message }}"
mode: single

Sharpie is the name of my phone.

Now I can curl from any device anywhere and receive a notification on my phone!

curl -i -H "Content-Type: application/json" -d "{\"message\": \"Sup?\"}" https://<home-assistant-server>/api/webhook/<wehook-id>

I can plop this down at the end of any command chain or script and be notified when it completes.

But I figured out all this after the fact. During the fact (is that even a phrase?), I had already started a long-running process that I wanted to monitor and be informed of when it terminated. So there was no way for me to append a notification curl at the end of the chain without stopping the process.

Enter the ps command with the -p flag. This can be passed a process ID and gives you information about the process.

➜  ~ echo $$
266386
➜  ~ ps -p 266386
    PID TTY          TIME CMD
 266386 pts/1    00:00:00 zsh
➜  ~ echo $?
0

$$ is the PID of the current shell process.

Note that the exit code returned by the ps -p command is 0. If we were to try a PID that does not exist, we will get a 1.

➜  ~ ps -p 42069
    PID TTY          TIME CMD
➜  ~ echo $?
1

Thus, without knowing the exact details or output of using ps -p on our target process, we can check the exit code and find out if the process is running. This leads us to this helper function.

# Check if the process with PID exists
process_exists() {
    ps -p $1 > /dev/null 2>&1
}

Checking if our target process is still running, now becomes a matter of a sleepy loop (to avoid spamming the command).

# Loop until the process no longer exists
while process_exists $PID; do
    sleep 1
done

echo "Process $PID has stopped."

# Send the notification
send_message $MESSAGE

The send_message function can take advantage of our webhook and notify us when the process completes!

send_message() {
  curl -i -H "Content-Type: application/json" -d "{\"message\": \"$1\"}" https://<home-assistant-server>/api/webhook/<wehook-id>
}

And with that we have a script that can monitor a process and let us know when it completes.

Wrapping it Up

I hope you enjoyed this brief exploration of figuring out how to monitor a live process. If you were to try something like this yourself, you could, no doubt, switch out the notification part with any number of other services. Perhaps send yourself an SMS or blink the lights in your house on and off 21 times.

Note: I will be using the terms amd64, x86_64, and Intel interchangeably in this post. They all refer to the same architecture. Similarly arm64 and Apple Silicon are used interchangeably.

I recently got handed a new MacBook at work. It’s a MacBook Pro with an M3 chip and my first encounter with Apple Silicon. I have been hearing many good things about Apple Silicon and having experienced it for myself for the past day I must say that my old Intel MacBook can’t compare. I know that comparing my 2020 Intel MacBook, which has been used and abused for 4 years, to a brand new, out-of-the-box 2023 M3 Mac is not an Apples to Apples comparision. So I will take the word of tech reviewers through the years as well as my own biased experience (I no longer have to wait several seconds for a compilation step) and say that Apple Silicon is a game changer.

The point of this post is not to talk up Apple’s chips but to talk about the challenges I faced getting up and running after moving my stuff from an Intel Mac to an Arm Mac.

Migration Assistant

While not specific to the Intel -> Arm transfer, I was reminded once again that I should probably be running Migration Assistant over a cable instead of WiFi. It took all night and the way the numbers kept going up instead of down, sometimes, gave me a ton of anxiety. The next time I do this I have to remind myself that taking 5 extra minutes to get that cable out and connected will save me hours later.

It Does Not Come with Rosetta

I’m not entirely sure if this is always the case or if it ended up being the case because I performed a Migration, but Rosetta was not already installed on my system. Instead, any time I tried to run a process that was Intel based, I got a pop-up that asked me if I want to install Rosetta.

I put it off for as long as I could. I wanted to have everything running as natively Apple Silicon as I could possibly manage. I finally did cave when I discovered a few tools did not have an Arm counterpart, but I think I managed to reinstall a lot of things before I hit that stage.

You Can See What’s Running on Rosetta

If you open the Activity Monitor, you can check which apps are running on Rosetta. In the CPU tab, look at the Kind column. It will either say Intel or Apple.

I managed to discover several things running on Rosetta and then went and found Apple Silicon versions of them. For example, I have some bash scripts that open tunnels to my company’s internal network. For managing these multiple tunnels the script uses tmux. I discovered in Activity Monitor that I had several bash processes running as Kind Intel. I suspected tmux spawned these processes, so I did a quick check:

➜  which tmux
/usr/local/bin/tmux
➜  file /usr/local/bin/tmux
/usr/local/bin/tmux: Mach-O 64-bit executable x86_64

It looks like tmux is still an x86 executable.

➜  which bash
/bin/bash
➜  file /bin/bash
/bin/bash: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/bin/bash (for architecture x86_64):	Mach-O 64-bit executable x86_64
/bin/bash (for architecture arm64e):	Mach-O 64-bit executable arm64e

bash on the other hand, is a universal binary, an app format that encapsulates multiple architectures and acts as one of the bridges between Intel and Arm Macs.

But since these bash processes were spawned by an x86_64 tmux, it was the x86_64 version that got spawned. Once I replaced tmux with the arm version, the bash processes also spawned with their arm counterparts. So far so makes sense!

A Scriptological Journey

Warning: This section ended up being longer than anticipated and is filled with arcane unix tools and commands. It was also the most fun section to write!

I wanted to find other binaries that were still x86_64 and see if we could clean them up. So I went into the /usr/local/bin/ directory and ran the following command:

➜  find -L . -type f -exec file {} + | grep 'x86_64' | grep -v 'arm64'

Breaking it down:

• find -L . -type f searches in the current directory and subdirectory, limited to items that are files. The -L option follows symbolic links as well.
• -exec file {} + executes the file command on each file.
• grep 'x86_64' finds us instances of x86_64 appearing in the output (the output for x86_64 binaries is Mach-O 64-bit executable x86_64).
• grep -v 'arm64' excludes instances where there is a universal binary (like bash, as mentioned before), which has both x86_64 and arm variants available in one package.

That last grep is to explicitly ensure that if I have a universal binary with both x86_64 and arm64 architectures, we keep that universal binary.

Side Note: The + modifier is an interesting one. It instructs find to accumulate the found files and execute the command passed to exec (in this case file) command on them in a single invocation. So while without + each of the found files would have file executed against them one by one, with + the files are passed to the command in one go.

The result of running this command:

./vagrant-go (for architecture x86_64):	Mach-O 64-bit executable x86_64
./dummy:                                        Mach-O 64-bit executable x86_64
./taste:                                        Mach-O 64-bit executable x86_64
./python3.10-intel64:                           Mach-O universal binary with 1 architecture: [x86_64:Mach-O 64-bit executable x86_64]
./python3.10-intel64 (for architecture x86_64):	Mach-O 64-bit executable x86_64

If you look closely, you’ll notice that the command is not quite completely giving us the result we want. For ./python3.10-intel64 (which is a universal binary with only 1 architecture) we are getting two lines of output. And for ./vagrant-go (which we can deduce to be a universal binary with both architectures), we are getting the x86_64 line of output when we actually want none at all. Recall that the ultimate goal is to delete the Intel-only binaries, for which we want a clean list of files.

awk to the rescue!

For universal binaries with two architectures, file gives us the following.

➜ file vagrant-go
vagrant-go: Mach-O universal binary with 2 architectures: [arm64:Mach-O 64-bit executable arm64] [x86_64]
vagrant-go (for architecture arm64):	Mach-O 64-bit executable arm64
vagrant-go (for architecture x86_64):	Mach-O 64-bit executable x86_64

The first line tells us that this is a universal binary with 2 architectures and the next two lines give more details on those architectures, one line being for each architecture.

We can’t treat this as a line-by-line problem, anymore. We need to process the whole 3-line block. We need to recognize that we’re processing a 2-architecture universal binary and then figure out a way to exclude the whole block.

➜  find -L . -type f -exec file {} + | \
    awk '/Mach-O universal binary with 2 architectures/ && /arm64/ && /x86_64/ { getline; getline; next } { print }' | \
    grep 'x86_64' | \
    grep -v 'arm64'

We’ve added an awk invocation after listing our files via find (and file) and before we start grepping them. Breaking down the command:

• awk is our invocation of the awk tool.
• /Mach-O universal binary with 2 architectures/ && /arm64/ && /x86_64/ is the pattern that matches the first line output by file.
• { getline; getline; next } is the action block that executes on the lines matched by our pattern. We call getline twice to skip over the two architecture lines. next stops processing the current line since we don’t want to do anything with the architecture lines.
• { print } the default action block to apply to anything that does not match our pattern. It lets them pass through and be printed.

So to put it all together, our awk invocation looks for lines that indicate a universal binary with two architectures and skips over the architecture lines, essentially ignoring the whole block.

The result of this invocation is:

./dummy:                                        Mach-O 64-bit executable x86_64
./taste:                                        Mach-O 64-bit executable x86_64
./python3.10-intel64:                           Mach-O universal binary with 1 architecture: [x86_64:Mach-O 64-bit executable x86_64]
./python3.10-intel64 (for architecture x86_64):	Mach-O 64-bit executable x86_64

Now we no longer get ./vagrant-go as part of our results, but ./python3.10-intel64 still appears twice. This is easily remedied by tacking on another grep -v "universal binary" to the end of our commands.

➜  find -L . -type f -exec file {} + | \
    awk '/Mach-O universal binary with 2 architectures/ && /arm64/ && /x86_64/ { getline; getline; next } { print }' | \
    grep 'x86_64' | \
    grep -v 'arm64' | \
    grep -v 'universal binary'
./dummy:                                        Mach-O 64-bit executable x86_64
./taste:                                        Mach-O 64-bit executable x86_64
./python3.10-intel64 (for architecture x86_64):	Mach-O 64-bit executable x86_64

At this point our command chain is getting a bit too long. For the sake of sanity I will put it all in a file called find86_64.sh.

Side Note: If you’re wondering what dummy is, let me show you the trick to creating a dummy Mach-O x86_64 executable file for testing purposes.

➜  printf '\xcf\xfa\xed\xfe\x07\x00\x00\x01\x03\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > dummy

This file won’t execute, but running file on it gives us

dummy: Mach-O 64-bit executable x86_64

Which is good enough for testing.

The next step is to extract just the name of the file. We can leave in the leading ./ since ls (which we will run next) will accept that just fine.

awk to the rescue again!

➜ find86_64.sh | awk -F':| ' '{print $1}'
./dummy
./taste
./python3.10-intel64

Breaking it down:

• awk -F':| uses either : or space as a field separator to split the line into multiple fields.
• {print $1} prints the first field, which, in our case, is the file name.

Our updated script now looks like this.

#!/bin/sh
find -L . -type f -exec file {} + | \
    awk '/Mach-O universal binary with 2 architectures/ && /arm64/ && /x86_64/ { getline; getline; next } { print }' | \
    grep 'x86_64' | \
    grep -v 'arm64' | \
    grep -v 'universal binary' | \
    awk -F':| ' '{print $1}'

Pipe that through xargs ls -l.

➜ find86_64.sh | xargs ls -l
-rw-r--r--  1 shak  admin  48  9 Mar 12:47 ./dummy
lrwxr-xr-x  1 shak  admin  81  9 Mar 12:57 ./python3.10-intel64 -> ../../../Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10-intel64
-rw-r--r--  1 shak  admin  48  8 Mar 23:50 ./taste

Note that we have a mix of regular files and symlinks (the l at the beginning of the permission string indicates a symlink, which we can also see by the arrow pointing from ./python3.10-intel64 to what it is linked to). This is not unexpected. Package managers and installation scripts often write their binaries to a location of choice and make them reachable for execution by symlinking from places like /usr/local/bin/.

Side Note: If we were to pipe anything into ls by invoking | ls -l, it would ignore the piped input and run ls -l in the working directory, which is clearly not the behavior we want. This is because ls does not accept input piped into it. xargs converts the piped in input to arguments for ls to consume.

If we were to pipe the output of find86_64.sh directly into xargs rm, we would remove the files stored in /usr/local/bin. Which works well for the binaries that are stored in this directory but for the ones that are symlinked, we will only have their symlinks removed and not the underlying files.

After a bit of research I found the readlink command which can be used to find the target of a symbolic link.

➜ readlink ./python3.10-intel64
../../../Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10-intel64
➜ echo $?
0

On regular files it outputs nothing and exits with a 1.

➜ readlink ./dummy
➜ echo $?
1

With this knowledge in mind, we can extend our script to read each line and do something different based on whether it’s a symlink or a regular file.

#! /bin/bash
find -L . -type f -exec file {} + | \
    awk '/Mach-O universal binary with 2 architectures/ && /arm64/ && /x86_64/ { getline; getline; next } { print }' | \
    grep 'x86_64' | \
    grep -v 'arm64' | \
    grep -v 'universal binary' | \
    awk -F':| ' '{print $1}' | \
    while read -r file; do
        if [ -L "$file" ]; then 
            target=$(readlink "$file")
            echo "$file is a symlink targeting $target"
        else
            echo "$file is a regular file"
        fi
    done

Breaking it down:

• while read -r file reads each line, the -r prevents any backslashes from being processed as escape sequences, preserving the original content of the line.
• if [ -L "$file" ] checks if the file is a symlink

And the rest is just echos. You can imagine how this could be replaced with a bunch of rms to clean up our unwanted binaries. Disinclined to take responsibility for shenanigans apart from my own, I will leave that final bit of scripting to the reader.

Easier Than I Imagined

I talked to a colleague who got his Apple Silicon Mac more than three years ago when these things were fresh out of the oven. His tales of trial and tribulations made me grateful that I am making the move now when so many projects have adopted Apple Silicon and have arm versions released. I code primarily in Golang, and it was almost trivial to clean up my old Golang versions and install Apple Silicon compatible ones.

Wrapping it Up

I started a draft for this post when I began my migration journey. I imagined I would have to go through a lot more trouble than I actually ended up going through. Almost every app had an Apple Silicon version that was a simple download+replace operation and the story held true for nearly everything I needed to install via brew. I was up and running in a matter of hours.

Having said that, the scripting journey was a lot of fun. I had to dust off some old Unix tools and learn a few new ones. I also got to play with some of the more arcane features of these tools. I hope you had as much fun reading about it as I did writing about it.

I thank everyone who have been experimenting with Apple Silicon these last few years. They walked with hobbling steps, so people like me could run.

I’m making my way through Robert Nystrom’s Crafting Interpreters as well as doing 2023’s Advent of Code. I am late to the latter but I am enjoying myself regardless. I am using Rust as my language of choice for both endeavours.

I am on the Representing Code chapter of Crafting Interpreters where Robert mentions using a separate tool for generating the classes for various Expression types instead of hand-writing each since they share a tedious amount of boilerplate. Had I been writing this in Go, this is the path I would have followed. However, with Rust, we have the power of macros which presents me with the perfect opportunity to learn how they work.

In Java (which is the language of choice for the first of the two interpreters we write with Robert, in the book), Robert recommends structuring Expressions as subclasses of an abstract Expression class. The following code shows this, along with the Binary class that would extend the abstract Expression class:

package com.craftinginterpreters.lox;

abstract class Expr { 
  static class Binary extends Expr {
    Binary(Expr left, Token operator, Expr right) {
      this.left = left;
      this.operator = operator;
      this.right = right;
    }

    final Expr left;
    final Token operator;
    final Expr right;
  }

  // Other expressions...
}

Since Rust does not have inheritance, we will need to look to something else. With the help of Github Copilot, I decided to go with using enums representing expressions like so:

use crate::token::Token;

pub enum Expr {
    Binary(Binary),
}

pub struct Binary {
    pub left: Box<Expr>,
    pub operator: Token,
    pub right: Box<Expr>,
}

impl Binary {
    pub fn new(left: Expr, operator: Token, right: Expr) -> Self {
        Binary {
            left: Box::new(left),
            operator,
            right: Box::new(right),
        }
    }
}

But similar to Robert’s approach, instead of writing these out by hand, we shall automate. I’m thinking of a macro which might look like this:

create_expr!(Binary, left: Expr, operator: Token, right: Expr);

Which will, hopefully, result in the definition of the Binary struct as shown above as well as a constructor function. We will define the Expr enum by hand and Token is already available as a type I’ve previously defined.

Note: Our macro accepts two arguments, the first being the name of the expression type and the second being a variadic list of its fields. In the Binary case, Binary is the name of the expression and left, operator and right would be its fields. For a different Expression type, say, Unary this would be create_expr!(Unary, operator: Token, right: Expr);.

Now let’s read up on how macros work in rust. My first stab at it involved this page on the rust-by_example site. While it reads like a perfectly passable statement of facts, I find the language difficult to parse.

When I decided to give learning Rust another go I looked for articles which could explain the magic behind Rust in an intuitive way. One of my first stumbling blocks was the module system and it kept baffling me until I came across this article and the secrets were finally laid bare. Something in it clicked in my head. I recommend reading it if you have a few minutes to spare. It’s not long but very informative.

The tutorial from the LogRocket blog told me that I could use repeating arguments for macros and have repeating blocks of code generated for those arguments. It also reminded me that I can’t write the enum Expr by hand, nor can I do multiple invocations of my create_expr macro with for each of my Expression types. The base enum Expr needs to know all its variants right at the beginning. So taking that into account, I need to reach for something like this:

macro_rules!(
  Binary, left: Expr, operator: Token, right: Expr;
  Grouping, expression Expr;
  Literal, value: Literal;
  Unary, operator: Token, right Expr;
)

This is the analog of this block in Robert’s Java code:

defineAst(outputDir, "Expr", Arrays.asList(
  "Binary   : Expr left, Token operator, Expr right",
  "Grouping : Expr expression",
  "Literal  : Object value",
  "Unary    : Token operator, Expr right"
));

After reading through another article that did not help me much (I find the examples too simplistic), I finally came across A Practical Intro to Macros in Rust 1.0 by Daniel Keep which has just the kind of real-world use case with which I like to learn things, not only because it provides the necessary complexity to dive deeper into a system but also for the personal insights the author has gleamed from having attempted a task.

Side Note: Daniel only has one other article on his blog, Rust Iterator Cheat Sheet which is also a fun read. He seems to be one of those amazing writers who only have the one or two pieces out but from whom you’d wish to read a lot more. I’ve let him know via a Github issue since that’s the only way I could see to reach him. I hope he writes more.

This was the first stab at defining the create_expr macro:

macro_rules! create_expr {
    ($($name:ident, $($field:ident : $typ: ty),+);+) => {

        pub enum Expr {
            $(
                $name($name),
            )*
        }

        $(
            pub struct $name {
            }

            impl $name {
                pub fn new() -> Self {
                    $name {
                        $($field: $typ),+
                    }
                }
            }
        )*
    };
}

Let’s break this down, more for my own understanding than yours.

We are capturing a repeating list of items, separated by a ;. That looks like $(<something>);+ where <something> can be replaced by one of Rust’s designators. Whatever <something> turns out to be, we want to capture one or more of it, that’s what the + is for (just like in regex!).

Our <something> happens to be $name:ident, $($field:ident : $typ: ty),+. There are two pieces here:

1. $name:ident
2. $($field:ident : $typ: ty),+

The first is for capturing our syntax for defining the name of the expression type, while the second is for capturing a list of one or more fields. You’ll notice the second piece also has the form $(<something>),+ but this time with a , separator. And our <something> in this case happens to be $field:ident : $typ: ty which matches syntax of the form left: Expr (for an explanation of what ty and ident mean, I recommend reading through some of the blog posts I’ve already linked).

Trying to compile it, it works!

But trying to use it with an invocation

create_expr!(Binary, left: Expr, operator: Token, right: Expr;
             Grouping, expression: Expr;
             Literal, value: Token;
             Unary, operator: Token, right: Expr
            );

does not work!

error: expected expression, found `Expr`
  --> src/expr/mod.rs:19:35
   |
18 |                       $name {
   |                       ----- while parsing this struct
19 |                           $($field: $typ),+
   |                                     ^^^^ expected expression
...
27 | / create_expr!(Binary, left: Expr, operator: Token, right: Expr;
28 | |              Grouping, expression: Expr;
29 | |              Literal, value: Token;
30 | |              Unary, operator: Token, right: Expr
31 | |             );
   | |_____________- in this macro invocation
   |
   = note: this error originates in the macro `create_expr` (in Nightly builds, run with -Z macro-backtrace for more info)

I have no idea what I did wrong here. So I will turn to Github Copilot and see if it can help me. Copilot Chat to be specific; it seems to be a version of ChatGPT restricted to answering programming-related questions.

It’s answers pointed out that I had made some glaring initial mistakes. I blame my girlfriend’s vacuuming (that was a joke, don’t @ me, I love her very much).

This is the offending piece of code:

$(
    pub struct $name {
    }

    impl $name {
        pub fn new() -> Self {
            $name {
                $($field: $typ),+
            }
        }
    }
)*

I forgot to use $($field: $typ),+ in the struct definition block and then accept arguments for the new method that are then assigned to the struct declaration, like so:

$(
    pub struct $name {
        $($field: $typ),+
    }

    impl $name {
        pub fn new($($field: $typ),+) -> Self {
            Self {
                $($field),+
            }
        }
    }
)*

Oh and I need to call the macro with Boxed Expr types to avoid having to allocate infinite heap 👼.

create_expr!(Binary, left: Box<Expr>, operator: Token, right: Box<Expr>;
             Grouping, expression: Box<Expr>;
             Literal, value: Token;
             Unary, operator: Token, right: Box<Expr>
            );

And this finally compiles!

Random Aside: I’ve always struggled with understanding Lifetimes in Rust. However, I recall an evening when a former colleague and I were walking to a guy’s place to play some DnD. We had taken a tram to the nearest tram stop to said guy’s place and were walking from the tram stop to his house. We were both eager and beginner rustaceans and were talking to each other about our own intuitions regarding various Rust systems. Somewhere along that walk I had an epiphany and I felt in my bones that I finally understood Rust’s lifetime system. Too bad I did not immediately make a note because, for the life of me, I cannot recall what the epiphany was and here I am, back to not really understanding Lifetimes.

Now let’s try to run this.

let exp = expr::Unary::new(
    token::Token::new(
        token::TokenType::Minus,
        "-".to_string(),
        1,
        None,
        0,
    ),
    Box::new(
        expr::Expr::Literal(
            expr::Literal::new(
                token::Literal::Number(123.0),
            )
        )
    ),
);

println!("Welcome to rlox! {:?}", exp);