Protobuf for Everything!! (or: How I Stopped Writing the Same Struct in Three Languages)

The first thing I wrote in this project wasn't a line of Go. It was a .proto file.

I'm currently working on a microservice-based project with three languages talking to each other, because why not? Go services do the heavy lifting, a couple of Nuxt dashboards sit on top, a Svelte widget runs on customer sites, and, because Go still doesn't have a torch runtime, a few Python sidecars do the ML work. That's three languages, four kinds of boundary, and one deceptively simple question: what does a Site look like?

In most projects I've worked on, the honest answer is "depends who you ask." The Go struct says one thing. The TypeScript interface someone hand-wrote six months ago says something slightly different. The Python dataclass hasn't been touched since the week it was born. And the queue payload? That's whatever json.Marshal felt like emitting on the day.

It all works fine. Until it doesn't.

So I made a slightly unhinged rule: every byte that crosses a boundary is described by a .proto file. HTTP bodies. Queue messages. Sidecar calls. Form validation. Even the list of labels I hand to an LLM.

Protobuf for everything. Two exclamation marks. Let me explain.

The Problem with Three Copies of the Truth

A boundary type written by hand is a promise that someone has to keep in sync manually, forever, in every language that touches it.

Nobody keeps that promise. Not because people are lazy, but because the drift is invisible. You rename tracking_key to site_key in Go, the TypeScript still compiles (it's just any with extra steps), and the first person to find out is a customer whose widget quietly stopped loading.

I'd been bitten by this enough times that this time I wanted the compiler to catch it. In every language. At the same time.

The OpenAPI Trap

The obvious path is JSON everywhere, with an OpenAPI spec as the "source of truth." In practice the spec is either generated from the Go code, which means Go is the real source of truth and everyone else is a second-class citizen, or written by hand, which means it's a fourth copy of the truth that drifts too.

And then there's tRPC, which a certain corner of the internet will tell you solves all of this. Here's what tRPC actually is: TypeScript interfaces, shared between two TypeScript projects, with a nice router on top. That's it. The "end-to-end type safety" is the TypeScript compiler, which, let's be honest, is an over-engineered linter. It's very good at crying in your editor. It's much less good at stopping anything at runtime, because by then every type has been erased.

And the moment a second language shows up, the whole thing just... stops. My Go services don't care what your AppRouter type says. My Python sidecars have never heard of it. "End-to-end" turns out to mean "end-to-end, as long as both ends are TypeScript." In a three-language system that isn't a contract. It's a pinky promise.

(Yes, you can hang zod validators off your procedures and get real runtime checks. Congratulations: you now have a schema language that exists in exactly one of my three languages.)

Because I needed a contract and not a pinky promise, I went the other way: one proto/ tree at the root of the monorepo, with buf generate fanning it out into Go, TypeScript (protobuf-es) and Python. The generated code is committed, and a freshness check (make proto && git diff --exit-code proto/gen) catches anyone who edits a .proto and forgets to regenerate.

As I write this, that tree has 22 packages, 41 files, 255 messages and 33 enums, all generated by one command running in one pinned Docker image.

Nothing runs on the host. No protoc version roulette. No "works on my machine."

"Why Didn't You Just Use..."

Now, I know what you're thinking. tRPC isn't the only contender, and before I fully joined the cult of buf generate I did look at the others. They all solve part of the problem. They also all break something else once you've got three languages in the mix.

TypeSpec. Microsoft's API design language is honestly brilliant. You write something that looks a lot like TypeScript, and it fans out into OpenAPI, JSON Schema, or even Protobuf. But notice that last one. If the best path to Go and Python structs is TypeSpec → .proto → buf generate, then TypeSpec is a generator for my generator. I'd rather just write the .proto.

GraphQL. A genuine cross-language contract... right up until you leave HTTP. It's fantastic for dashboards, and GraphQL Code Generator makes lovely frontend types. But when a Go service needs to drop a job on a queue? Or talk to an ML model over a Unix socket? GraphQL has nothing to say. I'd end up running GraphQL for the frontend and something else for everything behind it. Two contract systems. No thanks.

Avro. The king of data pipelines and message queues, and rightly so. Schemas that travel with the data (or sit in a registry next to it) are genuinely elegant on the backend. But it's infrastructure tech, not edge tech. I'm not shipping an Avro decoder to a tiny Svelte widget on someone else's homepage.

FlatBuffers / Cap'n Proto. If you're building a trading engine or a game loop, zero-copy serialization is magic. For "here's a list of sites, please render it," paying for extreme CPU efficiency with a famously clunky developer experience is a terrible trade.

Protobuf was the only one that hit all three marks. It's strict enough to be a real contract. It's flexible enough to speak JSON to a browser when it has to. And it's boring enough that Go, Python and TypeScript all have mature, battle-tested toolchains for it.

Boring, in this case, is a compliment.

One Tree, One Owner Per Message

The first thing that goes wrong with a shared schema repo is that it turns into a junk drawer. Everything ends up in common.proto, nobody owns anything, and every change turns into a meeting.

So every message gets exactly one home, and picking it comes down to one question: is it an order, or an announcement?

• Orders ("do this") live with whoever receives them. The crawler sends an EmbedRequest, but it lives in the embedder's package, because the embedder decides what it's willing to accept.
• Announcements ("this happened") live with whoever sends them. PageCrawled lives with the crawler.

That's it. No common.proto. I did try a "three services use it, so it goes in shared" rule early on. It was the junk drawer again, just with extra steps.

Every Boundary Speaks Proto (Just Not the Same Dialect)

Here's the honest bit: "protobuf for everything" doesn't mean "binary protobuf for everything." It means the schema is protobuf everywhere. The encoding depends on the edge:

On the HTTP side, a Go handler looks like this:

var req sitesv1.RegisterSiteRequest
if err := c.Bind(&req); err != nil {
    return err
}
if err := c.Validate(&req); err != nil {
    return err
}
// ...
return negotiation.Respond(c, http.StatusCreated, site)

No hand-written request struct, no decode helper, no validation if block. The binder speaks protobuf, and the validator reads the rules straight off the message:

message RegisterSiteRequest {
  string domain = 1 [(buf.validate.field).string.hostname = true];
}

On the dashboard side, the same message gets encoded with toBinary and the response is decoded with fromBinary:

const site = await sites.apiSend(SiteSchema, '/v1/sites', {
  method: 'POST',
  reqSchema: RegisterSiteRequestSchema,
  body: event.data,
})

That's the network boundary. The sidecar boundary is a different animal: no network at all, just a Go process and a Python process sharing a Unix socket on the same host. Same protos, though.

The Python sidecars (a headless crawler, and a guardrail service running a few small classifier models) implement a generated gRPC servicer. The Go host launches them with HashiCorp's go-plugin, the same machinery Terraform uses for its providers. The crawler even gets a bidirectional stream, so the Go side can veto a URL mid-crawl before Python fetches it.

Same types in every direction.

Then the Rule Started Paying Rent

For the first few weeks, protobuf-for-everything was pure overhead. More files, more codegen, more "why can't I just use a struct?"

Then the rule started paying rent.

The first payment was validation. Those buf.validate annotations turned out not to be a Go thing at all. The dashboards wrap the proto descriptor as a Standard Schema and hand it straight to the form component, so the "invalid hostname" under a text box comes from the exact same rule the server enforces. I deleted a whole layer of zod schemas the day that landed, and I haven't missed them once.

Then the API docs started writing themselves. A small package walks the proto descriptors, validation rules and all, and spits out an OpenAPI document. For free. ("For free" here means "after I wrote the annoying adapter once," which is the software-engineering definition of free.) The spec stopped being another copy of the truth and became a view of it.

And then came the one I'm proudest of. When a model classifies a page into a funnel stage or tags a visitor's intent, the list of labels it's allowed to use isn't a hand-written string array sitting in a prompt somewhere. It's projected straight from the proto enum:

// the taxonomy comes from the proto descriptor, not a hand-written list
var intents = protoenum.Project[intentv1.Intent]()

Add a value to the enum and the model can use it on the next deploy. Remove one and the model can't emit it any more. Before this, nine of those vocabularies were plain string fields. Converting them to enums was a deliberately breaking change: make proto-breaking flagged exactly nine fields, and I shipped it anyway, with a note to drain the queue first.

"But I Can't Read Binary in DevTools!"

I can already hear you typing in the comments.

You're right. That was the single biggest cost, and I knew it going in. You can't eyeball a binary body in the network tab, and curl hands you gibberish.

The fix turned out to be boring, which is the best kind. Every service does content negotiation: send Accept: application/protobuf and you get binary; leave the header off and you get proto-as-JSON. The dashboards have a NUXT_PUBLIC_WIRE_FORMAT=json switch for when I want readable bodies in devtools. The end-to-end test harness uses the JSON path too.

Production traffic is binary. Debugging is JSON. Both come from the same schema.

Where It Bit Me Anyway

This isn't a victory lap. Proto doesn't remove every class of bug. It just moves them somewhere more interesting.

Casing. Proto field names are snake_case, but protojson emits camelCase by default. For a while my OpenAPI docs said tenantId, the services returned tenant_id, and the TypeScript codec sent tenantId. It all "worked", because the proto JSON parsers in every language happily accept both spellings. That's great for resilience, and terrible for noticing you're wrong. Now the rule is explicit: senders emit proto names, receivers accept both.

Enums in two dialects. Protojson represents enums as names ("FUNNEL_STAGE_AWARENESS"), but the binary encoder wants numbers. One dashboard fed JSON-shaped data into the binary encoder and got a delightful invalid int32: NaN. The tests passed because they checked the shape of the object rather than running it through the codec. Lesson learned: test through the codec.

Decoding the wrong dialect. A scheduled-job payload stored as proto-as-JSON got decoded with fromBinary. The Edit button on every scheduled crawl was broken. Oops.

The widget is the exception. The Svelte widget that runs on customer sites speaks proto-as-JSON and pulls in the generated types only, with no protobuf runtime at all, because I refuse to ship a protobuf runtime to someone else's homepage. The cost is some manual mapping of snake_case fields and enum names inside the widget. It's a trade I'd make again, but it's a trade.

The tooling has opinions. buf breaking compares against git, and inside a git worktree .git is a file pointing outside the container mount, so the check can't run there. And buf generate fetches a dependency from the Buf registry on every run, which fails with a very unhelpful "remote is unavailable" when Docker's network is having a moment.

None of these were dealbreakers. All of them cost me an afternoon.

Why This Works for Me

• One source of truth. A field exists in the .proto or it doesn't exist.
• Breaking changes are loud. buf breaking tells me before a customer does, and removed fields get reserved so a number can never be reused by accident.
• Validation lives in one place, enforced in Go, in the dashboard forms and in Python.
• New services are cheap. Write the proto, run make proto, and the handler, client, form and docs already agree on the shape.

Is it more ceremony than slapping json:"name" on a struct? Yes, on day one. On day ninety, when you're renaming a field that eight services and two dashboards depend on, it's the difference between a compiler error and a 3 AM incident.

And here's the funny part. Look back at that table: the queue speaks JSON, the widget speaks JSON, the errors speak JSON. Half the wire isn't even binary.

Turns out this was never about using binary protobuf everywhere. It was about refusing to describe the same idea three times.

Two exclamation marks. Fully earned.