How I Taught My Services to Talk to Each Other (Without a Message Broker)

So, picture this. You've built a slick Go service. It takes some data from Azure Blob Storage, does some heavy lifting to compile it into a regexp object, and uses that for string matching. The compilation is the slow part, so you, being the clever developer you are, slap an in-memory LRU cache in front of it. The first request is slow, but every request after that is lightning-fast. Beautiful.

Life is good. The service is humming along.

Then comes the day every developer dreams of and dreads in equal measure: "We need to scale it."

No problem, you think. You containerize it, throw a few instances behind a load balancer, and call it a day.

And that's when the chaos begins.

The Problem with Talking to Yourself

My perfect little system had a fatal flaw. When a list of domains was updated, the instance that handled the request would dutifully update the blob in Azure and then, crucially, clear its own local cache. On the next request, it would pull the fresh data, recompile, and bada bing, bada boom, it was fully up-to-date.

But what about the other instances? The load balancer was happily sending requests to three other processes that had no idea the data had changed. They were still serving matches based on the old, stale regex living in their local caches.

Oops.

This is the classic distributed systems problem: how do you get a set of independent processes to agree on the state of the world?

The Obvious Path vs. The Path I Took

The textbook answer here is to use a message broker with a Pub/Sub model. When one instance updates the data, it publishes a tiny "invalidation" message to a channel (like Redis Pub/Sub or NATS). All the other instances are subscribed to that channel, receive the message, and clear their local caches.

It's a great pattern. It's robust, scalable, and decouples your services.

But... it also means adding another piece of infrastructure to the stack. Another thing to manage, monitor, and pay for. For my use case, it felt like using a sledgehammer to crack a nut. I wondered, couldn't they just... talk to each other directly?

It turns out, they can. And modern container platforms make it surprisingly easy.

DNS is Your Friend

The core challenge of direct communication is discovery. How does Instance A know the IP addresses of Instances B, C, and D? Yelling into the void isn't very effective.

This is where the magic of built-in service discovery comes in.

Both Docker Compose and Kubernetes have their own internal DNS systems. When you run multiple instances (replicas or pods) of a service, the platform knows the IP address of every single one. And they let you ask for the whole list.

In Docker Compose, if your service is named my-go-service, you can do a DNS lookup for a special hostname within the container: my-go-service. Instead of one IP, Docker's DNS will hand you back a list of all the IP addresses for every container running that service. In Kubernetes, a "Headless Service" accomplishes the exact same thing.

Suddenly, my instance wasn't an island anymore. It had a phonebook.

The "Hey, Update Your Stuff!" API

With the discovery problem solved, the rest was straightforward. I added a new, internal-only HTTP endpoint to my Go service:

POST /invalidate-cache

This endpoint does one simple thing: it purges the specified key from the local LRU cache.

To keep things secure and prevent just anyone on the network from telling my service to dump its cache, I secured it with a simple pre-shared key. The calling service has to include a header like Authorization: Bearer <my-secret-key>. If the key is missing or wrong, it gets a 401 Unauthorized and that's the end of that.

The final workflow now looks like this:

1. A request to update a domain list hits Instance A.
2. Instance A updates the blob in Azure Storage.
3. Instance A then performs a DNS lookup for my-go-service to get the IPs of all its peers (including itself).
4. It loops through that list of IPs and sends a POST /invalidate-cache request to each one, with the secret key in the header.
5. Every single instance, upon receiving this request, clears the relevant key from its cache.

The next time any instance gets a match request for that data, it will be a cache miss. It will fetch the fresh data from Azure, recompile, and be fully up-to-date.

Problem solved. All my services are now in sync, and I didn't have to add a single extra piece to my infrastructure. Sometimes the simplest solution is the most elegant one. Hopefully. 🤞