The Polyglot Contract: Taming Microservices Chaos with Protobuf

In the early days of the Pomai Ecosystem, communication between services was simple. The Realtime Hub (Go) would emit a JSON event, push it to Kafka, and the AI Hub (Python) or a Node.js worker would consume it.

JSON is human-readable, flexible, and native to almost every language. It felt like the perfect choice. Until a seemingly harmless update brought the entire pipeline to its knees at 2 AM.

The "Silent Assassin" of Microservices: Schema Drift

I had updated a Node.js service, changing a simple property name from userId to user_id to match a new database convention. I deployed the service. Everything looked fine—no compiler errors, no immediate crashes.

However, the downstream Data Hub (written in Go) was still strictly expecting userId in its struct tags. When it pulled the new JSON messages from Kafka, the field unmarshaled as an empty string. The Data Hub didn't crash; it just quietly started writing corrupted, orphaned records into the MinIO Data Lakehouse.

By the time I caught the issue, thousands of records were malformed. This is the inherent danger of JSON in a distributed system: It fails at runtime, often silently.

I realized that in a polyglot architecture (Go, Python, Node.js), sharing code is impossible, but trusting developers to manually sync JSON shapes across three different repositories is a recipe for disaster. I needed a Single Source of Truth.

Enter Protocol Buffers (Protobuf)

I decided to rip JSON out of the Kafka pipeline and migrate entirely to Protocol Buffers.

Instead of defining types in Go structs or TypeScript interfaces, I created a central repository: api/proto/v1/events.proto. This file became the absolute legal contract for the entire Pomai Ecosystem.

message KnowledgeUpdate {
  string document_id = 1;
  string action = 2; 
  string content = 3;
  // userId was safely deprecated using reserved tags
  reserved 4; 
  reserved "userId";
}

The "Aha" Moment: Compile-Time Safety Across Languages

The magic wasn't just in the binary compression (though saving network bandwidth was a nice bonus). The real breakthrough was the developer experience.

I wrote a unified Makefile that acts as the heartbeat of the ecosystem. Whenever the .proto contract is updated, running make generate simultaneously compiles:

1. Idiomatic Go structs for the Realtime and Data Hubs.
2. TypeScript interfaces (via ts-proto) for the Node.js services.
3. Python classes for the AI Hub.

Now, if someone renames or removes a field in the .proto file, the Go compiler literally refuses to build the Data Hub, and the TypeScript compiler throws red squiggly lines in the Node.js IDE.

We moved the error from Runtime (production crash) to Compile-Time (local dev environment).

The Lesson: From Code Sharing to Contract Sharing

Migrating to Protobuf added boilerplate upfront. Setting up protoc, managing plugins for three languages, and learning backward-compatibility rules (like never reusing a field tag) took a few days of heavy engineering.

But the peace of mind is invaluable. Today, when a message flows from Go through Kafka to Python, I don't have to guess if the data shape is correct. The contract guarantees it.

If you are building a system with more than one programming language, stop relying on JSON and implicit trust. Establish a contract. Your future self (especially at 2 AM) will thank you.