The semantic engine

It doesn't compress your data. It stops sending the parts that haven't changed.

When nothing has changed, a request and its answer cost twenty-one bytes each way — in place of a full HTTP request and a full JSON response. Nothing is being squeezed to get there. The payload simply isn't sent, because both ends already agree on what it is and can prove that agreement cheaply.

SAME / DIFF / FULL against a learned template

Serial request/reply leaving the wire idle

One execution answering many waiting callers

Semantic Compression — the byte accounting on camera: BLDC-1 against FNWP-1.

On camera: the Proxy and Daemon video shows a 21-byte SAME and a 31-byte reconstructed DIFF in journalctl, with the byte accounting live — this page's whole argument. Watch the series →

Current implementation: the shared-RAM contract and engineering record

Learning and reuse.

A full representation establishes the structure and state needed at both ends. SAME represents an unchanged answer; DIFF represents changed values against learned structure. A permanent session and library-held semantic cache avoid paying the same representation cost on every operation.

Application views have another purpose.

A participant holds a view so it can compute and answer from its present knowledge. The semantic cache reduces the bytes required to maintain that knowledge. The application should not recreate transport caching just to avoid repeated representation.

Published measurements.

The earlier pipe_workload campaign reports 2.3× to 8,741× ratios for different shapes. The large result is mostly static megabyte structure with six changing fields. Churning telemetry has a different result. A 21-byte SAME frame excludes the rest of the concrete network exchange.

The hot echo comparison averaged 118 ms against 201 ms direct Apache in the recorded run. Neither number is a portable service latency bound. Read the accounting context.

Protocol versions and clients.

The older HTTP proxy/daemon path and the modern stand-alone session client are different entry points. FNWP-1/BLDC-1 descriptions from the original specification must not be merged with FNW1 session layouts without reading their code.

The Kotlin member intentionally uses RAW frames. Interoperability establishes its protocol contract; it does not claim semantic compression for that client.

Not a security control.

Structure suppression is not cryptography. Region authentication, authorization and confidentiality are the operator’s or vendor’s policy. Read the separate boundaries.

§01The claim, stated precisely

Twenty-one bytes, and here is every one of them.

A steady-state exchange on the FrogNet plane is this and nothing more. It is worth spelling out because the obvious reading — "clever entropy coding" — is wrong.

4bytes

Protocol magic. Names the frame as FNWP-1 so nothing else can be mistaken for it.

1byte

The opcode. Which of the wire's verbs this is — repeat, diff, full, same.

16bytes

The identifier. Binds this exchange to a request and an answer both ends already hold.

21 bytes out. 21 bytes back. For what would otherwise be a full HTTP request and a full JSON response. The saving is structural, not statistical — which is why there is no ratio to quote and no benchmark to lose.

Compression ratios are the wrong frame entirely. The right frame is that most network traffic is repetition — status polls, sensor readings, health checks, dashboards refreshing — and repetition can be named instead of transmitted.

§02How it works

Four moves, and the last one is the one people get wrong.

The node becomes the network

The semantic engine takes the well-known HTTP port on every FrogNet node. The web server moves to a private port behind it and becomes an origin only the local machinery talks to.

The effect: every HTTP client on the box gets this without knowing anything exists. A script, a browser, a sensor, an application you shipped years ago. No library to link, no SDK to adopt, no code to change. This is the difference between a fabric and a library, and it's the single most important architectural fact about the system.
It learns the shape of your traffic

The first time a given kind of request crosses a link it goes uncompressed, and both ends learn a template from it — which parts identify the request, and which parts vary from call to call. That template is stored and shared. Every subsequent request of that kind is described as a difference against it.
It sends only what changed

Each end keeps a reference: the last set of values it successfully exchanged with that peer for that kind of request. Three outcomes follow. Nothing sent before — send it once, in full. Some fields changed — send only those, by index. Nothing changed at all — send the identifier and nothing else.
The answer is proven, not assumed

When a repeated request arrives, the far end re-executes it against the real origin. It compares the new answer to the previous one, and only if they are genuinely identical does it reply "same" — with a token cryptographically bound to both the request and the response content, so it cannot be forged or guessed. If the data changed, the change comes back as a difference.

This is the part most people expect to be a cache

It is not a cache. A cache serves an old answer and hopes it is still true. This re-runs the work every time and only says "unchanged" once it has confirmed that from the origin itself.

Stale answers are not possible by construction — because no answer is ever served without being recomputed. That is a deliberate and more expensive choice than caching, and it is the reason the system can be trusted with data that matters. Correctness before compression, in that order.

§03Why it holds up at scale

Two mechanisms stop the mesh asking the same question twice.

They are keyed differently and they remove different costs, so neither substitutes for the other.

Protecting the link

When several parts of one node ask the same question at the same moment, one request goes on the wire and everyone shares the answer. The link is the scarce resource on a slow bearer.

Protecting the origin

When several different nodes ask the same question at once, the receiving node executes it once and serves them all from that single execution. The origin is the scarce resource on a small computer.

§04Telling the mesh

When something changes, word spreads — with three brakes on it.

A node tells its immediate neighbours, each of them tells theirs, and the mesh converges. Left alone that is a broadcast storm, so it is bounded three ways.

Brake one

Seen it, dropped it

Each wave carries an identity, and a node that has already seen that identity drops it. This is what makes an epidemic terminate instead of ringing forever.

Brake two

Neighbours, not the fleet

Each node notifies only its direct neighbours rather than every node it knows of — turning fan-out from fleet-sized to neighbour-sized, while re-propagation still covers everything.

Brake three

A burst is one event

Arrivals are debounced, so a flurry of notifications collapses into a single reconciliation instead of a queue of redundant ones.

And the notifications are themselves ordinary HTTP requests — so they ride the engine like everything else. Only the event identifier differs between them, so after the first one to a peer, every subsequent notification carries a single changed field. The gossip layer is nearly free because it rides the compression layer.

For the guild · open problem

The thresholds are a decision, not a discovery.

The boundaries between SAME, DIFF, and FULL were set by judgement and have held up in practice — but nobody has measured where they should sit for a given traffic shape. That is a bounded, self-contained piece of work with a clear way to prove you were right. Get the source → — open source under the GPL, nothing to request. If you want to work on this, say so in Discussions and say which; that is a courtesy, not a condition.

§05Three things worth taking away

What this actually changes.

Repetition, not entropy

Conventional compression asks how few bits can represent this message. FrogNet asks whether the message needs to be sent at all. Those are different questions, and the second has far better answers on the traffic real networks actually carry.

No adoption cost

Because the node takes the port, existing software gets the benefit unmodified. Most efficiency technologies require you to rewrite something to use them. This one requires you to install it.

And the honest boundary

This is the bridge, not the destination. Making REST dramatically cheaper does not make REST the FrogNet programming model. Semantic transport exists partly so existing applications can migrate without an architectural flag day; the destination is eliminating exchanges that only exist because the application was written around messages. Semantic exchange is what makes a network you own affordable to run — but the reason to want one is UnREST and network shared memory: exchanging memory instead of messages, so the plumbing stops existing rather than getting cheaper. The engine is how you afford the prize; it isn't the prize.

Get the source →Read the build manual →

§06Open work · BLDC-1

Language detection, and languages we do not have

We need: compression people, format and parsing people, anyone who has built a schema inference engine or a domain-specific encoder.

The seam is language detection and language handling. Today the classifier is deliberately empirical — sniff_body_mode never trusts Content-Type, treating it as a weak hint and letting the body decide. It resolves to one of five modes: json, xml, html, text, raw. Each has a handler, and each handler learns structure from traffic and diffs against what the far end already holds.

Better detection. The classifier is a set of _looks_like_* predicates over a decoded body. That is honest, and it is also the crudest part of the pipeline. Misclassification costs a template: a body that falls through to raw learns nothing and diffs against nothing, so every send is a FULL.

More languages, and custom ones. Five modes is what the general case needed. A protobuf handler, a CBOR handler, a CSV handler, a handler for a specific industrial or medical format — each is self-contained and touches nothing else. A domain-tuned codec can beat the general one badly, because it knows what the fields mean rather than only what shape they are. Someone who owns a data format professionally is the right person to write its handler, and they never have to read the election code.

How you would know it worked. A corpus of real bodies in the target language, with measured FULL/SAME/DIFF ratios before and after. The compression win is the proof, and it is directly measurable.

None of these is hidden behind a fallback. Each one fails visibly, says what it is, and is written down — which is the only honest way to ship something unfinished. The codec seam → · Get the source →

§07Prove a better one

This is an initial implementation, and I don't claim it is the best.

What I claim is that it appears to work, and work correctly. That is a smaller claim than it sounds, and it is deliberately falsifiable.

BLDC-1 is one codec, arrived at by one person. The template learning, the SAME/DIFF boundary, the decision to run stock LZ4 over an already-reduced payload rather than tune a compressor — every one of those is somewhere a better answer could be demonstrated. The codec is a working group in the guild, open to anyone who wants to look at it, and the simulator is how a proposal gets settled: run it, show the numbers, and the argument is over.

Contributions do not have to be a replacement algorithm. A payload the template learner handles badly is useful. So is a case where SAME is returned and should not be. Somebody will eventually take responsibility for this subsystem, and it will be whoever has done the work rather than whoever asked for the title.

I welcome contributions that can prove a better algorithm. Programmers don’t work alone.

Join the guild →The ten subsystems →Get the source →

§Measured, 17 August 2026 · Seattle 6 to New York 1, internet tunnels

One megabyte of JSON, six dynamic fields: 92 KB moved, not 31 MB.

The payload was what made it slow, and the payload was almost entirely structure both ends already had. Six fields moved. That is one fact, not two — and it is the row that cannot be dismissed as a small-message trick.

Workload	Direct to Apache	Through FrogNet	Bytes on the wire
1 MB JSON, 6 dynamic fields	0.9 req/s	4.4 req/s	92 KB, not 31 MB
1 MB, burst 300	—	4.4 req/s	38 KB, not 347 MB
echo ×300	4.0 req/s	4.7 req/s	1 KB, not 39 KB
echo, burst 300	—	65.3 req/s	12.3 KB, not 478 KB
chat ×25	2.7 req/s	4.6 req/s	6.4 KB, not 414 KB
chat, burst 300	—	20.4 req/s	28.8 KB, not ~4 MB

On the 300-way burst, 279 of 300 succeeded; 21 hit a client-side deadline. A failure to wait is not a failure to serve, and saying so is why the rest is believable. Other published sessions (this site's earlier run, the book's Seattle-to-New-York session) give different absolute rates on different hardware — read the ratios, not the targets; these numbers are from the 17 August session.