The Dither Project

• The Dither Project
  • What is it?
• Core Design Tenets
• Structure
  • Core Process
  • Service Swarm
  • User Interface
  • Interface Layer
• Other Links
  • Inspirations for Dither

What is it?

Dither is a project with the ultimate goal of creating a decentralized and privacy-respecting Internet. It is a repository of libraries, tools, ideas, and applications that allow people to communicate privately, distribute data, manage accounts, much more. It is currently being developed by @Zyansheep and (for the time being) hosted via GitHub under the libdither organization.

The aim for Dither is to replace existing centralized applications with decentralized alternatives that are unified through their use of a singular, modular protocol.

See the application document for outlines of various applications that could be built using Dither.

Core Design Tenets

It seems helpful for projects to have guidelines to help aid design and collaboration. These are the ones I've chosen for now, as with everything, they are subject to change.

Dither should be useful

• Dither is a project. Projects should be useful.
• This one probably doesn't even need to be said, but it is always important while thinking about design, to keep in mind that you eventually want to create something useful.

Dither should be modular and modelable

• Once you build something, it often gets harder and harder to add features to it unless it is clear how the parts of the system connect and you can separate your concerns. To ensure extensibility and comprehensibility for future Dither developers, modularity and modelability are key.

Dither should be interoperable

• The goal of Dither is to replace existing services and standards. This is very hard to do (see: xkcd #927). To try to avoid this fate and make transition as easy as possible, Dither should do anything necessary to make the experience the same, or better, than existing platforms and services.
  • A real-world example of this working is PipeWire acting as a single program unifying nearly all existing audio APIs on linux. (and being backwards compatible with programs that use those other audio APIs).
• See Dither Interoperability for more details.

Dither should rely on itself

• This tenet is simply a reminder of the end-goal of Dither: to replace the centralized internet. So long as the other tenets are satisfied, this is the ultimate goal. (Because who doesn't want to reinvent the wheel!)

Structure

Following the first tenet of Dither, in the future the lines between these layers will blur and everything will be a module. However, the design of current operating systems don't easily allow for shared code and data, requiring a more formal structure. In the future this layered structure will be replaced with a more flexible system.

Core Process

The Core Dither Process is the part that deals with all operating system-facing operations such as data storage, establishing peer-to-peer connections with other computers, in addition to managing all Dither services and connections between them.

This "Core Process" provides a few core APIs that only certain services running in the "Service Swarm" are allowed to use for security purposes. The idea behind a "Core Process" is to create a sandboxed environment for services to run in a safe manner.

peer-to-peer connections with other computers running Dither. Currenly in the dither-sim program, this is implemented via a simple TCP stream. In the future this layer will be implemented using existing libraries such as libp2p transports, Pluggable Transports, something else, or some amalgamation of all three. The idea behind this layer is to provide as many methods of communication as feasibly possible.

Service Swarm

The service layer provides all functionality related to routing, encryption, data storage, user management and everything else. Each of these services are split up into separate modules each of which runs its own processes and communicates with other services through inter-process communication.

All these processes are managed as child processes under one "main process". The main process contains the Transport Layer implementation, the routing protocol API and APIs for managing the child services as well as managing inter-process communication between child processes.

User Interface

The application layer contains services just like the service layer that are registered under the main process. These registered service's APIs can be used by other applications.

The application layer also refers to the application's core API, which is used by the interface layer. This "core application API" can be built into the applicaiton's executable, or it can run as a service under the main process and used by multiple interfaces. This is left up to the application developers. Applications with multiple interfaces should prefer to register application APIs under the main process.

Existing planned applications may be found here.

Interface Layer

The final layer is the interface layer. This just refers to standalone applications that provide some kind of interface to the user using the services running under Dither.

These interfaces can be implemented however, but it is recommended for them to follow Dither's application design philosophy for some level of standardization.

Other Links

Inspirations for Dither

As with any creative endeavor, Dither takes inspiration from many other projects. This is a list of what parts of Dither have been inspired from other projects.

Structure

This document outlines the major structure of Dither.

System Manager

At the core of Dither is the system manager. This will be written in Rust and should only be run once on any given user account. The system manager provides a sandbox for Dither protocols and is built in a modular fashion to support any kind of setup or platform Dither might run on.

Core Services

In order to sandbox Dither services, the system manager provides certain core services such as access to storage, network, or other Dither services.

• Service Manager
  • Provides any service with access the ability to organize other service's permissions, manage storage, as well as stop and start services at will.
• Network Service
  • Provides any service with access the ability to establish arbitrary TCP or UDP connections.
• Storage Service
  • Provides any service with access the ability to fetch and store data unique to that service.

Other services may be added as needed.

Service Swarm

The system manager acts as a kind of sandbox for all services running on Dither and facilitates communication between different services.

Services

This is a list of planned Dither services and their dependencies, note these may be split up into smaller sub-services.

• Distance-Based Routing, DBR (Network, Storage)
• Directional-Trail Search, DTS (Distance-Based Routing, Storage)
• Reverse-hash-lookup, RHL (Directional-Trail Search)
• User Manager (DBR, DTS, RHL, Storage)
• Dither Chat (DBR, DTS, RHL, User Manager, Storage)
• Dithca (DTS, RHL, User Manager, Storage)
• Protocol of Truth (DTS, RHL)

Applications

Applications in Dither will be external programs that communicate with specific services in the Dither System Manager. i.e. Dither Chat will use the Dither Chat Service Dithca will use the Dithca service.

Or applications can use a multitude of different services as needed.

Dither Anonymous Routing

Dither Anonymous Routing (DAR) is a peer-to-peer protocol for efficiently and flexibly obfuscating connections between computers.

It improves on speed and versatility over existing solutions (i.e. I2P and TOR) by incorporating latency and bandwidth estimation techniques so that intermediate relays may be (optionally) optimally chosen. It exposes this option to applications built on DAR, allowing the developers and users to explicitly dictate the trade-off they prefer between speed and anonymity for each networked application they use.

In addition, to align itself with the philosophy of Dither, DAR aims to be as generic as possible, so that different encryption schemes and transport types may be used for different connections to hedge against the risk of one of them breaking, as well as easily allowing for novel schemes that may not be rigorously tested, but have desirable properties such as the stateless encryption protocol described in HORNET.

Network Modeling

In order to figure out what paths through the network are ideal for the given application and user (trading-off latency, bandwidth, anonymity, and cost). We need to be able to model the effects of connecting and routing through different nodes. "Will connecting to 245.14.973.23 to hide my connection still reach my latency or bandwidth targets?", "To what degree will it gain me anonymity from various threat models?" "What will it cost?" etc. Users should be able to set QOL goals that they are happy with, minimum performance and anonymity guarantees on a per-application basis and have the protocol automatically figure out what nodes to select and route through. This is a general RL task however, and thus to solve it it seems likely that we'll need some general world modeling algorithms.

A world model should be able to:

• Take two IP addresses, or a series of IP addresses and a timestamp, and guess the latency and bandwidth between them.
• Take a desired latency and bandwidth and generate likely single or multiple IP chains that satisfy the desired latency, bandwidth, and cost. Or are as close to satisfying them as possible.

Ideally this world model should be trainable in a federated / decentralized fashion. It is an open problem for how this could be done without divulging local information. (Perhaps it could be incorporated into the loss function for it to be bad at predicting certain "sensitive" metrics, at least for versions of the model sent to other nodes. This would somehow have to be balanced well with the game theory of the other nodes.)

Once you do have a world model, you need to RL it. What are the rewards we are maximizing?

• Predictive accuracy of metrics outside of yourself. (Predict results of actions)
  • Measurements, cost requests
  • Path selection metrics

What about the incentive layer here? Lets assume we have an out-of-band currency exchange system that is anonymous.

Nodes are trying to maximize their own income from acting as proxies. Cost is similar to latency or bandwidth, its a metric you receive from pinging a node and it can be predicted.

Nodes get requests for proxy and send back price, they have some baseline resource usage and want to maximize cumulative money over time. When they get a proxy request they can either accept or send back their own price which the sending node can either accept or not. RL algorithms will then need to learn how to bargain with each other automatically within their constraints.

Peer Discovery

To form a network, there needs to be a process for new nodes to connect to existing nodes. Dither Anonymous Routing aims to allow peers to be as anonymous as possible, and thus the peer discovery method aims to expose as little as possible about nodes on the network. Specifically, as little information as possible about nodes that don’t want to be discovered, and nodes far away (latency-wise) from the new node.

TLDR: Peer Discovery must expose information about some nodes, but should only expose information about nodes that are nearby and want to be known.

For more information, check out the Discovery section.

To assign routing coordinates to nodes, there is a process of peer-discovery that functions as follows. This process happens whenever a new node joins the network.

1. New node bootstraps onto the network by initiating connection to one or more existing nodes.
2. New node tests response times (latency) to connected nodes (peers).
3. New node requests from some subset of lowest-latency (closest) peers that it would like more peers.
4. New node’s peers notify some slice of their peers that a new node would like more connections.
5. Notified nodes initiate connection if they are configured to do so and measure latency to new peers.
6. Notified nodes initiate connection with new nodes and new nodes measure latency to new peers.
7. New node takes note of the smallest latencies of its peers and goes back to step 3 until there are no closer nodes who want to peer.
8. After a certain number of closest nodes are found whose latency measurements are stable, the new node then calculates routing coordinates and is records its currently connected nodes so that it may reconnect when if went offline.

Through this process, a distributed network is formed that reflects the physical topology of the relative orientations of the nodes.

Process

A packet with an RRC can be routed to its destination via the following process:

1. Node chooses the peer that will receive the packet next by comparing RRC directions
2. Node subtracts next peer’s RRC from packet’s RRC
3. Node forwards modified packet to next peer

The process continues until the packet’s RRC is all zeroes and the last node it reaches either is the destination node, knows the destination node, or is the wrong node in which case the packet is dropped or sent back depending on the packet type.

Usage

Compared to the global routing tables and complicated peering and address space allocation protocols that the existing internet uses. Routing Coordinates are much better for peer-to-peer applications because they are pretty much infinitely scalable.

That said, RC in some ways give away more information than traditional IP addresses do. Since the self-organized networks that use RCs reflect real-world network topologies, just knowing someone’s routing coordinate relative to you could be akin to knowing roughly where they live. This is an acceptable risk because it is much easier to do efficient onion routing on networks with RCs than those without meaning that there is no reason not to have all connections onion-routed to some degree, providing better privacy overall.

Other benefits of routing coordinates are that they have the potential to almost completely prevent denial-of-service attacks. To even attempt such an attack, the attacker must find the routing coordinate of their target. Disregarding user error, this kind of attack is essentially impossible since everything is onion-routed by default. Even if the attacker does have the target’s routing coordinate, trying to DOS a routing coordinate is like trying to DOS the entire expanse of network between the attacker and the target, the attacker(s) will be ineffective or blocked by other nodes automatically for overuse of the network. Even distributed denial of service attacks can be mitigated with additions to the protocol allowing the victim to notify the network that they are being attacked and to rate limit the attackers.

Anonymous Routing (Onions, Garlic, and all the others…)

Conventionally, anonymous routing is an incredibly slow ordeal because of how intermediate peers are selected from the network. Due to this inefficiency, onion routing protocols have been somewhat limited in what kind of privacy they can provide because low data rates and high latency was a concern. This is no longer the case with DBR, which may support all kinds of anonymous routing schemes:

• Onion Routing
  • The simplest routing of them all. Simply select a list of peers and establish a route from beginning to end.
• Garlic Routing
  • Similar to onion routing, but when sending packets to multiple peers at once, send them together for them to be split apart at some mid-point in the path.
• Multi-path routing
  • Maintain multiple Onion routes throughout the network and randomly send packets along all or some subset of them.
• Pool Routing
  • Create a group that nodes can join. All nodes randomly send randomly-sized data packets to all other nodes at random intervals, sending real data (padded) if there is data to send and sending random bytes if not.

DBR plans to use a modification of the HORNET protocol for setting up fast onion-routed links.

Preventing Network Abuse

Routing protocols that rely solely on people voluntarily hosting nodes typically only have a relatively small number of peers willing to route packets through themselves (i.e. TOR). This is why protocols like BitTorrent, I2P and IPFS have systems in place that incentivize peers who use the network to contribute back for the benefit of all.

To accomplish this behavior for DBR there must be some way to limit packets going through nodes that either don’t use the network that much or don’t have a lot of bandwidth capacity and speed up packets through nodes that contribute greatly to the network. Also, to take into consideration are the management of nodes that have inconsistent uptime or inconsistent routing.

When talking about incentives, we are talking about game theory. So lets analyze the game theoretical situation at the level of an individual node.

Constraints:

• Each node is directly connected to a fixed number of other nodes at a varying latencies and bandwidth.
• Each node wants to send traffic through other nodes to use the network.
• Each node wants to establish onion proxies with other nodes for privacy.
• Each node has set of parameters that may change over time:
  • Percentage of the time it will immediately respond and route a packet.
  • Amount of traffic per unit time it is willing to route on average.
  • Max amount of traffic per unit time it can route.

The goal is to allow for unrelated nodes to route and establish proxies through each other in proportion to how much each node contributes in some way to the network.

Ideas:

• Each node keeps track of the amount of traffic (bytes) flowing through its itself from directly connected nodes.
• Each node only sends traffic through direct nodes it knows it has received traffic from.
• There is more to theorize about here for future research :)

Conflicts with ISP Load Balancing

(This section is WIP)

Implementing an alternative routing protocol on top of regular IP routing may pose issues for ISP routing (i.e. forcing utilising of certain links too much, causing major slowdowns). ISPs don't optimize for latency or bandwidth, they optimize for load balancing to prevent too much link utilisation. TODO: Dither should take this into account by implementing its own second-layer load balancing system that makes sure ISP links aren't overloaded.

Questions for this problem:

• How might peer-to-peer overlay networks (of various kinds) effect an ISP’s ability to do load balancing well?
  • ISP simply have to do more buffering, slowing queue times for specific links, making those links become unattractive, shifting the route prioritization to ones that are less desirable -> problem is this makes those routes unusable for regular (nearby) people and are instead hijacked by through-traffic.

Research

• Dawn - Selfish overlay compensate for careless underlay

• If overlay networks can or do conflict with ISP load balancing effort, how can that conflict be reduced via design of the protocol?

  • Maybe reimplement load balancing into the network protocol?
    • What might this look like in a peer-to-peer setting?

Some set of nodes in the network:

• Sending along certain local paths (need to be careful to not overwhelm bandwidth along local link: not likely)
• Sending along mid-range path, some room for path diversity, but if everyone is doing it, it may create hotspot: need

Main issue: A is sending large file to B, anonymously. If they route through one path, this will be a problem, if they route along multiple paths, ISPs have more ability to distribute the load.

• More paths = more chance for paths to be surveiled: is it obvious who is chatting with who? -> More obvious than just one path
  • Mixnets ? -> How fast would these be for large data transfers?

Research, Inspirations & Similar Work

(List in rough order of reading of notable articles I am using to implement Dither Routing. If anyone knows of any similar work not listed here, please let me know on Matrix!)

[1] HORNET: High-speed Onion Routing at the Network Layer

• Stateless Onion Routing, improves establishment of onion routes as well as speed of forwarding.

[2] Vivaldi - pdf, video

• The paper that pretty much started the distributed Network Coodinate System field of research.

[3] Coordinate-Based Routing for High Performance Anonymity

• Applying Vivaldi to Anonymous Routing

[4] Phoenix: A Weight-Based Network Coordinate System Using Matrix Factorization

• Improvements on Vivaldi, uses Matrix Factorization instead of Euclidian Embedding

[5] NCShield: Protecting Decentralized, Matrix Factorization-Based Network Coordinate Systems

• Threat modeling on Network Coordinate Systems. Prevents Frog-Boiling attacks.

[6] DMFSGD: A Decentralized Matrix Factorization Algorithm for Network Distance Prediction

• Improvements on Phoenix paper’s algorithm.

[7] Application-Aware Anonymity, Sherr et al.

Directional Trail Search

Directional Trail Search (DTS) is a protocol for efficiently fetching a piece of stored data from a network given its hash.

It is intended to be a vast improvement over existing protocols like IPFS by removing the use of poor-latency Distributed Hash Tables (DHT) and giving up some amount of data persistence in exchange for fast anonymous retrival of popular documents and flexibly anonymous hosting, and fixing the scalability issues to create a true "inter-planetary" file system.

The Main Idea

DTS works via two forces:

• The desire of people (nodes) who want to host information and want their information to be quickly accessible.
• The desire of people (nodes) who want to access information and want to access it as fast as possible.

These two forces are corralled by DTS to create an efficient system to host and find data. First let us talk about the desire of a node who wants to host some data. This desire manifests in the form of a "data trail".

A data trail in DTS is a trail left in the network with the sole purpose of leading to a specific piece of data. Specifically, a trail is a chain of peered nodes that store a mapping between the hash of a specific piece of data, and the id of the next peer in the chain.

A data trail is formed with the following process:

• A node that wants to host information broadcasts a "trail-laying" packet that travels to a specific relative coordinate on the network.
• All the nodes that this "trail-laying" packet encounters on the way to its destination will do one of the following:
  • Reject to be a part of the trail, sending the packet to the previous node in the chain. This will reflect poorly on the node if it wants to host data of its own.
  • Register a connection between the hash contained within the packet and the id of the node the packet came from and forward the packet on to another node of a consistent distance away that it thinks will agree to be apart of the chain.
• Allowing nodes to either host data & be apart of trails or not host data and not be apart of trails makes sure packets flow along computers that actually host data and are likely to be relatively stable and with resources to spare.

Once a data trail is formed, it may be encountered by a "trail-tracing" packet which, once on the trail, is routed directly to the root of the trail.

However, this trail of nodes may be very thin, and thus trail-searching packets may have a hard time finding trail because they skip over too many nodes. To fix this issue, trail-laying packets will broadcast out to all known peers within a certain range that they are a part of a trail related to a specific hash. Peers receiving this broadcast will use a counting bloom filter to make a record that they are nearby a trail for a specific hash. "trail-searching" packets that come across these "nearby-trail" nodes can ask the nodes to ask all their peers within a certain radius if the peers are apart of a real trail. This casts a larger net and makes it easier to find trails without putting undue burden on too many nodes.

WIP: Section about the requesters role in the network and the expectations of the network for requesters to contribute to hosting for some time. (like bitorrent or ipfs)

Specific Structure (WIP)

Note: This is an insanely hard problem to solve well: finding the node hosting a piece of data that matches a given hash on a network. The protocol here is a formulation of a protocol that might work to solve this problem.

Every Dither node implementing DTS contains the following state:

• A hashmap that maps multihashes to its corresponding data stored on disk, so that the node may retrieve its own data.
  • The size of this map should be primarily up to the node owner's discretion, depending on how much data they would like to store.
  • Data may be temporarily be added to this hashmap to help with the caching of other node's data.
• A hashmap that maps multihashes to peers.
• A bloom filter containing a set of multihashes of which there are trails nearby.

Every node will store the following information about their peers:

• WIP

Downsides

While Directional Trail Search is in theory much faster and much more efficient than DHTs, it is likely not as good when considering rare data. With a DHT, as long as there is at least one node hosting the data, it will be found eventually. With DTS, there is no guarantee that a piece of data will be found (i.e. if the data trails are too far away from the requesting node to be encountered by a searching packet).

There are multiple potential solutions in order of feasibility:

• Use a DHT in addition to DTS by default (this may have privacy implications)
• Figure out how to get DTS to work better for rare files, perhaps by making it so that trails form circles around the earth and thus are nearly impossible not to encounter.
• Store routing coordinates / routing areas on the Reverse Hash Lookup.
• Implement a Network Coordination feature that tells all nodes in the network to notify a node when they find the requested data. (induces denial of service vector, probably a bad idea)

Reverse Hash Lookup (WIP)

Directed Acyclic Graph data structures (DAGs) may be elegant for storing and linking pieces of data, but they don't provide any kind of mutability on their own. This is the purpose of the Reverse Hash Lookup (RHL). RHL allows you to create a piece of data that links to two or more other pieces of data (via hash-linking) and then lookup the link given one of the linked pieces of data.

Structure

RHL has many different ways to solve the two problems of distributing links and finding links. (Links here being pieces of data containing the hash of some other object that is being "linked to"). Links could be shared only with friends or trusted individuals, links could be broadcast all over the network and re-stored by other nodes, links could be stored in blockchains or link maps maintained by centralized servers, etc. All these different use-cases may be useful for different applications, so RHL tries to generalize over all possible use-cases.

Ideas:

• A Link defines its own methods of distribution and search. (Perhaps embedded as a hashtype).

Methods of Distribution & Search

• A link can be broadcast to some set of trusted nodes (friend group), queries are done by asking friends if they have registered any links to an object.
• A link can be registered in some sort of global consensus
  • Central server(s) store any links that are uploaded and respond to queries
  • Blockchain storing links (all "full" nodes store all links), queries done locally
  • Global Binary Tree mapping hashes to links (less space, more network activity when searching)
• A link can be broadcast via a publish-subscribe system and exist ephemerally, find beliefs by constantly listening on a topic (or to the entire network).

Potential Applications

This kinda of system is useful as a basis for other systems that need to link disparate pieces of data together for the purposes of querying or consensus. Here are just a few examples of the systems that RHL could enable:

• Comment systems. Each comment is a piece of data signed by some individual. Links can created over the comments that allow for querying some subset of all the comments, i.e. top comments or set of all comments or comments from friends. These links themselves may be immutable, but can be joined together via other links in a chain where the newest link in the chain is the most up-to-date view of the comment system.
• Web of Beliefs. A justification / rule links two beliefs together and beliefs may be private or publicly shared and distributed. Beliefs justified by some rule sets may be more propagated than others (i.e. scientific beliefs). Or beliefs can be accepted or rejected based on arbitrary social conditions like "what proportion of my friends hold belief x?" (Such as is the case with names). Multiple conflicting beliefs can be held at once, such as if it is not clear which one should be accepted as default (or there are uses for both beliefs in different context, such as with definitions).
• Anonymous Interactions. Links could be generated ephemerally to prove that a public key in some trusted set of public keys interacted with a piece of content in some manner, and then added to some hyperloglog or statistical counter.

Specification Ideas

Disp type Link<Type> is implemented for whatever hash type is used in the specific link needed. i.e.

For a link to be type-valid, the objects it links to must also be the correct type. Zero-knowledge proofs could be used for this application in the future to avoid unnecessary fetching.

struct Link<T: Type> {
  hash: Multihash,
  valid_link: (fetch(hash) : T)
}

User Magagement

Desirable Properties

• Flexible Provenance
  • A user should be able to choose how strong the link between the data they publish and themselves is. (i.e. optional / zk proof / ring signature signing)
• Flexible Privacy
  • A user should be able to choose to a reasonable extent who can see the data they publish. (i.e. one-to-many group-based publishing)
• Multi-device support & Realistic recovery mechanisms from compromised devices.
  • If a user looses access to their devices or a mechanism of authentication (password) or device gets compromised, there should be realistic mechanisms to recover the damage both to the public and privately.
• Anonymous Collective Feedback
  • For mechanisms of collective feedback (like/dislike counters, analytics), the data associated with the feedback event should not in general be attributable to a particular group, but should be verifiable that it was a unique member of the group that did the feedback.
• Flexible Storage Permissions
  • Encrypted data associated with a user, stored across many different computers, should support permission hierarchies based on the sensitivity of the data. I.e. requiring more factors of authentication.
• Web-of-trust Identity
  • Data associated with a user's identity should be vouchable by other users (in a web-of-trust) or by mechanistic processes. The identifying data should

Inspiration

• Scuttlebutt
• Polycentric

Identity

Identification is needed when one party must store and retrieve data associated with another party for the purpose of specific interaction.

Formally, identification is the process by which one party (identifier) under some assumptions internally associates some external stimulus with the known identity of another party, thus changing the identifying party's behavior.

For Example:

• A service (identifying party) must identify a user (identified party) using a username and password (external stimulus) to allow the user to log in (changed behavior).
  • Main Assumption: The user's password or computer they are using has not been compromised.
• An individual must identify a fellow human being using their senses before treating them not as a stranger.
  • Main Assumption: The fellow human is not attempting to disguise themselves as someone else.
• When receiving an encrypted message from a friend, the receiver must identify the friend by verifying the signature of the message using the friend's known public key.
  • Main Assumption: The public key truly does belong to the friend and the friend's private key has not been compromised.

Design

In Dither, there will be two general categories of identification.

• Cryptographic identification
• Characteristic-based identification.

Cryptographic identification will occur for when one needs to identify an party on other other end of a specific communication channel.

Characteristic-based identification will occur when one needs to find publically-declared associated data (i.e. public keys) of another party given only knowing certain characteristics of the other party.

Disp

The key to decentralization is effective coordination of collaborating computational components.

Disp is a programming language where programs, types, and the type checker itself are all represented as regular programs in a self-reflective combinator calculus called the tree calculus. It is being developed to solve some of the fundamental problems in Dither's development, and in doing so should solve programming language design, subsuming all other languages. (Yes I know, big claims, I promise this makes sense)

The problems to solve are:

• For a distributed system you need to be absolutely sure your code has no bugs and never will. To do so you need formal verification against the most rigorous set of bug-denying constraints you can come up with, ideally with a compiler that is itself verified in the same way.
• We need a language that can allow users to prove arbitrary equivalences and for these equivalences to be actively applied as optimizations, ideally directly to assembly.
• For a language to not ossify over time and new ideas/designs to be able to be invented and be automatically safely adopted new designs need to be able to be formally proven to apply and safely rewrite old code dynamically at each individual user's desire.
  • As a subpoint here, the surface level details of the language should be user-customizable and fit to their preferred language syntax styles, whether that be block coding, ML-syntax, C-style, pythonic or what have you.
• In order to create things fast, it is generally infeasible to spend copious amounts of time worrying about specific algorithms and it would be ideal if programmers could simply write constraints and have an optimizer satisfy them. i.e. programming-language-native program synthesis, a.k.a. specification-driven development.

The Core Idea: Types as predicate functions

The Core™ idea of disp is that type systems (the feature of compilers that handle whether or not a given program you've written fits some constraints) should be definable in the language itself. Thinking about type systems for a second, you essentially have a program in the meta-language that looks like this: check(term: Term, typ: Typ) where Term and Typ are some special datastructures. If you think about this though, specifically the case where you partially-apply typ to check, you get a function that just checks if a given term is of a particular type. But what if you could just define this as its own function?

This is what disp does and it makes it so that types are fundamentally first-class objects in the language, i.e. simply just functions that inspect some encoding of some data, and return true or false. More on this in Universal System of Types.

The key part to make this happen being "inspect some encoding of some data" and this is where the backend combinator calculus disp is built on comes in. It's called the tree calculus (Invented by Barry Jay, with development help by Johannes Bader) and it allows you to (similar to quote in lisp) directly inspect arbitrary trees (which can include datatypes and functions).

The Solutions

Each of the problems above can be pretty easily solved downstream of types-as-predicates plus the tree calculus:

• Formal verification: A type can encode any constraint you can compute, up to and including full specifications ("this function sorts its input"). And because the type checker is itself disp code, it can be checked by the same machinery it implements, instead of being a pile of trusted compiler internals.
• Provable optimization: Equality between programs is just another type, so "these two programs always produce the same result" is a proposition you can prove, package up, and share. Combined with hardware modeling to judge when a rewrite is actually faster, optimizations become a library anyone can contribute to rather than a compiler release.
• No ossification: Since the type system is a library, new disciplines can be adopted (or not) per user without forking the language. And since programs are nameless trees identified by hash, code deduplicates naturally across Dither, names and syntax become personal rendering layers on top, and proven equivalences let new designs safely rewrite old code.
• Program synthesis: Partially apply the type checker to a specification-type and you get a function that accepts or rejects programs. Add scoring functions for speed and size and you have exactly the thing an optimizer can search against.

Current State

There is a working prototype at github.com/libdither/disp. A small TypeScript runtime evaluates trees; everything else (the kernel, the standard types, the test suite) is written in Disp itself. See Implementation for what exists and what doesn't yet.

Inspirations

• Tree calculus (Barry Jay) for a substrate where programs can inspect programs.
• Lisp for treating programs-as-data in the first place.
• Idris, Coq, Agda, Lean & Friends for dependent types.
• Unison & IPFS for content-addressed, deduplicated code.
• Rust for its safe & fast zero-cost abstractions.

Universal System of Types

a type system is a logical system comprising a set of rules that assigns a property called a type to every "term"

Who chooses the rules that assign types to terms? What makes one set of rules better than another? Some type systems are more general than others, some are more easily compiled into efficient bytecode, some allow for writing complex mathematical proofs. Exactly how large is this design space?

Disp captures the entire design space of possible type systems by generalizing the concept of types:

To make a long story short, a type is simply a program that returns true, false, or loops forever when given the source code of another program.

This allows the programmer to define their own type system in Disp. Or just use a pre-made one.

The Implementation

In Disp, every program is a tree, and trees are data that other programs can inspect. So a type can literally be a function: give it a program, and it tells you whether that program belongs to the type. Bool is a function that accepts exactly two trees (true and false). Nat accepts the trees that represent counting numbers. Checking x : Nat means running Nat on x and seeing what comes back.

The tricky part is function types. To check that some f has type Nat -> Nat, you can't just run f on every number (there are a lot of them). Instead, the checker mints a hypothetical number: a sealed token that answers "I am a Nat" when asked its type, but reveals nothing else. It feeds this token to f and watches what happens. If f tries to peek inside the token, checking fails. If f produces something that checks as a Nat anyway, then f works for any number, because it never depended on which one it got.

Since types are just programs, things that would normally require new language features come for free:

• A dependent type (a type that depends on a value) is just a function that returns a type.
• A refinement like "even numbers only" is just a stricter checker.
• A proposition is a type whose inhabitants are proofs, so the same machinery checks programs and theorems.

The standard library comes with all the familiar types (Bool, Nat, function types, pairs, equality, etc.), but none of them are built in. They are ordinary definitions, and you can read them, swap them out, or write your own.

The mechanics of how checking stays sound (what the sealed tokens are, and what programs are allowed to do with them) are sketched in Implementation.

Syntax

Disp currently has one concrete syntax, and there isn't much of it. A .disp file is a sequence of definitions, tests, and imports. Here is a representative chunk:

open use "lib/prelude.disp"

// The identity function. {x} -> x is a function literal.
id := {x} -> x
test id t = t

// A typed definition: addition on natural numbers.
add : Nat -> Nat -> Nat := {n, m} -> nat_rec ({_} -> Nat) m ({pred, ih} -> succ ih) n
test add 2 3 = 5

The pieces:

• name := expr defines and exports name. let name = expr defines something private to the file.
• name : Type := expr attaches a type. The annotation is not just a comment for the reader, it makes the compiler run the type (which is a program) against the definition when the file loads.
• {x, y} -> body is a function of two arguments. Binders can carry types, as in {x : Nat} -> body, and a function type is written the same way: {x : Nat} -> Nat, or just Nat -> Nat when the result doesn't depend on the argument. This one form covers ordinary functions, generic functions, and dependent types.
• test expr = expr is an assertion checked every time the file loads. The test suite for the language is mostly files like this.
• use "path" loads another file as a record of its exported names; open dumps those names into scope.
• { a : Nat, b := double a } is a record type with a derived field: give it an a and b is filled in for you.

There is also match for case analysis, if/then/else, and a hole marker _ for "let the compiler figure this out". That is most of the language. There are no keywords for classes, interfaces, modules, or macros, because records and functions end up covering those jobs.

Fair warning: the syntax is still in flux. It exists to get trees into the computer, and the trees are the real program, so syntax decisions are intentionally low-stakes. The authoritative grammar lives in SYNTAX.typ in the repo.

Syntax Agnosticism

Syntax is one of the most visible barriers separating different languages from each other. It is easy to distinguish between Lisp and C, BASIC and APL, Haskell and Fortran. All these languages can pretty much do the same things, but their different syntax plays a big role in preventing programmers experienced in one from trying out another. Syntax is also where language communities waste the most energy: arguments about braces and keywords (bikeshedding) stall features that everyone otherwise agrees on.

Disp's position is that syntax is not what a program is. A Disp program's identity is not its text. Source text is parsed into a tree, and the tree is the program: it is what gets type checked, what gets evaluated, and what gets hashed for naming and sharing. Format the same definition differently, or rename every variable in it, and you get the same tree, byte for byte. The text is just one rendering.

Today Disp has just the one concrete syntax described above, so in practice you are looking at one particular rendering. But nothing downstream of the parser knows or cares what the source looked like, which leaves the door open for things that are very hard to retrofit onto text-based languages:

• Alternative grammars. A different surface syntax that parses to the same trees is a frontend, not a fork. Code written in one style stays usable from any other.
• Personal rendering. Since names live outside the trees, your editor could show you the same library with your preferred naming conventions, or in your native language.
• No more bikeshedding. When people disagree about how something should look, both spellings can coexist as long as they mean the same tree. Defaults can be picked (and re-picked) by something like persistent voting without breaking anyone's code.

The eventual goal is for Disp code to be stored and shared as trees, with text generated on demand in whatever style the reader prefers. Switching between styles should be as simple as a click of a button.

Names

A Natural Idea

A word in a language is a label to which we assign a definition. These definitions may be physical or abstract, but in a programming language, a definition is a piece of code.

id := {x} -> x - We are fitting the label "id" to the program {x} -> x.

Words in natural languages may have more than one definition, yet in conversation we usually intend only one of them, and the listener works out which from the context. If we are talking about garnishing food, you mean adding something, whereas if we are talking about garnishing wages, you mean taking something away. Nobody has to announce which dictionary entry they intend.

Contexts also exist in programming languages: namespaces, modules, classes. But unlike real conversation, the computer can't just "figure it out" (yet). The programmer has to spell out exactly where in a meticulously organized hierarchy of modules every name lives. This puts a special strain on the programmer, who must keep that whole pre-defined hierarchy in their head just to program effectively.

The goal: remove this burden by resolving names from the context of the program, the way a listener does.

Structure

In Disp, names and programs are kept strictly separate. A program compiles to a tree that contains no names at all: variables are compiled away into the tree's structure, so two definitions that differ only in naming (or formatting, or syntax style) produce the identical tree. The tree's hash is the program's true identity.

Names are then just labels that people attach to hashes. A .disp file is, in effect, a record mapping labels to trees. This has a few nice consequences:

• The same program may be named differently by different people, in different styles or different human languages, without anyone forking anything.
• Renaming is free. It touches the label layer, never the code.
• When code is shared over Dither, identical definitions deduplicate automatically, no matter who wrote them or what they called them.

Resolving from Context

A label-to-hash map is the simple version. A perfect way to model the richer version is a knowledge graph: a name may resolve to many different programs, and the scope of possibilities is narrowed down by the surrounding context (the types in play, the file's imports, specifically declared contexts), much like a conversation does. Type information makes this more tractable than it sounds, since most candidate meanings of a name simply won't fit where it is being used.

Programs from Specifications

The long-term goal behind Disp: I want to be able to synthesize the best possible program given a specification.

Unpacking that sentence: a specification is a type, and in Disp a type is a program that checks other programs. So the type checker can be turned into a function that takes a candidate program and returns 1 or 0. Combine that with functions that measure other things you care about (speed, memory usage, code size) and you have a scoring function: a single program that looks at another program and tells you how good it is.

Once "how good is this program?" is itself a program, writing software can be turned inside out. Instead of writing the program, you write the scorer, and an optimizer searches for a program that scores well: one that is formally correct (the type checker accepts it) and efficient along whatever axes you measured.

For this to work, a few things have to hold:

• Programs must be data, so that the scorer can inspect candidates and the optimizer can build them. This is what tree calculus gives Disp.
• The specification language must be strong enough to say what you actually mean. That is what the dependent type system is for: "a sorting function" can be specified precisely, not just "a function from lists to lists".
• Performance must be measurable without guesswork, which is what the rest of this page is about.

Provable Optimization

The whole business of computing faster in general comes down to finding new structures and algorithms that contain the same information as the old ones, but can do certain computations faster.

For example: multiplying by 2 can be done two ways on a CPU, through the multiplication instruction (slow) or bitshifting (very fast!). Same answer, different cost. A bigger example might be swapping one data structure for an equivalent one with better O(n) behavior, which only pays off past a certain dataset size. Real programs are towers of choices like these.

Today, these substitutions happen in two ways. Compilers apply a fixed bag of rewrites that their authors hand-picked and trust. Programmers do the rest incrementally by hand, using benchmarks as a form of proof. Ideally all optimization could be done by the compiler, but that compiler would have to have knowledge of every possible implementation of every algorithm and when each one wins.

Disp aims to solve this by making optimization a library instead of a compiler internal. Since Disp programs are data, and equality between programs is something the type system can state and check, an optimization can be packaged up as: here are two programs, here is a proof that they always produce the same result, and here is the evidence about when one is faster. Such a package can be:

• applied manually, like importing any other dependency,
• found automatically, by a compiler searching the library for rewrites that apply to your code (with results recorded for subsequent runs),
• shared safely, because the equivalence proof is checked on arrival. You don't have to trust whoever sent it.

The "when is it faster" half is the harder part, since benchmarks are noisy and machine-specific. That is what hardware modeling is for.

Hardware Modeling

How do we generate fast code?

Optimizing compilers generate fast code through intermediate representations and various techniques for figuring out what code is needed and what code is not. These strategies are mostly arbitrary and not easily improvable: a programmer has to go through and figure out which optimizations are worth the compiler computation cost and which aren't, and there are an immeasurable number of trade-offs and potential improvements for any given piece of hardware.

How do we create fast algorithms?

We typically write algorithms we think will run fast and then compare them to other algorithms using benchmarks. However, benchmarks have so many confounding factors that it is difficult to compare two of them and make definitive conclusions. The same algorithm may run faster or slower depending on a bazillion different factors, down to the layout of the compiled object file the compiler happened to pick.

To create efficient code, we must have a model of how that code is run. Typically this model is held in the programmer's mind and expressed through compiler and algorithm design. Instead of holding it in our forgetful, biased brains, it might be a good idea to have a software-defined model of our hardware, so that we can prove (or at least have really good heuristics that check whether) our optimizations and algorithms are actually faster.

What such a model should account for:

• Modern CPU features: pipelining, superscalar and out-of-order execution, branch prediction.
• Differences between generations of the same architecture.
• Everything else that moves the needle: RAM latency, cache behavior, GPU parallelism.

Simple models could be hand-built from published data like Agner Fog's instruction tables. For open hardware (RISC-V, say), more sophisticated models could be derived from the actual logic design.

Benefits of this approach:

• No more annoying benchmarks. A faster algorithm can be provably faster for a given model. (Benchmarks still get used to compare different models of a hardware system, but those benchmarks can be much more sophisticated and account for more confounding factors.)
• Faster optimization adoption. With models and equivalence proofs, optimizations can spread through a public codebase and pushed to users without a compiler release in between:
  • Bob finds a new optimization for a certain pattern of expressions.
  • Bob proves it is faster for a well-trusted model of x86_64 CPUs.
  • Bob broadcasts the optimization to everyone.
  • Everyone's computers check the proof.
  • Everyone's programs get faster.

Closing the Loop

The really interesting part is what happens once all of this feeds back into itself. The type checker is a Disp program, so it too can be scored and optimized. The optimizer can be pointed at its own components. The hardware models can be refined by the very search they accelerate. And a large enough library of proven-equivalent programs is a perfect dataset for training an AI to find new equivalences. Every piece of the system is a candidate for the system to improve, which is why so much of Disp's design is about keeping everything (checker included) inside the language rather than bolted onto it.

This is all a long way off, to be sure. What exists today is the foundation it requires: a calculus where programs are data, and a self-hosted type checker on top of it. The optimizer is future work, and an external search process (combinatorial, neural-guided, or both) will probably have to bootstrap it before it can take over its own improvement.

Implementation

Disp has a working prototype, developed at github.com/libdither/disp. This page is a snapshot of how it is built and how far along it is. There is also an interactive walkthrough that builds everything below from scratch with runnable examples, though fair warning: it describes an older version of Disp, and it was largely AI-generated, so it reads like it.

The Substrate

Everything bottoms out in tree calculus: programs are binary trees, and there is a single reduction rule for applying one tree to another. A small TypeScript runtime implements this rule, along with a parser for .disp source files and a compiler that turns parsed definitions into trees.

One implementation detail matters a lot: trees are hash-consed, meaning structurally identical trees are stored exactly once, and comparing two trees for equality is a single pointer comparison. This is what makes "same code = same hash" cheap enough to use as the language's notion of identity.

The Kernel

The type system is not in the TypeScript. It is written in Disp, as a small kernel of .disp files, under a strict discipline:

The in-language code is the specification. Host code is only allowed to be an optimization of it.

Since types are programs that check other programs, most of the type system is plain library code. The kernel proper is just the machinery that plain code can't be trusted to build: it mints the sealed "hypothetical value" tokens used to check functions, and runs candidate programs under a watcher that keeps them from peeking inside those tokens. A program that checks out on a hypothetical input is sound for every input, and the watcher is what makes that argument hold. Everything else (Bool, Nat, function types, records, equality and its proof rules) is ordinary library code built on top, and the test suite is itself written in Disp.

What Exists / What Doesn't

Working today: the runtime, the parser and compiler, the kernel, a standard library (numbers, lists, options, results, pairs, sets), dependent records with derived fields, and a few hundred in-language tests, including ones that pin down the soundness boundary by checking that known attacks on the checker fail.

Not yet built: erasing checks from verified code so it runs at full speed, a proper effects story for talking to the outside world, better error messages, and the optimizer. The full design, including the parts that are still on paper, lives in the spec documents in the repo (TYPE_THEORY.typ, SYNTAX.typ, COMPILATION.typ).

Dither Application Index

List of application Ideas

• Dither Chat - Community Chat application aiming to replace Discord. Provides e2ee encrypted DMs, voice chat, servers, voting, and integration with most other chat protocols.
• Dithca - Comprehensive & Versatile decentralized comment system where anyone can comment on any type of data structure on Dither. Can interface with most other centralized comment systems and deal with misinformation & crediting using a comprehensive community flagging system.
  • Can be used to create Reddit / Twitter Replacement. Can be integrated into other Dither applications or ported to web.
• Dithgit - Github on Dither
• Dithix - Dither Resource Manager: Manage and cache any kind of resource, interface between the Merkle Tree and the Filesystem.
• Nomia on Dither - Nomia on Dither,
• Tree of Math - Directed Acyclic Graph linking a standardized data structure for defintions and proofs together based on set theory creating a comprehensive tree of knowledge.
• Dither Coin → Money - The currency is now a core stack concern, not a standalone application: a portfolio of demurrage-decaying shares in local resource pools, with exchange rates discovered at zone boundaries. (Supersedes the old fixed-supply "Dither Coin" sketch.)
• Dither DEX - A all-faceted decentralized exchange to facilitate trade any kind of real or virtual asset. Supports meeting up in real life or exchanging other virtual assets anonymously and securely. Also supports moderation, karmic filtering
• Protocol of Truth

Other Application Ideas:

• Manga & Reading App
  • UI will be similar to Tachiyomi, but will also support book reading. Pulls content from various websites and stores on Dither. Written in Flutter, desktop & mobile versions. Supports comments through Dithca. Has built-in feature for paying for translation & replacing bad translations with better translations if they are made.
• YouTube replacement
  • pulls and stores videos in a decentralized, uncensorable manner from any site that youtube-dl supports
  • Built-in chat (Using Dithca protocol)
  • Community-generated captions, sections, sponsor segments (pulled from sponsorblock) flagging etc.
  • Support for likes, view counting (and congregating), and "Hearts" (method of giving Dither Coin to creators).
  • Automatic community flagging of stolen/used content (including music, other people's videos, meme origins, pretty much anything)
  • Community misinformation flagging of content (interfaces with Dithca)
• Peer-to-Peer Exchange
  • Allow for the exchange

Dither Chat

• Dither Chat
  • What is it
  • Users & Sync
  • Chat Events
  • Event Storage - Storage of a sequence of events in memory or storage
  • Trusted Friends Application API
  • Chat Interface
  • Direct Messaging
  • Group messaging
  • Servers
    • Roles, Tags & Colors

What is it

Dither Chat is a decentralized communication application using the Dither protocol. Servers are communally hosted with local consensus. Bots and plugins will be supported and also communally hosted. tl;dr Discord but decentralized and better.

Users & Sync

• Each user may have multiple Peers (devices that Dither is installed on)
• Chat Event history can be optionally synced across Peers.
• A peer may host multiple users
• Each UserId must have at least 1 peer that hosts it.

Chat Events

• All Events are signed with the private key of the person who sent it (these will be verified with a config option to let through or ignore unsigned or incorrectly signed events)
• Chat Message structure
  • Date sent, last edited, markdown data / embed json, UserID mentions, emoji reactions
• Rich Presence (updating custom statuses and online/offline status)
  • Optional storage - can store presence update history (off by default)
  • Optional sending, extracts information about what you are currently doing and updates your friends. (on by default)
  • Customization options to only share with certain friends

Event Storage - Storage of a sequence of events in memory or storage

• Stored as hash-linked local blocktree that messages are added to and new blocks are created when a certain amount of time elapses between the last message sent or max block size exceeded. Block size can be set to 1 to prevent messages being ordered out-of-order.
• Indexing can be done on a by-block level (TODO: more customization options needed)
• Block structure can allow for thread branching & thread conversation movement across users. (e.g. create a group dm on top of an ongoing conversation)

Trusted Friends Application API

• Option to rank friends manually or by how much you chat with them
• Can mark friends as “Trusted, Neutral or Untrusted”
• Friend rank can be used by other applications
• e.g. for Stellar Consensus Protocol quorum selection

Chat Interface

• Built-in markdown formatting + advanced chat box (similar to discord)
• Link Displaying
• Metadata can be sent so receiver doesn’t have to send request to web pages
• TODO: Do we need to worry about invalid metadata being sent, tricking the user? Perhaps just scanning for suspicious domains is enough.
• Receivers can choose if they want to fetch link data, only fetch commonly used sites (e.g. youtube, twitter, soundcloud etc.) or not fetch anything at all and only display sent link metadata

Direct Messaging

• Simply sending JSON-encoded message/other events to UserID on Dither

Group messaging

• Messages are broadcast over gossipsub and conflicting blocks are ordered by time.

Servers

• Servers are communially hosted by the moderator's computers. However, the owner has full control over the server and can choose who can assist
• Red Nodes are hosting nodes, blue nodes are members. Blue node with yellow stroke is proxying it's connection to the server Image of a possible dither-chat server arrangement

Roles, Tags & Colors

• To distinguish people in a server, there are roles, name colors, and tags
• Tags are shown next to tagowner's name as a small icon (like in Discord)
  • By default there is only 1 tag enabled for a server: the Owner tag.
  • There are other tag presets such as Donator, Moderator & Member
  • Custom tags can be created and be attached to a specific role.
  • Tags should be displayed in order of importance
• Roles can be made for anyone and can have permissions attached to them
  • Roles and can also be organized into hierarchies where users can be ranked-up to posses a higher role (usually with more permissions)
• Each role has a color and color priority assigned
  • This is used to determine what color a user's username should be assigned

Protocol of Truth

This is a protocol of back and forth debate with the aim to evaluate the quality and truthfulness of content posted to the internet.

If you have ever seen a photoshopped post on social media, or a video citing a badly done study you will know that it is incredibly hard to figure out how well-researched a piece of content is at first glance. This Dither protocol aims to inform through discourse the truthfulness of a given piece of media to anyone who might come across it.

The Process

The process for the Protocol of Truth is carried out by trusted individuals (chosen by a karma system). The steps are as follows:

1. The Media is examined for a set of assertions it makes as well as deductive and inductive reasoning made using the assertions. This creates a "Assertion Graph".
2. Once the graph is constructed and properly cited (i.e. each assertion should provide some link to the part of the media it was extracted from) assertions may be supported or criticized.
3. Assertions in "Assertion Graphs" may be shared between different media and thus the supporting evidence presented by one media may be used in the justification of another. This supporting evidence may be found independently or through the author's cited sources.
4. Each Assertion in the graph may be labeled through a persistent voting scheme under the following categories of deduction type, truthfulness and standard of evidence.

• Deductive / Inductive - What kind of assertion is it.
• Truthiness - How the trusted community rates the assertion in terms of truthfulness.
  • Inherited - The assertion is deduced from the assumptions, therefore its truthfulness is inherited from the truthfulness of the assumptions.
  • Likely True - The assertion is likely true
  • Likely False - The assertion is likely false
  • Ambiguous - The truth of the assertion cannot be satisfyingly asserted either way.
• Standard of Evidence
  • Statistically Significant - The assertion is supported by sound scientific observation or experiment.
  • Personal Experience - The assertion is a personal observation, it is subject to individual bias.
  • Common Knowledge - The assertion is generally held to be true, but there is no specific evidence to support it. i.e. stuff like "thing X exists" or "hammers are generally used to hammer in nails". There is no hard evidence for it, but it is generally held to be true. This is the least powerful standard of evidence and should be replaced with a more powerful form if at all possible.

Fractional Funding

This idea aims to solve the issue of effectively funding peer-to-peer creative endeavors.

Rough Description

One really big idea which I think would improve the content economy (i.e. remove dependence on ads or sponsorships and make it more feasible for small creators to make a living) would be to have a system similar to YouTube premium where you pay a monthly subscription and the money gets distributed among the creators you watch, but instead of it being distributed among everyone you watch, its distributed only to people who make videos that you click a certain button on (lets say its like a heart-shaped button next to the "like" button), and you can click it multiple times to give multiple "shares" of your monthly subscription. Then these shares are distributed to all these videos at the end of the month, but instead of simply going to the uploader of the video, there is a system that tries to identify either via user feedback or AI or something else parts of the video/content that were made by the original uploader, or that where taken from elsewhere to create a "fair" monetary distribution for that video. (i.e. to pay the artists and thumbnail drawers and anyone else) and then depending on each user's preferences the money from each video can either go to everyone equally (weighted by distribution) or they can add further weights to prioritize people who have less income or things like that.

Formal Description

Users

• People who consume content and set aside some fixed amount of money to give to creators.

Content

• Pieces of data (videos, images, writing, etc.) by credit for its creation can be proportioned to some set of humans.

Creators

• People who upload/remix content.

Goal: Take money from users and give it to creators in some fair manner.

Idea for Process to achieve Goal:

• Each user watches a video and gives to it some fixed number of shares of monthly money they've set aside.
• At the end of each month the user's preferences on how money should be distributed for each video is sent to creators.

The Decentralization Stack

🚧 Draft, and openly a rabbit hole. One more thing, up front: 🤖 nearly every document in this section was drafted by an AI collaborator, working from the author's designs and direction — read what that means before trusting a fluent sentence. The math lives in the Reference; open problems are marked ⚠️ next to the claims they qualify; the rawest new material is in the working notes.

"If you wish to make an apple pie from scratch, you must first invent the universe." — Carl Sagan

We wished to make a chat app.

That is not a framing device. The oldest design document in this repository, written around 2020, describes the whole ambition in one line: "tl;dr Discord but decentralized and better." The history file next to it has three year-headings and a single bullet point that says "Conception," which tells you roughly how the last few years have gone. This book is what we found at the bottom of that one-line plan, arranged so that you can climb down with more dignity than we did.

The descent

Picture the community you'd actually build this for. Ours, for the sake of the book, is a knitting circle: forty people who argue about cable patterns with the intensity of a border dispute, five years of running jokes, and a pinned message in which Sarah finally concedes that the scarf was cursed. All of it lives in a Discord server, which is to say: all of it exists at the pleasure of a company that has never heard of these people and would not notice their disappearance. One policy change, one bad quarter, one wrongly-banned moderator, and the whole shared memory of the place is gone. You have probably watched this happen to a community you loved. Most people online have, by now.

So you decide to build them something better. How hard could it be?

The first attempt is the obvious one: rent a server, run the chat yourself. Congratulations — you are now the company. You pay the bill, you hold the logs, you can be subpoenaed, and the community's five years of history now has a bus factor of exactly you. Federation (the Matrix move) improves this less than you'd hope: a smaller landlord is still a landlord, and "trust your homeserver admin" is a sentence with the word trust doing regrettable work.

The second attempt is real peer-to-peer, messages living on the members' own machines, and this is where the trapdoor opens. Each problem you solve turns out to have been standing on another one:

1. Laptops sleep. For the archive to survive the night it has to live on machines that aren't yours — which means strangers have to hold the knitting circle's data.
2. Strangers will not store five years of scarf discourse out of love. You have to pay them. Now you need a market for storage, and while you're at it bandwidth and compute, because a chat app with search needs all three.
3. Pay them with what? A card processor is the somebody-in-charge again, wearing a fee schedule. A token is whales and volatility. You need money that works here, which turns out to be its own abyss.
4. Before any of that: the free tier, the spam, the vote-kicks. What stops one bored teenager from being forty enthusiastic new members? You need to count people, and there is no passport desk.
5. "Anonymous" was in the original pitch, so the network can't leak who talks to whom — the routes themselves have to keep secrets.
6. That stranger holding the archive: is she actually holding it? She says the search index she computed is correct. Is it? You need to check work you didn't do, done by people you'll never meet.
7. Moderation. Somebody posts something awful, or merely wrong, and the group needs a way to decide what it believes without appointing a mod-king — because the mod-king is the somebody-in-charge again, at the smallest possible scale, and he is tired.
8. The group fund, the rule changes, the eternal argument about whether the crochet people may join: deciding together, without the decision being buyable.
9. And someday — there is always a someday — the schism. Half the circle storms off to found a rival network. What happens to the shared history, the shared money, the shared reputation? Forks and mergers need to be survivable, which means they need to be designed for.

Somewhere around the fifth item you have stopped building a chat app. What you are building is a small economy with an immune system and a shared memory — the least evasive word for it is an organism. We know how that sounds. The rest of this book is us earning that sentence slowly.

The bet

There is one precedent for strangers cooperating at scale with nobody in charge, and it has been in production for four billion years. Biology's trick is specific enough to copy: ship shared priors, refine them at runtime, select between whole systems. Every cell carries the genome — the agreements pre-loaded before any interaction. Lifetime learning tunes behavior against evidence. And selection between organisms retires the priors that stopped working. Nothing decides; everything is decided anyway.

So that is the design: every institution in this book — the market, the money, the identity machinery, the governance — is a shipped prior, a good default with an explicit update path, not a claim of correctness. The network pays its participants to refine the priors against reality (that's the market), refines deliberately what evidence can't settle (that's governance), and keeps exit cheap so better-run networks can win (that's evolution, and it is load-bearing, not decorative).

The walls from the descent, and where each one falls:

Five ways it dies

Everything below is written against five failure modes — formally, viability conditions V1–V5 (Mathematical Core §7). In plain words: it starts believing itself (forecasts feed the verdicts that score them, and the shared model detaches from reality); one actor wears a thousand faces (everything that counts gets counterfeited); wealth pools until the richest set the truth; leaving gets too expensive to matter (and with exit dies the selection pressure that keeps any of this on course); and it mistakes correlation for cause and acts on it. Each chapter will tell you which of the five it is defending against, because knowing exactly how your design dies is most of designing.

How to read it

Front to back; the book grows the thing from one node to two to an economy to something that can fork and survive it. The knitting circle will keep showing up — them, the friend with the NAS in her closet, and Dana, whose relationship to voting will become mathematically interesting around chapter 6. If you want theorems instead of yarn, the Mathematical Core is the same book with the jokes removed, and the roadmap, glossary, open questions, and working notes round out the reference shelf.

One structural confession before we start: there is no clean foundation under all this, no single primitive the rest bolts onto. The nearest thing to a root is a habit — prediction — and the nearest thing to a keystone is the correlation machinery of chapter 5, which is maintained by the running system rather than installed before it, the way a cell membrane is maintained by the metabolism it makes possible. Books about towers get to start at the ground floor. This is a book about an organism, so we start with a cell.

🤖 About These Docs

This notice is itself AI-drafted. It seemed only fair.

Every document in the Decentralization Stack section that carries the 🤖 marker — which, at the moment, is all of them — was drafted by an AI collaborator (Anthropic's Claude), working from the human author's designs, critiques, and direction across many iterative sessions.

The division of labor, as accurately as we can state it: the human supplied the goals and tenets, the core ideas (the currency-as-local-shares intuition, the predictive-memory idea, the zone-merging crux, the protocols-as-priors reframe, among others), every architectural decision, the critiques that drove each rewrite, and the final call on what ships. The AI supplied synthesis, structure, formalization sketches, connections to existing literature, and most of the actual sentences, including the jokes.

Why tell you before you click: fluent prose can smuggle unverified claims, and these documents are fluent. Several results are stated with proof sketches that no referee has checked. Several mechanisms are design candidates, not commitments. Citations to real literature (session types, consensus numbers, scoring rules) were made from the AI's training knowledge and spot-checked, not systematically verified. An AI's confidence and a claim's reliability are uncorrelated exactly at the margins that matter, so read the ⚠️ blocks as load-bearing rather than decorative.

The full marker legend for this section:

Documents outside this section — the dither/ routing designs, the disp/ language docs, the application sketches — are largely human-written unless individually marked.

If you find an error, that is the system working as described; the whole design is about paying for error correction. Until the machinery in chapter 3 exists to pay you properly, consider filing an issue at libdither/dither-spec an act of unpaid forecasting.

1 · The Agent

🤖 AI-drafted, human-directed — what that means.

Strip the problem down to one machine: yours. Before the knitting circle can have a shared archive, your node has to be able to do two things entirely alone — remember, and check. That sounds too small to be a chapter. It's the chapter everything else stands on, and the place where we can introduce, in miniature, the single habit the whole book keeps returning to.

Here it is. An agent is a predictor with a boundary, and it survives by running one loop:

Spend resources to reduce uncertainty about something, until you are confident enough to act — where "confident enough" scales with what's at stake.

Remembering is this loop pointed at "what were those bits." Verifying is the loop pointed at "is this result real." Later chapters will point it at stranger targets — who am I talking to, what will storage cost here next month, what will careful people eventually conclude about this claim — and each time it will be the same loop wearing different clothes. If the book has a thesis, it's that this observation can be taken embarrassingly far.

Certainty has a price list

Say a peer hands your node a computed result — a search index over the archive, a rendered preview, the resolution of some heavy query you didn't want to run yourself. On today's internet you'd believe it because of who sent it; the logo does the verifying. Alone, with no logos available, your node has to ask a different question, and it's a budget question rather than a trust question: how much is it worth spending to be how sure?

Because checks come at every price point, and they form a single curve of certainty-per-cost:

• A sniff test. Does the result look like what a cheap local model expects? Nearly free, and worth nearly what you paid.
• A predicate check. disp types are predicates on results, and checking is usually far cheaper than computing (the whole of NP is built on this asymmetry). Cheap, strong, the workhorse.
• Sampled re-execution. Ask k independent executors. Undetected fraud now requires every one of them to collude on the same wrong answer, so confidence compounds like (1−h)^k — exponential certainty at linear cost, which is the kind of deal you should distrust and this time is real.
• Bisection. Two executors disagree; reduction traces are Merklizable; a referee finds the first wrong step in O(log T) checks. The expensive path, walked only on disagreement.
• Full re-execution. The top of the curve. Always available, rarely worth it.

The classical name for shopping along this curve is sequential hypothesis testing (Wald, 1945): keep buying evidence while the marginal value of certainty beats its marginal cost, then stop at a threshold set by the stakes. A signature on a meme gets the sniff test. A signature on the deed to the group's treasury gets the top of the curve. And some claims never reach threshold at any price you're willing to pay — those get served as uncertain, or not at all, which will matter more than it seems when we reach moderation.

What makes the curve's exact end reachable at all is disp. Programs there are content-addressed trees — the hash names what a program is, not where it lives — and computation is deterministic, confluent reduction, so two honest executors agree bit for bit, with "number of reduction steps" falling out as a machine-independent unit of work. (File that last fact away; it becomes the gas meter of chapter 3.) Exactness exists. The loop's job is deciding when it's worth paying for.

Memory is a policy, not a warehouse

Now the other solo job. The naive picture of storing the archive — bits in a file system — turns out to be one point in a space of options, and not always the best one. When your node "stores" something, what it actually commits to is a policy for materializing that value later, and policies trade off four things at once: the bits you hold (rent, paid over time), the compute you'll spend reconstructing (paid at request time), the latency you can tolerate (paid in impatience), and the fidelity you actually need — which depends on the use, not the data.

That last axis is the interesting one. Consider the options for a chunk of the archive:

The last row is new, and it's where this design parts company with ordinary storage systems. A model of the knitting circle's five years of chat can regenerate most of any given month — the running jokes have structure, Sarah's scarf saga follows an arc — and the parts it can't predict are precisely the residual worth paying raw-bit prices for. Which policy wins is not a global choice; it's decided pointwise by demand. The pinned messages stay raw. The archive's middle years live as model-plus-residual. The ambient texture of the place can live in the model alone, served with a confidence label, and for "what was the vibe of March 2022" a labeled approximation is the right product.

Retrieval, under this picture, is the certainty loop again: a request is a partial context, and the node either completes it above the caller's confidence threshold or escalates toward exact bits. Hash-addressed data isn't a different system from generative memory — it's the same system at the infinite-certainty end, kept for what justifies the cost.

One rule here is important enough to state as law, because everything downstream depends on it: a generated completion never silently substitutes for canonical bits. The confidence label is the entire difference between memory and hallucination. Drop it and you haven't built lossy compression; you've built a machine for corrupting archives politely.

⚠️ What we can't defend yet: disp is a working prototype (kernel and elaboration stages 0–3 self-hosted; effects, erasure, and the optimizer pending), and its networking story is the next chapter's typed-sessions program, which is presently a design note with theorems where the code should be. The economics of generative memory — who trains the models, how staleness gets priced — waits on chapter 3, and the privacy regression it causes (a partial context confesses far more about you than a hash does) waits on chapter 2's transport. Risks and details: the predictive-materialization note.

📐 Formal treatment: Mathematical Core §1, §10.4.

So: one node, alone, that can remember an archive and check a stranger's work. We set out to build a chat app and so far have built a machine that audits its own memory — which is progress, actually, though it doesn't feel like it. The next thing it needs is the thing no amount of solo cleverness provides: somebody else. That's chapter 2.

2 · Coupling

🤖 AI-drafted, human-directed — what that means.

Enter the friend with the NAS in her closet. Every peer-to-peer project has one — the person whose spare machine and misplaced faith carry the whole early network — and ours arrives now, because your node has hit the wall that ended chapter 1: laptops sleep, and an archive with one copy is a countdown.

Nobody is in charge of the pair of you. So why does cooperation happen at all, and what do you actually send each other? The first question has a boring answer with interesting consequences: cooperation happens because it has surplus. She holds your residuals overnight and you hold hers; her node is good at maps and yours at code; her closet is physically near data you keep fetching. The second question has a four-part answer, and the four parts turn out to be seeds of everything else in the book:

• Capability advertisements — "here's what I can predict, serve, and check, and roughly what it costs me." The seed of routing, and of the market.
• Attestations — signed receipts: "I saw the hash of your message at my clock time." The seed of ordering, timestamps, and (give it three chapters) the entire notion of before.
• Credit — a running IOU ledger: you owe her three map-lookups, she owes you a night of backup. The seed of money. The exchange rate you settle on — lookups per gigabyte-night — is barter, which is what every exchange rate is before it grows up.
• Fate — standing claims on each other: stakes, commitments, skin in the game. The seed of zones, pools, and chapter 4.

Notice the stance. Nothing here is mandated; the design's whole posture toward cooperation is make it explicit, priced, and reversible — cheap to enter where surplus exists, cheap to leave when it stops. You do not defeat Moloch by ordering people to cooperate. You starve him, by making defection and exit boring, legible, and affordable.

Couplings you can typecheck

"Explicit" can be taken literally. A coupling between two nodes is a protocol agreed in advance, and a protocol can be a type — a session type, declaring who may send what, in what order, under what obligations. This is mature theory rather than speculation (multiparty session types: well-typed participants provably follow the protocol, never deadlock, always make progress), and disp is a suspiciously good host for it. A protocol spec is a content-addressed tree, so protocols have hashes and versions diff like anything else, with no registry to capture. disp types are predicates, so a message slot can require things — "a transfer, signed by the sender, whose balance covers it." And since every message cites the hashes of what preceded it, a protocol run leaves a tamper-evident transcript, which means disputes about behavior get settled the way chapter 1 settled disputes about computation: bisect the transcript, find the first ill-typed step, done.

A contract, in this world, is a session type plus collateral plus an arbitration clause. What makes a protocol good has a four-part answer we develop in a working note — sound, incentive-compatible, self-measuring, evolvable — with one caveat worth carrying everywhere: types give you safety, but only incentives give you liveness. A counterparty can always go silent. Only stakes make silence expensive.

Receipts become time

Here's the quiet magic trick in that second message type. Once the hash of your record appears inside your friend's signed messages, and hers inside third parties', your record is sandwiched: it provably existed before everything that cites it and after everything it cites. To backdate anything you'd have to rewrite the signed histories of every independent witness in the web — and there stands the whole timestamping design. No blockchain. Just gossip, signatures, and causality, with precision set by how fast receipts circulate.

Payments ride the same insight further than you'd expect. A payment from a single-owner account needs only the sender's ordering of their own spends — that's all double-spending prevention is — and the sender can just provide it. This is a theorem (consensus number 1; Guerraoui et al., 2019), not an optimization: reliable broadcast suffices, and the entire apparatus of global consensus turns out to be unnecessary for moving money. The one thing that genuinely needs strangers to agree is contested allocation — two buyers, one name — and that's rare, local, and small enough for a modest quorum drawn from the neighborhood.

From pairs to a neighborhood

Run the coupling move a few more times and institutions start arriving on a schedule, none of them designed, all of them implied:

• Four nodes. Your friend vouches for her cousin — the first transitive trust. The pairwise IOUs start netting against each other — the clearing function is born, unnamed. Three nodes can now witness for two — the first quorum.
• Eight. The first true stranger shows up, and for the first time it matters whether "three enthusiastic newcomers" are three people or one bored one — Sybil pressure begins, and with it everything in chapter 5. One node ends up in the middle of most trades simply by being central, and the hub is born, useful and dangerous in equal measure. Unanimity stops scaling. Voting appears, and immediately misbehaves in ways we'll spend two chapters on.

A cluster that couples densely — trading, witnessing, vouching, netting — earns a name: a zone. Pool where the coupling surplus is high; deal at arm's length where it's low; the boundary falls where the surplus does. Hold that thought through chapter 4, because the boundary is where money changes, in both senses.

Underneath all of it sits one shared piece of geometry. Nodes embed themselves in a latency coordinate space — near in the space means cheap to reach in the world — and this one map gets reused so relentlessly the book calls it the body map: quorums are drawn from it, currency zones live on it, newcomers are vouched into it, and chapter 5's statistics lean on it (correlation between neighbors is expected; correlation between strangers is a tell). Transport across the map is onion-wrapped, each relay learning only its neighbors, so fetching the archive doesn't confess your interests to whoever watches the wire. And retrieval composes with chapter 1's memory: a request is a partial context, each hop moves it toward nodes advertising lower expected loss on that kind of context, and the exact-bit fetch is the expensive last resort. (Descent on an uncertainty gradient, if you want it in one phrase.)

Identity, at last

Now notice what this chapter never needed: a registry. At two nodes, identity is perception — you know her; the NAS is in her closet. The question "who is real" only grows teeth as strangers arrive, and the answer that scales is not a credential but a posterior: each node, and each zone for its own purposes, maintains its own working estimate of how distinct and how trustworthy each counterparty is. What feeds it is a portfolio of evidence streams, individually weak, collectively expensive to fake in bulk:

1. Locality — latency triangulation, physical presence. An attacker can fake nodes; faking nodes everywhere is another matter.
2. Authorizers — and here the passport desk returns, demoted from foundation to witness. A government office, an employer, the knitting circle's founder: each is just an attestor with an auditable track record, weighted like any other evidence.
3. Stake-backed vouching — vouch for a puppet, lose your stake.
4. Behavioral streams — the paid work of later chapters: service patterns, forecast records, transaction rhythms.
5. Continuity — hash-entangled history. Old identities are expensive by construction, which is the property forgers hate most.
6. Hardware — a disk only commits once.

Two properties of this posterior matter more than any stream feeding it. It's stake-scaled: a two-dollar trade needs a shrug of confidence, a treasury vote needs the full portfolio, and you buy exactly the certainty the interaction warrants — chapter 1's loop, pointed at personhood. And it composes along paths: you'll never compute a fine-grained estimate of a stranger nine hops away, and you don't need to; confidence flows through vouching chains with attenuation, exactly the way exchange rates will flow through trading chains in chapter 4. Identity confidence, trust, exchange rates: one family of objects, fields on the body map, composing along routes. The book will lean on that rhyme more than once.

⚠️ What we can't defend yet: the quorum design for contested allocation (member selection, cross-zone disputes) is unwritten; the relay-incentive game — who pays whom for carrying traffic — waits on the market; timestamp precision under real gossip is unmeasured (reviving the old dither-sim work is still the cheapest de-risking available, and still undone); and the typed-sessions program over disp is a note with ambitions, not code.

📐 Formal treatment: Mathematical Core §4.4, §10.3; the coupling-and-merging note.

Two nodes, then eight; receipts that became timestamps; IOUs that will become money; a map that will become the skeleton of everything. The circle's archive now survives the night. What nobody can do yet is price anything beyond barter — and the moment a stranger's disk enters the picture, barter stops scaling. A market, then. Chapter 3.

3 · The Market

🤖 AI-drafted, human-directed — what that means.

The circle grows, as circles do, and acquires a video channel — someone's cable-cast tutorials, forty gigabytes and climbing — and suddenly the friend with the NAS is not enough. You need strangers' disks. You also need their bandwidth on tutorial night, and their compute when someone finally demands search over five years of arguments. The obvious move is to build three markets (storage, bandwidth, compute), and if you ask around, that's the standard advice; there are whole projects for each. Then, a little later, you'd discover the fourth market hiding behind them — because "is this rumor about the merino shortage true" and "which client fork deserves the treasury's money" also need pricing, and for those, today, you hire an editor or exhaust a moderator.

We'd like to talk you out of building four markets. It's one market, selling one thing.

Everything sold here is evidence — a verified reduction in someone's expected loss. Compute manufactures evidence (a reduction trace is evidence about f(x)). Storage preserves it. Bandwidth delivers it. Forecasters, whom you'll meet shortly, scout it early. And human judges certify the kind no machine can. Different production processes, one product, and one payment rule shared by all of them: you get paid the amount by which you beat the shared expectation. Keep that rule in view; it's the spine of the chapter, and by the end it will connect a cache hit to a moderation call.

The fast side: one flow problem

Physically, any request the circle makes is MATERIALIZE(value, spacetime-region) — make this value available there, then — and it's satisfied by chaining three kinds of edge:

The same value can be produced many ways — fetch a cached copy, recompute from inputs, or, with chapter 1's generative memory, predict it and correct with residuals, a fourth family of routes whose distinguishing attribute (a confidence label) gets priced like any other. What looks like three policy debates (cache or not? replicate or not? memoize or not?) is one threshold, applied pointwise: hold value v at location x whenever local demand times delivery price exceeds the cost of holding. A CDN, a DHT's replication factor, and a memo table are that inequality wearing three costumes.

Who runs this? Nobody, which is the point, and also you, which is the mechanism. There's no scheduler; there's the owner of each disk predicting where demand will appear and pre-positioning for it. Get it right and you serve fast and earn; get it wrong and you bleed rent on bits nobody wants. And under one pricing rule — delivery priced at the marginal cost of unmet demand — a small theorem does a lot of work: each node's profit from any local action equals the reduction in total unmet demand that action caused. Every selfish caching decision becomes a step of gradient descent on the network's collective error. The price field is the prediction-error field; where the network is wrong, prices spike, and whoever corrects it first gets the difference. That's the fast half of the payment rule above (Mathematical Core §10.1 makes the "one rule" claim precise).

Since generation is now a production route, predictive models are capital equipment, and it's worth splitting them by their economics. Coordination models — the shared predictors of prices, routes, demand that participants must roughly agree on for the market to clear at all — are Schelling infrastructure, like the protocol itself: they want to be open, and funding them is a public-goods problem the coalition should just pay for. Content models — the domain predictors that give a node its edge — can stay private, because the market never needs to see weights; it buys verified outputs, and chapter 1's certainty curve prices the verifying. When should the network buy a private model open? When the estimated value of diffusion beats the buyout price — a question the market's own slow half can price, pleasingly enough.

The slow side: truth

Which brings us to the tired moderator. Someone posts the merino-shortage rumor; someone else posts a takedown; the circle wants to know what it believes, and appointing one person to decide is how you get a mod-king (the smallest possible somebody-in-charge, and the most tired). Scale the same problem up and it's the oldest allocation problem there is: you have money but not expertise, the experts have knowledge but no money, and anyone appointed judge of the truth becomes the most profitable person in the system to corrupt. Science funding, grant-making, open-source bounties, and the circle's treasury are all this problem in different hats.

Prediction markets almost solve it, and each existing form breaks somewhere specific: play-money attracts no serious effort; real-money needs a single, objective, point-in-time resolution, which is a central point of exploitation with money on the line; and real-money aggregation degrades exactly as wealth concentrates, which is exactly when you need it. One root cause underneath all three — resolution is a single irreversible event coupled directly to payout — so the mechanism decouples it:

• Each forecaster $i$ publishes timestamped probability distributions $p_{i,q,t}$ on open questions. Timestamps come pinned by chapter 2's receipts, so "I said this in March" is unforgeable. Forecasters stake information, not money.
• Each resolver $j$ (a capital-holder who wants the answer) may, whenever the evidence ripens, privately declare a verdict $r_{j,q}$ — their own judgment of what happened. Declaring is optional and revisable, and a resolver can simply discard a question that turned out ambiguous. There is no bell that rings.
• A proper scoring rule pays forecaster $i$ from resolver $j$'s budget for having moved belief toward $j$'s eventual verdict, relative to the consensus at the time they spoke:

$${\rho }_{i,j,q}=\sum _t{w}_t\left[S(p_{i,q,t},r_{j,q})-S(p_{q,t}^{\text{ref}},r_{j,q})\right]$$

Echo the crowd, earn nothing. Move belief the wrong way, pay. The reward goes to right, different, and early ($w_t$ favors early), and if the shape of that rule looks familiar, it should: the reference $p^{\text{ref}}$ is the shared prior, and you're being paid for beating it — the fast market's rule again. The two halves of this chapter differ in only one respect, what verifies the purchase and how long that takes. A hash verifies a computation in milliseconds. A human verdict verifies a research bet in years. Everything distinctive about the slow side — reputations, retroactivity, the fine print below — exists to make years-later verification safe to pay against.

Watch it run once. "Will resolver $j$ judge paper X high-impact by 2030?" The consensus sits at 0.30. Alice has actually read the paper and publishes 0.80, early. Bob echoes the crowd at 0.30. Carol, contrarian by temperament, bets 0.10. Come 2030 the resolver, holding five years of citations and replications, declares: yes.

Alice gets paid well for information the crowd lacked. Bob contributed nothing and collects the same. Carol was different and wrong, and the penalty is not cruelty — it's what keeps confidence calibrated, because had the verdict gone the other way, Alice's bold 0.80 would have cost her 1.25. (Reputation has a cold start, so real deployments run a play-money phase to seed track records before budgets attach.)

For a forecaster who can't influence resolutions, the optimal strategy is exactly one thing: report your best prediction of the capital-weighted future verdicts of the resolvers — which, under the shared-reality assumption below, is a proxy for future common knowledge. Out the other end come two artifacts: a live world-model over every open question (including, yes, the merino shortage), and a skill ranking of who sees ahead, which chapter 6 will put to work. And notice what an attacker finds to attack: no oracle, no resolution instant, no single judge. A corrupted resolver corrupts only their own channel and degrades their own future allocations, since resolvers consume the rankings their verdicts produce. Bribing one is remarkably poor value for money (📐 §2.4 prices it).

The fine print

Five assumptions hold the slow side up. Each gets its defense somewhere specific, which is most of why the rest of the book exists:

• A1 · Shared reality. On most questions, resolvers eventually agree; otherwise "consensus" has no referent and the target dissolves. Kept plausible by keeping the perception layer global (ch. 7).
• A2 · Dispersed capital. No whale holds most of the budget, or the money-optimal forecast quietly becomes "predict the whale." Chapter 4 exists to enforce this one structurally (V3).
• A3 · Repeated game. Stale beliefs must cost reputation, or nobody updates.
• A4 · Tamper-evident timestamps. No backdating, or every skill ranking is fiction. Chapter 2.
• A5 · Non-reflexivity. Forecasts must not cause the verdicts they predict. The fragile one; chapter 5 is mostly about it.

Open problems

Where this chapter can break, and where each break is handled: reflexivity and the dark room (chapter 5); resolver honesty once verdicts gate money (reduced to a stake-versus-bribe inequality, 📐 §2.4); reference gaming, where a badly chosen baseline pays for contrarianism as such (lagged references and frozen budgets, 📐 §2.5); Sybil forecasters and coordinated resolver blocs (chapter 5 again); and causal validity — verdicts must be counterfactual contrasts, not raw conditionals, or the scoring rule lends its considerable authority to a confounded claim (V5, Futarchy and Causality). On the fast side: the alignment theorem assumes price-taking and convex costs, and lumpy disks and monopoly relays break both, so mechanism design is owed there, along with the per-term cost constants of the materialization objective.

📐 Formal treatment: Mathematical Core §1, §2, §10.1.

The circle can now buy storage from strangers, verify what it gets, and even buy judgment without crowning a judge. All of it, though, is denominated in something — and that something is the sharpest design decision in the book, because pick a unit that concentrates and whoever ends up holding the pile will buy the slow side's verdicts and, with them, the network's beliefs. Money. Chapter 4.

4 · Money

🤖 AI-drafted, human-directed — what that means.

A month passes. You've served ten thousand fetches of the tutorial channel, held a terabyte of other people's residuals, sold compute overnight. What lands in your wallet?

A confession before the design, because it's instructive. Our first answer to this question — it's in the git history, and we're leaving it there — bolted IOTA's block structure to Stellar's consensus protocol, obfuscated everything with zk-STARKs, and pegged the result to the US dollar through a scheme where you'd mint new coins by provably burning other cryptocurrencies. It seemed reasonable at the time. It had every feature except a reason the pieces belonged together, and it survived exactly one contact with the actual question, which is not "how do I make a coin" but: what should a unit of value even be, here?

Because the obvious answers fail in instructive ways. Dollars reinstall everything the earlier chapters removed — a bank to hold them, a settlement network to move them, a state to define them. A fixed-supply token fails slower but harder: such tokens concentrate (first movers, compounding returns), and chapter 3 just showed you what concentration buys in this system. The slow market's verdicts. The network's beliefs. So the currency has one requirement beyond functioning as money, and it's non-negotiable: wealth must be structurally unable to pool into a weapon.

And money quietly has a second job here, one that usually goes unstated: it's the fate-coupling instrument, the thing that makes strangers' welfare depend on each other. Hold that thought; it's how this chapter ends up answering a question you didn't ask it — how networks merge.

Shares of a pool

Apply the book's standing refusal — no global anything — to value itself. There is no global value, only value to someone: a cached tutorial near the people who watch it, committed disk, a relay's bandwidth at eight in the evening. So money here isn't a token conjured from nowhere. A zone (chapter 2's densely-coupled cluster — the circle plus its storers is about to become one) has a pool: its real productive capacity, valued in the units the market already prices (byte-months, transfers, reduction steps). Your money is a share of the pools you participate in: a slice of your own community's capacity, plus slivers of a dozen others, earned by trading with them. Wealth is the portfolio w_i = Σ_z θ_{i,z}·V_z.

Two things quietly vanished in that paragraph. Minting: a zone's money supply is V_z, growing when disks join and shrinking when they leave, so the question central banks exist to answer — how much money should exist — doesn't arise; the answer is "however much capacity there is," by construction. And settlement: paying someone inside your zone is a share transfer between single-owner accounts, which chapter 2 showed needs no consensus at all.

Exchange rates are discovered, not decreed

The hard part is between zones. Your shares are claims on your community's capacity — why would a stranger three networks away accept them? It helps that history has run this exact experiment every time two disconnected economies met, and the sequence is stable enough to copy. First contact is barter; the rate is born as a goods ratio, haggled at the interface. Then some common good both sides value — silver, salt — becomes a bridge, and arbitrage pins each money's price in terms of it, within a band set by the cost of physically moving the stuff. Then credit replaces shipment: merchants net claims through clearing fairs and correspondent banks. Long-run, rates drift toward purchasing-power parity, and the residual gap is not noise but information about local conditions.

The design lesson we take from this: nobody, in several thousand years of monetary history, ever built a merge mechanism. Exchange rates are shadow prices of boundary trade, discovered where the boundary is. Build the boundary and the rate shows up on its own.

Here the bridge good is built-in and physical. A byte-month is the same good in every zone, but its cost is local — power, hardware, demand — so the rate between two zones is anchored by resource arbitrage nobody can push without providing or consuming real capacity. Rates form a field χ(x→y) over the body map (near 1 between neighbors, wide between strangers), and spending far from home is a path payment: hop by hop, each intermediary applying its local rate and taking a spread. The accumulated spread is the true cost of distance — and note the rhyme with chapter 2, where identity confidence composed along vouching chains the same way. Money, trust, and identity are one family of fields on one map, all composing along routes, all attenuating with distance. We did not plan this; the design kept insisting.

⚠️ The clearing layer deserves suspicion, and earlier drafts of this book swept it under the rug, so here it is in the open. A clearing node holds reserves in neighboring zones, quotes χ at the boundary, and earns the spread — a miniature bank, with a miniature bank's attack surface. Who gets to be one? Anyone with reserves; the position is contestable, not appointed. What stops quote manipulation? Arbitrage against the resource bridge, plus reputation — though thin boundaries with little volume remain the manipulable case, and we don't have a full answer there. What stops reserve theft? Collateral and the typed contracts of chapter 2. Each of those answers is a sketch, not a spec. So is V_z itself: valuing a pool presupposes a live resource market with real price discovery, a bootstrap dependency we can't remove, only order around (market first, then the money denominated in it).

Demurrage: local decay, global dispersion

Now for the non-negotiable requirement, and the mechanism is almost disappointingly boring. Each zone, independently, lets idle shares decay toward the equal share, and recycles the decay into a basic income drip for members — weighted by each member's independent-mind measure, so a farm of fresh accounts shares one drip (chapter 2's posterior, formalized next chapter). Per member, per zone:

ẇ_{i,z} = −δ_z·w_{i,z} + δ_z·b_z·ν_{i,z} + (earn − spend)

Every term is local; nothing crosses a boundary. But sum over a member's whole portfolio with a common rate δ and something global falls out anyway:

w_i*  =  B_i  +  Φ_i / δ

Steady-state wealth is your baseline plus your sustained net contribution, scaled by 1/δ. What you piled up melts. What you keep doing persists. And δ turns out to be a dial that directly bounds how concentrated the slow market's budgets can get — assumption A2 from last chapter, converted from a hope into a control knob (V3). The money⇄truth cycle everyone warns you about is real here, and closed on purpose: the thing that needs dispersion sets the dial that enforces it.

Two behaviors emerge unbidden, one welcome, one dangerous. Gresham's law — usually a pathology — works for us: people spend the melting shares and save the stable basket, which is precisely the circulation demurrage wants. And a reserve hub emerges: the most central zone is cheapest to route through, so its shares become everyone's intermediary (the dollar's position, re-derived by geometry). A hub is tolerable exactly as long as it's contestable — earning its spread through actually-cheaper routing, with rivals and bypass paths in reach. Extraction without contestability is this book's working definition of cancer, and the treatment is the same everywhere: competition, plus the cheap exit that demurrage itself provides, since stock is a weak hostage and shares and reputation travel.

One catch, though, and it's sharp enough to drive the rest of the chapter: with heterogeneous rates, wealth flees to the slowest-decaying zone and pools there. A haven. Local monetary sovereignty and global dispersion are in genuine tension, and the resolution is a floor — zones choose their own δ_z, above a network-wide δ_min. Which invites the question this whole design keeps circling: who sets the floor, with nobody in charge?

Zones, merging, and the floor

The live frontier. Read it as a proposed resolution, not a settled one.

Where do zones come from? Not from a committee with a map. Chapter 2's coupling surplus κ accumulates wherever agents trade, witness, and vouch densely; pool where κ is high, deal at arm's length where it's low, and the zone boundary falls where κ drops — which is, not coincidentally, exactly where an exchange rate becomes worth its upkeep. ⚠️ The measurement subtlety here is the design's central open problem: κ is read from the same correlation structure chapter 5 reads for Sybil detection, with the sign flipped — the immune system discounts correlation, the money layer rewards it. Telling cooperators from puppets is one estimation problem wearing two hats, and both hats matter.

Why would networks merge? This is the question we promised at the top, and the answer is that they don't decide to; it happens to them, via the cap table. Every trade under pool-equity money leaves you holding a little of your counterparty, and cross-holdings mean your welfare depends on their survival — fitness alignment, implemented by accounting rather than exhortation. Trade is merging in slow motion. As cross-holdings deepen, shared quorums and a shared floor start paying for themselves, χ drifts toward 1, and one day the boundary is a formality. De-merging is the same movie backwards. When the knitting network and the crochet network finally federate — and they will, whatever the hardliners say — nobody will sign a treaty; their members will just have been trading long enough that the seam stops mattering. (The coupling note gives the surplus condition G − C > 0 and the optimal-coalition-size consequence.)

Who sets δ_min? Nobody picks the number; three nested mechanisms regulate it. Locally: boundary reputation. Zones price each other's shares partly by policy, so a haven finds its spread widening and its shares refused as reserves. Fled capital gets in and can't cheaply get out — the medieval law merchant's boycott, no global vote required. Globally: homeostasis. The organ that concentration damages is the slow market, so its measured accuracy is the sensor, and governance raises the floor when perception reports creeping capture. Ultimately: selection. A network that tolerates havens gets a captured world-model, worse decisions, and emigration (chapter 7).

The homeostatic loop wants engineering care, because its sensor lags by months (verdicts resolve slowly) while its plant relaxes at 1/δ, and a naive controller under lag oscillates — monetary boom and bust as a control-theory failure, not a moral one. The working sketch: move δ_min slowly, from a low-pass-filtered signal, with slew limits, and let the fast boundary-reputation loop absorb shocks. Mapping where that loop is stable is the first simulation this design owes anyone (open question Q8).

Newcomers

A newcomer needs a starting share to transact at all, and chapter 2's identity posterior is weakest at exactly that moment — no history, nothing to weigh. Geography closes the loop: locally present members vouch the newcomer into a zone, the starting share is bounded by the strength of that vouching, and the drip ramps toward a full share as their behavioral signature individuates. It saturates at one share. A ramp, not a head start.

⚠️ Still owed by this chapter: the primary-issuance rule (adding capacity earns how many new shares, diluting whom?); thin-boundary manipulation; and the compliance-observation problem — enforcing the floor means observing a zone's δ_z and concentration, which collides head-on with transaction privacy and points toward a zero-knowledge proof-of-compliance primitive. Tracked as open questions Q7–Q10.

📐 Formal treatment: Mathematical Core §4, §10.3.

So the circle's treasury holds shares that melt if hoarded, buy real capacity, and quietly entangle its fate with everyone it trades with. Wealth is handled — it can no longer pool into a weapon. What can still pool is correlation: one bored person wearing forty faces, a market that believes itself, a bloc that buys the floor. That's chapter 5.

5 · The Immune System

🤖 AI-drafted, human-directed — what that means.

Three things go wrong with the circle, in its third year, in the same month.

First, the spam wave: forty enthusiastic new members join in an hour, and every one of them has opinions about the treasury. Second, subtler: the cursed-scarf theory — which began, you'll recall, as Sarah's joke — has hardened into orthodoxy, not because new evidence arrived but because everyone keeps citing everyone else citing Sarah, and by now expressing doubt feels vaguely rude. Third, quieter still: three members with large stakes have started coordinating their moderation verdicts, and the votes have begun going their way slightly too often.

A puppet attack, an echo chamber, a capture. They look like three problems, which is why platforms staff three departments against them. They are one problem. The puppet's forty accounts are correlated because they share a controller. The echo chamber's members are correlated because they share a deference habit. The bloc is correlated because it shares a payoff. Mismanaged correlation, three costumes — and one immune system, whose whole method is: measure the correlation, decompose it, and make weight follow independence.

Counting strangers without a registry

Start with the puppets, because the counter to them is the toolkit's core. "Is this account a distinct person?" is a question only an authority can answer, and we have no authority — so ask the answerable question instead: how much independent information does this participant contribute?

Watch behavioral residuals: what remains of forecast errors, transaction timing, service patterns after subtracting everything explainable by public information. Distinct strangers' residuals wander independently, because they come from distinct private lives. A puppeteer's accounts stay correlated no matter how carefully scripted, because they share the one thing that matters — a single controller's private state. For a cluster of k accounts with mutual correlation ρ, the number of voices actually present is

n_eff = k / (1 + (k−1)·ρ)

Perfect puppets (ρ → 1) collapse to one voice however many accounts they open; genuine strangers count in full. Note what this isn't: there's no verdict, no ban, no personhood tribunal. Just a weight, graduated and recoverable. And one rule makes the weight bite: everything the system counts is issued per unit of n_eff, never per account — votes, the drip, aggregation influence, witness power. The spam wave arrives, gets weighed, and discovers that forty of it adds up to roughly one.

(There is no global Σ matrix behind this, incidentally — that would be its own somebody-in-charge. Each zone estimates over its own members, with the body map supplying the prior on where correlation is expected: neighbors correlating is normal, strangers correlating is a tell. This is the formal version of chapter 2's stake-scaled posterior, not a new mechanism.)

The remaining attack is to script the puppets into looking independent. But the monitored dimensions are the paid dimensions — forecast accuracy, storage served, bytes routed — so passing the independence test means running forty separately maintained streams of real, individually rewarded work. At the limit, the cheapest way to fake k agents is to be k agents, at which point, from the network's side of the boundary, they simply are. Identity, here, is an accumulated signature of work; the forgery and the original cost the same.

And the worry that keeps well-meaning designers up at night — won't a close-knit community that truly thinks alike get punished as colluders? — dissolves once you notice the system never punishes anyone. Correlated voices really do carry less independent information, discounting them is accurate inference rather than judgment, and the weight recovers the moment members act on their own signals. The feared autoimmune disease was an artifact of demanding a yes/no classifier. A continuous weight never has to ask the unanswerable question.

The market that believes itself

Now the scarf problem, which is the same disease wearing its epistemic costume, and the one place chapter 3 can genuinely die.

Model a resolver's verdict as a blend: a fraction $\lambda$ of deference to the published forecast $p$ (through some response curve $g$), the rest driven by their private signal. Forecasters, optimizing honestly, will publish the fixed point of $p=\lambda g(p)+(1-\lambda )\pi$, where $\pi$ is the evidence posterior. Solve it, and the first result is a genuine relief: if deference is linear and $\lambda <1$, the fixed point is exactly $\pi$. Influence, by itself, does no damage — it flows around the loop and cancels. We expected worse.

The trap needs conformity: response steeper than proportional, the social-pressure regime where a 0.9 consensus doesn't just inform your verdict but drags it. Define the deference slope $L=\lambda \cdot \sup |g^{\prime }|$. Below $L=1$, truth is the unique equilibrium (quality degrades smoothly as $\lambda$ grows — deferring resolvers are correlated resolvers, worth fewer voices by the accounting above, and yes, the two costumes just became one). At $L=1$ the system bifurcates, and beyond it live stable fiction equilibria: forecasts near certainty, evidence nowhere in sight. The scarf theory, formalized. Active inference calls this failure the dark room. The whole risk compresses into one scalar and one safety condition, $L<1$, and it's the first of the five deaths (V1).

A measurable scalar is a controllable one, and all three control levers are correlation management by other names:

1. Blind resolution. Don't show the consensus while a verdict is being recorded. An interface decision with a theorem inside it: you can't make $\lambda$ zero (public belief leaks through the world), but you can cut the highest-bandwidth channel.
2. Reality coupling. Pay resolvers a bonus on their own verdicts scored against evidence that arrives later, making the private signal profitable and pulling $\lambda$ down.
3. Epistemic routing. Weight question budgets toward uncertainty that matters for live decisions, so optimization power sits where reality still pushes back.

Better: $L$ can be measured in production — randomize what consensus resolvers see at verdict time, regress verdicts on the display. The first pilot's primary measurand is $\hat{L}$, the system's distance from its own cliff. Few institutions can tell you how far they are from believing their own propaganda; this one is designed to publish the number.

One consequence deserves promotion from footnote to principle. An echo chamber is, epistemically, a single agent. A coalition whose members defer to each other is a high-correlation bloc, so any outside observer running n_eff weights collapses it toward one voice automatically — no echo-chamber detector required, no one has to declare the scarf people a cult. Internal decorrelation (blind resolution) keeps a community's own perception sharp; external weighting quarantines what remains; and chapter 2's coupling dynamics starve the rest, since a dark room's predictions lose value against reality and its members' surplus increasingly lies outside. Three defenses, one toolkit.

The bloc that buys the floor

The third costume is correlation of payoff — the verdict-coordinating stakeholders, or a zone selling low demurrage to attract fleeing capital. Chapter 4 already installed the treatments: resolver-side n_eff weighting discounts coordinated verdicts exactly as it discounts puppets, boundary reputation makes havens illiquid, and the homeostatic floor rises when the slow market's measured accuracy sags. What this chapter adds is just the diagnosis that it's the same disease — shared payoff is one more component of correlation, managed by the same accounting.

The one estimation problem

⚠️ And now the thing we cannot yet defend, stated as plainly as we can. Everything above reads correlation with a sign. The immune system discounts it: shared source, shared deference, shared payoff. The money layer rewards it: shared fate is what zones and mergers are made of. And some of it is neither attack nor asset but simple honesty — two careful observers of the same world should agree. Decomposing measured correlation into shared-source, shared-deference, shared-payoff, shared-fate, and shared-truth, at finite monitoring richness, against adversaries who shape their own statistics: that is the load-bearing open problem of the entire design. The mimicry-cost bound (faking costs what being costs) is asymptotic comfort, not a finite-sample guarantee; the estimator, with the body map as its prior, is unwritten; and every chapter in this book leans on it. If this design has a heart to aim at, it's here.

📐 Formal treatment: Mathematical Core §2, §3.

The spam wave weighs one, the scarf theory faces a published $\hat{L}$, and the bloc's verdicts discount themselves. Counts and beliefs are worth acting on again — which leaves acting: choosing what the circle actually wants, and never, ever wiring a forecast straight into a decision. Chapter 6.

6 · Governance

🤖 AI-drafted, human-directed — what that means.

The crochet question has been open for two years.

Not for lack of facts. The circle's slow market can tell you, with calibrated confidence, how admitting the crochet people would change membership, treasury flow, and message volume — those are forecastable questions with eventual answers, and chapter 3 prices them continuously. What it cannot tell you is whether the circle wants to be that kind of place. No evidence will ever arrive for that. In Schelling's terms, some focal points are pinned — reality eventually enforces them, and chapter 5's machinery exists to keep the pin in — and some are free: equilibria among many, where a community simply has to choose. Facts are pinned. Values are free. Governance is the discipline of choosing your free focal points on purpose, instead of inheriting them from whoever shouted first.

(In the three-timescale picture from the overview, this is the middle loop: the market refines beliefs automatically, evolution replaces whole rulebooks slowly, and governance is the deliberate refinement in between — the priors that evidence can't update, updated anyway.)

Split the decision where it wants to split

• Facts go to the market. "To what degree would policy X advance goal G" is forecastable, resolvable in hindsight, and already has a price.
• Preferences stay personal, and quadratic. Which goals matter is voted, with intensity: casting v votes costs v² credits, so caring more counts more but shouting scales badly. Quadratic voting's textbook weakness — split your credits across k puppets and mint √k extra influence — dies at the door here, because credits issue per unit of n_eff. The spam wave from last chapter gets, collectively, one member's worth of credits, which rather takes the fun out of it.

A member who cares about goal G backs it with credits and needs no expertise; the market supplies the live estimate of which policies serve G; spending follows. Values from people, facts from the machine, each guarded by the machinery built for it.

The Dana problem

Every community has a Dana. Dana reads the proposals, Dana understands the treasury, and somewhere in year two, half the circle quietly adopted the policy of voting however Dana votes. Liquid-democracy designs treat this as a feature to be maximized; independence accounting treats it as a correlation to be priced; both, it turns out, are right, and the pricing writes itself.

Under QV, k independent aligned voters with budget c each cast k√c votes. A bloc that pools everything into Dana casts √(kc). Those are the two endpoints of the independence spectrum, and one formula interpolates:

votes(bloc)  =  √( n_eff(bloc) · pooled credits )

At ρ = 1 — one mind, many wallets — it yields √(kc). At ρ = 0 — many minds, one banner — it yields k√c. Delegation needs no new mechanism at all: it's voluntary correlation, priced by the same accounting that prices the involuntary kind. And the bloc's position on the spectrum is measurable, because delegators leave statistical fingerprints: the ones who occasionally override Dana, or split their delegation, or diverge on unrelated questions, demonstrably formed their own preferences and pull the bloc toward k√c; pure rubber stamps converge to √(kc). Dana's influence levels off exactly as fast as Dana's followers stop thinking. There is something almost pastoral about the incentive this creates. ⚠️ Almost: the estimator mustn't reward performative disagreement, and designing it so it doesn't is open (coupling note).

Two axes fell apart in that analysis, and keeping them apart resolves an old confusion. Distinctness — is this one source or several — is chapter 5's question. Trust — do I want shared fate with you — is chapter 2's κ, accumulated from history. You delegate to those you trust; the system prices what that does to distinctness. Someone can be verifiably singular and still not worth co-governing with, and the machinery has always known the difference even when our prose didn't.

The prohibition

One rule in this chapter is load-bearing, and it's a thou shalt not: a market estimate must never be wired mechanically into allocation. "Fund whatever the conditional forecast favors" rebuilds futarchy, and futarchy's flaw is structural, not incidental — conditional forecasts measure correlation, decisions need causation, and for the branch not taken there is never an observation. So estimates stay evidence, weighed by humans who keep the final judgment; resolution prompts ask for counterfactual contrasts ("did X help, relative to no-X?"); and a resolver's right to discard a hopelessly confounded question is causal hygiene, not laziness. The full argument, including exactly where the impossibility theorem does and doesn't bite this design, lives in Futarchy and Causality; its residue is viability condition V5.

Governance as maintenance

Strip the drama away and most governance, most of the time, is tending set-points: the demurrage floor against measured concentration (chapter 4), question budgets against decision-relevance, funding for the open coordination models (chapter 3). Each is a feedback loop with a slow sensor — verdicts take months — and the engineering rule from the money chapter generalizes: move set-points slowly, from filtered signals, with slew limits, and let fast local mechanisms absorb the shocks. A constitution, on this view, is a controller specification, and most constitutional crises are oscillation. ⚠️ Stability regions unmapped; open question Q8 is the first simulation owed.

The right first deployment is the mirror test: the network governing its own development. Quadratic votes over roadmap goals, the market estimating which work advances them, contributor funding following the estimates as evidence. If the machinery can't govern its own repository, it has no business governing anything else.

📐 Formal treatment: Mathematical Core §10.2; Futarchy and Causality.

The crochet question finally gets its vote (they're in; Dana abstained, to everyone's surprise), the treasury moves by rules nobody can buy, and the circle can steer. One question left, and it's the oldest one in evolution's book: what happens when the rules themselves — this book included — turn out to be wrong? Chapter 7.

7 · Evolution

🤖 AI-drafted, human-directed — what that means.

The schism arrives in year five, as schisms do. It isn't even about crochet in the end — it's about the demurrage floor, of all things, and a faction that wants it lower, and a weekend of arguments that produce no agreement but do produce a fork announcement. Half the circle is leaving to run their own network, with different money and looser rules.

On every platform you've ever used, this is where the story turns tragic: the archive stays with whoever holds the keys, the schismatics start from nothing, and both halves spend a year poorer. Here, something else happens, and the fact that it can is the last thing this book has to explain. The leavers take their reputations (portable logs, recomputable by anyone). They take their shares (the treasury splits along the cap table, since demurrage made stock a weak hostage). They take the archive (content-addressed; it was never anyone's to withhold). The two networks keep trading — a boundary appears where the coupling thinned, with an exchange rate on it — and eighteen months later, when the low-floor experiment has produced exactly the concentration this book predicted, most of the leavers drift back through that boundary without ceremony, because trade had quietly kept the door open. Nobody wins the schism. Both sides survive it. That's the design working.

What we actually built

Look back at the whole descent and notice how little machinery there is. One loop: spend to reduce uncertainty until confident enough for the stakes. One move: couple where surplus beats cost, explicitly and reversibly. One payment rule: reward equals verified improvement over the shared reference. One accounting: weight follows independence, and correlation gets decomposed rather than ignored. Seven chapters of chat-app yak-shaving compress into four sentences — and that compression is not a flourish, it's the survival strategy. Small things can be reasoned about. Things that can be reasoned about can be safely changed. And changed is the point, because every mechanism in this book is a shipped prior, and priors are meant to be outgrown.

The organism frame from the overview can now be cashed out organ by organ:

Perception and action are one calculus pointed opposite ways — the slow market changes beliefs to match the world, governance changes the world to match beliefs — and chapter 5's L < 1 is the condition keeping that motor coupled to reality rather than idling in a dark room.

Why does the commons hold? For the reason it holds in biology, which solved Moloch several times before we named him. The tragedy of the commons is a multipolar trap, and every escape evolution found was the same move — a major transition: a higher-level individual emerges whose own fitness depends on the commons, and which can discipline its constituents. Genes into chromosomes, cells into bodies, insects into colonies. The stability conditions for that move read like this book's table of contents: align constituent fitness with the whole (the market pays for work the network demands; cross-holdings entangle welfare), suppress internal defection (the immune accounting), and keep selection alive between wholes. That last one is this chapter, and it is the one modern platforms fail hardest.

Forks are typed; mergers are gradual; exit is geometry

A protocol version is a hyperprior — the genome cells carry before any interaction. Framing it that way makes protocol change precise instead of catastrophic:

• Backwards compatible means a typed coercion exists at the boundary (chapter 2's session types), so old and new agents still transact — versions interoperate the way currency zones do, through an adapter with a spread. A hard fork is a boundary with no coercion: χ undefined, the door actually closed.
• Forward-pressuring means the new version strictly dominates for its adopters without demanding global switchover. Adoption should be a ramp; network effects build cliffs by default, and ramp-engineering is first-class protocol work, not marketing.
• Merging, as chapter 4 showed, needs no mechanism at all. Networks that trade accumulate each other's shares; fitness aligns through the cap table; the boundary thins until it stops mattering. Our schismatics came home this way, and never signed a thing.

Selection between networks is then real selection: a network whose priors detach from reality (dark room), concentrate (capture), or extract (cancer) loses predictive surplus and sheds members to rivals — provided exit stays cheap. That proviso is V4, and it has a geometry worth memorizing: a healthy focal point sits in a basin that is deep against unilateral defection and shallow against coordinated migration. Portable reputation, portable shares, and content-addressed archives keep the migration wall low; demurrage keeps stock from becoming a hostage. ⚠️ And exit has a dark side we can only flag, not dismiss: selection between networks can race to the bottom, and network effects raise the migration wall silently, year by year, with no announcement. Keeping exit cheap is an engineering requirement with no finish line — the treadmill is the job.

The guardrails, revisited

Earlier editions of this book kept a section called where the frame could deceive us. The frame has since earned an update to every entry, which we record partly as progress and partly as a warning about how good this frame is at explaining things:

• "Free energy explains everything, which is suspicious." Fixed by making it operational: the Alignment Invariant (Math Core §10.1) is a spec rule a mechanism either satisfies or fails. The metaphor stopped carrying load; the invariant carries it.
• "The metabolism unification is abstraction-level." Downgraded, not closed: representations-as-policies reduces storage to amortized compute with a fidelity attribute, leaving one objective with constants owed rather than three unrelated markets.
• "Two irreducible problems: reflexivity and the autoimmune limit." Unified into one — the correlation-decomposition problem (ch. 5) — which is still open, but singular, with a measurement program attached.
• "The identity trilemma is the bottleneck." Dissolved rather than solved: strong-cheap-private constrained a single global credential, and no such credential exists anymore. What remains is estimator design.
• "Exit's dark side." Restated as basin geometry, above. Measurable now. Finished never.

And what is not revisable, stated once, plainly: the priors are defaults — demurrage, log scoring, quadratic voting, all of it — but the conditions on the refinement loop are the book's actual claims. Sound protocols, incentive-compatible play, self-measurement, evolvability, and the five deaths those imply. A network that violates them detaches from reality or gets captured, whatever mechanisms it runs. A network that satisfies them can outgrow every default we shipped, and should.

📐 Formal treatment: Mathematical Core §7 — the viability envelope and the grand conjecture, the coupled stability proof that remains the design's open centerpiece — and §10.

Six years ago the plan was one line long: Discord but decentralized and better. What it took was a machine that audits its own memory, receipts that became clocks, a market with one commodity, money that melts, an immune system for correlation, a voting rule that prices Dana, and a theory of schisms. The circle got its chat app, for whatever that's worth now; the archive will outlive the company that once hosted it, and the next forty-person community won't have to invent the universe first, because the priors ship in the box. Whether the organism lives — whether the coupled whole is stable — is one conjecture with five named conditions, and there is only one way to find out. It was never just a chat app. But it did start as one, and we think that's the right way around.

The Mathematical Core

🤖 AI-drafted, human-directed — what that means.

Reference. The formal companion to the chapters: every open question raised in the book is either answered, reduced to a measurable quantity, or stated precisely as open. Notation follows chapter 3; every symbol is in the glossary.

0. The object, defined

A Dither organism is a tree of agents (node ⊂ zone ⊂ network), where each agent a maintains a boundary and solves one control problem:

minimize  F_a  =  E[ priced unmet demand inside a's boundary
                     + maintenance cost of a's internal states
                     − revenue from serving demand across a's boundary ]

Agents are coupled by exactly three operators, and the entire stack is these three operators applied at every scale:

plus one vertical flow: weight issuance downward (credits, UBI, witness power, all gated by n_eff) and aggregation upward (world-models, preference tallies, all weighted by n_eff).

The claim made precise in this document: the organism is viable iff five inequalities hold (§7), each corresponding to a load-bearing open problem from Evolution.

1. The substrate: MATERIALIZE as priced flow

1.1 The optimization problem

Demand is a field D(v, x, t) — intensity of requests for value v at location x (locations = points in the latency coordinate space). The network chooses a flow over three edge types:

TRANSPORT-TIME(v, x, [t,t']):   cost = c_s(x) · |v| · (t'−t)        (storage)
TRANSPORT-SPACE(v, x→y, t):     cost = c_b(x,y) · |v|               (routing)
TRANSFORM({vᵢ} ↦ f({vᵢ}), x):   cost = c_c(x) · work(f)             (compute)

subject to: every demanded (v, x, t) is satisfied by some DAG of edges terminating in a materialization of v at (x, t) within its latency bound. This is a min-cost multicommodity flow problem in a spacetime-value graph; "fetch vs. recompute" is route choice within one graph.

work(f) has a canonical unit. Tree calculus is deterministic and confluent, so "number of kernel reduction steps" is a machine-independent work measure — the natural gas metric. disp gives the substrate market its metering for free.

1.2 The local decision rule (caching = replication = memoization)

For a price-taking node at x, holding value v is profitable iff

λ_v(x) · p(v, x)   >   c_s(x) · |v|

where λ_v(x) is local demand intensity and p(v,x) the posted price of delivering v at x. One inequality; instantiated with v = a file it is a CDN/replication rule, with v = a function result it is memoization, with v = a hot route's session state it is caching. This is the formal content of "storage, compute, routing are one thing": one flow problem, one threshold rule.

1.3 Prices are prediction errors (de-metaphorizing the FEP)

Define network free energy as total expected priced shortfall plus maintenance:

F  =  Σ_{v,x}  D(v,x) · shortfall_cost(v,x)  +  Σ_nodes maintenance

Proposition 1 (alignment). If delivery is priced at marginal shortfall cost and nodes are price-takers, then a node's profit from any local action equals the decrease in F that the action causes. Individual profit-seeking is distributed gradient descent on F.

Sketch. Marginal-cost pricing makes each served request transfer exactly its social shortfall-saving to the server; maintenance is borne locally; sum over actions. This is the first welfare theorem specialized to a convex flow problem. ∎

This answers the "FEP is slippery" objection from Evolution by choosing the free-energy functional and deriving the alignment, rather than asserting it: the price field is the prediction-error field. (Honest scope: Proposition 1 needs convex costs and no market power; lumpy storage and monopoly relays break it — see §8, Open-5.)

1.4 Verification (the layer-1 ↔ layer-4 bridge, made exact)

Because tree-calculus reduction is deterministic and confluent, two honest executors of f(x) agree bit-for-bit. Hence:

• k-replication: sample k executors from a pool with honest fraction h; undetected fraud requires all k sampled to collude on the same wrong hash: P[fraud] ≤ (1−h)^k. Exponential security for linear cost.
• Bisection disputes: hash-consed reduction traces are Merklizable; a referee resolves a disputed run by binary search over the trace in O(log T) checked steps (Truebit-style), so the expensive path is only taken on disagreement.
• Predicate checking: when the disp result-contract P is cheaper than f (NP-style asymmetry), the buyer verifies directly and no replication is needed.

1.5 The disp ↔ network boundary (roadmap Q4 — answered)

The language's effect signature and the network's service API are the same three-operation algebra:

Eff_net  ::=  store : Tree → Duration → Eff Receipt        (TRANSPORT-TIME)
           |  send  : Tree → Coord    → Eff Receipt        (TRANSPORT-SPACE)
           |  eval  : Tree → Tree     → Eff Tree           (TRANSFORM)
           |  price : Query           → Eff PriceQuote     (read the field)

Serialization is canonical: hash-consed subtree encoding with content-addressed chunking (dedup is free, since equal subtrees are pointer-equal already). The "TC-Net backend" in disp's EVALUATOR_PLAN is precisely an implementation of this algebra. There is no separate "integration layer" to design; the substrate market is disp's evaluator.

2. The epistemic engine: reflexivity solved-in-principle

This section addresses the gating question (roadmap Q1, truth-markets §8.1): does the perception↔action loop converge to truth or fiction?

2.1 The model

One binary question, true state θ. Forecaster's honest posterior given evidence: π. Resolvers see private signals of quality q and may also defer to the published consensus forecast p. Model resolver verdicts as

r  =  λ · g(p)  +  (1−λ) · σ(θ)

where σ(θ) is the signal-driven verdict probability, g(p) the deference response, and λ ∈ [0,1] the deference weight — the fraction of resolution behavior driven by the forecast itself rather than by independent perception. Forecasters paid by proper scoring against r will publish the fixed point

p*  =  λ·g(p*) + (1−λ)·π

2.2 The convergence results

Proposition 2 (linear herding is harmless). If g(p) = p (resolvers defer linearly) and λ < 1, the unique fixed point is p* = π. The forecast remains exactly the honest posterior; properness is preserved.

Proof. p* = λp* + (1−λ)π ⟹ (1−λ)p* = (1−λ)π ⟹ p* = π. ∎

Reflexivity does not bias the forecast at all in the linear regime: the influence passes through and cancels. The beauty-contest pathology requires nonlinearity, not mere influence.

Proposition 3 (bifurcation threshold). The fixed point is unique for any g with λ · sup|g′| < 1. If λ · sup|g′| > 1 (e.g. conformity response g(p) = sigmoid(κ(p−½)) with λκ/4 > 1), multiple stable fixed points exist — including fiction equilibria where p* is near certainty while π is not. This is the dark room, as a bifurcation.

Sketch. Banach contraction for uniqueness; for the sigmoid case, plot p ↦ λg(p)+(1−λ)π: slope > 1 at the crossing creates the classic S-curve with three intersections, outer two stable. ∎

Proposition 4 (continuous degradation below threshold). Even when unique, the verdict's information about θ degrades: verdict precision scales as (1−λ)², and deference correlates resolvers through the common channel, so m resolvers are worth only

m_eff  =  m / (1 + (m−1)·ρ_λ)

independent ones, where ρ_λ is the deference-induced verdict correlation. Aggregation quality decays smoothly in λ long before the bifurcation.

2.3 The design consequences (now concrete mechanisms)

The whole reflexivity question reduces to one measurable scalar: the deference slope L = λ·sup|g′|, with safety condition L < 1. Three mechanisms control it:

1. Blind resolution (information design). The resolution interface does not display the current consensus p when a resolver records a verdict. You cannot make λ = 0 (public knowledge leaks), but you remove the highest-bandwidth channel from p to r. This is a UI decision with a theorem behind it.
2. Reality coupling (incentive design). Pay resolvers a small properness bonus on their own verdicts scored against later-arriving information; this rewards using the private signal, directly lowering λ.
3. Perturbation audits (measurement). On a random subset of questions, randomize what consensus value is displayed to resolvers (or randomize blind vs. shown). The regression of verdicts on displayed consensus estimates λ·g′ directly. The pilot's primary measurement target is L̂ — the system's distance from the bifurcation.

Proposition 5 (effort routing preserves properness). Modifying the score with an uncertainty bonus breaks properness (it pays for confidence per se). But scaling each question's reward budget by any forecast-independent factor u_q preserves per-question properness. Choosing u_q ∝ expected value of information (decision-relevance × current entropy under a lagged consensus model) routes optimization power toward uncertainty that matters — the "epistemic value" requirement from the living-network essay, implemented without touching the scoring rule. (Lagged/frozen u_q so no individual forecast can move its own question's budget.)

2.4 Resolver honesty (truth-markets §8.2 — reduced to an inequality)

Per-resolver scoring contains corruption: resolver j lying corrupts only R_{i,j}, no one else's channel (unlike a global oracle, where one corruption poisons everything). The remaining question is whether j lies in their own channel. But j consumes their own rankings: their allocations are guided by forecasters selected via R_{i,j}. Mis-resolving toward a lie r̃ selects forecasters skilled at predicting r̃-shaped verdicts rather than θ, degrading j's own future allocation utility. Honesty is dominant when

∂U_j/∂(ranking fidelity)  >  side-payment available for lying

— a skin-in-the-game inequality satisfied for resolvers who actually allocate the budgets they score with (and not for pure influence-buyers, who should be down-weighted in the public aggregate exactly as §3 prescribes — the same independence machinery applies to the resolver side: correlated resolver blocs get GLS-down-weighted in p*).

2.5 The "extrapolated intention/utility" operator (truth-markets §8.5 — defined)

Define the retroactive value of forecaster i's information to resolver j as decision value:

EU(i→j)  =  E_j[ U_j(allocation with p_i) − U_j(allocation without p_i) ]

evaluated under j's own posterior world-model. This is implementable by self-application of the engine: "will j, in hindsight, judge that i's forecast improved j's allocation?" is itself a question with a per-resolver retroactive verdict. The regress terminates because its base case — j's realized allocation outcomes — is directly observed by j. The previously-undefined desideratum is thus an ordinary question type inside the same market.

3. The immune system: independence accounting

This section addresses Sybil resistance, collusion, the autoimmune dilemma, and the QV attack at once (roadmap Q2, truth-markets §8.4, living-network §4 & §8).

3.1 The reframe: from personhood to precision

Stop asking "is this a distinct person?" Ask: how much independent information does this constituent contribute? Let x_i be agent i's behavioral residual stream — forecast errors, transaction timing, verdicts — after conditioning on public information c. For honest distinct agents, residuals are (approximately) independent; for puppets of one controller, they remain coupled given c, because they share private state.

Let Σ be the residual correlation matrix. Define the effective population:

n_eff  =  1ᵀ Σ⁻¹ 1

— the precision of the optimally-weighted (GLS) average of the streams. For an equicorrelated cluster of k accounts with correlation ρ: n_eff = k/(1+(k−1)ρ). Perfect Sybils (ρ→1) collapse to n_eff = 1 no matter how many accounts; genuine independents (ρ→0) count fully.

All weight in the system is issued per unit n_eff, not per account:

• Aggregation (world-model, resolver consensus): GLS weights w = Σ⁻¹1 / 1ᵀΣ⁻¹1 — correlated blocs automatically discounted.
• Voting credits: QV is √k-vulnerable to Sybils (split budget c across k accounts: votes go from √c to √(kc)). Issue credits proportional to each agent's marginal n_eff contribution and the attack yields zero: a fully-correlated cluster receives one agent's credits regardless of account count.
• UBI / perfusion: newcomer drip ramps with established marginal n_eff (a farm of fresh accounts shares ≈ one UBI until the accounts behaviorally diverge — §4.3).
• Witness power (timestamping, §5): attestations weighted by n_eff, so "many witnesses" means many independent witnesses.

n_eff is the single security parameter of the entire stack. Votes, aggregation quality, UBI farming, witness security, and resolver-bloc resistance are all the same number.

3.2 The autoimmune dilemma — dissolved, not solved

The feared false positive — "an honest local community that genuinely agrees gets punished as colluders" — is not an error under this accounting. Agents correlated through a private shared channel genuinely contribute less independent information; weighting them as n_eff < k is accurate inference, not injustice. There is no binary verdict to get wrong: weight is graduated, continuous, and recoverable (diverge behaviorally and your weight grows). The autoimmune problem was an artifact of demanding a yes/no Sybil classifier; precision accounting never asks the unanswerable question.

What remains open is estimation, not semantics: Σ is n×n and needs regularization. The geographic/vouching graph is the prior on Σ's sparsity structure — locality's fourth job (after latency, birth-vouching, currency zones) is statistical: it tells the estimator where correlation is expected, so deviations are informative. (Privacy: residual correlations are computable over pseudonymous streams via secure aggregation; no real-world identity required — the trilemma's privacy corner holds.)

3.3 The mimicry bound (why this composes with metabolic identity)

The remaining attack is adversarial decorrelation: a puppeteer runs k accounts that simulate independence on every monitored dimension while coordinating on the payload.

Proposition 6 (mimicry cost). As the monitored behavioral dimensions approach the dimensions in which the system pays for work (forecast accuracy on diverse questions, storage/relay/compute service, transaction patterns), maintaining k streams that pass conditional-independence tests requires ≈ k independent streams of real predictive/metabolic work. In that limit, the cheapest way to fake k agents is to be k agents — at which point, from the network's perspective, they are k agents.

Sketch. Passing independence tests on paid dimensions means producing k decorrelated, individually-rewarded performances; decorrelated competent forecasting requires k separately-maintained information positions; decorrelated service requires k resource commitments. The controller's preference correlation can survive — but preferences are exactly where weight is √-dampened (QV) and capped by credits ∝ n_eff earned on the paid dimensions. ∎

This is the formal content of "identity = metabolic signature": deterrence = making mimicry-cost ≥ honest-cost, and the inequality tightens as more of the economy's real work feeds the monitor. Honest scope: this is an asymptotic/arms-race bound, not a closed-form guarantee at any finite monitoring richness (§8, Open-1).

4. Money: demurrage as a control law

4.1 The wealth dynamics

Money supply M, population N (measured in n_eff!), demurrage rate δ. Each wallet decays continuously; the entire decay flow is recycled as the UBI drip δM/N per capita. Agent i earns e_i and spends s_i per unit time:

ẇ_i  =  −δ·w_i  +  δM/N  +  e_i − s_i

Proposition 7 (egalitarian attractor). The steady state is

w_i*  =  M/N  +  (e_i − s_i)/δ

with relaxation time 1/δ. Wealth converges to equal plus net-contribution-flow scaled by 1/δ. Demurrage converts unbounded stock inequality into bounded flow inequality: you can only be richer than baseline by (your sustained net flow)/δ.

Generalization (Part B). Money replaces the scalar wallet w_i with a portfolio of shares in local pools, w_i = Σ_z θ_{i,z}·V_z, and runs demurrage per-zone (δ_z). Summing the per-zone dynamics over a portfolio with uniform δ reproduces exactly the attractor above — local dispersion dials agglomerate into this global one. Heterogeneous δ_z makes the lowest-demurrage zone the binding constraint (a concentration "haven"), which is why the design needs a global floor δ_min.

4.2 The δ-dial: assumption A2 becomes a theorem

The truth machine's aggregation quality requires dispersed resolver budgets (assumption A2). If budgets are proportional to wealth and net earning advantages are bounded by ē, then the largest steady-state budget share is

max_j B_j/ΣB  ≤  1/N + ē/(δM)

To enforce a target concentration bound β, set δ ≥ ē / (M·(β − 1/N)). The demurrage rate is the knob that enforces the epistemic layer's soundness condition: monetary policy and truth-machine validity are one equation. (Caveat: this bounds steady-state stock, not instantaneous flow-through; a high-flow actor can still spend heavily in bursts — burst-spend caps on resolver budgets close that gap.)

4.3 Perfusion (new-user bootstrapping, quantified)

UBI to account i is δM/N · ν_i where ν_i is i's marginal n_eff (§3.1), with the geographic vouching graph supplying the newcomer's prior ν. Consequences:

• A genuine newcomer, locally vouched, starts with modest nonzero perfusion and ramps to full share as their behavioral signature individuates. Saturates at 1 — a ramp, not rich-get-richer.
• A k-account farm shares ≈ one UBI until (per Prop. 6) it performs k agents' worth of real independent work.
• Zones run their own (M_z, δ_z) with exchange at boundaries: local cost-of-living tracking, organ-level metabolic autonomy.

4.4 What needs consensus (roadmap Q3 — answered, with citations)

• Payments need no consensus. Asset transfer with single-owner accounts has consensus number 1 (Guerraoui, Kuznetsov, Monti, Pavlović, Seredinschi, The Consensus Number of a Cryptocurrency, PODC 2019): only the sender's own transaction order matters, and the sender provides it. Byzantine consistent broadcast suffices (implemented in FastPay et al.). Shared k-owner accounts need consensus only among the k owners.
• Demurrage needs no consensus at all. w(t) = w(t₀)·e^{−δ(t−t₀)} + flows is locally verifiable arithmetic.
• Timestamps need only causal entanglement. A forecast hash h, once referenced inside other agents' signed messages, is sandwiched: h existed before everything that cites it and after everything it cites. Backdating requires rewriting the signed causal cone of independent witnesses — and witness independence is, again, n_eff. Gossip rate sets timestamp precision; minutes-level precision is ample for scoring weights w_t. No total order required.
• Only contested allocation needs ordering — auctions for the same scarce item, name registries, zone-level governance execution. These run on small zonal BFT quorums (or, for slow decisions, on the truth machine itself). The stack needs a blockchain nowhere; it needs zonal BFT in one narrow place.

5. Architecture theorem: perception global, action zonal

Proposition 8 (reputation is intrinsically portable). R_{i,j} is a pure deterministic function of two public, self-certifying logs: i's signed timestamped forecast stream and j's signed verdict stream. Therefore any network, zone, or fork can recompute any reputation from portable data. No platform can custody reputation; epistemic switching cost ≈ 0 by construction.

This resolves the shared-reality vs. exit-discipline tension of Evolution with an architectural rule:

The perception layer (signed forecast/verdict logs, the world-model) is global, content-addressed, and portable. The action layers (currency, governance, resource allocation) are zonal and exitable. Reality stays shared at the top (assumption A1 gets its large population); discipline and exit operate at the bottom (between-zone and between-network selection stays live); demurrage already makes currency stock a weak lock-in (wealth melts; what persists is flow and portable reputation).

Exit cost is then dominated by social re-coupling, which the portable logs minimize. This is "make exit cheap" as a design invariant rather than an aspiration.

6. What the centralized pilot must measure (roadmap Q5 — specified)

The pilot is now an experiment with defined estimands, not a demo:

1. L̂ — the deference slope (the bifurcation distance, §2.3). Randomize, per question × resolver, whether current consensus is displayed at verdict time (blind vs. shown), and at what displayed value (within honest jitter). Regression of verdicts on display estimates λ·g′. Success criterion: L̂ measurably < 1 under blind-resolution UI; dose-response visible when shown.
2. Σ̂ — forecaster residual correlation structure — feasibility of independence accounting: do residuals (conditional on public info) actually separate known-distinct individuals from deliberately-planted sock-puppet pairs (plant some, pre-registered)?
3. Properness in practice — do participants report calibrated beliefs under the reference-relative log score (calibration curves, sharpness)?
4. VOI routing — A/B question-budget weighting u_q (flat vs. VOI-weighted, §2.5): does optimization power follow the subsidy without distorting calibration?

Scale for signal: on the order of 30–100 resolvers, 100–300 questions with staggered horizons (science-impact claims fit: 6–24-month resolvability), every forecast and verdict signed and hash-entangled from day one so the timestamping layer (§4.4) is exercised by the same pilot.

7. The viability envelope

The load-bearing problems from Evolution are now five inequalities. The organism is viable iff:

(V1) Contraction:    L = λ·sup|g′| < 1            — perception dominates action
                                                     (else: dark room / fiction equilibria)
(V2) Immunity:       cost(mimic k agents) ≥ cost(be k agents)
                                                     — boundary integrity; n_eff sound
(V3) Dispersion:     δ ≥ ē/(M·(β−1/N))             — metabolic turnover enforces A2
                                                     (else: whale capture of the epistemic layer)
(V4) Exit:           switching cost < tyranny premium
                                                     — between-network selection stays live;
                                                       held by Prop. 8 (portable perception layer)
                                                       plus zonal action layers

(V5) Causal resolution:  verdicts are ex-post counterfactual contrasts, not raw
                         conditional outcomes  — else the rule lends proper-scoring
                         authority to a confounded judgment (futarchy's flaw).
                         See futarchy-causality.md. Dual to V1: both keep the
                         market an evidence instrument, never a mechanical decision rule.

These are not independent: V2 (n_eff) is a parameter inside V1's resolver count, V3's population N, and V4's witness security. Boundary integrity is the load-bearing wall, exactly as the anatomy predicted.

The grand conjecture (the system's existence theorem, still open): the coupled dynamics — price field (§1), scoring fixed point (§2), weight estimation (§3), wealth flow (§4) — possess a stable joint fixed point whenever V1–V4 hold strictly, and lose it when any is violated. Each subsystem's result above is a lemma toward this; the coupled proof (or agent-based demonstration) is the centerpiece of the Phase-1 theory track.

8. Scorecard: every open question, dispositioned

From roadmap.md "Open questions before Phase 2":

From the truth-markets synthesis §8:

Genuinely still open (the honest residue):

1. The arms race floor (V2 at finite richness). Prop. 6 is asymptotic; how much monitored behavioral diversity suffices in practice is empirical.
2. The shape of g. L < 1 is measurable, but we don't know human deference response curves in this setting until the pilot runs.
3. Preference-side depth. n_eff-gated QV fixes Sybil/collusion, and the fact/preference split (delegate facts to the market, keep preferences quadratic and personal) is justified — but full liquid preference delegation semantics under independence accounting (delegation is voluntary correlation) needs its own treatment. Current recommendation: delegation operates on the epistemic side only.
4. Open-economy monetary dynamics. Inter-zone exchange rates, speculative pressure on a demurrage currency, and the (dubious) external peg all need real macro analysis.
5. Substrate market under non-convexity. Prop. 1 assumes price-taking and convex costs; lumpy storage commitments and relay market power need mechanism design (posted-price vs. auction hybrids).
6. The grand conjecture (§7): the coupled fixed-point/stability proof — the actual "does the organism live" theorem. The Phase-1 agent-based simulation should target exactly the V1–V4 phase boundaries.

9. The whole design in five equations

(1) Substrate:    cache/replicate/memoize iff   λ_v(x)·p(v,x) > c_s(x)·|v| ;
                  profit = −∇F under marginal-cost prices.            (§1)

(2) Perception:   p* = λ·g(p*) + (1−λ)·π ;  truth-tracking iff L < 1. (§2)

(3) Identity:     n_eff = 1ᵀΣ⁻¹1 ;  all weight (votes, UBI, witness,
                  aggregation) issued per unit marginal n_eff.        (§3)

(4) Metabolism:   ẇ = −δw + δM/N + e − s ;  w* = M/N + (e−s)/δ ;
                  δ ≥ ē/(M(β−1/N)) enforces dispersion (A2).          (§4)

(5) Architecture: perception global & portable (R = pure fn of logs);
                  action zonal & exitable.                            (§5)

Five equations, five viability inequalities (V1–V5), one conjecture (§7).

10. Working extensions (drafts)

Formal seeds from the working notes; each is a candidate section, stated compactly, not yet integrated with §§1–7.

10.1 The Alignment Invariant

Both payment rules already in this document are one rule. The market scoring rule's expected payment is a Bregman improvement toward the verdict: for the log score, E[ρ] = KL(θ‖p_ref) − KL(θ‖p_new). The substrate's Proposition 1 pays profit = −ΔF for a convex expected-shortfall functional. Aggregation weight (§3) is marginal precision — the same quantity inside the estimator. Unifying spec rule:

Every payment is a verified marginal reduction in an identified stakeholder's convex expected loss, relative to the shared reference. A mechanism whose payment cannot be written in this form is out of spec. (The reference p_ref is the shared prior; the whole economy pays for prior improvement.)

10.2 Delegation under independence accounting

Under QV, k independent aligned voters with budget c each cast k√c votes; a fully pooled bloc casts √(kc). One formula interpolates and is n_eff-consistent by construction:

votes(bloc) = √( n_eff(bloc) · pooled credits )

Endpoints: ρ=1 → √(kc), ρ=0 → k√c. Delegation is voluntary correlation, priced identically to involuntary correlation; the bloc's position on the spectrum is estimated from preference residuals (overrides, split delegations). Open: estimator design that does not reward performative disagreement.

10.3 Coupling surplus and merging

Coalitions A, B couple/merge iff G(A,B) − C(A,B) > 0, where G is exploitable shared structure (F_{A⊕B} = F_A + F_B − G: shared caching, gains from trade, risk pooling, netting) and C is coordination cost (joint-boundary consensus latency, policy compromise, capture exposure). κ = accumulated observed G − C. Diminishing G with distance plus growing C with size ⇒ optimal coalition size (Coase, Mundell as instances). Cross-holdings from pool-equity trade implement fitness alignment (∂U_A/∂ survival(B) > 0), so merging is the χ → 1 limit of trade rather than a mechanism.

10.4 Verification as optimal stopping

A claim with prior b₀; checks are purchasable evidence with cost and expected likelihood-ratio contribution; the optimal policy is Wald's SPRT — buy evidence while marginal value of certainty (stake-dependent) exceeds marginal cost, stop at threshold τ(stake). §1.4's menu is one curve: model-agreement checks at the cheap end, predicate checks, k-replication (P[fraud] ≤ (1−h)^k: multiplicative confidence at linear cost), bisection, re-execution at the exact limit. Claims that never cross τ are served as uncertain, with the confidence label load-bearing.

Futarchy and Causality

🤖 AI-drafted, human-directed — what that means.

Reference. A deep dive on the strongest objection.

Evaluates the retroactive consensus market (and its formal core, The Mathematical Core) against Dynomight's "Futarchy's fundamental flaw" (dynomight.net/futarchy). Question: does the conditional-vs-causal critique of futarchy sink our design, and if not, exactly which parts survive? This is the causal twin of the reflexivity problem.

1. Dynomight's flaw, stated precisely

Futarchy = make decisions with conditional prediction markets: run P(value | do action A) and P(value | do action B), take the action with the better conditional. The flaw:

Conditional markets reveal probabilistic relationships P(Y | X=x), but decisions need causal ones P(Y | do(X=x)). These differ whenever there is reverse causality or confounding.

Five objections, in increasing depth:

1. Reverse causality — the conditioning event may be an effect, not a cause (a falling stock causes the firing).
2. Confounding via revelation — taking action A reveals information about the decision-maker (the board that fires Musk reveals hostility → predicts other value-destroying acts), so the conditional price reflects the action's signal, not its effect.
3. Real-policy confounding — same structure for policy (a no-fly-zone declaration prices in what it reveals about the leader's temperament, not just the policy's direct effect).
4. Pre-commitment doesn't save you — when a market activates conditionally on the decision, bidders condition on activation and order is not preserved (the trick-coin example: you bid more on the branch that cancels-and-refunds when unfavorable, because you're insured).
5. Impossibility theorem — no payout function of (bid, final price, outcome) can both force truthful bids and preserve causal information, given conditional cancellation.

Dynomight's own prescription: conditional markets are not worthless — treat them "like observational statistics: one piece of evidence, considered skeptically," never a standalone causal decision rule. The flaw can be fixed with causal-inference machinery, but "none are free."

2. Why our base mechanism is not the object the theorem is about

The impossibility theorem (objections 4–5) is a theorem about decision markets with conditional cancellation: one branch is taken, the other is refunded, and the selection-on-activation distorts bids. Three features of our design break that frame at the base layer:

• Decoupling. Forecasters publish timestamped distributions and are scored by a proper rule against resolver verdicts. They do not bet in a pool with refunds. There is no zero-sum activation branch to be insured against. For any unconditional question ("will Health be high in 2030?") this is pure forecasting; proper scoring is truthful; the theorem is simply irrelevant.
• Retroactive, optional, per-resolver resolution. There is no single point-in-time mechanical resolution. Resolution happens later, by a human, who may decline.
• The target is resolver consensus, not an observed outcome. The market predicts what thoughtful resolvers will eventually judge, not a mechanical readout of a price or statistic.

So the base layer (elicitation + aggregation) is not a decision market and does not inherit the impossibility result. What it produces is exactly the object Dynomight endorses: an aggregated observational/expert-judgment instrument. The flaw can only re-enter where we close the loop — wire a conditional forecast mechanically into a decision. That is the governance application, §4.

3. The key move: retroactivity converts an impossibility into a competence requirement

Dynomight assumes resolution is mechanical and ex ante: the market resolves on an observed outcome (the realized stock price, the realized death rate), and that observed outcome is confounded. Against mechanical resolution, the flaw is a theorem — provably unfixable by any payout function.

Our resolution is judgmental and ex post: a resolver, with hindsight and data, declares a verdict. The decisive consequence:

The causal question is relocated from ex-ante market pricing (where Dynomight proves it cannot be solved) to ex-post human judgment (where it is merely hard — and where causal inference tools actually work: RCTs, natural experiments, synthetic controls, difference-in-differences all operate ex post).

Concretely, the resolver is asked a counterfactual contrast, not a raw conditional: not "was Health high after X?" but "relative to the no-X counterfactual, did X advance Health?" A resolver in 2030, with five years of data and comparison jurisdictions, can attempt P(Health | do(X)). A forward market pricing P(Health | X) in 2025 cannot. Retroactivity is not incidental to the mechanism; it is the structural feature that sidesteps the impossibility result, by deferring the causal question to the time and the agent where evidence exists.

The price of this move is honest and specific: we trade a provable impossibility for a load-bearing assumption — that resolvers judge counterfactually and competently. Impossible becomes hard. That is a good trade, but it must be named as a soundness condition (§6).

4. Where the flaw fully survives: the governance coupling

The Ultimate Governance application says voters back "policies that require sub-policies to be structured around the market's estimates" — e.g. "to what degree does policy X advance goal Health?" Read mechanically — fund X iff the conditional market says X advances Health — this is futarchy and inherits objections 1–5 in full, including the impossibility theorem, because:

• It is a conditional question (Health given X).
• For the untaken branch (Health given ¬X, when X is enacted) the counterfactual world is never observed, so that branch is structurally unresolvable-by-observation — exactly the cancellation asymmetry the theorem formalizes. Our design does not refund it (so the trick-coin insurance distortion is removed), but it must resolve it by resolver best-guess, importing resolver judgment error in place of a clean order-violation.

The remedy is a hard design rule, and it is Dynomight's own prescription:

The market output is an observational instrument feeding human causal judgment, never a mechanical decision rule. Voters set preferences (QV over goals) and use market estimates as skeptically-weighted evidence; the human preference/judgment loop stays open. The moment governance hard-wires "allocate ∝ conditional estimate," we recreate the flaw.

This is the same loop-closure danger as reflexivity (§5), and the same fix: keep perception (market) and action (allocation) coupled through human judgment, not through a mechanical identity.

5. Dynomight's flaw and our reflexivity condition are the same shape

Both critiques say: a market measures what it is scored against, which need not be what you want.

• Dynomight (against observed outcomes): you get P(Y|X), you wanted P(Y|do X).
• Ours (against resolver verdicts, mathematical-core.md §2): you get predicted resolver consensus, you wanted truth — and if resolvers defer to the market (deference slope L→1), the loop self-fulfills.

They compound. Our causal validity is upper-bounded by resolver causal competence and degraded below it by reflexivity:

causal validity  ≤  (resolver counterfactual competence)  −  (reflexivity loss, growing in L)

The system can faithfully aggregate and forecast expert causal judgment (valuable — a continuous, incentivized, n_eff-weighted panel of careful retrospective evaluators); it cannot exceed the causal quality of those evaluators, and it can fall short of it if the deference loop closes. It is not a causal oracle, and we should never market it as one.

6. New soundness condition: causal resolution (V5)

The viability envelope (mathematical-core.md §7) gains a fifth inequality, dual to V1:

(V5) Causal resolution:  verdicts are counterfactual contrasts judged ex post with evidence,
                          not raw conditional outcomes.
     Failure mode: the rule perfectly incentivizes predicting a CONFOUNDED judgment,
     lending false proper-scoring authority to a correlation. Worse than no system.

V5 has concrete, testable mechanisms — and several fall out of features the design already has:

1. Counterfactual question framing. Resolution prompts ask "effect relative to the counterfactual," and the resolution UI supplies the counterfactual scaffolding (comparison units, pre-trends).
2. Optionality as causal hygiene (already in the mechanism). Resolvers may discard questions they judge hopelessly confounded (truth-markets §3.1). Forecasters won't be paid for predicting confounded conditionals because thoughtful resolvers won't resolve them → causally-hopeless questions are endogenously deprioritized. Double-edged: the system goes quiet exactly on the hardest causal questions, which may be the most decision-relevant.
3. Per-resolver independence as robustness (already in the mechanism). No single confounded resolution; you predict a distribution of independent counterfactual judgments, n_eff-weighted (§3 of the math core). Independent errors wash out; shared confounders (every resolver fooled the same way) do not — so V5 failures are correlated-error failures, and the n_eff immune machinery partially detects them (a confounded consensus looks like a low-independence bloc).
4. Slow payout buys causal evidence (the user's "distributed slowly over time"). Because resolution is not pinned to an instant, the resolver can wait for causal evidence to arrive — the natural experiment to mature, the RCT to publish — before declaring. Slow distribution does not fix confounding at the mechanism level; it grants the time and removes the single attackable resolution instant that makes ex-ante markets fragile.

7. Scorecard: each objection against our design

8. Bottom line

Dynomight is right, and the critique improves our design rather than refuting it:

• Vanilla futarchy resolves mechanically and closes the loop → it hits the impossibility wall. Our design resolves judgmentally ex post and keeps the loop open → it lands, by construction, in the "useful observational instrument, considered skeptically" regime that Dynomight explicitly endorses. In a real sense the retroactive consensus market is futarchy built the only way Dynomight says it could work.
• The single most important design rule that follows: never let governance convert a conditional market estimate into a mechanical allocation. Market = perception (evidence); humans = action (preference + final causal judgment). This is identical to the V1 anti-reflexivity rule and should be enforced as one principle.
• The single new soundness condition (V5): the system's causal validity is capped by resolver counterfactual competence. The proper-scoring machinery is dangerous precisely because it can lend rigorous-looking authority to a perfectly-predicted confounded judgment. Resolution methodology (counterfactual framing, discard rights, waiting for evidence, independence weighting) is therefore not a detail — it is a first-class soundness layer, co-equal with the scoring rule.

What we cannot claim: that the market produces causal knowledge. What we can claim: that it incentivizes, aggregates, and forecasts the counterfactual judgments of an independent panel of careful ex-post evaluators, at scale, with no single attackable resolution instant — and that this is the best a market can do, given that Dynomight proved the mechanical alternative impossible.

Notation & Glossary

🤖 AI-drafted, human-directed — what that means.

Appendix A. Every symbol used in the book, defined once.

The chapters introduce symbols where they're first needed; this is the single place they're all collected. A few letters are reused with different meanings in different layers — those collisions are flagged explicitly at the bottom, because they trip up first-time readers.

Populations and objects

The scoring engine

Reflexivity

Independence (the immune system)

Coupling (the merge field)

Money (the metabolism)

The substrate

Viability inequalities

The system is viable iff these hold (see Math Core §7):

The five assumptions A1–A5 (shared reality, dispersed capital, repeated game, tamper-evident timestamps, non-reflexivity) are defined in The Five Assumptions.

⚠️ Notation collisions

The same letter carries different meanings across layers. The four to watch:

• F = the resolver set in the engine, but the free-energy functional F_a in Math Core §0. (Context always disambiguates: a set of people vs. a quantity to minimize.)
• ρ = the reward contribution ρ_{i,j,q} in the engine, but a pairwise correlation in the n_eff formula.
• λ = the deference weight in reflexivity, but λ_v(x) is local demand intensity in the substrate.
• p = a forecast distribution p_{i,q,t} in the engine, but p(v,x) is the price field in the substrate.

These overlaps are inherited from each layer's own conventional notation; they were kept rather than invent non-standard symbols.

Roadmap: The Decentralization Stack (DRAFT)

🤖 AI-drafted, human-directed — what that means.

Appendix B. The engineering / build-plan view. For the conceptual development, read the chapters in order starting from the overview.

Working draft. The goal: map the path from programming language → routing → currency/incentives → data & compute trading → governance, identify how the layers depend on each other, and call out what's missing at each step.

The stack at a glance

The truth machine in one line: forecasters publish timestamped probability distributions; capital-holding resolvers privately and retroactively declare what they believe happened; a reference-relative proper scoring rule pays forecasters for moving belief toward eventual resolver consensus. Output: a live world-model and a skill ranking, with no exploitable point-in-time oracle. Governance bolts this onto liquid quadratic voting: voters set goals, the market estimates which policies advance them, skill rankings can weight estimates.

How the layers interact

These interactions are the actual argument for building this as one stack rather than five projects:

1. The truth machine fills the currency's biggest hole. The currency's recycled flow (demurrage → UBI, plus any communal pot) needs an "intelligent democratic mechanism" to direct it — and to set the global demurrage floor δ_min that keeps capital dispersed. Liquid QV + retroactive consensus markets is that mechanism. Conversely, governance needs a native unit for resolver budgets B_j, QV credits, and forecaster payouts — the pool-equity shares.

2. The currency's anti-hoarding mechanic supports the market's key assumption. The mechanism's aggregation quality degrades as capital concentrates (assumption A2, dispersed capital). A demurrage/redistribution currency structurally pushes against concentration. These two designs reinforce each other and should be co-designed, not bolted together.

3. Identity is upstream of everything quadratic. QV is meaningless without Sybil resistance; the prediction market's reference-subtraction can be farmed by Sybil forecasters; routing incentives and data markets need persistent pseudonyms for reputation. Dither's web-of-trust identity layer (dither/user-management.md) is currently an idea sketch but is a hard dependency of layers 3–5. It deserves promotion from "application concern" to core protocol.

4. Tamper-evident timestamps are a shared primitive. Forecasts must not be backdatable (assumption A4); the currency needs transaction ordering; RHL needs some consensus on link sets. One minimal ordering/timestamping/data-availability primitive serves all three — likely the only global consensus the stack needs. This reframes the consensus question (old_ideas/consensus/): we don't need general smart-contract consensus, we need cheap verifiable timestamping plus a payments ledger.

5. disp is the verification substrate for compute trading. Selling computation requires the buyer to trust the result. disp's program-as-data + types-as-predicates + provable-equivalence story is exactly a verifiable-compute story: a compute offer is a content-addressed program plus a disp predicate the result must satisfy; disputes resolve by re-execution or proof checking. No design for this exists yet — it's the main missing bridge between layers 1 and 4.

6. disp can make markets machine-resolvable. A question's outcome space and resolution criterion can be a disp predicate over published data. Resolvers can then delegate verdicts to programs for the objective subset of questions, reserving human judgment for ambiguous ones. This also unifies the quantitative truth machine with the qualitative Protocol of Truth assertion graphs: assertions become market questions; debate structure becomes evidence attached to them.

7. Per-resolver subjectivity matches Dither's polycentric philosophy. Reputation R_{i,j} is relative to each resolver's worldview — no global truth oracle, just per-perspective rankings. Same shape as web-of-trust identity and the polycentric model. The stack is philosophically coherent: subjective-but-aggregable all the way down.

8. "Pay for predictive accuracy" recurs at every layer. DAR's network world-models reward predicting latency/bandwidth; data markets reward predicting demand for caching; the truth machine rewards predicting consensus. A shared scoring/staking framework could serve all three.

What's missing — per layer

Layer 1 — disp

• Effects (Eff monad) and the erasure (strip) pass — needed before disp programs can do real I/O efficiently.
• Self-hosted parser + module resolution (elaborator stages 4–5) — last host-trusted pieces.
• Optimizer / synthesis engine (the long-term payoff; not on the critical path for the stack).
• Stack-specific gap: serialization + content-addressed code distribution format, and any networking story at all. Nothing connects disp to Dither today beyond intent.

Layer 2 — routing & data

• DAR implementation; incentive game theory unresolved (pricing/bargaining for relays).
• DTS rare-data problem; RHL is three sentences of architecture.
• Validation gap: no simulation results backing the routing-coordinate + HORNET design. Reviving dither-sim against the current spec is the cheapest de-risking step.

Layer 3 — currency

• No concrete consensus/ledger design (Stellar+IOTA hybrid is a preference, not a spec).
• Demurrage/redistribution mechanics, zone design, and the proof-of-external-destruction USD peg all need real economic analysis — the peg in particular is likely unsound and should be re-examined.
• No threat model (eclipse attacks, zone capture, fee-less spam).

Layer 4 — data & compute trading

• Essentially everything: storage market protocol, bandwidth settlement (micropayments per relayed byte?), verifiable compute protocol (see interaction 5), pricing discovery, comparison against Filecoin/Arweave/Golem to know what to copy vs. reject.

Layer 5 — governance / truth machine

• Reflexivity (A5) is the central theoretical gap: when skill rankings feed back into the votes that constrain resolvers, does the loop converge to truth or to a stable fiction? Needs agent-based simulation and/or a formal treatment before any high-stakes deployment.
• Resolver honesty is unmodeled once verdicts gate public money.
• "Retroactive reward by extrapolated intention/utility" is a desideratum, not a defined operator.
• Hierarchical/liquid structure of the QV layer is a TLDR, not a design (delegation mechanics, credit issuance, collusion resistance).
• The truth-markets notes live outside this repo and should be merged as a spec chapter. Done — merged as Retroactive Consensus Markets (mechanism, assumptions A1–A5, open problems §8).

Cross-cutting

• Identity/Sybil-resistance primitive (blocks 3, 4, 5).
• Timestamping/ordering primitive (blocks 3, 5, RHL).
• A stack-wide threat model document.
• A dependency-ordered build plan — which this document attempts below.

Sequencing principle

Two observations drive the ordering:

1. The truth machine does not need the rest of the stack to be tested. A centralized pilot (a web service with signed, timestamped forecasts and a handful of resolvers) tests the mechanism, the scoring rule, and empirically probes reflexivity — long before Dither routing or currency exist. The science impact-market is the natural pilot (low stakes, measurable outcomes, motivated forecasters).
2. Each layer should ship something independently useful (the "Useful" tenet), while producing the primitive the next layer needs.

Phased roadmap (draft)

Phase 0 — Consolidation (weeks)

• ✅ Merge truth-markets notes into this spec (governance/ chapter): mechanism, assumptions A1–A5, open problems — see Retroactive Consensus Markets.
• Write the cross-layer dependency map (this doc) and a first threat-model outline.
• Decide the identity and timestamping primitives' requirements (consumers: currency, markets, governance).

Phase 1 — Independent kernels (months, parallelizable)

• disp: land effects + erasure + stages 4–5 → a language someone can actually write a tool in. Then: content-addressed module distribution format.
• Routing: revive dither-sim; validate routing coordinates + DAR relay selection in simulation; publish results. Implementation only after simulation says the design works.
• Truth machine: build the centralized MVP (forecast publication, per-resolver retroactive scoring, skill rankings). Run the science impact-market pilot. Instrument it to study reflexivity and reference-belief gaming.
• Theory: agent-based simulation of the reflexive governance loop; formalize or refute convergence.

Phase 2 — Shared primitives (after Phase 1 signals)

• Web-of-trust identity protocol (design + prototype) — promoted to core.
• Minimal timestamping/data-availability layer (this is "the consensus decision," scoped down).
• Currency testnet on top of both; routing incentives denominated in it (first real economic loop: pay relays).
• Truth machine v2: forecasts anchored to the timestamping layer; identity-backed forecaster accounts (Sybil-resistant reputation).

Phase 3 — Markets

• Storage/bandwidth trading over DAR + currency (DTS trail hosting becomes a paid service).
• Verifiable compute protocol: disp predicates as result contracts; dispute resolution by re-execution.
• Machine-resolvable market questions (disp predicates as resolution criteria) feeding the truth machine.

Phase 4 — Governance

• Liquid QV pilot in a small, real community — the obvious candidate is the Dither project itself: fractional-funding contributions allocated by QV, informed by truth-machine estimates of "which roadmap item most advances goal X." Recursive self-governance satisfies the "Self-reliant" tenet and is the cheapest honest test.
• Scale outward (open-source funding, DAO treasuries, scientific funding bodies) only after the reflexivity question has empirical answers.

Open questions to resolve before Phase 2

Status update: all five are dispositioned — answered, reduced to measurable quantities, or precisely scoped — in mathematical-core.md (see its §8 scorecard).

1. Does the reflexive loop converge? → Answered in principle (math core §2): linear influence is harmless; danger is a bifurcation at deference slope L = 1; L is measurable and controllable (blind resolution, reality-coupling).
2. Minimal identity primitive for Sybil-resistant QV + anonymity? → Reframed (math core §3): weight ∝ effective independence n_eff = 1ᵀΣ⁻¹1 over pseudonymous behavioral residuals; no personhood classifier needed.
3. Is global consensus avoidable? → Yes (math core §4.4): payments are consensus-number-1; demurrage is local arithmetic; timestamps by causal entanglement; only contested allocation needs zonal BFT.
4. disp ↔ network boundary? → Answered (math core §1.5): the network API is disp's effect algebra (store/send/eval/price = the three MATERIALIZE edges + price field).
5. Smallest meaningful pilot? → Specified (math core §6): estimands L̂, Σ̂, calibration, VOI-routing; ~30–100 resolvers, ~100–300 questions.

The remaining genuinely-open list (arms-race floor, deference-curve empirics, preference-delegation semantics, open-economy monetary dynamics, non-convex market design, and the coupled stability proof) is in math core §8.

Open Questions — Research Scratchpad

🤖 AI-drafted, human-directed — what that means.

Raw questions that seeded the analytical notes. Each is tagged with its current status; several were taken up and answered (or reduced to measurable quantities) in Evolution and mathematical-core.md. Kept here as a provenance trail and a list of what genuinely remains.

1. Unification vs. anonymization. How does the unification of storage/compute/routing square with the anonymization of routing? What is a node in this context? Is there a connection to cryptography here? → Partially addressed. "What is a node" is answered: a node is a Markov blanket / agent solving one control problem (Evolution, mathematical-core.md §0), and the unification is formalized as min-cost flow over the three MATERIALIZE edges (mathematical-core.md §1). Still open: the tension between that unification and anonymous routing — anonymization deliberately hides the spacetime path that the min-cost-flow optimizer wants to expose — and whether there is a clean cryptographic framing of it.

2. Connections to compiler theory. Is there a relation between the unification and optimizing compilers / superoptimization? What is the existing mathematical research (topological / categorical connections)? Can we flesh this out mathematically, perhaps in disp syntax? → Still open. Gestured at — tree-calculus reduction steps give a machine-independent work metric and disp is the value algebra (mathematical-core.md §1.1, §1.5) — but the superoptimization / categorical connection is not developed.

3. When are Markov blankets a useful abstraction? Node-level blankets (inputs/outputs) feel natural, but the others feel more constructed / drawable in many ways. When is the abstraction load-bearing rather than decorative? → Partially addressed. The honest answer is that the blanket framing is a design heuristic and organizing language, not free predictive math (Evolution, "FEP is slippery"). mathematical-core.md §1.3 de-metaphorizes it by choosing the free-energy functional and deriving alignment (Prop. 1). The non-arbitrary blankets are exactly the three nesting levels node ⊂ zone ⊂ network (§0); other drawings are not claimed to be load-bearing.

4. The dark-room problem. Is there LessWrong / FEP literature on the dark-room issue, with obvious solutions / regularizers? Or does it solve itself by running into the real world? → Resolved (in principle). The dark room is the FEP failure mode where an agent minimizes surprise by predicting a trivial niche. It is formalized as a bifurcation at deference slope L = 1 (mathematical-core.md §2, Prop. 3 — viability inequality V1) and countered by epistemic-value rewards and reality-coupling (Evolution). "Solves itself by hitting reality" is exactly the reality-coupling / between-network-selection answer (Evolution).

5. Is Sybil resistance elegantly solvable? Is there a theoretically clean solution, or only best-approximation? If approximate, who wins, and are there provable bounds? → Largely resolved / reframed. Stop asking "is this a person?"; ask how much independent information the constituent contributes — n_eff = 1ᵀΣ⁻¹1 (mathematical-core.md §3, viability inequality V2). Provable content: the mimicry-cost bound (Prop. 6) — faking k agents costs ≈ being k agents — but it is asymptotic. Genuinely-open residue: the arms-race floor at finite monitoring richness (mathematical-core.md §8, Open-1).

6. A framework for coupling Markov blankets. Is there a mathematical framework for the general nature of coupling between blankets — correspondences between the modeled ("demiurge") world and real life? → Partially addressed. Cooperation is formalized as a continuous coupling field over the latency-trust coordinate space (Evolution, mathematical-core.md §5): edge weight = degree of fate-sharing. A general theory of coupled blankets — and the model-vs-reality correspondence the "demiurge" phrasing was reaching for — is not yet developed; it overlaps with the still-open grand-conjecture stability proof (mathematical-core.md §7).

From the money-layer redesign (pool-equity currency)

Raised by the Money rework and its proposed crux resolution. These are the honest residue after that resolution — what it still owes.

7. Measuring the cooperation field κ. Zone boundaries and the δ-coupling are defined by fate-sharing κ, read off the residual-correlation matrix Σ — the same object the immune system reads as Sybil-correlation, but with opposite sign. Can honest dense cooperation (common-shock exposure, autonomous action) be separated from puppetry (shared private controller state, mimicked independence) at finite monitoring richness? → Open. The same arms-race floor as the mimicry-cost bound (mathematical-core.md §8, Open-1) — but now load-bearing for where zones are drawn, not only for voting weight.

8. Stability of the δ_min homeostatic loop. The global baseline is set by a controller that senses concentration through the truth machine's (lagged) accuracy and adjusts demurrage. Plant lag ~1/δ plus sensor delay τ make a naïve bang-bang / high-gain-integral controller oscillate — monetary boom-bust. A damped PI controller with slew-limited δ over a low-pass-filtered signal should be stable, but the stable-vs-limit-cycle boundary is unmapped. → Open — first thing to simulate. Sweep (k_p, k_i, τ, δ-slew) in the coupled agent-based model; this is the money-layer slice of the grand-conjecture stability question (mathematical-core.md §7). Controller sketch in Money.

9. Bootstrapping the set-point. Homeostasis (Q8) needs a working truth machine as its sensor — but the truth machine needs dispersed capital (A2), which needs δ_min. Chicken-and-egg. Likely answer: start δ_min as a conservative constant and hand control to the loop once the truth machine is live and calibrated — but the hand-over criterion is unspecified. → Open.

10. Transparency vs. privacy of zone metabolism. Boundary-reputation enforcement (illiquidity-punishing havens) requires neighbours to observe a zone's δ_z and concentration. That transparency fights the currency's transaction-privacy goal (the zk / secure-aggregation story). What is the minimal disclosure — perhaps a zero-knowledge proof of "δ_z ≥ δ_min and my Gini ≤ β" — that enables enforcement without exposing the cap table? → Open. Points toward a proof-of-compliance primitive rather than raw disclosure.

Working Note · Predictive Materialization

🧪🤖 Working note (AI-drafted — provenance). Source material for the rewritten Agent and Market chapters. Denser than book prose; claims here are design candidates, not settled spec.

The idea in one paragraph

Storage, retrieval, routing, and verification are one activity: spending resources to reduce uncertainty about a value until you are confident enough to act, where "enough" scales with the stakes. A stored representation is a policy for future materialization, not a pile of bits. A retrieval query is a partial context to complete. A route is a step toward someone whose model completes it better. A verification is the same purchase pointed at "is this real." Exact bits and hash equality are not a different system; they are the infinite-certainty limit of this one.

Storage: the four-way trade

The naive form of the idea ("storage = model + residuals") optimizes description length only. The real objective has four terms, and the MATERIALIZE flow problem was already carrying three of them:

Representations are points on the Pareto frontier over these four:

• raw bits — max storage, zero decode, zero distortion, min latency;
• compressed bits — fewer bits, some decode, zero distortion;
• model + residual — few bits, more decode, still zero distortion (the residual corrects the model);
• model only — minimal bits, decode cost, known nonzero distortion carried as a confidence label.

Which point is optimal is decided pointwise by the price field: near hot demand latency dominates (hold raw); mid-field, model+residual; in the far tail, model-only. Storage tiers, CDN behavior, and semantic caching fall out as geometry over one field. The objective, informally:

minimize  Σ demand-weighted [ bits·t  +  decode-compute  +  latency-penalty  +  E[distortion]·stake ]

The generative option does not replace the market's flow problem; it widens the action set with edges that carry a new attribute (distortion/confidence), and the market prices that attribute like any other.

Retrieval: uncertainty descent

A request is a partial context. The serving node either completes it above the caller's confidence threshold (cheap, possibly lossy, labeled), or routes toward nodes advertising lower expected loss on this context type. Routing becomes mixture-of-experts gating over the network; the old data-trail-search breadcrumbs generalize to descent on an uncertainty gradient. The residual store holds what no model predicts, priced higher; hash-anchored canonical data is the expensive exact tail.

"Content addressing" then generalizes: agreement among independent models is probabilistic content addressing; hash equality is its deterministic limit, reached when the stakes justify the cost.

Verification: one curve, not a menu

The old substrate chapter offered a verification menu (re-execute, sample k executors, bisect disputes, check a predicate). These are points on one curve of certainty per unit cost, and the right frame is sequential evidence purchase (Wald's SPRT: optimal stopping for hypothesis testing):

• each check is an action with a cost and an expected likelihood-ratio contribution;
• spend while marginal value of certainty (stake-dependent) exceeds marginal cost;
• stop at threshold τ(stake), or when the budget runs out — then serve with the achieved confidence and its label.

k-replication was already on this curve (P[fraud] ≤ (1−h)^k: multiplicative confidence at linear cost). Model-agreement checks extend the curve downward into cheap territory; re-execution and hash equality sit at the top. Some claims never cross the threshold; they are served as uncertain or not at all.

Soundness rules and open risks

• The confidence label is load-bearing. A generated completion must never silently substitute for canonical bits. The label is what separates memory from hallucination; dropping it is data corruption, not lossy compression.
• Query privacy regresses. A partial context is a far richer surveillance surface than a hash request. The anonymity layer has new work (onion-wrapped inference? query splitting?), though generative serving also adds deniability in some regimes. Unresolved either way.
• Model economics. Who trains, who pays, how staleness is priced. Connects to the open/closed split in the market chapter: coordination models (routing, pricing — the shared expectations the market needs to clear) are Schelling infrastructure and want to be open, funded as public goods; content models can stay closed and sell verified inference, since the certainty market never needs to see weights. The network buys a model open when the estimated value of diffusion exceeds the buyout price — retroactive public-goods funding, priced by the truth machinery.
• Adversarial models poisoning the cheap end of the verification curve; the immune machinery (independence-weighting of model agreement) is the intended counter, undesigned.

Working Note · Coupling and Merging

🧪🤖 Working note (AI-drafted — provenance). Source material for the rewritten Coupling, Money, and Governance chapters.

The move

Two agents couple when the predicted surplus of coupling exceeds its cost, through instruments that are explicit, priced, and reversible. The instruments, and what each one is the seed of:

Nothing forces cooperation. The protocol's job is to make coupling cheap to enter, visible in the ledgers, and cheap to exit, so it happens exactly where surplus exists — the anti-Moloch stance is "easy and explicit," never "mandatory."

The growth story (1 → 2 → 4 → 8)

Every institution has a birthday in this sequence:

• n=1. A node alone: its storage is its model plus residuals, its truth is its posterior, its money is meaningless.
• n=2. Coupling for redundancy, specialization, latency. Attestations begin ordering; the credit line begins money; comparing notes begins truth. Governance is unanimity.
• n=4. Transitive vouching (the first independence inference); credit lines net multilaterally (the clearing layer is born); a 2-of-3 witness quorum becomes possible; the two pair-ledgers get an exchange rate.
• n=8+. The first stranger arrives and Sybil pressure begins; a hub emerges because it is central; the coalition notices hoarding and reaches for demurrage; unanimity stops scaling and voting appears.

Merging two established networks is the n=2 story again with bigger participants. Nothing new happens; the messages are larger.

Coupling surplus, formally (sketch)

Couple/merge when G(A,B) − C(A,B) > 0, where

• G = exploitable shared structure: common demand (shared caching), complementary specialization (gains from trade), risk pooling (variance reduction), netting (liquidity savings). In free-energy terms F_{A⊕B} = F_A + F_B − G.
• C = coordination cost: consensus latency across the joint boundary, policy compromise, capture exposure.

κ(A,B) is the accumulated, observed G − C. Diminishing G with distance/heterogeneity plus growing C with size gives an optimal coalition size — Mundell's optimal currency areas and the Coasean firm boundary as two instances of one inequality. Self-similar: the same formula at node, cluster, and network scale.

Money is the merge protocol

Under pool-equity money, every trade leaves you holding shares of counterparties' pools. Cross-holdings make ∂(my welfare)/∂(your survival) > 0 — fitness alignment in the multi-level-selection sense, implemented by the cap table. So networks do not decide to merge: they trade, and trade is merging in slow motion. As cross-holdings deepen, κ rises, shared quorums and a shared demurrage floor start paying for themselves, and the exchange rate drifts toward 1. The merge is the limit of entanglement; de-merging is symmetric (sell down, exit). This answers "why would nodes merge" with an accounting identity instead of an argument.

Identity is subjective, local, and stake-scaled

There is no global registry and no global correlation matrix — another instance of "no global anything." Each node (and, in practice, each zone for its own drip) maintains a local posterior over each counterparty, fed by evidence streams:

1. locality — latency triangulation, physical presence: attackers fake nodes, not nodes everywhere;
2. trusted authorizers — the desk returns, demoted from root to evidence: an attestor whose track record is itself auditable;
3. stake-backed vouching — vouchers lose stake if the vouchee proves to be a puppet;
4. behavioral residuals — the paid-work streams;
5. continuity — hash-entangled history; aging an identity is costly;
6. hardware commitment — a disk commits once.

Required confidence scales with value at risk: a small trade needs a weak posterior, a large vote a strong one. For distant agents you never compute fine-grained n_eff at all — confidence composes along trust paths with attenuation, exactly like exchange rates. Identity confidence, trust, and χ are one family: fields over the body map that compose along routes. This dissolves the identity trilemma (strong/cheap/private) — it constrained a single global credential that no longer exists.

Delegation, resolved by the same accounting

Under quadratic voting, k independent aligned voters with budget c each cast k√c votes; a bloc pooling kc into one delegate casts √(kc) — the two endpoints of the independence spectrum. One formula interpolates:

votes(bloc)  =  √( n_eff(bloc) · pooled credits )

ρ=1 gives √(kc) (one mind, pooled purse); ρ=0 gives k√c (k minds). Delegation is voluntary correlation, priced by the machinery that already prices involuntary correlation — no new mechanism. And the position on the spectrum is measurable: delegators who sometimes override or split demonstrate independent preference-formation and earn toward k√c; blind followers converge to √(kc).

Two distinct axes fall out, answering "real but not trusted": distinctness (how many sources — Σ's shared-noise component) vs. trust/alignment (do I want shared fate — κ, built from history). Co-governance requires both; neither implies the other.

Open

• Separating the Σ components (shared source vs. shared fate vs. honestly-shared truth) at finite monitoring richness — the load-bearing estimation problem of the whole design.
• Clearing-layer mechanics: who runs a clearing node, its incentives and attack surface, quote manipulation at thin boundaries.
• Preference-independence measurement for delegation without incentivizing performative disagreement.

Working Note · Protocols as Priors

🧪🤖 Working note (AI-drafted — provenance). Source material for the rewritten Overview, Coupling, and Evolution chapters, and the frame for the whole v4 book.

The reframe

An institution is a shared prior over joint behavior. Two humans who have never met can cooperate because evolution and culture pre-loaded them with agreements: reciprocity norms, fairness instincts, what counts as salient. Schelling focal points work only because salience is prior-relative. A protocol is exactly this artifact for machines — shipped, pre-agreed expectations about what messages mean, what counts as defection, and which equilibrium to play. Cooperation capital, delivered before any interaction.

This changes the book's epistemic posture. The mechanisms (demurrage, log scoring, quadratic voting, the δ floor) stop being claims of correctness and become good shipped priors with explicit update paths. Even the protocol is not global truth; it is a shared prior that pays to become a better posterior.

The three-timescale loop

The biology maps exactly: genome = hyperprior, development = prior instantiation, lifetime learning = posterior. The five viability conditions redistribute as the ways each loop's refinement breaks.

The scoring rule already says the fast loop's half: p_ref is the prior, and every payment is payment for beating it. One invariant covers the whole economy:

The Alignment Invariant. Every payment in the system is a verified marginal reduction in some identified stakeholder's expected loss, relative to the shared reference. (Market scoring rule: expected payment = Bregman-divergence improvement toward the verdict. Substrate market: profit = −ΔF. Aggregation weight: marginal precision. If a proposed mechanism's payment cannot be written this way, it is out of spec.)

Then: markets pay for beating the shared prior; governance chooses the priors that evidence cannot pin (values, parameters); evolution replaces the prior-generating machinery itself.

Protocols as typed message-passing programs

The theory this wants exists: multiparty asynchronous session types. A protocol is a global type, projected to local types per role; well-typed processes provably satisfy session fidelity, deadlock-freedom, and progress. The Curry–Howard bridge (Caires–Pfenning; Wadler's Propositions as Sessions) makes session types linear-logic propositions and message-passing programs their proofs.

disp is an unusually good host:

• A protocol spec is a global session type = a content-addressed tree: protocols have hashes, versions diff, no registry needed.
• disp types are predicates, so sessions can be dependent: a message slot can require "a share-transfer signed by the sender whose balance covers it." Types-as-predicates-on-results extends to types-as-predicates-on-transcripts.
• The network API is already disp's effect algebra (store/send/eval/price); session types over that signature are the missing disp networking story — serialization and interaction discipline defined together.
• Causal entanglement makes every run's transcript hash-entangled: a protocol run is an auditable proof object, and behavioral disputes resolve by bisection over the transcript, exactly as computation disputes bisect reduction traces. A contract = session type + collateral + arbitration over the typed transcript. The κ field gets a concrete carrier: the set of live typed sessions between two parties.

What makes a protocol good — four levels

1. Sound (logic): well-typed under the session discipline — fidelity, deadlock-freedom, progress; transcripts checkable against the type. Caveat carried openly: session types give safety, not liveness against rational silence; timeouts and abort-with-collateral are where incentives must take over.
2. Incentive-compatible (game theory): in the typed game (deviations within the type, plus silence/abort), intended play is an equilibrium; deviations are detectable (typed transcripts) and punishable (collateral, reputation). Distributed algorithmic mechanism design, with the Alignment Invariant as the bridge.
3. Self-measuring (statistics): the protocol's parameters are identifiable from its own transcripts — it emits the estimands needed to refine it (L̂, Σ̂, controller signals) — and online refinement has bounded regret. A protocol that cannot measure itself cannot serve as a refinable prior.
4. Evolvable (selection): new versions relate to old by typed coercions at boundaries. Backwards compatible = a coercion exists, so old and new agents still transact (protocol versions interoperate like currency zones; a hard fork is a boundary with no coercion — χ undefined). Forward-pressuring = the new type strictly dominates in surplus for adopters without requiring global switchover: adoption is a ramp, not a cliff.

What stays fixed (the relativism defense)

If everything is a prior, is anything load-bearing? Yes: the conditions on the refinement loop. Priors vary and are meant to be outgrown; soundness, incentive-compatibility, self-measurement, and evolvability — with V1–V5 as their runtime expressions — are what the book actually asserts as true. The claim is never "demurrage is correct"; it is "any shipped prior whose refinement loop violates these conditions detaches from reality or gets captured."

Open

• Instantiating MPST over disp concretely (projection, subtyping/coercion calculus, the timeout/abort discipline).
• The synthesis direction: given a session type as spec, synthesize the agent strategy (connects to disp's program-synthesis agenda).
• Formalizing "forward pressure" (surplus dominance along adoption gradients) game-theoretically; when do ramps exist at all?

Application Design Philosophy

Application APIs should be future-proofed as much as possible

Application GUIs should be generally be as cross platform as possible.

There should be different modes of use for different user skill levels to try and accomidate as many people as possible.

• These modes should enable/disable various aspects of configurability in the application
• Simple mode should be super easy to use for anyone and come with a quick tutorial
  • Configuration options should be assumed as much as possible,
• Default mode should come with a tutorial and explain how modes work and how to change them
• Advanced mode assumes the user already knows how to use the app or can figure it out on their own. All configurability functionality should be enabled.
  • Application developers should strive to implement as much configurability as possible into every aspect of their application for complex users.
  • For desktop applications, configuration on a small level (for specific contextual buttons or features) could be shown through right-click menus.
  • For mobile applications these configuration items can be shown through a long-press.

Why Copyright is Cancer

Copyright is ineffective in the internet age. Any piece of content can be pirated on a massive scale without the original creator even knowing. The only thing copyright does is suppress derivative creation on forums with strict copyright adherence. See these videos on why the current copyright system is broken and why it makes no sense. This document outlines an alternate system of funding creation that Dither aims to create.

Have a decentralized & democratized forum of communication and publication that no institution or individual can meaningfully obstruct. Dither is a protocol for creating decentralized applications.

Provide a direct connection of support between the consumers and the creators allowing people to directly support creators for the content they produce (Paying for Production, not Distribution). Even huge projects like Marvel movies or video games can be supported through community fundraising. Each production raises the reputation of the artist(s), allowing them to raise more money from their fans for the next project.

Preventing artists from having control over their art allows all artists to use and adapt and recreate the work of other artists. Imagine all the games and movies that were super hyped up but totally flopped. If anyone was allowed to make anyone else's work but better, artists have an incentive to make good stories so that they can raise money for their next project, instead of making something crappy and making all their money off of preorders. This would also prevent the fragmentation of distribution services as no one entity can have exclusive control over a piece of content.

But What IS a Monad?

This is an article that defines what a Monad while trying to stay as close as possible to its roots in Category Theory.

If you've ever heard of monads in the context of functional programming, you've likely also heard that they are just "Monoids in the Category of Endofunctors". But what does that even mean?

Well for one, this quote could use some more detail. The more descriptive quote is: "a monad in $\mathcal{X}$ is just a monoid in the category of endofunctors of $\mathcal{X}$, with product replaced by composition of endofunctors and unit set by the identity endofunctor."

Throughout this article, we will define "Category", "Endofunctor" and "Monoid" in a fictional programming language, and put these definitions together to define a Monad.

A Category (designated using curly letters, i.e. $\mathcal{C}$), for the purposes of this article, is simply some defined set of mathematical objects (numbers, values, types, terms, or even categories themselves) as well as some set of "morphisms" defined between the objects. A category also has some conventions such as:

• Every object has a morphism to itself (called the identity morphism)
• Morphisms can be combined together into other morphisms much like function composition.
• Morphism composition is associative.

Functors in category theory are simply morphisms in a "category of categories". Where categories are objects and functors are just morphisms that map one category to another. This can be visualized at either a high level of abstraction where the functor $f$ is just one arrow between two categories ($f:\mathcal{A}\to \mathcal{B}$). Or at a lower level where a functor could be represented as many parellel arrows uniquely mapping every object and morphism in category $\mathcal{A}$ to an object or morphism in category $\mathcal{B}$.

Endofunctors are just the special case where the two categories $\mathcal{A}$ and $\mathcal{B}$ are the same category. Endofunctors are essentially just a mapping between each object and morphism in a category to another object/morphism in the same category.

Endofunctors come up often when working in a specific Categories. For example all the types in a programming language, form a category, where the morphisms are functions defined between them. This category is called Type, and can be thought of as the "type of types".

There are many programming languages that have endofunctors. Option in Rust, Maybe in Haskell, or really any kind of wrapper type in any language is technically an endofunctor. While most languages have endofunctors, most don't have a powerful enough type system to formalize the definition of endofunctors itself. (For rustaceans, think of it like a trait that you can implement for type constructors like Option or Vec, not Option<T>, just Option by itself. this is not yet possible in rust at the time this article was written).

When defining what it means to be an Endofunctor formally within a given Category of objects (i.e. a Type System), one needs a powerful dependently-typed lagnuage. For the purposes of this article, we will be using a fictitious dependently-typed language. The following is an outline of some of the general features of the language.

Pseudocode Definitions (skip this if you think you can grok the psuedocode first try):

#![allow(unused)]
fn main() {
//This binds some name to a value and an optional type in some context. The value may be a type (like Vec(A)) or a value (like `3`)
let <name> [: <type>] := <value>
}

#![allow(unused)]
fn main() {
// A name bound to a value may be given a valid type, or one will be inferred.
// This will have an inferred type of `Nat` because 3 is most likely to be a natural number
let three := 3
// The type of `four` is specified to be `Int` (i.e. an integer). This is valid because while 4 is inferred to be Nat, Nat is a subset of Int (integers).
let four : Int := 4
// This is a type declaration. `Likelyhood` is defined to be a set of two numbers of type `Nat` and `Float`. Optionally, two names have been assigned to these numbers: `count` and `probability`. Curly braces represents un-ordered sets of objects. The inferred type here is the type of all types: `Type`.
let Likelyhood := { count: Nat, probability: Float }
// The above definition is semantically equivalent to this definition:
let Likelyhood := { probability: Float, count: Num }
// But the above two expressions are semantically different from this: (Brackets represent an ordered set, i.e. a list)
let LikelyhoodOrdered := [ count: Num, probability: Float ]

}

# If you are used to haskell-based currying the first two definitions of `Likelyhood` would be similar to providing two constructors for the same object like the following:
Likelyhood_a : Num -> Float -> Likelyhood
Likelyhood_b : Float -> Num -> Likelyhood
# But the 3rd definition would only be equivalent to Likelyhood_a

#![allow(unused)]
fn main() {
// type constructors can be defined like this, with an implicit `->` between `{A:Type}` and `{ x:A, ... }` for structure types.
let Vector3 := { A : Type } { x : A, y : A, z : A } -> { x, y, z }
// or like this using the `Struct` constructor.
let Vector3 := { A : Type } -> Struct { x : A, y : A, z : A }

// type constructors may also be defined for enum types.
let Option := { A : Type } -> Type;
let Some : {A : Type} { a : A } -> Option(A);
let None : {A : Type} -> Option(A);
// or like this using the `Enum` constructor.
let Option := { A : Type } -> Enum {
	Some { a : A },
	None,
}

// Terms can also be named and typed, but left undefined (like how functions in rust traits don't need to be defined immediately)
type add_one : Num -> Num

// This is defining add_one in accordance with its previous defined type. It would be an error to define add_one twice with different implementations in the same context, or for the definition to fail to typecheck with the defined type.
let add_one { a } -> a + 1 
}

Okay now that we have pseudocode out of the way, we can get to defining an endofunctor :D

In this language, Endofunctors can be defined as a dependent collection of two functions: obj_map: A -> F<A> which maps the object of the category, and a function func_map that takes an arbitrary function A -> B where B can be anything and returns a function F<A> -> F<B>, mapping the morphisms from the initial object. When this definition is used as a "type class" (think trait or interface), it allows the user to abstract over a specific endofunctor definition and refer to all endofunctors as a collective.

#![allow(unused)]
fn main() {
// An Endofunctor is a class of type functions that takes a type constructor F of type `Type -> Type` (i.e. a type constructor / wrapper). Classes are essentially partially defined functions where the return value (implementation) is only valid if it has been implemented. 
let Endofunctor := { F : Type -> Type } -> {
	// Specifies how the endofunctor maps objects of the category `Type` to other objects in `Type` using the type constructor `F`
	type obj_map : { A: Type } -> { obj : A } -> F(A);
	// Specifies how the endofunctor maps functions between objects in the category `Type` to other functions in the "extended" F(Type) category.
	type func_map : { A : Type, B : Type} { func : A -> B } -> F(A) -> F(B);
}
}

Examples of various type constructors and specifications (implementations) of the Endofunctor class.

#![allow(unused)]
fn main() {
// Option is a generic enum, i.e. a function of type `Type -> Enum`
let Option := { T : Type } -> Enum {
	Some[T],
	None,
}
// Defines the Endofunctor "function" for a specific type constructor.
let impl_option_endofunctor : Endofunctor(Option) := {
	let obj_map := {A} Option{A}::Some
	// Returns Option{A} -> Option{B} from `func : A -> B`
	let func_map := {A, B} {func} -> (
		{ fa : F(A) } -> match fa {
			Some[t] => Some(func(t))
			None => None
		}
	)
}
}

#![allow(unused)]
fn main() {
// List is a generic List, i.e. a function from Type -> Enum.
let List { T : Type } -> Enum {
	Cons { list : List T, value : T }, // Recursion defined implicitly
	Nil,
}
// Defines a term of type Endofunctor(List).
let impl_list_endofunctor : Endofunctor(List) := {
	// Creates a constructor that takes a `value : T` by partially applying the List::Cons variant.
	let obj_map := {A} List{A}::Cons { list: List{A}::Nil }
	let func_map := {A, B} {func} -> ({fa : F(A)} -> match fa {
		Cons{list, value} => Cons(func_map{A,B}{func}{list}, func(value)) // func_map is recursively defined for list
		Nil => Nil
	})
}
}

#![allow(unused)]
fn main() {
// Generic Identity Function Constructor, we need this to define unit for the Monad.
let identity : [T : Type] [T] T := [T] [v] v
}

NOTICE: New Syntax / Convention, identity is the name assigned to the expression [T] [v] v. But that expression has also been associated with a type ([T : Type] [T] T). The type of identity is automatically assigned the name (translated from snake_case to PascalCase): ^Identity.

There is analogous syntax for finding a value defined for a given type. This may not always resolve because multiple values could be defined for a given type.

Example: to refer to impl_option_endofunctor, you can instead infer the definition via: _ : Endofunctor(Option).

This also works when defining "implementations". You can use _ to not define a name, and instead let type inference do its job.

#![allow(unused)]
fn main() {
let _ : Endofunctor(Identity) := {
	// object map returns the identity function on `A` (i.e. A -> A).
	let obj_map := {A} identity(A)
	// func_map returns the function as-is because identity does nothing.
	let func_map := {A, B} {func} -> func;
}

// The above is somewhat equivalent this in rust :)
impl Endofunctor for Identity {
	fn obj_map<A>() -> impl Fn(A) -> A { |a| a }
	fn map<A, B>(func : impl Fn(A) -> B) -> { func }
}
}

OKAY, Now we can get back to defining the Monad! According to category theory, a monad in a category is a monoid in the category of endofunctors of some other category. We know what endofunctors are, what what is a Monoid?

Wikipedia describes a monoid (the algebraic version) as an set ($M$) equipped with a binary operation we will call "multiplication" ($\ast$) and particular member of the set $M$ called the "unit".

To translate this into a typed programming language, we will define a monoid as a type class (a.k.a. trait / interface) that can be implemented for objects. The definition of the type class is parameterized on the set $M$ and includes a set of two morphisms, unit and multiplication as well as the two monoid laws of associativity and identity: $\forall a,b,c:(a\ast b)\ast c=a\ast (b\ast c)$, and $\forall a:a\ast unit=unit\ast a$

#![allow(unused)]
fn main() {
let Monoid := [ M : Type ] {
	type unit : M,
	type multiplication : [a : M, b : M] -> M
	type associativity_proof : { a : M, b : M, c : M } multiplication[multiplication[a, b], c] = multiplication[a, multiplication[b, c]]
	type identity_proof : { a : M } multiplication(a, unit) = multiplication(unit, a)
}
}

Now that we have a monoid, we can talk about types that are monoids! (Assume Nat is the type of natural numbers)

#![allow(unused)]
fn main() {
// The natural numbers under addition form a monoid
let naturals_addition_monoid : Monoid(Nat) := {
	let unit := 0,
	let multiplication := add,
	// The actual type theory proofs will be left because I haven't figured out how they work yet lol.
	// addition is associative
	let associativity_proof := { a, b, c } <proof of associativity> 
	// 0 + anything = anything + 0
	let identity_proof := { a } <proof of identity>
}
}

Alright lets do another one! This time on a type we've actually seen before (I'll add type annotations for unit and multiplication so its more clear what is going on here)

#![allow(unused)]
fn main() {
// This is a generic monoid definition for all possible Option types. i.e. `impl<A> Monoid for Option<A>`.
let impl_option_monoid : {A : Type} Monoid(Option(A)) {
	// The `unit` must satisfy the identity law with respect to `multiplication`
	let unit : Option(A) := Option(A)::None
	// The combination is essentially `Option::or`, it combines two Options together and returns the first if it contains a value, or the second.
	let multiplication := [a, b] -> match a {
		Some(a) => Some(a),
		None => b
	} : [Option(A), Option(A)] -> Option(A)
	let associativity_proof := { a , b , c } <proof of associativity>
	let identity_proof := { a } <proof of identity>
}
}

Oh wait, since Option is a endofunctor, isn't this just a Monad? (A monad is a monoid defined on an endofunctor after all) Lets try and pinpoint a more exact definition of a Monad :)

#![allow(unused)]
fn main() {
let Monad [ M : Type -> Type, M_is_functor : Endofunctor(A) ] {
	unit : M,
	multiplication : [a : M, b : M] -> M
	associativity_proof : { a : M, b : M, c : M } multiplication[multiplication[a, b], c] = multiplication[a, multiplication[b, c]]
	identity_proof : { a : M } multiplication(a, unit) = multiplication(unit, a)
}
}

If you followed all that, You should now have a good understanding of what a Monad is! (Specifying it is essentially the same as a monoid above)

How to do Node Discovery while Limiting Structural Data Leakage?

Debate: When requesting for peers from a connected node, should the node send back a list of publically available IPs? Or should the node forward the request to the peers in question for them to initiate the connection?

• Sending back a list of public IPs
  • Pros:
    • Faster to implement (kinda)
    • Used by most networks
    • Possibly easier NAT tunneling? (because it doesn't inherently require an open listening port)
  • Cons:
    • Allows for easy enumeration of all nodes in the network by just requesting lists and connecting to nodes.
• Notifying peers of the request for them to connect.
  • Does this strategy actually prevent peer enumeration?
    • People with a single computer can just request for a given node and ask for new connections to get IPs. If they repeatedly do this from different IPs, they can get a good sense of all the peers of a given node.
      • Fix: Instead of just forwarding request to all peers. Forward request to one peer at a time and have that peer report back the measured latency. Forward request to another peer if first peer reported greater latency than request receiver. If peer was in fact closer to requester, that peer connects to the requester and the process starts anew.
    • Even if the peer selection is random and only ~2-3 peers will be contacted at any given peer request, you can still repeatedly ask the node to eventually get a list of all peers.
      • Fix: Request is only forwarded to peers of a similar latency to the requesting node. This also likely makes for quicker convergence to a "local group" as well as prevents closeby peers from needlessly connecting to requesters.
      • Also: any node after the initial requested node can use network coordinates to more accurately recommend closer and closer nodes to forward the peer request to.
    • If someone can place nodes relatively close to a given node (<30ms), they can do peer requests at various distances and eventually sus out all the peers.
      • One way to lessen this attack would be for some peers to simply not participate in the discovery process. Its not a process that needs a whole lot of participants anyway. These peers could initialize connections only if arbitrary (user-defined based on threat profile) conditions are met such as:
        • Predefined trust between the requester and the peer (via any kind of trust-system like web of trust, trusted certificates, or simply a set of trusted public keys)
        • A lighter requirement might be that the node must be connected to some subset of their peers.
        • A third requirement could be that the node must be within a certain latency radius as predicted by routing coordinates.
      • No matter how many times unknown IPs request for peers from a given node, it will never forward the request to these peers.
      • Another way to lessen this kind of attack might be to require new nodes to have knowledge of multiple bootstrap nodes before new peers may be requested. These bootstrap nodes could then collude to decide which of their peers is closest to the requesting node and forward the request on to them (passing along all the latency measurements and routing coordinate estimate data for even more accurate positioning).
    • At this point, I think only an ISP would be able to figure out peer relationships via faking nodes and doing peer requests, and it would probably be easier for them to just do direct traffic analysis...
  • Pros
    • Provides wayy more protection from even relatively advanced adversaries trying to probe the structure of the network.
  • Cons
    • wayyy more complex
    • Slower for new nodes to bootstrap depending on design (although its likely fine once they already have established connections to nearby peers), could also potentially be speed up by web-of-trust stuff (i.e. bootstrapping off multiple friends).