Build a multiwriter database with Autobee

A Hyperbee is a key/value B-tree with one writer. Autobee gives it many. Each peer appends to its own local core, Autobee linearizes those cores, and an apply handler replays the result into a shared B-tree that every peer derives identically—so readers can get and range-scan instead of replaying a log.

This guide uses Corestore to hold the cores and Hyperswarm to find peers; see Work with many Hypercores using Corestore if those concepts are unfamiliar.

This guide consists of two applications:

• bee-base-writer-app - creates the database, writes records, and grants write access to other peers.
• bee-base-peer-app - joins by key, reads the view, and writes once it has been made a writer.

How Autobee differs from Autobase

If you have read Build a multiwriter app with Autobase, the shape is familiar—but four things change:

• The view is always a Hyperbee. There is no open handler to write in this guide; apply receives a writable B-tree and writes to it with view.write() batches.
• Values are raw bytes. Autobee has no valueEncoding, so both apps encode records as JSON buffers and decode them in apply.
• The raw 'update' event has different semantics. Autobase's update event fires after every apply cycle that changed the view. Autobee's does not: it fires only after an interrupt, or when db.writable flips—not on an ordinary view change. Use the update(view, changes) handler option (not db.on('update', ...)) to react to view changes, and db.on('writable', ...)—same as Autobase—to react to becoming a writer.
• Replicate through the database. db.replicate(connection) also registers the wakeup stream, which store.replicate(connection) would skip.

Create the bee base writer app

The bee-base-writer-app creates a new Autobee, announces it on Hyperswarm, and writes key/value records. It also accepts an add <writer-id> command that grants another peer write access.

Create the bee-base-writer-app directory and add dependencies

Start the bee-base-writer-app project with the following commands:

mkdir bee-base-writer-app
cd bee-base-writer-app
npm init -y
npm pkg set type="module"
npm install autobee@2.2.2 corestore hyperswarm b4a bare-pipe bare-process

This will install the following dependencies:

• autobee: A module for building multiwriter Hyperbee views.
• bare-pipe: A module for working with pipes, used here to read stdin.
• bare-process: A module for working with processes.
• hyperswarm: A module for working with Hyperswarm.
• corestore: A module for working with Corestore.
• b4a: A module for working with buffers.

Add the bee-base-writer-app logic

Create the bee-base-writer-app/index.js file with the following content:

The whole body of apply runs inside a try/catch (L19,L30–L32): apply replays this node on every peer forever, including on restart, so a malformed one—bad JSON, a typo, or a node from a hostile peer—must be skipped, never thrown. Inside that, apply decodes each node's bytes back into an operation (L20), routes an addWriter operation to host.addWriter (L22–L25), and writes everything else into the B-tree through a view.write() batch (L27–L29). The update handler is Autobee's post-apply hook, used here to print the whole view (L38–L44). Passing null as the key creates a new database rather than joining one (L47). Connections are replicated through db.replicate so the wakeup stream is registered too (L51), and the id is printed only once the topic has been announced to the DHT (L55–L58)—advertising it earlier races a peer that looks the topic up immediately. Reading stdin (L65–L76) splits input two ways: add <writer-id> appends an addWriter operation (L68–L72), and key=value writes a record (L73–L75).

import process from 'bare-process'
import Hyperswarm from 'hyperswarm'
import Corestore from 'corestore'
import Autobee from 'autobee'
import Pipe from 'bare-pipe'
import b4a from 'b4a'

const store = new Corestore('./bee-writer-storage')
const swarm = new Hyperswarm()
process.once('SIGINT', () => swarm.destroy().then(() => process.exit(0)))

// Reduce the linearized nodes into the Hyperbee view. Deterministic, and
// mutates only the `view` it is handed.
async function apply (nodes, view, host) {
  for (const node of nodes) {
    // apply replays this node on every peer, forever, including on restart.
    // A malformed value (bad JSON, a bad typo, or a hostile peer) must not
    // throw here — that would crash every peer that ever processes it.
    try {
      const op = JSON.parse(b4a.toString(node.value))

      if (op.addWriter) {
        await host.addWriter(op.addWriter)
        continue
      }

      const batch = view.write()
      batch.tryPut(b4a.from(op.key), b4a.from(op.value))
      await batch.flush()
    } catch (err) {
      console.error('skipping malformed node:', err.message)
    }
  }
}

// Autobee has no view-changed event: this handler is the hook that fires
// after apply, once per cycle that changed something.
async function update (view) {
  const entries = []
  for await (const entry of view.createReadStream()) {
    entries.push(`  ${b4a.toString(entry.key)}: ${b4a.toString(entry.value)}`)
  }
  console.log(`view (${entries.length} entries):\n${entries.join('\n')}`)
}

// key is null, so this call creates a new Autobee rather than joining one.
const db = new Autobee(store, null, { apply, update })
await db.ready()

// db.replicate, not store.replicate: it also registers the wakeup stream.
swarm.on('connection', (conn) => db.replicate(conn))

// Announce the topic before advertising the key, so a peer that looks it up
// straight away finds this writer.
const discovery = swarm.join(db.discoveryKey)
await discovery.flushed()

console.log('db key:', db.id)

await db.append(encode({ key: 'greeting', value: 'hello from the first writer' }))

const stdin = new Pipe(0)

// `add <writer-id>` grants write access. `key=value` writes a record.
stdin.on('data', (data) => {
  const line = b4a.toString(data).trim()
  if (!line.length) return
  if (line.startsWith('add ')) {
    const id = line.slice(4).trim()
    db.append(encode({ addWriter: id })).then(() => console.log('added writer:', id), console.error)
    return
  }
  const i = line.indexOf('=')
  if (i === -1) return console.log('usage: <key>=<value>  or  add <writer-id>')
  db.append(encode({ key: line.slice(0, i), value: line.slice(i + 1) })).catch(console.error)
})

function encode (op) {
  return b4a.from(JSON.stringify(op))
}

Run the bee-base-writer-app

In one terminal, run bee-base-writer-app with bare.

bare bee-base-writer-app

It prints the database id, then the view—which starts with the one record the app writes on startup:

db key: 9fa71efyo1am3xzaarpkx9n46nqahgutygmdhokaegfpinjadd7y
view (1 entries):
  greeting: hello from the first writer

Create the bee base peer app

The bee-base-peer-app takes the database id as a command-line argument and joins the same Autobee. It has no write access at first, so it replicates the view and waits.

Create the bee-base-peer-app directory and add dependencies

Create the bee-base-peer-app project with the following commands:

mkdir bee-base-peer-app
cd bee-base-peer-app
npm init -y
npm pkg set type="module"
npm install autobee@2.2.2 corestore hyperswarm b4a bare-process

This will install the following dependencies:

• autobee: A module for building multiwriter Hyperbee views.
• bare-process: A module for working with processes.
• hyperswarm: A module for working with Hyperswarm.
• corestore: A module for working with Corestore.
• b4a: A module for working with buffers.

Add the bee-base-peer-app logic

Create the bee-base-peer-app/index.js file with the following content:

apply is copied unchanged from the writer app (L18–L35)—both peers must derive the same view from the same nodes, and skip the same malformed ones rather than crashing on them. The database id arrives as a command-line argument (L7–L9) and is passed as the key, which joins the existing database instead of creating one (L48). db.local is this peer's own writer core; its id is what the writer app has to add (L52). After joining the swarm (L54–L55), the peer waits for the topic lookup to settle before drawing any conclusion from an empty swarm (L58) and runs a cycle with db.update() (L59). It then waits on the writable event if the grant hasn't already landed (L61–L64).

import process from 'bare-process'
import Hyperswarm from 'hyperswarm'
import Corestore from 'corestore'
import Autobee from 'autobee'
import b4a from 'b4a'

const key = Bare.argv[2]

if (!key) throw new Error('provide a db key')

const store = new Corestore('./bee-peer-storage')
const swarm = new Hyperswarm()
process.once('SIGINT', () => swarm.destroy().then(() => process.exit(0)))

// Identical to the writer app's handler: both peers must derive the same view
// from the same nodes. apply replays this node forever, including on
// restart, so a malformed value must not throw here — skip and move on.
async function apply (nodes, view, host) {
  for (const node of nodes) {
    try {
      const op = JSON.parse(b4a.toString(node.value))

      if (op.addWriter) {
        await host.addWriter(op.addWriter)
        continue
      }

      const batch = view.write()
      batch.tryPut(b4a.from(op.key), b4a.from(op.value))
      await batch.flush()
    } catch (err) {
      console.error('skipping malformed node:', err.message)
    }
  }
}

// Autobee has no view-changed event: this handler is the hook that fires
// after apply, once per cycle that changed something.
async function update (view) {
  const entries = []
  for await (const entry of view.createReadStream()) {
    entries.push(`  ${b4a.toString(entry.key)}: ${b4a.toString(entry.value)}`)
  }
  console.log(`view (${entries.length} entries):\n${entries.join('\n')}`)
}

// Passing the key joins the existing Autobee instead of creating one.
const db = new Autobee(store, key, { apply, update })
await db.ready()

// db.local is this peer's own writer core. Its id is what the writer app adds.
console.log('writer id:', db.local.id)

swarm.on('connection', (conn) => db.replicate(conn))
swarm.join(db.discoveryKey)

// Wait for the topic lookup to settle before deciding this peer is read-only.
await swarm.flush()
await db.update()

if (!db.writable) {
  console.log('read-only, waiting to be added as a writer')
  await new Promise((resolve) => db.once('writable', resolve))
}

console.log('now writable')

await db.append(b4a.from(JSON.stringify({ key: 'reply', value: 'hello from the second writer' })))

Run the bee-base-peer-app

In another terminal, run the bee-base-peer-app with bare and pass it the database id from the bee-base-writer-app.

bare bee-base-peer-app <SUPPLY THE DATABASE ID HERE>

It prints its own writer id, replicates the view read-only, and waits:

writer id: enwnbx883kxpngwhhyo3fmwds7qi4g17bmfi558xbr7apjexc4so
read-only, waiting to be added as a writer

Grant write access

Copy the peer's writer id and, in the bee-base-writer-app terminal, type:

add <SUPPLY THE WRITER ID HERE>

The writer app appends an addWriter operation. It replicates to the peer, whose own apply runs the same host.addWriter call and reaches the same conclusion—the grant is part of the shared, replayable history, not a side channel. The peer's writable event fires, and it writes its record:

now writable
view (2 entries):
  greeting: hello from the first writer
  reply: hello from the second writer

Within a moment the same two records appear in the bee-base-writer-app terminal. Both peers have converged on one B-tree built from two writers.

Type colour=green into the bee-base-writer-app terminal and watch the record show up in both views, sorted by key alongside the others. Only the writer app reads interactive input—the peer app writes its one scripted record and otherwise just replicates.

Where to go next

Both apps print the whole view on every cycle, which is fine for two records and wrong for a real database. The update handler's second argument is a changes object describing what moved—changes.get('view') returns the head from and to—so a real app diffs that range instead of rescanning. Key exchange is the other production gap: this guide passes ids through the terminal, whereas apps that pair over an invite use blind-pairing to hand the database id and the writer id over automatically.

See also

• Autobee reference—full API, including optimistic appends, anchors, and fast-forward.
• Build a multiwriter app with Autobase—the same pattern when the view is not a Hyperbee.
• Share append-only databases with Hyperbee—the single-writer B-tree this guide gives many writers.
• Work with many Hypercores using Corestore—the Corestore patterns this guide builds on.