🬟 🬑 🬂🬰🬂 🬂🬰🬂 🬒🬂🬟 🬒🬂🬟 ╭───────╮ 🬋 🬋 🬋 🬃 🬇 🬋🬋🬃 zig java │clojure│ typescript go 🬑 🬟 🬭🬰🬭 🬰 🬮🬭🬑 🬮🬭🬑 ╰───────╯ See codeboost/xitdb-clj on GitHub 🬳🬶 🬕🬧 🬳🬶 🬨🬕 🬳🬶 🬴🬗 🬳🬶 🬤🬒 🬴🬒 🬴🬒 🬴🬗 🬔🬧 🬺🬻 🬨🬕 🬲🬷 🬴🬒 🬴🬒 ▌▐ 🬨🬕 ▌▐ 🬴🬗 🬴🬒 🬞🬏 🬴🬗 🬳🬶 🬤🬒 🬨🬕 ▌▐ 🬲🬘 ▌▐ ▐▌ ▌▐ 🬲🬘 ▌▐ 🬱🬘 🬲🬭 ▌ 🬕🬢 🬣🬘 ▌▐ ▐▌ ▌▐ 🬲🬭 ▌ 🬣🬘 ▐▌ 🬣🬘 🬕🬢 🬲🬭 🬁🬀 ▌ ▌▐ 🬱🬘 ▐▌ xitdb is a database that... • Is embedded, like SQLite • Single file database • Runs as a library • Exposes data structures rather than a query language • SQL and other query languages are too high level • "Build your own database" with data structures instead • Is immutable, like Datomic • Lock-free reads • Easy replication • Has dependency-free variants in Zig, Java, TypeScript, and Go • Also a slick Clojure wrapper • Each is only 4-5k lines of code 🬤🬒 🬳🬶 🬣🬘 🬺▐ 🬔🬧 🬨🬕 🬔🬧 🬔🬧 ▌▐ 🬴🬒 🬴🬗 🬣🬘 ▌ 🬳🬶 🬺▐ 🬔🬰 ▌▐ 🬳🬶 🬔🬰 🬴🬒 🬤🬒 🬱🬘 ▌▐ ▐▌ ▌🬬 🬣🬘 ▐▌ 🬣🬘 🬈🬥 🬣🬘 🬲🬭 🬕🬢 ▐▌ 🬲🬭 ▌▐ ▌🬬 🬣🬷 🬣🬘 ▌▐ 🬣🬷 🬲🬭 🬱🬘 Pop quiz: is this query doing a full table scan? ╭────────────────────────╮ │SELECT * │ │FROM orders │ │WHERE customer_id = 42 │ │ AND status = 'shipped'│ │ORDER BY created_at DESC│ │LIMIT 10; │ ╰────────────────────────╯ It's impossible to tell without asking the query planner! In SQLite, that would be EXPLAIN QUERY PLAN SELECT. SQL is a high-level language, and the efficiency of a query can suddenly change without you noticing. 🬕🬧 🬳🬶 🬨🬕 🬳🬶 🬤🬒 🬨🬕 🬴🬗 ▌▐ 🬔🬂 🬨🬕 ▌▐ 🬴🬗 🬴🬒 🬤🬒 🬳🬶 🬴🬗 🬴🬒 🬳🬶 ▌ ▌ 🬣🬘 🬔🬧 ▌▐ 🬺▐ 🬴🬒 🬴🬒 🬕🬧 🬲🬘 ▌▐ ▐▌ ▌▐ 🬱🬘 ▐▌ 🬕🬢 🬣🬘 🬣🬭 ▐▌ 🬣🬘 🬕🬢 🬲🬭 🬱🬘 ▌▐ 🬕🬢 🬲🬭 ▌▐ 🬲🬭 🬲🬭 ▐▌ 🬣🬘 🬣🬘 ▌🬬 🬲🬭 🬲🬭 🬲🬘 ╭─────────────────────────────────────────────────╮ │(let [orders (:orders moment) │ │ index (:customer->status->orders moment)│ │ statuses (get index 42) │ │ shipped (:shipped statuses)] │ │ (doseq [[_ order-id] (take 10 shipped) │ │ :let [order (get orders order-id)]] │ │ ;; do something with the order │ │ )) │ ╰─────────────────────────────────────────────────╯ I can tell it's efficient just by reading the code! 🬨🬕 🬺🬻 🬴🬗 🬴🬒 🬕🬧 🬳🬶 🬺▐ 🬔🬂 🬴🬒 🬺🬻 🬨🬕 🬤🬒 🬺🬻 🬳🬶 🬨🬕 🬔🬂 🬲🬷 🬷🬲 ▌▐ ▌ 🬲🬭 🬲🬘 ▌▐ ▌🬬 🬣🬭 🬲🬭 ▌▐ 🬷🬲 🬱🬘 ▌▐ ▌▐ ▐▌ 🬣🬭 ▌▐ SQL returns flat rows, so you write glue code to rebuild your data: ╭──────────────────────────────────────────────────────╮ │(def rows (jdbc/execute! conn [" │ │ SELECT users.name, tags.tag │ │ FROM users LEFT JOIN tags ON tags.user_id = users.id│ │ WHERE users.id = 1"])) │ │ │ │(def user-name (:users/name (first rows))) │ │(def tags (keep :tags/tag rows)) │ ╰──────────────────────────────────────────────────────╯ xitdb stores your data in the shape you already use: ╭────────────────────────╮ │(def user (get users 1))│ │(def tags (:tags user)) │ ╰────────────────────────╯ 🬣🬘 🬨🬕 🬨🬕 🬕🬧 🬴🬗 🬁🬄 🬤🬒 🬕🬧 🬳🬶 🬨🬕 🬳🬶 🬤🬒 🬨🬕 🬴🬗 ▌▐ 🬔🬂 🬨🬕 ▌▐ 🬴🬗 🬴🬒 🬤🬒 🬔🬧 🬷🬲 ▐▌ 🬲🬘 🬲🬘 🬱🬘 🬲🬘 ▌▐ ▐▌ ▌▐ 🬱🬘 ▐▌ 🬕🬢 🬣🬘 🬣🬭 ▐▌ 🬣🬘 🬕🬢 🬲🬭 🬱🬘 Array mapped trie • ArrayList is a growable array • HashMap contains key-value pairs stored with a hash • HashSet contains unique values stored with a hash B+tree • LinkedArrayList is a growable array that can be efficiently sliced and concatenated • SortedMap contains key-value pairs whose keys are byte arrays kept in lexicographic order • SortedSet contains byte arrays kept in lexicographic order 🬺▐ 🬴🬒 🬤🬒 🬨🬕 🬳🬶 ▌ ▌ 🬣🬘 🬔🬧 ▌▐ ▌▐ 🬳🬶 🬺▐ 🬨🬕 ▌🬬 🬲🬭 🬱🬘 ▐▌ ▌▐ 🬲🬭 🬲🬭 ▐▌ 🬣🬘 🬣🬘 🬬🬝 ▌▐ ▌🬬 ▐▌ ╭───────────────────────────────────────────────────────╮ │[{"users": {"alice": "Alice Jones", "bob": "Bob Lee"}}]│ ╰───────────────────────────────────────────────────────╯ In xitdb you can nest data like in the JSON above easily: ╭─────────────────────────────────────────────────────╮ │;; the top-level list is the db's history, │ │;; so get the hash-map at the end of it (same as @db)│ │(def moment (xdb/deref-at db -1)) │ │ │ │;; get the "users" sorted map inside of that │ │(def users (:users moment)) │ │ │ │;; get "bob" │ │(def bob (get users "bob")) │ ╰─────────────────────────────────────────────────────╯ It's just like how you'd do it using in-memory data structures! 🬴🬗 ▌▐ 🬨🬕 ▌ 🬕🬧 🬣🬘 🬔🬧 ▌▐ 🬴🬗 🬔🬧 ▌▐ 🬺▐ 🬕🬧 🬳🬶 🬨🬕 🬳🬶 🬴🬗 🬳🬶 🬤🬒 🬴🬒 🬲🬘 🬣🬘 🬷🬲 🬲🬭 🬲🬘 ▐▌ 🬣🬘 🬣🬘 🬕🬢 🬣🬘 🬬🬝 ▌🬬 🬲🬘 ▌▐ ▐▌ ▌▐ 🬲🬘 ▌▐ 🬱🬘 🬲🬭 ╭──────────────────────────────────────────────────────────────────────────╮ │CREATE TABLE users (id INTEGER PRIMARY KEY, username TEXT, name TEXT); │ │CREATE INDEX users_username ON users (username); │ │ │ │INSERT INTO users (id, username, name) VALUES (1, 'alice', 'Alice Jones');│ ╰──────────────────────────────────────────────────────────────────────────╯ In xitdb, an index is just another map: ╭───────────────────────────────────────────────────────────────╮ │(swap! db (fn [moment] │ │ (-> moment │ │ (assoc-in [:users 1] {:username "alice" │ │ :name "Alice Jones"})│ │ (update :username->id (fnil assoc (sorted-map))│ │ "alice" 1)))) │ ╰───────────────────────────────────────────────────────────────╯ With these data structures you can "build" your own database. 🬤🬒 🬴🬒 🬳🬶 🬴🬗 🬔🬂 🬲🬷 🬱🬘 🬲🬭 ▌▐ 🬕🬢 🬣🬭 ▌▐ Another nice built-in feature of SQLite is search: ╭─────────────────────────────────────────────────────────────────────╮ │CREATE VIRTUAL TABLE posts USING fts5(title, body); │ │INSERT INTO posts (title, body) │ │ VALUES ('hello world', 'xitdb is a database from the future-past');│ │SELECT title FROM posts WHERE posts MATCH 'database'; │ ╰─────────────────────────────────────────────────────────────────────╯ In xitdb you need to manually set it up: Build an "inverted index": a SortedMap from each word to a SortedSet of the posts containing it. ╭────────────────────────────────────────────────────────╮ │{"database": [1, 7], "future": [1], "past": [1, 3], ...}│ ╰────────────────────────────────────────────────────────╯ • One word: a single lookup in the SortedMap returns every matching post • Prefixes: the words are sorted, so seek to "data" and walk forward while words still start with it • Multiple words: walk each word's SortedSet in step, seeking past posts the others don't have • Updates: when a post changes, only add or remove the words that changed • Build your own: haxy implements all of this in about 230 lines of Zig 🬨🬕 🬨🬕 🬁🬄 🬤🬒 🬳🬶 ▌ ▌ 🬨🬕 🬺▐ 🬔🬂 🬴🬗 🬴🬒 🬺🬻 🬴🬒 🬺▐ 🬨🬕 🬳🬶 ▌ 🬷🬲 ▐▌ 🬱🬘 ▌▐ 🬲🬭 🬲🬭 🬷🬲 ▌🬬 🬣🬭 🬕🬢 🬲🬭 ▌▐ 🬲🬭 ▌🬬 ▐▌ ▌▐ 🬲🬭 This isn't "object serialization". Every language has a way to write objects to disk, but that only works for tiny datasets that fit in memory. A xitdb database can be terabytes in size and still perform well, because all reads and writes are incremental. 🬨🬕 🬨🬕 🬺🬻 🬴🬒 🬨🬕 🬴🬗 🬳🬶 ▌▐ 🬴🬒 ▌ ▐▌ 🬷🬲 ▌▐ 🬲🬭 ▐▌ 🬕🬢 ▌▐ 🬧🬔 🬲🬭 🬲🬭 Revert to any previous state of the database: ╭───────────────────────────────────────────────────────────────╮ │;; the db counts like the list of all transactions made in it, │ │;; so save the current index so we can revert back to it │ │(def history-index (dec (count db))) │ │ │ │;; ...add some new transactions here │ │ │ │;; read the old copy of the db from before the new transactions│ │(def moment (xdb/deref-at db history-index)) │ │ │ │;; revert back to the old copy of the db │ │(reset! db moment) │ ╰───────────────────────────────────────────────────────────────╯ ▌ 🬔🬧 🬔🬂 🬲🬅 🬴🬒 🬴🬗 🬴🬒 🬴🬒 🬲🬭 🬣🬘 🬣🬭 🬕🬢 ▌ 🬕🬢 🬲🬭 🬲🬭 Read xitdb from multiple threads without locks: ╭──────────────────────────────────────────────────────────────╮ │;; all threads can share one db: each thread reads through │ │;; its own read-only file handle, so readers never take a lock│ │(def db (xdb/xit-db "main.db")) │ │ │ │;; read from any thread, even while another thread is writing │ │(def moment @db) │ ╰──────────────────────────────────────────────────────────────╯ Mutable databases like SQLite must use locks. (Yes, even in WAL mode.) 🬴🬗 🬴🬒 🬴🬗 ▌ 🬨🬕 🬔🬂 🬳🬶 🬨🬕 🬨🬕 🬔🬧 🬺▐ 🬕🬢 🬲🬭 ▌ 🬲🬭 🬷🬲 🬣🬭 ▌▐ ▐▌ 🬷🬲 🬣🬘 ▌🬬 Append-only databases are trivial to replicate: ╭────────────────────────────────────────────────────────────────────╮ │;; on the replica: describe what it already has │ │(def state (xdb/replica-state replica)) │ │ │ │;; on the source: open a stream of only the bytes the replica lacks,│ │;; then copy the new bytes from the source to the replica │ │(with-open [in (io/input-stream source :since state) │ │ out (io/output-stream replica)] │ │ (io/copy in out)) │ ╰────────────────────────────────────────────────────────────────────╯ SQLite requires complicated external tools like LiteStream (20k+ LOC) 🬴🬗 🬔🬧 ▌ 🬣🬘 🬔🬰 ▌ 🬔🬧 🬨🬕 ▌ 🬣🬘 🬲🬭 ▐▌ 🬣🬷 🬲🬭 🬣🬘 ▐▌ There are 4 dependency-free implementations: • Zig: xit-vcs/xitdb • Java: xit-vcs/xitdb-java • TypeScript: xit-vcs/xitdb-ts • Go: xit-vcs/xitdb-go Each one is 4-5k lines of code, compared to 150k lines in SQLite. Yes, SQLite is more featureful. But the impedance mismatch often leads you to write more code despite the extensive built-in features. And a nice Clojure wrapper around xitdb-java as well: • Clojure: codeboost/xitdb-clj