| Data Structures | Source Location | |
|---|---|---|
| |
Caution: the Architecture Guide is not updated in lockstep with the code base and is not necessarily correct or complete for any specific release.
WiredTiger's cross-checkpoint caching feature reduces redundant network reads and unnecessary memory usage on disaggregated standbys by sharing disk images in memory across checkpoints. When a standby picks up a new checkpoint (see Checkpoint), pages that are unchanged from the previous checkpoint can be served directly from memory instead of being re-fetched from remote storage. This feature is only active on disaggregated standby nodes; see State machine for when it is enabled and disabled.
The shared disk cache is implemented as a connection-level hash table keyed on the address cookie. Because different files can hash to the same bucket, lookup finds the correct disk image by walking the bucket's linked list and matching on file ID, address size, and address cookie. Each entry holds a single disk image buffer shared across all in-memory pages that reference the same block. Normally each page owns its disk image buffer privately, but with the cross-checkpoint cache, the disk image is owned by the cache entry and pages share it instead. When a page is read, WiredTiger first checks the shared disk cache for an existing entry. On a miss, it reads the block from remote storage and adds the disk image to the shared disk cache. When the last page referencing an entry is discarded, the entry is removed and the buffer freed.
Pages sharing a disk image do not count those bytes toward connection-level cache totals (bytes_inmem). The shared disk cache counts them once when the entry is first inserted and drains them when the last reference is released. Each page's memory_footprint still includes the disk image size for per-page eviction decisions.
| File | Role |
|---|---|
src/include/cache.h | WT_SHARED_DSK_CACHE struct and state definitions |
src/include/btmem.h | shared_dsk_item pointer in the page's WT_PAGE_DISAGG_INFO |
src/btree/bt_read.c | call site: get on page fault-in, put after storage read |
src/btree/bt_discard.c | call site: release when a page is discarded |
src/cache/shared_dsk.c | get, put, release, init, and destroy |
src/cache/cache.c | cache creation and destruction wiring |
src/conn/conn_layered.c | step-up transition (ACTIVE to READONLY) and step-down transition (back to ACTIVE) |
src/conn/conn_sweep.c | sweep server flipping READONLY to DEAD after the grace period |
The cache is created in __wt_cache_create when a disaggregated connection opens on a standby, and it starts in WT_DSK_CACHE_ACTIVE. On connection close, it is destroyed in __wt_cache_destroy. State transitions on step-up and beyond are described in State machine.
__wt_shared_dsk_cache_get): looks up an entry by file ID and address cookie. On a hit the reference count is incremented and the caller receives a pointer to the cached disk image. On a miss the caller reads from storage as usual.__wt_shared_dsk_cache_put): inserts a newly read disk image. If an entry for that address already exists its reference count is incremented and inserted is set to false; the caller must free the redundant buffer. On a fresh insert inserted is set to true and the cache takes ownership of the data.__wt_shared_dsk_cache_release): decrements the reference count. When the count reaches zero the entry is removed from the hash table and its buffer freed.Two structs in src/include/cache.h implement the cache.
WT_SHARED_DSK_CACHE is the top-level object, embedded in WT_CACHE:
state: atomic uint8_t holding the current WT_DSK_CACHE_STATE (see State machine).readonly_since: atomic wall-clock timestamp (seconds) recording when the cache transitioned to READONLY on step-up (see Lifecycle of the cache).hash / hash_size: bucket array sized at roughly 0.2% of configured cache, with a floor of 1000 buckets.hash_locks / hash_lock_size: spinlock array, striped across buckets so that each lock guards a group of buckets.WT_SHARED_DSK_ITEM is a single cache entry:
fid + addr[]: composite lookup key: file ID and address cookie.data / data_size: cached disk image buffer and its length.ref_count: number of in-memory pages sharing this entry; incremented on get or put collision, decremented on release. Drops to zero causes the entry to be immediately removed and the buffer freed. Protected by the bucket spinlock.block_meta: WT_PAGE_BLOCK_META copied into the page's disagg_info on a cache hit, avoiding a storage round-trip.Each in-memory page that holds a reference stores a pointer to its WT_SHARED_DSK_ITEM in page->disagg_info->shared_dsk_item (see src/include/btmem.h).
Cross-checkpoint caching is only enabled on disaggregated standbys, because leader reads come from the live btree rather than older checkpoint images. The cache moves through the following states over the lifetime of a connection:
| State | Who enters it | Reads allowed | Writes (puts) allowed |
|---|---|---|---|
WT_DSK_CACHE_OFF | Leader that has never been a standby; non-disagg connections | No | No |
WT_DSK_CACHE_ACTIVE | Standby | Yes | Yes |
WT_DSK_CACHE_READONLY | Immediately after step-up to leader | Yes | No |
WT_DSK_CACHE_DEAD | After the sweep server's grace period elapses on a leader | No | No |
On step-up, the cache moves from ACTIVE to READONLY so that warm pages already in the cache continue to be served. A grace period (WT_DISAGG_OUTDATED_GRACE_SECS) controls how long the cache remains in READONLY: once the reuse window expires, the sweep server flips it to DEAD. The sweep server preserves outdated stable-checkpoint dhandles for that grace period instead of discarding them immediately, allowing the next checkpoint generation to reuse the shared disk image.
On step-down, the cache transitions back to ACTIVE. If the hash table was never initialized (i.e. the node started as a leader and was not previously a standby), it is created at this point, otherwise the existing table is reused.