Chunkcache

Pinned Content

This functionality allows users to provide a list of tables that should remain persistently cached, ensuring faster data access times and reducing the need to fetch data from potentially much slower block storage. If a user wishes to cache a table in the chunk cache (such that the cached chunks belonging to these tables are non-evictable), the pinned configuration is required.

Note: It is crucial to assess the memory implications of pinning large tables. Ensure the chunkcache has enough capacity to cache the pinned tables.

Example Configuration:

chunk_cache=[enabled=true,chunk_size=512KB,capacity=20GB,pinned=("table:chunkcache01", "table:chunkcache02")]

In the configuration above, the pinned configuration array contains the names of tables to be pinned. In this example, the tables chunkcache01 and chunkcache02 are specified. By marking tables as pinned in the chunk cache, all chunks belonging to these tables (when read into chunkcache) remain in the cache. These chunks won't be evicted, regardless of how infrequently they might be accessed.

Note: There could be circumstances when the pinned chunks can become unpinned (and evicted) for internal reasons.

This ensures that the pinned tables are always relatively cheap to access, minimizing the performance overhead of accessing these tables.

Newly inserted or modified content

Newer content is more likely to be accessed again, so we insert freshly-flushed data into the chunkcache. This minimizes latency and reduces resource usage (disk bandwidth and API access). This saves the chunkcache from having to read freshly-flushed data back from the object store soon after it was put there. When a tiered object is flushed and before it is deleted locally, we make an ingest call and the new or modified table will be inserted into the chunkcache. This function will also check for outdated pinned content in the chunk cache, ensuring the older versions of the pinned content can be cleaned up by chunk cache's eviction process.