The interface implemented by applications to provide custom compression. More...
Public Attributes | |
int(* | compress )(WT_COMPRESSOR *compressor, WT_SESSION *session, uint8_t *src, size_t src_len, uint8_t *dst, size_t dst_len, size_t *result_lenp, int *compression_failed) |
Callback to compress a chunk of data. | |
int(* | decompress )(WT_COMPRESSOR *compressor, WT_SESSION *session, uint8_t *src, size_t src_len, uint8_t *dst, size_t dst_len, size_t *result_lenp) |
Callback to decompress a chunk of data. | |
int(* | pre_size )(WT_COMPRESSOR *compressor, WT_SESSION *session, uint8_t *src, size_t src_len, size_t *result_lenp) |
Callback to size a destination buffer for compression. |
The interface implemented by applications to provide custom compression.
Compressors must implement the WT_COMPRESSOR interface: the WT_COMPRESSOR::compress and WT_COMPRESSOR::decompress callbacks must be specified, and WT_COMPRESSOR::pre_size is optional. To build your own compressor, use one of the compressors in ext/compressors
as a template: ext/nop_compress
is a simple compressor that passes through data unchanged, and is a reasonable starting point.
Applications register their implementation with WiredTiger by calling WT_CONNECTION::add_compressor.
static WT_COMPRESSOR my_compressor = { my_compress, my_decompress, my_pre_size }; ret = conn->add_compressor(conn, "my_compress", &my_compressor, NULL);
int(* WT_COMPRESSOR::compress)(WT_COMPRESSOR *compressor, WT_SESSION *session, uint8_t *src, size_t src_len, uint8_t *dst, size_t dst_len, size_t *result_lenp, int *compression_failed) |
Callback to compress a chunk of data.
WT_COMPRESSOR::compress is given a source buffer and a destination buffer, by default of the same size. If the callback can compress the buffer to a smaller size in the destination, it does so, sets the compression_failed
return to 0 and returns 0. If compression does not produce a smaller result, the callback sets the compression_failed
return to 1 and returns 0. If another error occurs, it returns an errno or WiredTiger error code.
On entry, src
will point to memory, with the length of the memory in src_len
. After successful completion, the callback should return 0
and set result_lenp
to the number of bytes required for the compressed representation.
If compression would not shrink the data or the dst
buffer is not large enough to hold the compressed data, the callback should set compression_failed
to a non-zero value and return 0.
[in] | src | the data to compress |
[in] | src_len | the length of the data to compress |
[in] | dst | the destination buffer |
[in] | dst_len | the length of the destination buffer |
[out] | result_lenp | the length of the compressed data |
[out] | compression_failed | non-zero if compression did not decrease the length of the data (compression may not have completed) |
/* * A simple compression example that passes data through unchanged. */ static int my_compress(WT_COMPRESSOR *compressor, WT_SESSION *session, uint8_t *src, size_t src_len, uint8_t *dst, size_t dst_len, size_t *result_lenp, int *compression_failed) { /* Unused parameters */ (void)compressor; (void)session; *compression_failed = 0; if (dst_len < src_len) { *compression_failed = 1; return (0); } memcpy(dst, src, src_len); *result_lenp = src_len; return (0); }
int(* WT_COMPRESSOR::decompress)(WT_COMPRESSOR *compressor, WT_SESSION *session, uint8_t *src, size_t src_len, uint8_t *dst, size_t dst_len, size_t *result_lenp) |
Callback to decompress a chunk of data.
WT_COMPRESSOR::decompress is given a source buffer and a destination buffer. The contents are switched from compress:
the source buffer is the compressed value, and the destination buffer is sized to be the original size. If the callback successfully decompresses the source buffer to the destination buffer, it returns 0. If an error occurs, it returns an errno or WiredTiger error code. The source buffer that WT_COMPRESSOR::decompress is given may have a size that is rounded up from the size originally produced by WT_COMPRESSOR::compress, with the remainder of the buffer set to zeroes. Most compressors do not care about this difference if the size to be decompressed can be implicitly discovered from the compressed data. If your compressor cares, you may need to allocate space for, and store, the actual size in the compressed buffer. See the source code for the included snappy compressor for an example.
On entry, src
will point to memory, with the length of the memory in src_len
. After successful completion, the callback should return 0
and set result_lenp
to the number of bytes required for the decompressed representation.
If the dst
buffer is not big enough to hold the decompressed data, the callback should return an error.
[in] | src | the data to decompress |
[in] | src_len | the length of the data to decompress |
[in] | dst | the destination buffer |
[in] | dst_len | the length of the destination buffer |
[out] | result_lenp | the length of the decompressed data |
/* * A simple decompression example that passes data through unchanged. */ static int my_decompress(WT_COMPRESSOR *compressor, WT_SESSION *session, uint8_t *src, size_t src_len, uint8_t *dst, size_t dst_len, size_t *result_lenp) { /* Unused parameters */ (void)compressor; (void)session; if (dst_len < src_len) return (ENOMEM); memcpy(dst, src, src_len); *result_lenp = src_len; return (0); }
int(* WT_COMPRESSOR::pre_size)(WT_COMPRESSOR *compressor, WT_SESSION *session, uint8_t *src, size_t src_len, size_t *result_lenp) |
Callback to size a destination buffer for compression.
WT_COMPRESSOR::pre_size is an optional callback that, given the source buffer and size, produces the size of the destination buffer to be given to WT_COMPRESSOR::compress. This is useful for compressors that assume that the output buffer is sized for the worst case and thus no overrun checks are made. If your compressor works like this, WT_COMPRESSOR::pre_size will need to be defined. See the source code for the snappy compressor for an example. However, if your compressor detects and avoids overruns against its target buffer, you will not need to define WT_COMPRESSOR::pre_size. When WT_COMPRESSOR::pre_size is set to NULL, the destination buffer is sized the same as the source buffer. This is always sufficient, since a compression result that is larger than the source buffer is discarded by WiredTiger.
If not NULL, this callback is called before each call to WT_COMPRESS::compress to determine the size of the destination buffer to provide. If the callback is NULL, the destination buffer will be the same size as the source buffer.
The callback should set result_lenp
to a suitable buffer size for compression, typically the maximum length required by WT_COMPRESSOR::compress.
This callback function is for compressors that require an output buffer larger than the source buffer (for example, that do not check for buffer overflow during compression).
[in] | src | the data to compress |
[in] | src_len | the length of the data to compress |
[out] | result_lenp | the required destination buffer size |
/* * A simple pre-size example that returns the source length. */ static int my_pre_size(WT_COMPRESSOR *compressor, WT_SESSION *session, uint8_t *src, size_t src_len, size_t *result_lenp) { /* Unused parameters */ (void)compressor; (void)session; (void)src; *result_lenp = src_len; return (0); }