org.lwjgl.util.zstd

Class Zdict

java.lang.Object

org.lwjgl.util.zstd.Zdict

public class Zdict
extends java.lang.Object

Native bindings to the dictionary builder API of Zstandard (zstd).

Field Summary

Fields

Modifier and Type	Field and Description
static int	ZDICT_CONTENTSIZE_MIN
static int	ZDICT_DICTSIZE_MIN

Method Summary

Modifier and Type	Method and Description
static long	nZDICT_finalizeDictionary(long dictBuffer, long dictBufferCapacity, long dictContent, long dictContentSize, long samplesBuffer, long samplesSizes, int nbSamples, long parameters) Unsafe version of: finalizeDictionary
static int	nZDICT_getDictID(long dictBuffer, long dictSize) Unsafe version of: getDictID
static long	nZDICT_getErrorName(long errorCode)
static int	nZDICT_isError(long errorCode)
static long	nZDICT_optimizeTrainFromBuffer_cover(long dictBuffer, long dictBufferCapacity, long samplesBuffer, long samplesSizes, int nbSamples, long parameters) Unsafe version of: optimizeTrainFromBuffer_cover
static long	nZDICT_optimizeTrainFromBuffer_fastCover(long dictBuffer, long dictBufferCapacity, long samplesBuffer, long samplesSizes, int nbSamples, long parameters) Unsafe version of: optimizeTrainFromBuffer_fastCover
static long	nZDICT_trainFromBuffer_cover(long dictBuffer, long dictBufferCapacity, long samplesBuffer, long samplesSizes, int nbSamples, long parameters) Unsafe version of: trainFromBuffer_cover
static long	nZDICT_trainFromBuffer_fastCover(long dictBuffer, long dictBufferCapacity, long samplesBuffer, long samplesSizes, int nbSamples, long parameters) Unsafe version of: trainFromBuffer_fastCover
static long	nZDICT_trainFromBuffer(long dictBuffer, long dictBufferCapacity, long samplesBuffer, long samplesSizes, int nbSamples) Unsafe version of: trainFromBuffer
static long	ZDICT_finalizeDictionary(java.nio.ByteBuffer dictBuffer, java.nio.ByteBuffer dictContent, java.nio.ByteBuffer samplesBuffer, org.lwjgl.PointerBuffer samplesSizes, ZDICTParams parameters) Given a custom content as a basis for dictionary, and a set of samples, finalize dictionary by adding headers and statistics.
static int	ZDICT_getDictID(java.nio.ByteBuffer dictBuffer) Extracts dictID.
static java.lang.String	ZDICT_getErrorName(long errorCode)
static boolean	ZDICT_isError(long errorCode)
static long	ZDICT_optimizeTrainFromBuffer_cover(java.nio.ByteBuffer dictBuffer, java.nio.ByteBuffer samplesBuffer, org.lwjgl.PointerBuffer samplesSizes, ZDICTCoverParams parameters) The same requirements as trainFromBuffer_cover hold for all the parameters except parameters.
static long	ZDICT_optimizeTrainFromBuffer_fastCover(java.nio.ByteBuffer dictBuffer, java.nio.ByteBuffer samplesBuffer, org.lwjgl.PointerBuffer samplesSizes, ZDICTFastCoverParams parameters) The same requirements as trainFromBuffer_fastCover hold for all the parameters except parameters.
static long	ZDICT_trainFromBuffer_cover(java.nio.ByteBuffer dictBuffer, java.nio.ByteBuffer samplesBuffer, org.lwjgl.PointerBuffer samplesSizes, ZDICTCoverParams parameters) Train a dictionary from an array of samples using the COVER algorithm.
static long	ZDICT_trainFromBuffer_fastCover(java.nio.ByteBuffer dictBuffer, java.nio.ByteBuffer samplesBuffer, org.lwjgl.PointerBuffer samplesSizes, ZDICTFastCoverParams parameters) Train a dictionary from an array of samples using a modified version of COVER algorithm.
static long	ZDICT_trainFromBuffer(java.nio.ByteBuffer dictBuffer, java.nio.ByteBuffer samplesBuffer, org.lwjgl.PointerBuffer samplesSizes) Train a dictionary from an array of samples.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ZDICT_CONTENTSIZE_MIN

public static final int ZDICT_CONTENTSIZE_MIN

See Also:

Constant Field Values

ZDICT_DICTSIZE_MIN

public static final int ZDICT_DICTSIZE_MIN

See Also:

Constant Field Values

Method Detail

nZDICT_trainFromBuffer

public static long nZDICT_trainFromBuffer(long dictBuffer,
                                          long dictBufferCapacity,
                                          long samplesBuffer,
                                          long samplesSizes,
                                          int nbSamples)

Unsafe version of: trainFromBuffer

ZDICT_trainFromBuffer

public static long ZDICT_trainFromBuffer(java.nio.ByteBuffer dictBuffer,
                                         java.nio.ByteBuffer samplesBuffer,
                                         org.lwjgl.PointerBuffer samplesSizes)

Train a dictionary from an array of samples.

Redirect towards optimizeTrainFromBuffer_fastCover single-threaded, with d=8, steps=4, f=20, and accel=1.

Samples must be stored concatenated in a single flat buffer samplesBuffer, supplied with an array of sizes samplesSizes, providing the size of each sample, in order.

The resulting dictionary will be saved into dictBuffer.

Note: ZDICT_trainFromBuffer() requires about 9 bytes of memory for each input byte.

Tips:

• In general, a reasonable dictionary has a size of ~ 100 KB.
• It's possible to select smaller or larger size, just by specifying dictBufferCapacity.
• In general, it's recommended to provide a few thousands samples, though this can vary a lot.
• It's recommended that total size of all samples be about ~x100 times the target size of dictionary.

Returns:

size of dictionary stored into dictBuffer (≤ dictBufferCapacity) or an error code, which can be tested with isError.

nZDICT_getDictID

public static int nZDICT_getDictID(long dictBuffer,
                                   long dictSize)

Unsafe version of: getDictID

ZDICT_getDictID

public static int ZDICT_getDictID(java.nio.ByteBuffer dictBuffer)

Extracts dictID.

Returns:

zero if error (not a valid dictionary)

nZDICT_isError

public static int nZDICT_isError(long errorCode)

ZDICT_isError

public static boolean ZDICT_isError(long errorCode)

nZDICT_getErrorName

public static long nZDICT_getErrorName(long errorCode)

ZDICT_getErrorName

@Nullable
public static java.lang.String ZDICT_getErrorName(long errorCode)

nZDICT_trainFromBuffer_cover

public static long nZDICT_trainFromBuffer_cover(long dictBuffer,
                                                long dictBufferCapacity,
                                                long samplesBuffer,
                                                long samplesSizes,
                                                int nbSamples,
                                                long parameters)

Unsafe version of: trainFromBuffer_cover

ZDICT_trainFromBuffer_cover

public static long ZDICT_trainFromBuffer_cover(java.nio.ByteBuffer dictBuffer,
                                               java.nio.ByteBuffer samplesBuffer,
                                               org.lwjgl.PointerBuffer samplesSizes,
                                               ZDICTCoverParams parameters)

Train a dictionary from an array of samples using the COVER algorithm.

Samples must be stored concatenated in a single flat buffer samplesBuffer, supplied with an array of sizes samplesSizes, providing the size of each sample, in order.

The resulting dictionary will be saved into dictBuffer.

Note: ZDICT_trainFromBuffer_cover() requires about 9 bytes of memory for each input byte.

Tips:

• In general, a reasonable dictionary has a size of ~ 100 KB.
• It's possible to select smaller or larger szie, just by specifying dictBufferCapacity.
• In general, it's recommended to provide a few thousands samples, though this can vary a lot.
• It's recommended that total size of all samples be about ~x100 times the target size of dictionary.

Returns:

size of dictionary stored into dictBuffer (≤ dictBufferCapacity) or an error code, which can be tested with isError.

nZDICT_optimizeTrainFromBuffer_cover

public static long nZDICT_optimizeTrainFromBuffer_cover(long dictBuffer,
                                                        long dictBufferCapacity,
                                                        long samplesBuffer,
                                                        long samplesSizes,
                                                        int nbSamples,
                                                        long parameters)

Unsafe version of: optimizeTrainFromBuffer_cover

ZDICT_optimizeTrainFromBuffer_cover

public static long ZDICT_optimizeTrainFromBuffer_cover(java.nio.ByteBuffer dictBuffer,
                                                       java.nio.ByteBuffer samplesBuffer,
                                                       org.lwjgl.PointerBuffer samplesSizes,
                                                       ZDICTCoverParams parameters)

The same requirements as trainFromBuffer_cover hold for all the parameters except parameters.

This function tries many parameter combinations and picks the best parameters. *parameters is filled with the best parameters found, dictionary constructed with those parameters is stored in dictBuffer.

• All of the parameters d, k, steps are optional.
• If d is non-zero then we don't check multiple values of d, otherwise we check d = {6, 8}.
• If steps is zero it defaults to its default value.
• If k is non-zero then we don't check multiple values of k, otherwise we check steps values in [50, 2000].

Note: ZDICT_optimizeTrainFromBuffer_cover() requires about 8 bytes of memory for each input byte and additionally another 5 bytes of memory for each byte of memory for each thread.

Returns:

size of dictionary stored into dictBuffer (≤ dictBufferCapacity) or an error code, which can be tested with isError. On success *parameters contains the parameters selected.

nZDICT_trainFromBuffer_fastCover

public static long nZDICT_trainFromBuffer_fastCover(long dictBuffer,
                                                    long dictBufferCapacity,
                                                    long samplesBuffer,
                                                    long samplesSizes,
                                                    int nbSamples,
                                                    long parameters)

Unsafe version of: trainFromBuffer_fastCover

ZDICT_trainFromBuffer_fastCover

public static long ZDICT_trainFromBuffer_fastCover(java.nio.ByteBuffer dictBuffer,
                                                   java.nio.ByteBuffer samplesBuffer,
                                                   org.lwjgl.PointerBuffer samplesSizes,
                                                   ZDICTFastCoverParams parameters)

Train a dictionary from an array of samples using a modified version of COVER algorithm.

Samples must be stored concatenated in a single flat buffer samplesBuffer, supplied with an array of sizes samplesSizes, providing the size of each sample, in order. d and k are required. All other parameters are optional, will use default values if not provided. The resulting dictionary will be saved into dictBuffer.

Note: ZDICT_trainFromBuffer_fastCover() requires about 1 bytes of memory for each input byte and additionally another 6 * 2^f bytes of memory.

Tips: In general, a reasonable dictionary has a size of ~100 KB. It's possible to select smaller or larger size, just by specifying dictBufferCapacity. In general, it's recommended to provide a few thousands samples, though this can vary a lot. It's recommended that total size of all samples be about ~x100 times the target size of dictionary.

Returns:

size of dictionary stored into dictBuffer (≤ dictBufferCapacity) or an error code, which can be tested with isError.

nZDICT_optimizeTrainFromBuffer_fastCover

public static long nZDICT_optimizeTrainFromBuffer_fastCover(long dictBuffer,
                                                            long dictBufferCapacity,
                                                            long samplesBuffer,
                                                            long samplesSizes,
                                                            int nbSamples,
                                                            long parameters)

Unsafe version of: optimizeTrainFromBuffer_fastCover

ZDICT_optimizeTrainFromBuffer_fastCover

public static long ZDICT_optimizeTrainFromBuffer_fastCover(java.nio.ByteBuffer dictBuffer,
                                                           java.nio.ByteBuffer samplesBuffer,
                                                           org.lwjgl.PointerBuffer samplesSizes,
                                                           ZDICTFastCoverParams parameters)

The same requirements as trainFromBuffer_fastCover hold for all the parameters except parameters.

This function tries many parameter combinations (specifically, k and d combinations) and picks the best parameters. *parameters is filled with the best parameters found, dictionary constructed with those parameters is stored in dictBuffer.

• All of the parameters d, k, steps, f, and accel are optional.
• If d is non-zero then we don't check multiple values of d, otherwise we check d = {6, 8}.
• If steps is zero it defaults to its default value.
• If k is non-zero then we don't check multiple values of k, otherwise we check steps values in [50, 2000].
• If f is zero, default value of 20 is used.
• If accel is zero, default value of 1 is used.

Note: ZDICT_optimizeTrainFromBuffer_fastCover() requires about 1 byte of memory for each input byte and additionally another 6 * 2^f bytes of memory for each thread.

Returns:

size of dictionary stored into dictBuffer (≤ dictBufferCapacity) or an error code, which can be tested with isError. On success *parameters contains the parameters selected.

nZDICT_finalizeDictionary

public static long nZDICT_finalizeDictionary(long dictBuffer,
                                             long dictBufferCapacity,
                                             long dictContent,
                                             long dictContentSize,
                                             long samplesBuffer,
                                             long samplesSizes,
                                             int nbSamples,
                                             long parameters)

Unsafe version of: finalizeDictionary

ZDICT_finalizeDictionary

public static long ZDICT_finalizeDictionary(java.nio.ByteBuffer dictBuffer,
                                            java.nio.ByteBuffer dictContent,
                                            java.nio.ByteBuffer samplesBuffer,
                                            org.lwjgl.PointerBuffer samplesSizes,
                                            ZDICTParams parameters)

Given a custom content as a basis for dictionary, and a set of samples, finalize dictionary by adding headers and statistics.

Samples must be stored concatenated in a flat buffer samplesBuffer, supplied with an array of sizes samplesSizes, providing the size of each sample in order.

Notes:

• dictContentSize must be ≥ CONTENTSIZE_MIN bytes.
• maxDictSize must be ≥ dictContentSize, and must be ≥ DICTSIZE_MIN bytes.
• ZDICT_finalizeDictionary() will push notifications into stderr if instructed to, using notificationLevel>0.
• dictBuffer and dictContent can overlap.

Returns:

size of dictionary stored into dictBuffer (≤ dictBufferCapacity) or an error code, which can be tested with isError.