Warning
Note: zDNN is not the same as ZenDNN.
- zDNN (this page): IBM's Deep Neural Network acceleration library for IBM Z & LinuxONE Mainframes
- ZenDNN: AMD's deep learning library for AMD EPYC CPUs (see ZenDNN documentation)
IBM zDNN (Z Deep Neural Network) is a hardware acceleration library designed specifically to leverage the IBM NNPA (Neural Network Processor Assist) accelerator located within IBM Telum I and II processors. It provides significant performance improvements for neural network inference operations.
The llama.cpp zDNN backend is designed to enable llama.cpp on IBM z17 and later systems via the IBM zDNN hardware acceleration library.
| Hardware Level | Status | Verified |
|---|---|---|
| IBM z17 / LinuxONE 5 | Supported | RHEL 9.6, IBM z17, 40 IFLs |
| IBM z16 / LinuxONE 4 | Not Supported |
| Data Type | Status |
|---|---|
| F32 | Supported |
| F16 | Supported |
| BF16 | Supported |
| Q8_0 | Supported |
Note: Quantized weights are requantized to int8 with one scale per group of up to 2048 values in each row (or 1024 with GGML_ZDNN_HIGH_PREC=1), after each input channel is divided by its RMS (the activations are multiplied by it instead), and each activation is quantized to two int8 values (about 15 bits). This requires NNPA_PARMBLKFORMAT_1 (IBM z17 / LinuxONE 5).
The IBM zDNN backend has the following CMake options that control the behaviour of the backend.
| CMake Option | Default Value | Description |
|---|---|---|
GGML_ZDNN |
OFF |
Compile llama.cpp with zDNN support |
ZDNN_ROOT |
"" |
Override zDNN library lookup |
| Environment Variable | Default Value | Description |
|---|---|---|
GGML_ZDNN_HIGH_PREC |
0 |
Set to 1 to requantize Q8_0 weights in smaller groups, for lower loss at lower speed |
Note: Using the zDNN library provided via apt or yum may not work correctly as reported in #15772. It is preferred that you compile from source.
git clone --recurse-submodules https://github.com/IBM/zDNN
cd zDNN
autoreconf .
./configure --prefix=/opt/zdnn-libs
make build
sudo make installgit clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -S . -G Ninja -B build \
-DCMAKE_BUILD_TYPE=Release \
-DGGML_ZDNN=ON \
-DZDNN_ROOT=/opt/zdnn-libs
cmake --build build --config Release -j$(nproc)