Reading the entries of a zip file in Node with node:zlib

A build script that downloads a zip archive often only needs one or two files out of it. If the download is gzipped then node:zlib covers that with gunzipSync. If it’s a zip archive, however, that’s a bit trickier, because a zip archive is a container format as well as containing compressed data. The native Node.js library only handles the compression half. Pulling one file out of a zip usually requires adding a dependency, which seems a bit wasteful if we only need to read a few bytes of headers.

The compression side of the problem is the more difficult part, and the native node:zlib can handle it. After that, all we need is to read a small number of offsets, and that is doable in about a hundred lines of code.

Zip archives are initially read backwards from the end of the file. Near the end of a zip file is an “End of Central Directory” record. That record describes how many entries there are in the zip archive and where to find the central directory containing metadata for them. A zip reader scans backwards to find this central directory record first and then uses the central directory as a kind of index to find the other parts of the archive.

import { inflateRawSync } from "node:zlib";

const END_OF_CENTRAL_DIRECTORY = 0x06_05_4b_50;
const CENTRAL_FILE_HEADER = 0x02_01_4b_50;
const END_RECORD_SIZE = 22;
const CENTRAL_HEADER_SIZE = 46;
const LOCAL_HEADER_SIZE = 30;
const MAX_COMMENT = 0xff_ff;
const STORED = 0;
const DEFLATED = 8;

function findEndRecord(zip: Buffer): number {
  const earliest = Math.max(0, zip.length - END_RECORD_SIZE - MAX_COMMENT);
  for (let at = zip.length - END_RECORD_SIZE; at >= earliest; at--) {
    if (
      zip.readUInt32LE(at) === END_OF_CENTRAL_DIRECTORY &&
      at + END_RECORD_SIZE + zip.readUInt16LE(at + 20) === zip.length
    ) {
      return at;
    }
  }
  throw new Error("not a zip file: no end-of-central-directory record");
}

The variable length comment right at the end of the zip can be up to 65,535 bytes, so the reader does not need to scan further than that to find the central directory record.

Once the reader has found the central directory, it walks across the entries it contains. There is one header for each entry, which gives the position in the zip of the local header for that file. The actual compressed data follows the local header, so the reader can use that information to efficiently read any individual entry.

export function readZipEntries(zip: Buffer): Map<string, Buffer> {
  const endRecord = findEndRecord(zip);
  const entryCount = zip.readUInt16LE(endRecord + 10);
  let at = zip.readUInt32LE(endRecord + 16);

  const entries = new Map<string, Buffer>();
  for (let index = 0; index < entryCount; index++) {
    if (zip.readUInt32LE(at) !== CENTRAL_FILE_HEADER) {
      throw new Error(`corrupt zip: bad central directory header at ${String(at)}`);
    }

    const method = zip.readUInt16LE(at + 10);
    const compressedSize = zip.readUInt32LE(at + 20);
    const nameLength = zip.readUInt16LE(at + 28);
    const extraLength = zip.readUInt16LE(at + 30);
    const commentLength = zip.readUInt16LE(at + 32);
    const localHeader = zip.readUInt32LE(at + 42);
    const name = zip.toString(
      "utf8",
      at + CENTRAL_HEADER_SIZE,
      at + CENTRAL_HEADER_SIZE + nameLength,
    );

    const localNameLength = zip.readUInt16LE(localHeader + 26);
    const localExtraLength = zip.readUInt16LE(localHeader + 28);
    const start = localHeader + LOCAL_HEADER_SIZE + localNameLength + localExtraLength;
    const body = zip.subarray(start, start + compressedSize);

    if (method === STORED) {
      entries.set(name, body);
    } else if (method === DEFLATED) {
      entries.set(name, inflateRawSync(body));
    } else {
      throw new Error(`unsupported compression method ${String(method)} for ${name}`);
    }

    at += CENTRAL_HEADER_SIZE + nameLength + extraLength + commentLength;
  }

  return entries;
}

Probably the trickiest part of this is correctly combining those four variables that together give the start position for one entry. Some information is duplicated across an entry in the central directory record and the corresponding entry itself, and the length can differ between the two. Getting the wrong start position means the reader will try to read binary data from the wrong position which will fail opaquely.

For example, it’s easy to try and calculate the start offset based on the central directory headers for that entry. Those actually differ from the headers on the entry, so the calculated position ends up a few bytes into the binary compressed data. Trying to call inflateRawSync from that incorrect position just errors, and the error message is not very helpful for figuring out what the problem is.

Anyway, once that is calculating the positions correctly, reading individual entries from a zip archive is then a relatively straightforward map lookup.

import { readFile } from "node:fs/promises";

const members = readZipEntries(await readFile("foobar.zip"));
const data = members.get("foobar/data.txt");
if (data === undefined) {
  throw new Error("foobar.zip is missing foobar/data.txt");
}

Note that this implementation is somewhat basic and does not cope with other compression methods, encrypted archives, non-UTF-8 filenames, ZIP64, archives over 4GB or archives with more than 65,535 entries. It’s supposed to be a quick-and-simple implementation to avoid a potentially heavy dependency when that’s not truly needed. In more complex situations or when dealing with larger zip archives, it’s probably worth just using a proper zip library to handle all the edge cases correctly.

Tech mentioned

• Node.js
• TypeScript