Catching stale dist files in an npm tarball before publishing

Most TypeScript packages publish only their build output from the dist/ directory. That looks like this in the package.json file:

{
  "files": ["dist"],
  "main": "./dist/index.js",
  "types": "./dist/index.d.ts"
}

That’s usually fine, but it can come back and bite you if stale build output sticks around under dist/. The tsc compiler does not delete any existing files under dist/, so if you delete or rename a directory under src/, for example, you can end up with a ghostly apparition haunting the build output in dist/. That then gets packed and shipped off to npmjs.com or wherever you are publishing the package, and from there on to users’ machines.

It can be difficult to spot that ghost, because dist/ is git-ignored and not covered by any linters or tests. The tests continue to import from the up-to-date src/, so there’s no apparent problem.

The fix and prevention is straightforward, at least.

{
  "scripts": {
    "build": "rm -rf dist && tsc -p tsconfig.build.json"
  }
}

That just wipes the whole dist/ directory before the build, so there’s no chance of ethereal apparitions hanging around from compilations past. The drawback with that simplistic solution is that it only covers builds which go through the build npm script. Anything that runs tsc from outside that might still forget to clear dist/ first.

The tarball that gets shipped off to npmjs is what actually ends up on users’ machines, so it would be better if we could check that rather than relying on dist/ being cleared out before each build. It’s not so straightforward to inspect the contents of a tarball, though.

A middle-ground solution is to put the generated tarball in a fresh temporary directory, then record the output of tar into a listing file which we can use for verification.

#!/usr/bin/env bash
set -euo pipefail

here="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
cd "$here"

work="$(mktemp -d)"
trap 'rm -rf "$work"' EXIT

pnpm pack --pack-destination "$work" >/dev/null
tarball="$(find "$work" -name '*.tgz' -maxdepth 1 | head -1)"
if [[ -z "$tarball" ]]; then
  echo "pack:check: no tarball was produced" >&2
  exit 1
fi

listing="$work/listing.txt"
tar -tzf "$tarball" >"$listing"

That uses pnpm pack, but npm pack works the same way. All the build paths inside the tar archive are prefixed with package/, and we use that prefix for matching during the verification step.

For verification, we iterate all instances of package paths from the listing, and check them against the equivalent paths under src/. If a path is present in the listing but not in src/, we report it and fail the verification step.

fail=0
note() {
  echo "pack:check: $1" >&2
  fail=1
}

while read -r compiled; do
  source_path="src/${compiled#package/dist/}"
  source_path="${source_path%.js}.ts"
  if [[ ! -f "$source_path" ]]; then
    note "$compiled has no source at $source_path"
  fi
done < <(grep -E '^package/dist/.*\.js$' "$listing")

That seems kind of backwards, but we’re trying to prevent stale files inadvertently being included in the build and shipped off to npmjs. Every compiled JavaScript file under dist/ should have a corresponding source file under src/. It’s a bit like going through the cargo hold of a ship and checking that each piece of cargo corresponds correctly to a line on the shipping manifest.

As we’re doing this verification of the package for shipping, we can fit in some extra checks. One is to confirm that the entry points specified in package.json are present and correct in the package. The other is to confirm that unwanted directories like src, test, scripts and docs are not present in the package.

for required in \
  package/dist/index.js \
  package/dist/index.d.ts \
  package/README.md \
  package/LICENSE; do
  grep -qxF "$required" "$listing" || note "missing $required"
done

if grep -qE '^package/(src|test|scripts|docs)/' "$listing"; then
  note "the tarball carries sources, tests or docs"
fi
if grep -qE '\.test\.(js|ts|d\.ts)$' "$listing"; then
  note "the tarball carries test files"
fi

if [[ "$fail" -ne 0 ]]; then
  exit 1
fi

echo "pack:check: $(wc -l <"$listing" | tr -d ' ') files, $(du -h "$tarball" | cut -f1)"

That logic is contained a shell script file, but we can make it easier to run by adding it as an npm script.

{
  "scripts": {
    "pack:check": "./scripts/sh/pack-check.sh",
    "check": "pnpm lint && pnpm build && pnpm build:check && pnpm test:coverage && pnpm pack:check"
  }
}

It’s also handy to get the file count and total size in the console output for every build check.

pack:check: 307 files, 4.1M