fileserver: parallelize directory listing to speed up large-directory browsing

[firefart]

Summary

file_server browse listings were slow for directories with many entries.
directoryListing() called entry.Info() for every entry sequentially - on Unix that's a real Lstat syscall per entry (Windows already gets this for free from FindNextFile), plus a second Stat (and optional Readlink) for every symlink. A directory with thousands of files paid thousands of sequential syscall round-trips.

This splits the work into three passes and parallelizes the expensive one:

1. Filter (sequential) — drop hidden entries, no syscalls involved.
2. Stat (parallel) — a bounded pool of workers (sync.WaitGroup.Go, Go 1.25+) each stat a contiguous chunk of the remaining entries. Each worker owns disjoint indices into a preallocated results slice, so there's no shared mutable state and no locking. Workers bail out on ctx.Err() per item, matching the previous cancellation behavior.
3. Aggregate (sequential) — sum NumDirs/NumFiles/TotalFileSize/TotalFileSizeFollowingSymlinks and build Items from the results. Cheap, no syscalls.

Final Items order doesn't matter here since applySortAndLimit always resorts before any output format (JSON/text/HTML) uses it.

Worker count defaults to 16 (browse.go's defaultDirListingConcurrency), deliberately conservative rather than tied to GOMAXPROCS: this is a syscall/IO-bound workload (Go already scales OS threads for blocked syscalls independent of GOMAXPROCS), and the bound is per request, not global — nothing caps concurrent syscalls across simultaneous browse requests, so a high default would multiply badly under concurrent load. It's now also configurable per-site:

file_server {
    browse {
        concurrency 32
    }
}

(0/unset uses the built-in default.)

RevealSymlinks now also checks whether the fs.FS implements a new readLinkFS interface (ReadLink(name string) (string, error)) before falling back to os.Readlink, so custom filesystem modules can resolve symlinks their own way instead of assuming an OS path.

Benchmark

go test -bench=BenchmarkDirectoryListing -benchmem -run '^$' ./modules/caddyhttp/fileserver/, real OS-backed directories (existing BenchmarkDirectoryListing harness), before vs. after, ns/op:

entries	before	after	speedup
100	178,349	138,127	~1.3x
1,000	1,773,131	772,169	~2.3x
10,000	20,534,202	4,599,638	~4.5x
50,000	96,676,685	22,168,099	~4.4x

Allocations rise modestly (e.g. ~56KB → ~80KB at 100 entries) from goroutine/closure overhead — an accepted tradeoff for the wall-clock win, and there's no regression even at small directory sizes since worker count is capped at min(concurrency, len(visible)).

Assistance Disclosure

I used Claude Code to design and implement this change (directory-listing parallelization, the Concurrency config option and its Caddyfile syntax, and the new test), working from my own investigation that large-directory listings were slow. I reviewed the design, benchmarked it, and made additional changes myself on top (the readLinkFS symlink-resolution interface, and the concurrencySet fix in the Caddyfile parser).

[Copilot]

Pull request overview

Reviewed by GitHub Copilot (agent).

This PR speeds up file_server browse directory listings for large directories by parallelizing per-entry filesystem metadata collection, while adding a configurable concurrency knob and improving symlink target resolution for custom filesystem implementations.

Changes:

• Parallelize the expensive per-entry Info/Stat/Readlink work using a bounded worker pool, then aggregate results sequentially.
• Add browse { concurrency <n> } Caddyfile support and a Browse.Concurrency JSON field with a conservative default.
• Add a new directory-listing unit test and an integration Caddyfile-adapt test for the new config.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
modules/caddyhttp/fileserver/caddyfile.go	Adds Caddyfile parsing for browse { concurrency ... }.
modules/caddyhttp/fileserver/browsetplcontext.go	Refactors directoryListing() into filter/stat/aggregate passes and introduces parallel statting plus readLinkFS.
modules/caddyhttp/fileserver/browsetplcontext_test.go	Adds a unit test validating aggregated listing results across concurrency settings and failure cases.
modules/caddyhttp/fileserver/browse.go	Adds Browse.Concurrency config and default concurrency constant.
caddytest/integration/caddyfile_adapt/file_server_browse_concurrency.caddyfiletest	Verifies Caddyfile adaptation outputs the expected JSON for browse.concurrency.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (3)

modules/caddyhttp/fileserver/browsetplcontext.go:218

• Custom filesystem ReadLink support is unreachable in real requests. Filesystems returned by fsrv.fsmap.Get are internal/filesystems.wrapperFs values (map.go:44), and that wrapper embeds an fs.FS-typed field, so optional methods on the underlying implementation are not in the wrapper's method set. This assertion therefore fails and falls back to os.Readlink, recreating the behavior this change is intended to fix. Forward Go 1.25's standard fs.ReadLinkFS capability through the wrapper (and use fs.ReadLink) so registered modules can actually resolve their links.

			if rlFS, ok := fileSystem.(readLinkFS); ok {
				symLinkTarget, err = rlFS.ReadLink(targetPath)
			} else {
				symLinkTarget, err = os.Readlink(targetPath)

modules/caddyhttp/fileserver/browse.go:79

• The native JSON API can set concurrency to a negative value because the rejection exists only in the Caddyfile adapter. Provisioning then succeeds and directoryListing silently treats the invalid value as the default, although this contract documents only zero as the default sentinel. Validate Browse.Concurrency < 0 during FileServer.Provision so every config adapter enforces the same range.

	// Concurrency sets how many directory entries are stat'd (and, for
	// symlinks, have their target resolved) at once while building a
	// listing. Gathering this per-entry info involves filesystem syscalls,
	// so raising this can speed up listings of large directories; lowering
	// it reduces the burst of concurrent filesystem calls a single listing
	// request can generate. If 0 (default), a built-in default is used.
	Concurrency int `json:"concurrency,omitempty"`

modules/caddyhttp/fileserver/browsetplcontext.go:104

• This chunk calculation can launch substantially fewer workers than requested. For example, with 17 visible entries and concurrency 16, chunkSize is 2, so only workers 0–8 have non-empty ranges and concurrency is limited to 9. Partitioning with lo := w * len(visible) / concurrency and hi := (w+1) * len(visible) / concurrency keeps all configured workers non-empty because concurrency is already capped to the entry count.

		chunkSize := (len(visible) + concurrency - 1) / concurrency

		var wg sync.WaitGroup
		for w := 0; w < concurrency; w++ {
			lo := w * chunkSize
			hi := min(lo+chunkSize, len(visible))