[DanielDarrenJones]

Warming the Stache makes every entry of a collection, keeps all of them in memory, and then has each index read through all of them. Peak memory is the whole collection, parsed, and it grows with the content.

Entries are now made and cached one at a time, and the indexes are filled from a single pass over them. An entry is let go of once it has been indexed, so what's held is the index values rather than the entries.

Opt-in on 6.x

This changes how every store builds its indexes, so on 6.x it is off by default:

// config/statamic/stache.php
'single_pass_indexing' => env('STATAMIC_STACHE_SINGLE_PASS_INDEXING', false),

A site whose published stache.php predates this needs that line added before the env variable does anything. With it off, paths(), warming and term associations run the code they run on 6.41.0: the old paths are kept beside the new ones rather than rewritten, and the numbers and the rendered output match 6.41.0 (below).

If you'd rather not carry a switch and two code paths in 6.x, I'm happy to hold this back, take the opt-in out and re-target it at 7.x as the only behaviour. That version is the first three commits here; the opt-in is the last one on its own.

What changes

With the option on:

• Store::paths() makes, caches and moves on, rather than building every item before caching any.
• Store::fillIndexes() fills every index that takes its value from one item at a time (Indexes\Value and its subclasses) in one pass. Index::update() goes through it, so the first index a cold request needs fills the rest of that store's indexes in the same pass, instead of each reading every item for itself later.
• Entries that need another entry are indexed in a second pass, read back one at a time: localizations (they take values from their origin), entries in a structure, and anything whose values go looking for other entries, such as a collection mounted on a page or a computed value that queries its own collection. If that happens while items are still being made, the entry and the ones after it wait for the second pass, and the index that was asked for too early is asked for again then.
• A structure's uri, parent and order indexes are still built the way they were, after the others. Finding an entry's place in a tree queries the collection, which needs its other indexes to exist. Entries of a structured collection are therefore still held, as they are today.
• Term associations are built from the entries' own indexes for the taxonomy field and site, rather than by loading every entry that has a term. On the site below this was the largest single holder of entries.
• The snapshots from [6.x] Don't keep two copies of every entry in memory while building the Stache #15795 are untouched and still used for structured collections. For other collections an entry's snapshot is dropped when the entry is, since it's indexed from the object that was just made.

An index that overrides getItems(), or a store that overrides getItemsFromFiles(), is left to build itself as before.

Benchmark

Against 6.41.0, with the option off and with it on, alternating, five rounds each, on an Apple M1 Pro with PHP 8.5 and the file cache store. Each run starts from a cleared Stache. Medians, with the range of the five runs.

A generated site of 9,780 entries (3,000 of them localizations across three sites, nested page builder content, two taxonomies):

	6.41.0	This PR, off	This PR, on	On vs 6.41.0
stache:warm, time	21.8 s (21.4 to 24.1)	21.4 s (19.9 to 22.2)	16.4 s (16.1 to 17.6)	−25%
stache:warm, peak memory	1,200 MB	1,201 MB	300 MB	−75%
First query on a cold Stache, time	9.9 s (9.8 to 10.5)	9.8 s (9.7 to 9.9)	10.9 s (10.7 to 11.1)	+10%
First query on a cold Stache, peak memory	609 MB	612 MB	245 MB	−60%

A second generated site of 2,700 files (two sites, a structured pages collection, a collection mounted on a page): stache:warm 5.1 s and 357 MB before, 5.4 s and 87 MB after. One run each, so the times are level and the memory is the result.

The first query on a cold Stache is slower. It's a query for twelve books ordered by title. Before, it built the four indexes it used and left the rest for later requests. Now it builds all of that collection's indexes in its one pass. The total work is less, but more of it lands on that first request. A site that runs stache:warm on deploy doesn't see this.

Profiling the warm: 23% of the time on 6.41.0 goes on reading entries back out of the cache, against 9% here.

Parallel warming

With STATAMIC_STACHE_PARALLEL_WARMING=true on the first site, three rounds each:

	6.41.0	This PR
Runs in parallel	No, falls back to sequential every time	Yes, 8 processes
Time	26 to 30 s	18.1 to 18.4 s
Peak memory, all processes together	1.1 to 1.2 GB, in the one process	about 505 MB, the largest about 270 MB

On 6.41.0 the worker processes run into PHP's 512 MB memory limit (Allowed memory size of 536870912 bytes exhausted), and the warm logs Parallel warming failed, falling back to sequential. Here each worker stays under the limit, so it runs as intended. The indexes and URLs are the same as a sequential warm on 6.41.0. It isn't faster than a sequential warm on this site, because one worker still does nearly all of the entries.

Checked

• Same result. Every index of every store and every entry and term URL is identical before and after on both sites (83 indexes and 12,840 URLs; 106 indexes and 2,884 URLs).

• Octane and queue workers. One long-running process, with a second process saving an entry, rebuilding the Stache and clearing it in between, run as a worker would handle requests (new container sandbox and facades each time) and as it would handle jobs (the between-jobs reset). Every value the worker read matched 6.41.0 at every step. Nothing is kept on a store between requests or jobs: the state of a pass is cleared when it ends, and the marker that stops a store being filled twice in one warm only exists while Stache::warm() is running.

• Entries held by that worker after first use went from 9,780 to about 30, and its memory from about 650 MB to about 265 MB.

• Off is 6.41.0. On a third generated site with deliberately awkward content (hand-written YAML, number and duplicate ids, computed values that query, scheduled entries, a thousand-term taxonomy, relations across sites, six-level trees), every page, Control Panel response, a fixed list of 136 indexes and the results of 44 queries are identical to 6.41.0 with the option off and with it on.

Not covered

• A site whose largest collection is structured gains less, because that collection's entries are still held for the three tree indexes.
• An index that's asked for during the second pass (by a computed value that queries, say) builds itself the way it used to, reading every entry of that collection. The result is right, but those entries are held while it does.
• With parallel warming, 6.41.0's own parallel time couldn't be measured, since its workers don't complete.

Tests

FillingIndexesTest (new) and three additions to EntriesStoreTest:

• with the option off, each index builds itself and nothing else is filled;
• asking for one index fills the others in the same pass;
• filled indexes hold what each index would build for itself, across four fixture collections including a structured one;
• a localization is indexed with the values it takes from its origin;
• entries of a collection mounted on another entry are indexed once every entry exists;
• a computed value that queries the collection is indexed once every entry exists;
• warming leaves indexes alone when updating them is disabled;
• term associations are built from the entries' indexes;
• entries aren't kept once they've been indexed, and are for a structure;
• rebuilding one index leaves the others alone;
• warming twice in one process fills the indexes both times, and a store another store needed during a warm isn't filled twice;
• a queue worker, and a later request in the same process, fill indexes from what is cached at the time.

The existing snapshot tests from #15795 pass unchanged. The full suite passes with the option off and with it switched on for every test (11,758 passed, 12 skipped, both ways).