Repository navigation
perf(build): stage IR with perl, a faster axiomcode-index, and AXIOMCODE_REINDEX=1 rebuilds; same graphs (#1835) - #1965
Open
swapnilpaliwal-sd wants to merge 4 commits into
Open
swapnilpaliwal-sd wants to merge 4 commits into
swapnilpaliwal-sd wants to merge 4 commits into
Conversation
swapnilpaliwal-sd
requested review from
JaredHLZhang,
Whua689 and
suyashpaliwal26
as code owners
October 15, 2026 03:15
swapnilpaliwal-sd
added
bug
Staging copies every IR CSV into a Souffle fact file and drops rows whose tab-separated field count differs from the header's. BSD awk spends most of that time splitting fields it never reads: on a ~1,300-file Python project the step took 16-19 s of wall time per build, 13.6 CPU-s of it on the largest IR file (expressions). perl counts tabs instead and stages the same bytes (identical md5 over every staged file of three projects, two Python and one Java) at about a tenth of the CPU. The filter moves to graph/pipeline/stage-rows.sh. It picks perl when it runs (macOS, mainstream Linux, and Git for Windows' bash, which is the shell the pipeline runs under on win32) and falls back to the same awk program where perl is missing, so no machine loses the step. AXIOM_STAGE_WITH=awk forces the fallback. graph/test/tools/stage-rows-test.sh holds both to the same stdout and stderr on the edge cases (trailing tabs, empty lines, CRLF, no final newline, torn rows) and runs as a Java suite preflight. Refs #1835
… nothing to reclaim, do not rebuild bundle indexes On a ~1,300-file Python project axiomcode-index took 12-14 s. Three parts of that bought nothing: - csv.DictReader over the IR. The reader is QUOTE_NONE, so a row is one line and a field is the text between tabs; ir_rows() does exactly that with str.split and dict(zip()) at about half the cost. It keeps the reader's semantics where they are not obvious: \r, \n and \r\n each end a row, blank lines are skipped, a short row gets None for the missing columns and a long one keeps the rest under the key None. tests/index_reader.py compares the two on fixtures (quoted tabs and newlines, CRLF, lone CR, repeated header names, invalid UTF-8) and on any IR directory given to it. - VACUUM of a database the bundle wrote seconds earlier. It now runs only when this run dropped a table it recreates (a re-index) or the file has free pages; a freshly bundled graph has neither. - ce_caller, ce_callee and fa_caller duplicate the bundle's own idx_call_edges_caller_id, idx_call_edges_callee_method_id and idx_field_access_caller_id. They are created only when the graph has no index on that column, so an older graph keeps getting them. graph.sqlite holds the same rows in every table (rowids included) on two Python projects and one Java project; only index_meta.built_at and the three duplicate indexes differ. Refs #1835
The build's messages, SKILL.md and ax_fresh all describe
AXIOMCODE_REINDEX=1 as a rebuild ("rebuilds it with this one", "to
force"). It skipped the newer-build shortcut and the remembered languages,
but both up-to-date checks still answered "graph up to date" whenever the
files matched the file table, so it rebuilt nothing. Both checks now step
aside when it is set.
tests/refresh.py checks that `index` with AXIOMCODE_REINDEX=1 rebuilds
over an unchanged tree; the existing check beside it, the same `index`
without it, is the control.
Refs #1835
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1975
Part of #1835.
Three build-time savings. Each one leaves the graph unchanged: on a ~1,300-file Python project, a ~700-file Python project and a Java project, every table has identical rows (rowids included) before and after. Only build metadata differs (
built_at, timings, engine paths).1. Stage IR rows with perl when it is there (
graph/pipeline/stage-rows.sh)The staging filter ran under BSD awk, which took 12 CPU-s across all IR files of the ~1,300-file project. The same filter in perl produces byte-identical output: the per-file and overall md5 match for every staged file on all three projects. It falls back to the same awk program when perl is absent; Git for Windows' bash, which the win32 path uses, ships perl.
AXIOM_STAGE_WITH=awkforces the fallback.Test:
graph/test/tools/stage-rows-test.sh, run as a preflight of the Java suite. It compares perl and awk output on 14 edge-case fixtures (47 checks), and fails if perl is present but not chosen. A mutant that drops the empty-line case fails 5 checks.2. axiomcode-index: a faster IR reader, no needless VACUUM, no duplicate indexes
ir_rows()replacescsv.DictReaderwith a line split. It keeps DictReader's semantics exactly: line endings, skipped blank lines, short and long rows. It returns the same dicts on every row of the real IR of all three projects (1.6 M rows).ce_callee,ce_callerandfa_callerare created only when the bundle did not already index those columns.axiomcode-index alone on the ~1,300-file project: 13.5 / 10.3 s → 9.3 / 7.7 s (wall / CPU), and the file is 7 MB smaller. Test:
tests/index_reader.py(21 fixtures, identical under Python 3.13 and 3.9; two mutants fail it).3.
AXIOMCODE_REINDEX=1rebuilds a graph whose files are unchangedIt was documented to force a rebuild, but both up-to-date checks in
axiomcode-buildstill answered "graph up to date". A new check intests/refresh.pyfails before the fix and passes after.End to end
Cold index of the ~1,300-file Python project: CPU 118.6 → 104.7 s (−12%). Wall time was measured under load, so it is noisy.
Suites
tests/run.py --lang java: 372/374.--lang python: 437/437 when re-run alone. The 2 Java failures fail the same way on the base.run-suite.sh python: 43/43.run-suite.sh java --no-torture: 79/0.--oracleleg needs JDK 24+, which this machine lacks, so CI is the first to run it.tests/surfaces.pyandtests/refresh.pyfail only on checks that also fail on the base.run-souffle.shchanges CI's engine cache key, so the first CI run recompiles the engines.