Index Kotlin with the Analysis API instead of a compiler plugin

[jupblb]

Replaces the scip-kotlinc compiler plugin with scip-kotlin-analysis, a standalone indexer built on the Kotlin Analysis API. The indexer bundles its own analysis engine, so the Kotlin version of indexed projects is no longer a binary-compatibility constraint.

• Output verified against scip-kotlinc on the snapshot corpus and a large-corpus A/B diff (no regressions; several old-indexer bugs fixed)
• Gradle integration now extracts per-task compile configs instead of injecting a plugin into kotlinc
• Kotlin toolchain and Analysis API engine updated to 2.4.0 (supersedes Update kotlin monorepo to v2.4.10 #871)

[ofir-blaus]

Thanks for this. We tried it (branch at a77e133) on several production Android/Kotlin codebases where the scip-kotlinc plugin can't run:

• Kotlin 2.3.21 / AGP 9.1 / Gradle 9.3: the plugin is rejected by the compiler (AbstractMethodError ... getPluginId()), and scipPrintDependencies hits the CME from Kotlin 2.3 / AGP 9 / Gradle 9 unsupported — both auto-mode and semanticdb-kotlinc fail #864
• Kotlin 2.2 (Gradle's embedded compiler, build-logic): NoSuchMethodError CheckerContext.getContainingFile()
• Kotlin 1.9: NoSuchFieldError MESSAGE_COLLECTOR_KEY

The standalone scip-kotlin-analysis indexer handled all of them, because the project's Kotlin version no longer matters. On a ~85K-LOC multi-module Android SDK it ran in under 30s. Passing each module's resolved compile classpath (we collect it with a resolution-only Gradle task, no compile) also resolved references into dependency artifacts, and they matched the producing repo's own definitions by descriptor.

We'd really like to see this merged and released. Is anything blocking it that we could help with, like testing on more real-world projects?

[christoph-sg]

@jupblb Thanks for yesterdays chat again! I'd like to review this PR and get it merged. I'm going to need to spend some time to build the necessary context, but as we discussed I agree this is the best path forward for indexing Kotlin code.

@ofir-blaus If you could test this on some more real world projects, I'd be thrilled. I'm especially curious about Android projects. If you could also compare the output you get on the projects that do build with the old indexer to the indexes you're getting from the new one that would be very helpful.

[ofir-blaus]

@christoph-sg thanks for picking this up! Happy to help with testing. We compared the old indexer (scip-java 0.13.1) with this branch on three projects:

• our Kotlin service;
• JetNews (an Android sample app);
• ktlint (363 files).

We had to use Kotlin 2.2.x, because 0.13.1 only loaded on 2.2.0 and 2.2.10. It failed on 2.1, 2.2.20+, 2.3 and 2.4. On Android it also produced no output until we attached the plugin with an extra Gradle script.

Overall: the results mostly match. Where both indexers mark the same code, 94–98% of symbols are identical. The new one finds 6–11% more references and runs 2–3x faster, without a build.

Better in the new indexer:

• Enum entries are defined. The old one left hundreds of references pointing at nothing.
• Imported constants and nested classes resolve. The old one wrote them as packages.
• Overloaded Java methods get the right (+N) suffix.

Missing or different in the new indexer:

• Imports of top-level functions have no reference.
• Implicit calls have no reference: calling a lambda, [] indexing, for loops, by delegates.
• Library constructors always get (). For example, Regex(String) should be (+1).
• Type aliases aren't expanded. HashMap() points to kotlin/collections/HashMap instead of java/util/HashMap.
• Code it can't resolve is dropped silently. Without a classpath, ktlint lost 23% of its references; a warning would help.

Same in both:

• extension properties with the same name on different types share one symbol;
• val isX gets getIsX();
• a private var gets a setter symbol that can clash with a hand-written one.

One crash, new indexer only: a top-level val (a, b) = ... in a .kt file crashes the indexer, and then it hangs. I opened #1038 with a fix, and I'd be glad to get a review on it.

Happy to run it on more projects, or share the scripts, if that helps.