Varka Notes
Vectorising Apache Spark's expressions inside the JVM.
9 November 2026Your sin() depends on your JDK and your instruction set
Eight of Spark's math functions can return a different last bit on a different machine:
why Math and StrictMath part ways, what changes the answer (the
instruction set and the JDK, not the chip, the JIT or codegen), where it bites, and what to do.
Two compilers, two queues
How HotSpot's five tiers, its two compilers and their two queues fit together, and why the length of C2's queue at one instant decides whether a method is ever profiled. With the policy's own trace of one method in a passing and a failing run, and what a warm-up has to do about it.
17 October 2026How a SQL engine is tested: ten methods, what each found, and what each missed
For testers: the oracle is vanilla Spark, errors included. Ten methods from solver proofs to mutation testing, each with an example, its cost and its yield, and the defects laid out by the method that found them. One was found by none of the ten.
17 October 2026How Spark compiles a query, and why it compiles it again
Why a dashboard query with a new date runs about twice as slow as a rerun with the same one: Spark writes the value into the code it generates, compiles the stage again, and keeps only about a hundred classes in its cache.
16 October 2026Nine numbers between your query and machine code
The limits a Spark SQL stage passes on its way to machine code: where each one acts, its default in Spark 4.2.0, and what happens past it, which is mostly a slower path with no error and no log line.
16 October 2026Why a warmed Vector API kernel can still allocate
A kernel C2 had compiled at the top tier allocated an array on every loop iteration, in a few warm-ups in every hundred: how C2 sums class-check traps over a whole compile, who leaves the traps, the JVM options that remove it, and the warm-up that all but does.
16 October 2026The class got cheaper. The first query got slower.
For people who write code generators: what changes when the generator writes bytecode instead of Java source, why a class three to six times cheaper to make did not speed up the first query, and where the saving that matters comes from: how classes are cached.
12 October 2026A large CASE WHEN, split only where it has to be
The sequel to the cliff: Spark 4.4.0 splits a large CASE WHEN inside a
stage only where its method would pass 8000 bytes; why splitting everywhere costs, how to
see the fix, the settings it adds, and what to do before you can upgrade.
When Spark stops compiling your query
The other places Spark's code generation gives up, and rarely says so: an operator that leaves its stage, a wide projection slower inside one, a limit logged where the shell hides it, a method interpreted instead of compiled; what each costs, and how to see it.
1 October 2026Under 8000 bytes by construction
Why a generator of Java source cannot keep a method under the JIT's limit, and how an engine that emits bytecode measures every method of the class it built and splits or declines before anything runs; with what that buys and costs, measured.
27 September 2026The 8000-byte cliff in Spark SQL
Why a query can get six times slower when a projection, a CASE WHEN or a
filter grows by a few terms, how to see it on your own cluster, what to set, and which
Spark release fixes which part.
Eight rows per instruction: Spark's TIME type on Varka's 64-bit lane
What a row-at-a-time loop leaves on the table, what one emitted vector loop does instead, and what the two cost per row on a machine whose datapath was measured rather than assumed.