LLMs are very good at writing Python.
That is partly a consequence of Python’s popularity: there is an enormous amount of Python in training data, its syntax is relatively compact, and many programming tasks can be expressed without much boilerplate. Ask a coding model to implement a parser, data transformation, feature extraction pipeline, simulation, or preprocessing stage and Python is often the natural result.
Consider a simplified request we might give an LLM when building an inference pipeline:
Given a stream of search events, normalize each query, map words to token IDs, incorporate per-user features, discard events that don’t contain enough recognized tokens, and return fixed-size feature vectors that can be passed to a model.
A plausible implementation might look something like this:
import re
VALID_EVENTS = {"search", "recommendation"}
MAX_FEATURES = 64
MIN_FEATURES = 4
def normalize(text):
text = text.lower()
text = re.sub(r"[^a-z0-9 ]", " ", text)
return text
def prepare_events(events, user_features, vocab):
batch = []
for event in events:
if event["type"] not in VALID_EVENTS:
continue
text = normalize(event["query"])
features = []
for token in text.split():
token_id = vocab.get(token)
if token_id is None:
continue
value = token_id * event["weight"]
value += user_features.get(
(event["user_id"], token_id), 0.0
)
features.append(value)
if len(features) < MIN_FEATURES:
continue
features = features[:MAX_FEATURES]
while len(features) < MAX_FEATURES:
features.append(0.0)
batch.append((event["user_id"], features))
return batch
This is ordinary Python. More importantly, it is the sort of application-specific code that often exists around optimized ML libraries: filtering records, manipulating strings, looking things up in dictionaries, applying business rules, computing small features, and assembling the result into the representation expected by the next stage.
There probably isn’t a library call that replaces prepare_events(), because its behavior is specific to the application.
An LLM can produce code like this quickly. We can run it, write tests around it, give the failures back to the model, change the requirements, and regenerate parts of the implementation. That makes Python an attractive intermediate representation between the developer and the coding model.
The interesting question comes later:
What happens if this code becomes part of the production system and isn’t fast enough?
Identifying the bottlenecks
Suppose prepare_events() eventually sits in front of a GPU inference worker. We run the system under a production-like workload and find that the model is not the bottleneck we expected it to be.
A simple profile might look roughly like this:
1000 batches
prepare_events() 4.7 s
model inference 1.8 s
other 0.4 s
--------------------------------
total 6.9 s
These are illustrative numbers (we’ll showcase measured results in a demo later) but the situation itself is straightforward. The GPU can process a batch faster than the CPU can prepare the next one.
A timeline might make the problem more obvious:
CPU: [ prepare ][ prepare ][ prepare ][ prepare ]
GPU: [infer] [infer] [infer]
^ idle ^ idle
At that point, making the model itself 20% faster will have limited effect on end-to-end throughput.
This is not because “Python is slow” in some general sense. Much of a modern Python ML application may already execute outside the Python interpreter: PyTorch operations execute native kernels; NumPy delegates numerical operations to compiled implementations; Hugging Face’s fast tokenizers are implemented in Rust; database and data-processing systems often do the expensive work in native code as well.
If our preprocessing function looked like this:
def preprocess(x):
return np.sin(x) + np.cos(x)
compiling the Python surrounding those NumPy calls would probably not be particularly interesting (even though there are still optimization opportunities for this seemingly simple function, like operator fusion).
Our example is different because substantial work is happening in Python itself:
for event in events:
if event["type"] not in VALID_EVENTS:
continue
...
for token in text.split():
token_id = vocab.get(token)
if token_id is None:
continue
value = token_id * event["weight"]
value += user_features.get(
(event["user_id"], token_id), 0.0
)
features.append(value)
Iteration, branches, lookups, and construction of the output all remain in the interpreter.
The profiler, rather than the programming language alone, tells us whether this matters.
Expanding on the pure Python solution
Discovering a CPU-bound Python function does not mean we immediately need another programming language.
There are usually simpler things worth trying first.
Perhaps some transformations can be cached. Maybe we are repeatedly parsing information that could be computed once. Perhaps batching can reduce overhead. Some of the work may be expressible through a library that already has a native implementation.
And if individual records are independent, multiprocessing is an obvious option.
For example, we could divide the workload across worker processes:
from multiprocessing import Pool
def prepare_chunk(args):
events, user_features, vocab = args
return prepare_events(events, user_features, vocab)
chunks = [
events[i:i + 10_000]
for i in range(0, len(events), 10_000)
]
with Pool(8) as pool:
batches = pool.map(
prepare_chunk,
[
(chunk, user_features, vocab)
for chunk in chunks
],
)
For many applications, this is a perfectly good solution.
It also changes the resource profile of the application. Data may have to be serialized between processes. Workers need to be managed. Large lookup structures can increase memory pressure. The machine may simply need substantially more CPU capacity to keep the downstream accelerator busy.
That means a useful comparison should measure more than execution time:
Throughput CPU Peak RSS
CPython ... ... ...
8 Python workers ... ... ...
If eight workers provide the required throughput at an acceptable cost, there may be no reason to do anything else.
But suppose they don’t, or suppose the resulting CPU and memory requirements are themselves becoming expensive. At that point, a common next step is to move the expensive part into native code.
The traditional native-code path
The core of prepare_events() is well suited to native execution. It consists largely of loops, branches, string processing, dictionary lookups, and arithmetic. An AI/ML team might therefore reimplement it in C++ or Rust and expose the resulting function back to Python.
The C++ version of just one part of our processing logic might look something like this
std::vector<float> prepare_features(
const Event &event,
const Vocabulary &vocab,
const UserFeatures &user_features) {
std::vector<float> features;
features.reserve(MAX_FEATURES);
for (const auto &token : split(normalize(event.query))) {
auto it = vocab.find(token);
if (it == vocab.end())
continue;
auto token_id = it->second;
float value = token_id * event.weight;
auto user_it =
user_features.find({event.user_id, token_id});
if (user_it != user_features.end())
value += user_it->second;
features.push_back(value);
if (features.size() == MAX_FEATURES)
break;
}
if (features.size() < MIN_FEATURES)
return {};
features.resize(MAX_FEATURES, 0.0f);
return features;
}
There is nothing inherently problematic about this implementation. In many systems, C++ or Rust is exactly the right choice. But the change is larger than the source code shown above:
- We also have to decide how
Event,Vocabulary, andUserFeaturescross the Python/native boundary. We need bindings. - We need a native build and packaging process.
- We need to verify that the native implementation behaves identically to the original Python as the preprocessing rules evolve.
Our workflow has gone from:
LLM generates Python
↓
run tests
↓
modify Python
↓
ship
to something more like:
LLM generates Python
↓
run tests
↓
profile
↓
identify hot path
↓
implement native version
↓
define Python/native boundary
↓
build + package extension
↓
differential testing
↓
maintain implementation
An LLM can help generate the C++ too. It can generate bindings, CMake files, and tests.
But that does not eliminate the transition. We still have to decide where the boundary belongs, validate a second implementation of the algorithm, debug problems that cross the boundary, and decide how future changes propagate between the Python and native versions.
This is where LLM-generated software makes the tradeoff more interesting.
LLMs changed one side of the equation
The traditional argument for prototyping in Python and rewriting performance-sensitive code later was reasonable because both stages involved substantial engineering work.
The first implementation might take days. If it proved valuable, spending additional time optimizing or rewriting it was simply another phase of development.
Coding models have changed the economics of the first phase.
A developer can now describe a transformation, generate a Python implementation, run it, give the model test failures, and iterate on the behavior very quickly. For many tasks, the initial implementation is no longer particularly expensive, but the production transition can still involve introducing another language, another representation of the algorithm, and another integration boundary.
In other words, LLMs have made this:
idea → working Python
much cheaper.
They have not necessarily made this:
working Python → production native implementation
equally cheap.
What if we compile the implementation instead?
Codon takes a different approach to this problem.
Codon is a compiler for Python-oriented code that generates native machine code. It does not attempt to execute every arbitrary Python program unchanged: highly dynamic behavior, arbitrary Python objects, and interactions with unsupported Python libraries can require changes.
But application code consisting primarily of typed data structures, loops, branches, strings, dictionaries, sets, and numerical computation often maps naturally to compiled code.
That means the production version of our processing function can remain much closer to the Python implementation we started with.
For example, the core processing logic might remain:
def prepare_events(events, user_features, vocab):
batch = []
for event in events:
if event.type not in VALID_EVENTS:
continue
text = normalize(event.query)
features = []
for token in text.split():
token_id = vocab.get(token)
if token_id is None:
continue
value = token_id * event.weight
value += user_features.get(
(event.user_id, token_id), 0.0
)
features.append(value)
if len(features) < MIN_FEATURES:
continue
features = features[:MAX_FEATURES]
while len(features) < MAX_FEATURES:
features.append(0.0)
batch.append((event.user_id, features))
return batch
The important comparison is not whether this syntax looks nicer than C++. It is how much of the original implementation we actually had to change.
That’s something we can measure.
def prepare_events(events, user_features, vocab):
batch = []
for event in events:
- if event["type"] not in VALID_EVENTS:
+ if event.type not in VALID_EVENTS:
continue
- text = normalize(event["query"])
+ text = normalize(event.query)
...
- value = token_id * event["weight"]
+ value = token_id * event.weight
The actual diff will depend on the workload and on which Python features it uses. Some programs require more adaptation than this; others can be compiled with very little modification. Rather than hide those changes, they should be part of the evaluation.
If the original LLM-generated implementation requires substantial restructuring to compile, that matters just as much as the runtime result.
Choosing the compilation boundary
A production system does not necessarily have to be converted wholesale.
One option is Codon’s Python JIT integration, which lets supported functions execute as compiled code while the surrounding application remains in CPython:
import codon
@codon.jit
def prepare_events(events, user_features, vocab):
# CPU-intensive processing
...
The rest of the inference worker can continue to look like ordinary Python:
while True:
events = queue.next_batch()
features = prepare_events(
events, user_features, vocab
)
predictions = model(features)
publish(predictions)
The boundary matters here.
Crossing from CPython into compiled code is not free, and Python values may need to be converted to native representations. Compiling a tiny operation that is repeatedly invoked from Python is therefore unlikely to be useful:
@codon.jit
def lookup_one_token(token, vocab):
return vocab.get(token, 0)
We would rather move enough computation across the boundary that the cost can be amortized:
@codon.jit
def prepare_batch(events, user_features, vocab):
# thousands of records
# normalization
# filtering
# lookups
# feature extraction
# packing
...
In another architecture, the boundary might move farther out. The entire preprocessing worker could be compiled, or Codon could be used to build an extension or standalone service.
There isn’t one correct integration pattern. The useful question is how much of the workload can remain in a compiled representation before control returns to Python.
Quantifying the improvement
Let’s actually run a concrete preprocessing example in both Python and Codon to showcase the difference:
import sys
import time
PUNCT = ".,!?"
def load_vocab(path):
vocab = {}
with open(path) as f:
for line in f:
token, token_id = line.rstrip("\n").split("\t")
vocab[token] = int(token_id)
return vocab
def load_blocked(path):
blocked = set()
with open(path) as f:
for line in f:
blocked.add(int(line))
return blocked
def load_user_bias(path):
bias = {}
with open(path) as f:
for line in f:
user_id, value = line.rstrip("\n").split("\t")
bias[int(user_id)] = float(value)
return bias
def load_events(path):
events = []
with open(path) as f:
for line in f:
user_id, event_type, weight, query = line.rstrip("\n").split("\t")
events.append((int(user_id), int(event_type), float(weight), query))
return events
def prepare_events(events, vocab, blocked, user_bias):
output = []
for user_id, event_type, weight, query in events:
if event_type != 2:
bias = user_bias.get(user_id, 0.0)
recognized = 0
score = 0.0
mix = 0
# Application-specific string processing + lookups + branching.
text = query.lower()
for raw_token in text.split(" "):
token = raw_token.strip(PUNCT)
token_id = vocab.get(token, -1)
if token_id >= 0 and token_id not in blocked:
recognized += 1
score += token_id * weight * 0.00001 + bias
if token_id % 7 == 0:
score += 0.125
elif token_id % 11 == 0:
score -= 0.075
mix = (mix + token_id * (recognized + 1)) % 1_000_003
if recognized >= 8 and score > 0.0:
output.append((user_id, score, mix))
return output
def checksum(output):
s = 0.0
for user_id, score, mix in output:
s += score + user_id * 1e-9 + mix * 1e-12
return s
def main():
base = "benchdata"
repeat = 3
if len(sys.argv) >= 2:
base = sys.argv[1]
if len(sys.argv) >= 3:
repeat = int(sys.argv[2])
if base.endswith("/"):
base = base[:-1]
print("loading...")
vocab = load_vocab(base + "/vocab.tsv")
blocked = load_blocked(base + "/blocked.txt")
user_bias = load_user_bias(base + "/user_bias.tsv")
events = load_events(base + "/events.tsv")
print("events:", len(events))
print("vocab:", len(vocab))
# Warm-up, useful for apples-to-apples process behavior.
warm = prepare_events(events[: min(1000, len(events))], vocab, blocked, user_bias)
best = 1e100
final_output = None
for i in range(repeat):
t0 = time.time()
out = prepare_events(events, vocab, blocked, user_bias)
dt = time.time() - t0
if dt < best:
best = dt
final_output = out
print("run", i + 1, "seconds:", f"{dt:.6f}",
"events/sec:", f"{len(events) / dt:,.0f}",
"accepted:", len(out))
print("best seconds:", f"{best:.6f}")
print("best events/sec:", f"{len(events) / best:,.0f}")
print("checksum:", f"{checksum(final_output):.12f}")
if __name__ == "__main__":
main()
The code contains some boilerplate for loading the application data (events, vocabulary, etc.) followed by the central prepare_events() function that comprises the actual benchmark. (If you want to try running this code, you can find a script for generating test data in the appendix below.)
Here are the results from running this exact code through both Python and Codon on a sample dataset of 2,500,000 events:
Python (python3 benchmark.py benchdata 3)
loading...
events: 2500000
vocab: 20000
run 1 seconds: 20.374985 events/sec: 122,699 accepted: 2142873
run 2 seconds: 20.300605 events/sec: 123,149 accepted: 2142873
run 3 seconds: 19.933118 events/sec: 125,419 accepted: 2142873
best seconds: 19.933118
best events/sec: 125,419
checksum: 35271842.443105116487
Python version: 3.14.3, Apple M1 MacBook Pro
Codon (codon run -release benchmark.py benchdata 3)
loading...
events: 2500000
vocab: 20000
run 1 seconds: 5.483989 events/sec: 455,873 accepted: 2142873
run 2 seconds: 5.151519 events/sec: 485,294 accepted: 2142873
run 3 seconds: 5.234167 events/sec: 477,631 accepted: 2142873
best seconds: 5.151519
best events/sec: 485,294
checksum: 35271842.443105116487
Codon version 0.20.0 (acf2b53), Apple M1 MacBook Pro
That’s roughly a 4x improvement in throughput on the same code, just from Codon compilation and execution.
But native compilation enables other avenues to improve performance as well, such as multithreading. Whereas Python has historically had significant limitations on threading due to its infamous global interpreter lock, Codon supports multithreading natively. We can update prepare_events() to take advantage of this:
def prepare_events(events, vocab, blocked, user_bias):
from threading import Lock
output = []
lock = Lock() # lock to allow multiple threads to update `output`
@par(num_threads=4) # Codon-specific parallel-loop decorator
for user_id, event_type, weight, query in events:
if event_type != 2:
bias = user_bias.get(user_id, 0.0)
recognized = 0
score = 0.0
mix = 0
# Application-specific string processing + lookups + branching.
text = query.lower()
for raw_token in text.split(" "):
token = raw_token.strip(PUNCT)
token_id = vocab.get(token, -1)
if token_id >= 0 and token_id not in blocked:
recognized += 1
score += token_id * weight * 0.00001 + bias
if token_id % 7 == 0:
score += 0.125
elif token_id % 11 == 0:
score -= 0.075
mix = (mix + token_id * (recognized + 1)) % 1_000_003
if recognized >= 8 and score > 0.0:
with lock:
output.append((user_id, score, mix)) # make sure only 1 thread appends
return output
Running this multithreaded implementation shows a further improvement:
loading...
events: 2500000
vocab: 20000
run 1 seconds: 3.998556 events/sec: 625,226 accepted: 2142873
run 2 seconds: 3.845620 events/sec: 650,090 accepted: 2142873
run 3 seconds: 3.734922 events/sec: 669,358 accepted: 2142873
best seconds: 3.734922
best events/sec: 669,358
checksum: 35271842.443104118109
The result is well over a 5x improvement over the original Python version. Beyond 4 threads, lock contention starts to dominate and there isn’t a measurable performance improvement.
The more important result is how little code changed. The single-threaded benchmark uses exactly the same source code between Python and Codon runs, improving throughput from roughly 125,000 to 485,000 events per second. Adding a small amount of Codon-specific parallelization brings that to roughly 669,000 events per second.
Of course, this does not mean every preprocessing workload will see a 5x improvement. Code that already spends most of its time in native libraries such as NumPy, PyTorch, or optimized tokenizers will have less Python execution to accelerate. But for application-specific code dominated by Python loops, strings, lookups, and branching, compilation can provide another option before resorting to a lower-level rewrite.
Where this approach makes sense
Compilation is most interesting when the profiler shows substantial time being spent executing application logic in Python itself.
That includes workloads such as custom parsing and transformation, string-heavy processing, loops with branches, repeated dictionary or set lookups, feature extraction, ranking or filtering logic, simulations, and other code for which there isn’t already an optimized native operation that does the work.
For example:
def score_candidates(candidates, weights, blocked):
result = []
for candidate in candidates:
if candidate.id in blocked:
continue
score = 0.0
for feature in candidate.features:
weight = weights.get(feature.name)
if weight is not None:
score += weight * feature.value
if score > candidate.threshold:
result.append((candidate.id, score))
return result
This is interesting from a compilation perspective because the application logic itself constitutes the computation.
By contrast:
def transform(x):
return np.linalg.svd(x)
probably isn’t. The expensive work already executes inside native numerical libraries.
There are also workloads that simply do not map cleanly to Codon. Highly dynamic Python, extensive runtime introspection, arbitrary Python objects, or code deeply coupled to unsupported libraries can make compilation impractical. And sometimes a dedicated C++ or Rust implementation provides the best long-term architecture.
The point isn’t that lower-level languages are obsolete. It’s that rewriting into one should not necessarily be the default consequence of discovering that application-specific Python has become CPU-bound.
The broader implication for LLM-generated software
ML preprocessing is only one example of a more general pattern. Many applications necessitate logic that follows this general pattern:
for item in items:
if should_skip(item):
continue
result = lookup[item.key]
for value in item.values:
result += transform(value)
output.append(result)
The particular application might be an inference pipeline, a financial simulation, a bioinformatics analysis, a database transformation, a parser, or a backend service.
Python is attractive in all of these cases because both the developer and the model can work with it easily, creating a natural development loop:
describe the task
↓
generate Python
↓
run + test
↓
give results back to the model
↓
modify the implementation
↓
repeat
The question is how far that loop can extend. If production performance eventually requires translating the implementation into another language, we end up with two representations of the same idea:
┌─ Python ── development
LLM → algorithm ────┤
└─ C++ ───── production
For workloads that can be compiled directly, there is another possibility:
LLM → Python-shaped implementation → native machine code
↑
same codebase
This becomes even more interesting for new software: if we know from the beginning that a component will need native performance, an LLM does not necessarily have to generate unrestricted CPython first and retrofit the code later. It can generate Python-shaped code with Codon as the intended target, so that the workflow becomes
describe the task
↓
LLM generates Codon-compatible Python
↓
compile + test
↓
profile
↓
LLM modifies the same implementation
↓
deploy native code
That requires the model to understand Codon’s differences from Python, just as it needs to understand the constraints of any target environment. But those constraints are comparatively easy to express to a coding model: which Python features are supported, which types are available, which libraries can be used, and which patterns should be avoided.
This suggests a broader role for Codon in AI-assisted software development. Python can serve not only as the language in which an LLM expresses an initial solution, but (where the workload is suitable) as a language much closer to what ultimately executes in production.
Runtime isn’t the only benchmark
When evaluating a compiler, the obvious benchmark is runtime:
CPython X records/sec
Python + workers Y records/sec
Codon Z records/sec
C++ ...
Those measurements matter, but LLMs make another set of measurements increasingly relevant:
- Lines of application code changed
- Amount of duplicated logic
- Integration code added
- Build complexity introduced
- Time spent validating a second implementation
- Cost of changing the algorithm six months later
This is ultimately the more interesting comparison. We already know that carefully written native code can outperform interpreted Python. The practical question for a developer is what they have to do about it.
If the answer is eight Python workers, and that works well, use eight Python workers.
If the expensive computation is already inside a native library, leave it there.
If the component belongs in C++ or Rust, write it in C++ or Rust.
But if the application contains a substantial amount of CPU-bound Python logic, there is another experiment worth running:
Take the Python you already have and see how much of it can become the native implementation.
LLMs have substantially reduced the cost of getting from an idea to working Python. A compiler can potentially reduce the cost of the next transition: from working Python to production code that meets the system’s performance requirements.
That transition, rather than a standalone microbenchmark, is really what is worth measuring.
Benchmark a real workload with Exaloop
Codon is open source, so the simplest way to evaluate it is to try it on the code that is actually consuming CPU time.
For teams considering Codon for a production workload, Exaloop can also help with that evaluation: profile the existing application, identify an appropriate compilation boundary, adapt the relevant code where necessary, and compare the result against the alternatives the team is already considering.
Sometimes the conclusion will be that Codon isn’t the right tool for that workload. Sometimes only a small function needs to be compiled. In other systems, moving an entire processing component out of CPython makes more sense.
The useful result isn’t a predetermined benchmark number. It’s finding out whether the Python implementation you already have can become the production implementation rather than the prototype that eventually gets replaced.
Appendix
The script generate_data.py below can be used to generate synthetic data for the demo above. Example: python3 generate_data.py --events 250000 --out benchdata
#!/usr/bin/env python3
import argparse
import random
from pathlib import Path
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--out", default="benchdata")
ap.add_argument("--events", type=int, default=250_000)
ap.add_argument("--vocab", type=int, default=20_000)
ap.add_argument("--users", type=int, default=50_000)
ap.add_argument("--min-tokens", type=int, default=18)
ap.add_argument("--max-tokens", type=int, default=42)
ap.add_argument("--seed", type=int, default=7)
args = ap.parse_args()
rng = random.Random(args.seed)
out = Path(args.out)
out.mkdir(parents=True, exist_ok=True)
vocab = [f"tok{i:05d}" for i in range(args.vocab)]
blocked = set(rng.sample(range(args.vocab), max(1, args.vocab // 50)))
with (out / "vocab.tsv").open("w") as f:
for i, token in enumerate(vocab):
f.write(f"{token}\t{i}\n")
with (out / "blocked.txt").open("w") as f:
for i in sorted(blocked):
f.write(f"{i}\n")
with (out / "user_bias.tsv").open("w") as f:
for user_id in range(args.users):
# Stable small per-user feature.
bias = ((user_id * 2654435761) % 1000) / 1000.0
f.write(f"{user_id}\t{bias:.6f}\n")
event_types = [0, 0, 0, 0, 1, 1, 2] # type 2 is ignored.
punctuation = ["", "", "", "", ".", ",", "!", "?"]
with (out / "events.tsv").open("w") as f:
for _ in range(args.events):
user_id = rng.randrange(args.users)
event_type = rng.choice(event_types)
weight = 0.5 + rng.random() * 1.5
n = rng.randint(args.min_tokens, args.max_tokens)
words = []
for _ in range(n):
if rng.random() < 0.12:
# OOV token.
token = f"unknown{rng.randrange(5000):04d}"
else:
token = vocab[rng.randrange(args.vocab)]
# Some uppercase and punctuation to force basic text normalization.
if rng.random() < 0.10:
token = token.upper()
token += rng.choice(punctuation)
words.append(token)
query = " ".join(words)
f.write(f"{user_id}\t{event_type}\t{weight:.6f}\t{query}\n")
print(f"Wrote {args.events:,} events to {out}")
if __name__ == "__main__":
main()
