What does a code embedding model measure?

It's my personal mission to make my local coding model better for me than anything commercially available. The big models are always going to be more capable and my consumer grade hardware will never be a match for a data center of compute. Still, the big models have to work for everyone, my local model only has to suit my workflow.

The biggest improvement I've seen with smaller models comes from the initial steering. If I can tell it which files, modules, classes or concepts a change will impact it does a better job.

An obvious choice is to reach for an embedding model to navigate the codebase.

More of these models are claiming to work with source code, from Qwen3 Embedding's release page:

Multilingual Capability: The Qwen3 Embedding series support over 100 languages, including various programming languages, and provides robust multilingual, cross-lingual, and code retrieval capabilities.

But what does that actually mean? Does the model understand structure? Naming? What?

I ran a small experiment, and here's what I found.

It's just names. They treat code like a big bag of tokens, structure makes no to minimal difference depending on model size.

Worse, comments and docstrings dominate, so a bad comment that doesn't reflect the implementation can be a better match than the correct implementation.

That doesn't mean these models are useless, I just need to do some other work first. But that's for another post, here's my experiment.

My experiment returned similar results across increasing model sizes:

• Qwen3 Embedding 0.6B
• Jina Code Embeddings 1.5B
• Nomic Embed Code 7B
• Qwen3 Embedding 8B

I took three reference Python functions, quicksort, a HTTP request handler and a larger (~30 line) pipeline function. I transformed each function in 5 ways:

• rename: kept the structure the same, but converted all identifiers to single letters
• rename_domain: kept the structure the same, but renamed all identifiers to different, misleading domain
• reorder: same tokens but scrambled the order of the lines (breaking the code, but keeping the same tokens)
• different_logic: same identifiers, but different implementation logic
• cross_language: same algorithm, ported to Javascript

I've put the quicksort sample below and the full experiment is on Github.

I had the embedding model produce vectors for all implementations, then calculated the cosine similarity between the reference and each variation.

The strongest matches (highest score) are where I kept identifiers the same, even when I scrambled the code structure.

--- Qwen3-Embedding-0.6B-Q8_0 ---

quicksort/reference.py:
  0.852 vs quicksort/cross_language.js
  0.929 vs quicksort/different_logic.py
  0.684 vs quicksort/rename.py
  0.298 vs quicksort/rename_domain.py
  0.961 vs quicksort/reorder.py

When I keep the implementation the same but change identifiers, the score drops. The worst score is for the same implementation with names from a different domain.

These scores are from Qwen Embedding 0.6B, but all models I tested produced the same pattern.

This gets worse when all implementations have the same comment describing the function. I added the same docstring to each variation:

def quicksort(items):
    """Recursive quicksort implementation; items must be a list of
    objects that support the <, == and > operators."""
    ...

The model weights the comment text much higher, with little difference between variations.

commented/reference.py:
  0.978 vs commented/different_logic.py
  0.950 vs commented/rename.py
  0.734 vs commented/rename_domain.py
  0.996 vs commented/reorder.py

If your comments are 100% accurate, this might be useful but that's not a bet I'd take.

Here's the quicksort sample:

def quicksort(items):
    if len(items) <= 1:
        return items
    pivot = items[len(items) // 2]
    left = [x for x in items if x < pivot]
    middle = [x for x in items if x == pivot]
    right = [x for x in items if x > pivot]
    return quicksort(left) + middle + quicksort(right)

def f(a):
    if len(a) <= 1:
        return a
    p = a[len(a) // 2]
    b = [c for c in a if c < p]
    d = [c for c in a if c == p]
    e = [c for c in a if c > p]
    return f(b) + d + f(e)

def parse_config(settings):
    if len(settings) <= 1:
        return settings
    default = settings[len(settings) // 2]
    overrides = [opt for opt in settings if opt < default]
    matches = [opt for opt in settings if opt == default]
    extras = [opt for opt in settings if opt > default]
    return parse_config(overrides) + matches + parse_config(extras)

def quicksort(items):
    right = [x for x in items if x > pivot]
    return quicksort(left) + middle + quicksort(right)
    pivot = items[len(items) // 2]
    if len(items) <= 1:
        return items
    left = [x for x in items if x < pivot]
    middle = [x for x in items if x == pivot]

Finally, a use for bubblesort.

def quicksort(items):
    for i in range(len(items)):
        for j in range(len(items) - i - 1):
            if items[j] > items[j + 1]:
                items[j], items[j + 1] = items[j + 1], items[j]
    return items

function quicksort(items) {
  if (items.length <= 1) {
    return items;
  }
  const pivot = items[Math.floor(items.length / 2)];
  const left = items.filter((x) => x < pivot);
  const middle = items.filter((x) => x === pivot);
  const right = items.filter((x) => x > pivot);
  return [...quicksort(left), ...middle, ...quicksort(right)];
}