google

/

embeddinggemma-300m

like 1.09k

Follow

Google 33.8k

Maybe this is known, but I ran into this today where produces different embeddings for sequences with length exactly 129 tokens (and similarly 193, 257; if you notice they all are (2**i)+1) when batched with shorter sequences. The embeddings differ by ~1% from the ground truth (individual encoding), despite proper attention masking.

It might be due to RoPE / Sliding Attention but I'm curious about it since it's always at 2**i+1 boundary. This doesn't happen with sentence-transformers/all-MiniLM-L6-v2 which might be due to architectural reasons (but it'll be nice to understand which part of architecture causes this).

Detailed Test Results

Quick Test

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("google/embeddinggemma-300m").to("cuda")

# Create texts with exactly 113 and 129 tokens
TEXT_113 = ("Hello " * 111).strip()  # 113 tokens
TEXT_129 = ("Hello " * 127).strip()  # 129 tokens

# Get baseline: encode TEXT_129 alone
baseline = model.encode([TEXT_129], batch_size=1)

# Batch with shorter text
batched = model.encode([TEXT_113, TEXT_129], batch_size=2)

# Compare
diff = np.abs(baseline - batched[1:2]).max()
print(f"Difference: {diff:.8f}")  # Shows ~0.01 (1% error!)

Here is a github gist too https://gist.github.com/praateekmahajan/17abaf7bfe435cd6cbb98ac6d0650377