Gemma 4 cheatsheet

#Hybrid Attention

• Local layers: sliding window (512 or 1024 tokens)
• Global layers: full context, interleaved with local
• Final layer is always a global attention layer

#PLE — Edge Models (E2B/E4B)

• Per-Layer Embeddings: each decoder layer has its own small embedding table
• Large static tables use fast lookups — not dense matrix multiply
• Effective compute parameters ≪ total loaded parameters
• Enables inference under 1.5 GB VRAM (with 4-bit quantization)

#p-RoPE & Shared KV Cache

• Proportional RoPE (p-RoPE) on global layers for long-range coherence
• Shared KV Cache across global layers reduces peak memory usage
• Supports stable 256K context without performance degradation

#MoE — 26B A4B

• 128 total experts + 1 always-active shared expert
• 8 experts activated per token at inference time
• Speed similar to a 4B dense model, quality near 30B
• Vision encoder: ~550M params (same as 31B)

Add <|think|> to the start of the system prompt:

messages = [
    {
        "role": "system",
        "content": "<|think|>You are a math expert."
    },
    {"role": "user", "content": "Solve: 3x + 7 = 22"}
]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True
)
outputs = model.generate(**inputs, max_new_tokens=2048)
response = processor.decode(
    outputs[0][input_len:], skip_special_tokens=False
)
result = processor.parse_response(response)
# result["thinking"]  → step-by-step reasoning
# result["response"] → final answer

#Thinking output structure

<|channel>thought
[Internal step-by-step reasoning — hidden from user]
<channel|>
[Final answer visible to user]

#Disabled thinking behavior

For 31B/26B A4B with enable_thinking=False, empty tags still emitted:

<|channel>thought
<channel|>
[Final answer]

E2B/E4B variants skip empty tags entirely when disabled.

Variable aspect ratio + configurable visual token budget:

# Image MUST come before text (required)
messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text",  "text": "Describe this chart."},
]}]
inputs = processor(
    text=text,
    images=image,
    return_tensors="pt"
).to(model.device)

Vision encoder: ~150M params (E2B/E4B) · ~550M params (31B/26B)

$ pip install -U transformers accelerate

#BF16 (default)

from transformers import (
    AutoProcessor, AutoModelForCausalLM
)
import torch

mid = "google/gemma-4-31b-it"
processor = AutoProcessor.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(
    mid,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

#4-bit quantized

from transformers import BitsAndBytesConfig

bnb = BitsAndBytesConfig(load_in_4bit=True)
model = AutoModelForCausalLM.from_pretrained(
    mid,
    quantization_config=bnb,
    device_map="auto"
)

Single 16 GB GPU (T4/free Colab or Kaggle):

from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    "google/gemma-4-E4B-it",
    load_in_4bit=True,
    max_seq_length=4096
)

model = FastModel.get_peft_model(
    model,
    r=16,
    lora_alpha=16,
    lora_dropout=0,
    target_modules=[
        "q_proj", "k_proj",
        "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
)

#Vision fine-tuning (E2B / E4B)

from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    "google/gemma-4-E4B-it",
    finetune_vision_layers=False,   # freeze encoder
    finetune_language_layers=True,
    load_in_4bit=True,
)