Live · scaled dot-productDrag the query

The word “it” carries a query. Drag it toward a word: the more they align, the more that word wins the ◇ blend (softmax of the dot-products).

This is what ÷ √dₖ controls. Big dot-products push softmax to the focused end and gradients die; dividing by √dₖ pulls it back toward the stable, blended end.

Live · self-attention · illustrativeClick a word

Click any word to make it the query; the rest light up by how much it attends to them. Two heads, two different jobs.