Scaled Dot-Product Attention

Q, K, V flow through the attention computation.

Template previewMermaid

Rendering…

Make it your own.

flowchart LR
Q[Q\nL × d_k] --> M1[matmul]
K[K\nL × d_k] --> M1
M1 --> S["÷ sqrt(d_k)"]
S --> Mask["+ mask<br/>(optional)"]
Mask --> SM[softmax]
SM --> M2[matmul]
V[V\nL × d_v] --> M2
M2 --> Out[Output\nL × d_v]