MultiheadCrossAttention#

class deeplay.ops.attention.cross.MultiheadCrossAttention(*args, **kwargs)#

Methods Summary

forward(*x[, batch_index])

Apply multihead self-attention to the input tensor.

Methods Documentation

forward(*x, batch_index=None)#: Apply multihead self-attention to the input tensor. Returns (y, attn) if return_attn is True, otherwise returns y. y is the output of the multihead self-attention layer, attn is the attention matrix, and x is the input to the multihead self-attention. If projection is nn.Identity, then x is the same as the input to the multihead self-attention. Otherwise, x is the output of the projection layer.