Write a Python function to implement the Positional Encoding layer for Transformers.
The function should calculate positional encodings for a sequence length (position) and model dimensionality (d_model) using sine and cosine functions as specified in the Transformer architecture.
The function should return -1 if position is 0, or if d_model is less than or equal to 0. The output should be a numpy array of type float16.
Examples
Example 1:
Input:
position = 2, d_model = 8Output:
[[[ 0.,0.,0.,0.,1.,1.,1.,1.,]
[ 0.8413,0.0998,0.01,0.001,0.5405,0.995,1.,1.]]]Explanation: The function computes the positional encoding by calculating sine values for even indices and cosine values for odd indices, ensuring that the encoding provides the required positional information.
Starter Code
import numpy as np
def pos_encoding(position: int, d_model: int):
# Your code here
pos_encoding = np.float16(pos_encoding)
return pos_encodingPython3
ReadyLines: 1Characters: 0
Ready