torch.nn

Intro

这篇文章是想记录一下 Pytorch 中 torch.nn 的一些自己会用到的用法，避免反复看 docs，:( 泪目。

pytorch scatter

https://pytorch-scatter.readthedocs.io/en/latest/functions/scatter.html#torch_scatter.scatter

torch.stack

比如我有一个 list[Tensor] len()=10 Tensor.shape=[768,] torch.stack(list, dim=0) -> Tensor shape=[10,768]

nn.Linear

nn.scatter

指定 index dim 留下需要的部分 求和或者取平均

nn.Embedding

可以把 nn.Embedding 作为一个查找表. 把 indices 嵌入到 word-embedding。

• 初始化时需要知道总共可能有多少 indices，需要嵌入到几维空间, 如 nn.Embedding(10, 3)代表最多有10个不同的indices，每个indices会被嵌入成3维度的Tensor。

import torch.nn as nn 

# vocab_size is the number of words in your train, val and test set
# vector_size is the dimension of the word vectors you are using
embed = nn.Embedding(vocab_size, vector_size)

# intialize the word vectors, pretrained_weights is a 
# numpy array of size (vocab_size, vector_size) and 
# pretrained_weights[i] retrieves the word vector of
# i-th word in the vocabulary
embed.weight.data.copy_(torch.fromnumpy(pretrained_weights))

# Then turn the word index into actual word vector
vocab = {"some": 0, "words": 1}
word_indexes = [vocab[w] for w in ["some", "words"]] 
word_vectors = embed(word_indexes)

ref from https://stackoverflow.com/posts/51060374/timeline