Получите длину каждого предложения перед заполнением в bucketiterator torchtext

Question

Можно ли получить длину каждого предложения перед заполнением в bucketiterator torchtext:

      train_loader = torchtext.legacy.data.BucketIterator(train_data, batch_size = 64, repeat=True, shuffle=True, sort_key = lambda x: len(x.text), sort=False, sort_within_batch=True, device = device)

bucketiterator загрузчик данных:

      inputs: tensor([[  34,   87,    2,  ...,  227,  239,  263],
        [ 138,    7, 1006,  ...,  840,  142,  665],
        [ 549,    4, 1028,  ...,   11,   14,    4],
        ...,
        [   1,    1,    5,  ...,   66,   23,   13],
        [   1,    1, 1062,  ...,  177,  252, 1587],
        [   1,    1,   66,  ...,  553,   52,   73]]), shape: torch.Size([64, 91])

Как при использовании загрузчика данных pytorch:

      train_loader = data.DataLoader(train_data, batch_size = 64, shuffle=True, collate_fn=padding)

def padding(batch):
 doc = [doc['input'] for doc in batch]
 len_doc = [len(doc['input']) for doc in batch]
 doc_pad = pad_sequence(doc, batch_first=True, padding_value=0)
 return doc_pad, len_doc

загрузчик данных pytorch:

      inputs: tensor([[   2, 1396, 2686,  ...,    0,    0,    0],
        [   2, 1391, 1396,  ...,    0,    0,    0],
        [   2, 2018, 2597,  ...,    0,    0,    0],
        ...,
        [   2, 1546, 1623,  ...,    0,    0,    0],
        [   2, 1435, 1396,  ...,    0,    0,    0],
        [   2, 1391, 1396,  ...,    0,    0,    0]]), shape: torch.Size([64, 40])

inputs_len_before_padding: tensor([18,  8, 21, 16, 16, 12, 40, 12,  9, 12, 17, 12, 17, 15, 16, 12,  8, 24,
        25, 10, 22,  8,  8, 13, 12, 22, 17, 14, 21, 14, 19, 13, 21,  8, 28, 16,
        31, 24, 23, 19, 10,  7, 16, 12, 16, 12, 17, 12, 18, 11,  8, 13, 17, 14,
        11, 13, 13, 20,  8, 12, 22,  7,  9, 11]), shape: torch.Size([64])

Answer 1

Вот минимальный пример, который используетtorchtext.data.Fieldиtorchtext.data.BucketIterator:

      import torchtext.data as data

# sample data
text = [
    'This is sentence 1.',
    'This sentence is a bit longer than the previous sentence.'
]

# define field -- notice include_lengths is set to True
text_field = data.Field(include_lengths=True, tokenize=lambda x: x.split())
fields = [('text', text_field)]

# create dataset and build vocabulary
examples = [data.Example.fromlist([t], fields) for t in text]
dataset = data.Dataset(examples, fields)
text_field.build_vocab(dataset)

# create iterator
data_iter = data.BucketIterator(dataset, batch_size=2, shuffle=False)

# the text field will now return both the data tensor and the length of the input text
for x in data_iter:
    print('Data:', x.text[0])
    print('Lengths:', x.text[1])

Это должно печатать (тензор данных сокращен для краткости):

      Data: tensor([[ 2,  2],
        ...
        [ 1, 10]])
Lengths: tensor([ 4, 10])