Bf/combine transformer embeddings

[helpmefindaname]

creates a transformer embedding that combines both TransformerWordEmbedding and TransformerDocumentEmbedding

it should be able to:

• still load all embeddings saved with previous versions
• should not need to load the tokenizer -> Cannot load NER model from local file #2445 should be fixed
• allow using document embeddings and token embeddings at once (might be usefull for Multitask Learning in flair #2508 ?)
• allow using long sequences / document context for DocumentClassification too (if the pooling is max or mean)
• make code better maintainable, as there is no duplicated code

current state:

• first implementation
• run test
• manually test training with all kinds of features
• test loading old embeddings / test backwardscompability

[alanakbik]

Thanks for adding this! Still testing, but found an error that appears with the following code:

embeddings = TransformerWordEmbeddings(model='xlm-roberta-base',
                                       layers="-1",
                                       subtoken_pooling="first",
                                       fine_tune=True,
                                       use_context=False,
                                       )
text = "."
sentence = Sentence(text)
embeddings.embed(sentence)

Suggestion to solve this (I think) added in-line.

[alanakbik]

Thanks a lot for refactoring this!

[alanakbik]

@helpmefindaname I found another error. It seems the fix for the previous error now broke sentences that are too long (over 512 subtokens).

Reproducible with this script:

from flair.data import Sentence
from flair.embeddings import TransformerWordEmbeddings

# example transformer embeddings
embeddings = TransformerWordEmbeddings(model='distilbert-base-uncased')

# create sentence with more than 512 subtokens
long_sentence = Sentence('a ' * 513)

# embed
embeddings.embed(long_sentence)

Throws the same assertion error as previously, i.e.:

  File ".../flair/flair/embeddings/base.py", line 769, in _add_embeddings_internal
    self._add_embeddings_to_sentences(expanded_sentences)
  File ".../flair/flair/embeddings/base.py", line 728, in _add_embeddings_to_sentences
    self._extract_token_embeddings(sentence_hidden_states, sentences, all_token_subtoken_lengths)
  File ".../flair/flair/embeddings/base.py", line 656, in _extract_token_embeddings
    assert subword_start_idx < subword_end_idx <= sentence_hidden_state.size()[1]
AssertionError

Any ideas how to fix this?

[isaac47]

Hi,
this merged request is done? How can use it?