Skip to main content
SearchLoginLogin or Signup

Position Encoding Schemes for Linear Aggregation of Word Sequences

Published onJun 08, 2021
Position Encoding Schemes for Linear Aggregation of Word Sequences
·

Abstract

Deep Averaging Networks (DANs) show strong performance in several key Natural Language Processing (NLP) tasks. However, their chief drawback is not accounting for the position of tokens when encoding sequences. We study how existing position encodings might be integrated into the DAN architecture. In addition, we propose a novel position encoding built specifically for DANs, which allows greater generalization capabilities to unseen lengths of sequences. This is demonstrated on decision tasks on binary sequences. Further, the resulting architecture is compared against unordered aggregation on sentiment analysis both with word- and character-level tokenization, to mixed results.

Article ID: 2021L28

Month: May

Year: 2021

Address: Online

Venue: Canadian Conference on Artificial Intelligence

Publisher: Canadian Artificial Intelligence Association

URL: https://caiac.pubpub.org/pub/wvnuon1p/

Comments
0
comment
No comments here
Why not start the discussion?