brainsteam.co.uk/brainsteam/content/annotations/2023/03/21/1679428744.md

66 lines
2.7 KiB
Markdown
Raw Normal View History

---
date: '2023-03-21T19:59:04'
hypothesis-meta:
created: '2023-03-21T19:59:04.177001+00:00'
document:
title:
- 2303.09752.pdf
flagged: false
group: __world__
hidden: false
id: 1MB9BMgiEe27GS99BvTIlA
links:
html: https://hypothes.is/a/1MB9BMgiEe27GS99BvTIlA
incontext: https://hyp.is/1MB9BMgiEe27GS99BvTIlA/arxiv.org/pdf/2303.09752.pdf
json: https://hypothes.is/api/annotations/1MB9BMgiEe27GS99BvTIlA
permissions:
admin:
- acct:ravenscroftj@hypothes.is
delete:
- acct:ravenscroftj@hypothes.is
read:
- group:__world__
update:
- acct:ravenscroftj@hypothes.is
tags:
- llm
- attention
- long-documents
target:
- selector:
- end: 1989
start: 1515
type: TextPositionSelector
- exact: "Over the past few years, many \u201Cefficient Trans-former\u201D approaches\
\ have been proposed that re-duce the cost of the attention mechanism over\
\ longinputs (Child et al., 2019; Ainslie et al., 2020; Belt-agy et al., 2020;\
\ Zaheer et al., 2020; Wang et al.,2020; Tay et al., 2021; Guo et al., 2022).\
\ However,especially for larger models, the feedforward andprojection layers\
\ actually make up the majority ofthe computational burden and can render\
\ process-ing long inputs intractable"
prefix: ' be applied to each input token.'
suffix: ".\u2217Author contributions are outli"
type: TextQuoteSelector
source: https://arxiv.org/pdf/2303.09752.pdf
text: Recent improvements in transformers for long documents have focused on efficiencies
in the attention mechanism but the feed-forward and projection layers are still
expensive for long docs
updated: '2023-03-21T19:59:04.177001+00:00'
uri: https://arxiv.org/pdf/2303.09752.pdf
user: acct:ravenscroftj@hypothes.is
user_info:
display_name: James Ravenscroft
in-reply-to: https://arxiv.org/pdf/2303.09752.pdf
tags:
- llm
- attention
- long-documents
- hypothesis
type: annotation
url: /annotations/2023/03/21/1679428744
---
<blockquote>Over the past few years, many “efficient Trans-former” approaches have been proposed that re-duce the cost of the attention mechanism over longinputs (Child et al., 2019; Ainslie et al., 2020; Belt-agy et al., 2020; Zaheer et al., 2020; Wang et al.,2020; Tay et al., 2021; Guo et al., 2022). However,especially for larger models, the feedforward andprojection layers actually make up the majority ofthe computational burden and can render process-ing long inputs intractable</blockquote>Recent improvements in transformers for long documents have focused on efficiencies in the attention mechanism but the feed-forward and projection layers are still expensive for long docs