66 lines
2.7 KiB
Markdown
66 lines
2.7 KiB
Markdown
|
---
|
||
|
|
date: '2023-03-21T19:59:04'
|
||
|
|
hypothesis-meta:
|
||
|
|
created: '2023-03-21T19:59:04.177001+00:00'
|
||
|
|
document:
|
||
|
|
title:
|
||
|
|
- 2303.09752.pdf
|
||
|
|
flagged: false
|
||
|
|
group: __world__
|
||
|
|
hidden: false
|
||
|
|
id: 1MB9BMgiEe27GS99BvTIlA
|
||
|
|
links:
|
||
|
|
html: https://hypothes.is/a/1MB9BMgiEe27GS99BvTIlA
|
||
|
|
incontext: https://hyp.is/1MB9BMgiEe27GS99BvTIlA/arxiv.org/pdf/2303.09752.pdf
|
||
|
|
json: https://hypothes.is/api/annotations/1MB9BMgiEe27GS99BvTIlA
|
||
|
|
permissions:
|
||
|
|
admin:
|
||
|
|
- acct:ravenscroftj@hypothes.is
|
||
|
|
delete:
|
||
|
|
- acct:ravenscroftj@hypothes.is
|
||
|
|
read:
|
||
|
|
- group:__world__
|
||
|
|
update:
|
||
|
|
- acct:ravenscroftj@hypothes.is
|
||
|
|
tags:
|
||
|
|
- llm
|
||
|
|
- attention
|
||
|
|
- long-documents
|
||
|
|
target:
|
||
|
|
- selector:
|
||
|
|
- end: 1989
|
||
|
|
start: 1515
|
||
|
|
type: TextPositionSelector
|
||
|
|
- exact: "Over the past few years, many \u201Cefficient Trans-former\u201D approaches\
|
||
|
|
\ have been proposed that re-duce the cost of the attention mechanism over\
|
||
|
|
\ longinputs (Child et al., 2019; Ainslie et al., 2020; Belt-agy et al., 2020;\
|
||
|
|
\ Zaheer et al., 2020; Wang et al.,2020; Tay et al., 2021; Guo et al., 2022).\
|
||
|
|
\ However,especially for larger models, the feedforward andprojection layers\
|
||
|
|
\ actually make up the majority ofthe computational burden and can render\
|
||
|
|
\ process-ing long inputs intractable"
|
||
|
|
prefix: ' be applied to each input token.'
|
||
|
|
suffix: ".\u2217Author contributions are outli"
|
||
|
|
type: TextQuoteSelector
|
||
|
|
source: https://arxiv.org/pdf/2303.09752.pdf
|
||
|
|
text: Recent improvements in transformers for long documents have focused on efficiencies
|
||
|
|
in the attention mechanism but the feed-forward and projection layers are still
|
||
|
|
expensive for long docs
|
||
|
|
updated: '2023-03-21T19:59:04.177001+00:00'
|
||
|
|
uri: https://arxiv.org/pdf/2303.09752.pdf
|
||
|
|
user: acct:ravenscroftj@hypothes.is
|
||
|
|
user_info:
|
||
|
|
display_name: James Ravenscroft
|
||
|
|
in-reply-to: https://arxiv.org/pdf/2303.09752.pdf
|
||
|
|
tags:
|
||
|
|
- llm
|
||
|
|
- attention
|
||
|
|
- long-documents
|
||
|
|
- hypothesis
|
||
|
|
type: annotation
|
||
|
|
url: /annotations/2023/03/21/1679428744
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
<blockquote>Over the past few years, many “efficient Trans-former” approaches have been proposed that re-duce the cost of the attention mechanism over longinputs (Child et al., 2019; Ainslie et al., 2020; Belt-agy et al., 2020; Zaheer et al., 2020; Wang et al.,2020; Tay et al., 2021; Guo et al., 2022). However,especially for larger models, the feedforward andprojection layers actually make up the majority ofthe computational burden and can render process-ing long inputs intractable</blockquote>Recent improvements in transformers for long documents have focused on efficiencies in the attention mechanism but the feed-forward and projection layers are still expensive for long docs
|