68 lines
2.8 KiB
Markdown
68 lines
2.8 KiB
Markdown
|
---
|
||
|
|
date: '2023-03-21T06:27:59'
|
||
|
|
hypothesis-meta:
|
||
|
|
created: '2023-03-21T06:27:59.825632+00:00'
|
||
|
|
document:
|
||
|
|
title:
|
||
|
|
- 'GPT-4 and professional benchmarks: the wrong answer to the wrong question'
|
||
|
|
flagged: false
|
||
|
|
group: __world__
|
||
|
|
hidden: false
|
||
|
|
id: hoqyasexEe2ZnQ_nOVgRxA
|
||
|
|
links:
|
||
|
|
html: https://hypothes.is/a/hoqyasexEe2ZnQ_nOVgRxA
|
||
|
|
incontext: https://hyp.is/hoqyasexEe2ZnQ_nOVgRxA/aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks
|
||
|
|
json: https://hypothes.is/api/annotations/hoqyasexEe2ZnQ_nOVgRxA
|
||
|
|
permissions:
|
||
|
|
admin:
|
||
|
|
- acct:ravenscroftj@hypothes.is
|
||
|
|
delete:
|
||
|
|
- acct:ravenscroftj@hypothes.is
|
||
|
|
read:
|
||
|
|
- group:__world__
|
||
|
|
update:
|
||
|
|
- acct:ravenscroftj@hypothes.is
|
||
|
|
tags:
|
||
|
|
- openai
|
||
|
|
- gpt
|
||
|
|
- ModelEvaluation
|
||
|
|
target:
|
||
|
|
- selector:
|
||
|
|
- endContainer: /div[1]/div[1]/div[2]/div[1]/div[1]/div[1]/article[1]/div[4]/div[1]/div[1]/p[6]/span[2]
|
||
|
|
endOffset: 42
|
||
|
|
startContainer: /div[1]/div[1]/div[2]/div[1]/div[1]/div[1]/article[1]/div[4]/div[1]/div[1]/p[6]/span[1]
|
||
|
|
startOffset: 0
|
||
|
|
type: RangeSelector
|
||
|
|
- end: 6591
|
||
|
|
start: 6238
|
||
|
|
type: TextPositionSelector
|
||
|
|
- exact: 'In fact, we can definitively show that it has memorized problems in
|
||
|
|
its training set: when prompted with the title of a Codeforces problem, GPT-4
|
||
|
|
includes a link to the exact contest where the problem appears (and the round
|
||
|
|
number is almost correct: it is off by one). Note that GPT-4 cannot access
|
||
|
|
the Internet, so memorization is the only explanation.'
|
||
|
|
prefix: the problems after September 12.
|
||
|
|
suffix: GPT-4 memorizes Codeforces probl
|
||
|
|
type: TextQuoteSelector
|
||
|
|
source: https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks
|
||
|
|
text: GPT4 knows the link to the coding exams that it was evaluated against but
|
||
|
|
doesn't have "internet access" so it appears to have memorised this as well
|
||
|
|
updated: '2023-03-21T06:27:59.825632+00:00'
|
||
|
|
uri: https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks
|
||
|
|
user: acct:ravenscroftj@hypothes.is
|
||
|
|
user_info:
|
||
|
|
display_name: James Ravenscroft
|
||
|
|
in-reply-to: https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks
|
||
|
|
tags:
|
||
|
|
- openai
|
||
|
|
- gpt
|
||
|
|
- ModelEvaluation
|
||
|
|
- hypothesis
|
||
|
|
type: annotation
|
||
|
|
url: /annotations/2023/03/21/1679380079
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
<blockquote>In fact, we can definitively show that it has memorized problems in its training set: when prompted with the title of a Codeforces problem, GPT-4 includes a link to the exact contest where the problem appears (and the round number is almost correct: it is off by one). Note that GPT-4 cannot access the Internet, so memorization is the only explanation.</blockquote>GPT4 knows the link to the coding exams that it was evaluated against but doesn't have "internet access" so it appears to have memorised this as well
|