brainsteam.co.uk/brainsteam/content/annotations/2023/03/21/1679380079.md

date	hypothesis-meta	in-reply-to	tags	type	url
2023-03-21T06:27:59	created document flagged group hidden id links permissions tags target text updated uri user user_info 2023-03-21T06:27:59.825632+00:00 title GPT-4 and professional benchmarks: the wrong answer to the wrong question false __world__ false hoqyasexEe2ZnQ_nOVgRxA html incontext json https://hypothes.is/a/hoqyasexEe2ZnQ_nOVgRxA https://hyp.is/hoqyasexEe2ZnQ_nOVgRxA/aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks https://hypothes.is/api/annotations/hoqyasexEe2ZnQ_nOVgRxA admin delete read update acct:ravenscroftj@hypothes.is acct:ravenscroftj@hypothes.is group:__world__ acct:ravenscroftj@hypothes.is openai gpt ModelEvaluation selector source endContainer endOffset startContainer startOffset type /div[1]/div[1]/div[2]/div[1]/div[1]/div[1]/article[1]/div[4]/div[1]/div[1]/p[6]/span[2] 42 /div[1]/div[1]/div[2]/div[1]/div[1]/div[1]/article[1]/div[4]/div[1]/div[1]/p[6]/span[1] 0 RangeSelector end start type 6591 6238 TextPositionSelector exact prefix suffix type In fact, we can definitively show that it has memorized problems in its training set: when prompted with the title of a Codeforces problem, GPT-4 includes a link to the exact contest where the problem appears (and the round number is almost correct: it is off by one). Note that GPT-4 cannot access the Internet, so memorization is the only explanation. the problems after September 12. GPT-4 memorizes Codeforces probl TextQuoteSelector https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks GPT4 knows the link to the coding exams that it was evaluated against but doesn't have "internet access" so it appears to have memorised this as well 2023-03-21T06:27:59.825632+00:00 https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks acct:ravenscroftj@hypothes.is display_name James Ravenscroft	https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks	openai gpt ModelEvaluation hypothesis	annotation	/annotations/2023/03/21/1679380079

In fact, we can definitively show that it has memorized problems in its training set: when prompted with the title of a Codeforces problem, GPT-4 includes a link to the exact contest where the problem appears (and the round number is almost correct: it is off by one). Note that GPT-4 cannot access the Internet, so memorization is the only explanation.

GPT4 knows the link to the coding exams that it was evaluated against but doesn't have "internet access" so it appears to have memorised this as well