brainsteam.co.uk/brainsteam/content/annotations/2022/11/23/1669233264.md

72 lines
3.4 KiB
Markdown
Raw Normal View History

---
date: '2022-11-23T19:54:24'
hypothesis-meta:
created: '2022-11-23T19:54:24.332809+00:00'
document:
title:
- 2210.07188.pdf
flagged: false
group: __world__
hidden: false
id: oTGKsmtoEe2RF0-NK45jew
links:
html: https://hypothes.is/a/oTGKsmtoEe2RF0-NK45jew
incontext: https://hyp.is/oTGKsmtoEe2RF0-NK45jew/arxiv.org/pdf/2210.07188.pdf
json: https://hypothes.is/api/annotations/oTGKsmtoEe2RF0-NK45jew
permissions:
admin:
- acct:ravenscroftj@hypothes.is
delete:
- acct:ravenscroftj@hypothes.is
read:
- group:__world__
update:
- acct:ravenscroftj@hypothes.is
tags:
- coreference
- NLProc
- data-annotation
target:
- selector:
- end: 5934
start: 5221
type: TextPositionSelector
- exact: Specifically, our work investigates the quality ofcrowdsourced coreference
annotations when anno-tators are taught only simple coreference cases thatare
treated uniformly across existing datasets (e.g.,pronouns). By providing only
these simple cases,we are able to teach the annotators the concept ofcoreference,
while allowing them to freely interpretcases treated differently across the
existing datasets.This setup allows us to identify cases where ourannotators
disagree among each other, but moreimportantly cases where they unanimously
agreewith each other but disagree with the expert, thussuggesting cases that
should be revisited by theresearch community when curating future unifiedannotation
guidelines
prefix: ficient payment-based platforms.
suffix: .Our main contributions are:1. W
type: TextQuoteSelector
source: https://arxiv.org/pdf/2210.07188.pdf
text: "The aim of the work is to examine a simplified subset of co-reference phenomena\
\ which are generally treated the same across different existing datasets. \n\n\
This makes spotting inter-annotator disagreement easier - presumably because for\
\ simpler cases there are fewer modes of failure?\n\n"
updated: '2022-11-23T19:54:24.332809+00:00'
uri: https://arxiv.org/pdf/2210.07188.pdf
user: acct:ravenscroftj@hypothes.is
user_info:
display_name: James Ravenscroft
in-reply-to: https://arxiv.org/pdf/2210.07188.pdf
tags:
- coreference
- NLProc
- data-annotation
- hypothesis
type: annotation
url: /annotation/2022/11/23/1669233264
---
<blockquote>Specifically, our work investigates the quality ofcrowdsourced coreference annotations when anno-tators are taught only simple coreference cases thatare treated uniformly across existing datasets (e.g.,pronouns). By providing only these simple cases,we are able to teach the annotators the concept ofcoreference, while allowing them to freely interpretcases treated differently across the existing datasets.This setup allows us to identify cases where ourannotators disagree among each other, but moreimportantly cases where they unanimously agreewith each other but disagree with the expert, thussuggesting cases that should be revisited by theresearch community when curating future unifiedannotation guidelines</blockquote>The aim of the work is to examine a simplified subset of co-reference phenomena which are generally treated the same across different existing datasets.
This makes spotting inter-annotator disagreement easier - presumably because for simpler cases there are fewer modes of failure?