---
date: '2022-11-23T20:01:42'
hypothesis-meta:
  created: '2022-11-23T20:01:42.722732+00:00'
  document:
    title:
    - 2210.07188.pdf
  flagged: false
  group: __world__
  hidden: false
  id: pn7SimtpEe2Rlr_TC_SBhg
  links:
    html: https://hypothes.is/a/pn7SimtpEe2Rlr_TC_SBhg
    incontext: https://hyp.is/pn7SimtpEe2Rlr_TC_SBhg/arxiv.org/pdf/2210.07188.pdf
    json: https://hypothes.is/api/annotations/pn7SimtpEe2Rlr_TC_SBhg
  permissions:
    admin:
    - acct:ravenscroftj@hypothes.is
    delete:
    - acct:ravenscroftj@hypothes.is
    read:
    - group:__world__
    update:
    - acct:ravenscroftj@hypothes.is
  tags:
  - coreference
  - data-annotation
  - NLProc
  target:
  - selector:
    - end: 20125
      start: 19879
      type: TextPositionSelector
    - exact: 'Procedure: We first launch an annotation tutorial(paid $4.50) and recruit
        the annotators on the AMTplatform.9 At the end of the tutorial, each annotatoris
        asked to annotate a short passage (around 150words). Only annotators with
        a B3 score (Bagga'
      prefix: asure inter-annotator agreement.
      suffix: 8The PreCo dataset is interestin
      type: TextQuoteSelector
    source: https://arxiv.org/pdf/2210.07188.pdf
  text: Annotators are asked to complete a quality control exercise and only annotators
    who achieve a B3 score of 0.9 or higher are invited to do more annotation
  updated: '2022-11-23T20:01:42.722732+00:00'
  uri: https://arxiv.org/pdf/2210.07188.pdf
  user: acct:ravenscroftj@hypothes.is
  user_info:
    display_name: James Ravenscroft
in-reply-to: https://arxiv.org/pdf/2210.07188.pdf
tags:
- coreference
- data-annotation
- NLProc
- hypothesis
type: reply
url: /replies/2022/11/23/1669233702

---



 <blockquote>Procedure: We first launch an annotation tutorial(paid $4.50) and recruit the annotators on the AMTplatform.9 At the end of the tutorial, each annotatoris asked to annotate a short passage (around 150words). Only annotators with a B3 score (Bagga</blockquote>Annotators are asked to complete a quality control exercise and only annotators who achieve a B3 score of 0.9 or higher are invited to do more annotation