brainsteam.co.uk/brainsteam/content/bookmarks/2022/04/02/models-official-projects-token-dropping-at-master-tensorflow-models1648877142.md

---
bookmark-of:
  title: models/official/projects/token_dropping at master · tensorflow/models · GitHub
  url: https://github.com/tensorflow/models/tree/master/official/projects/token_dropping
date: '2022-04-02T01:25:42.935585'
post_meta:
- date
tags:
- nlp
title: models/official/projects/token_dropping at master · tensorflow/models
type: bookmarks
url: /bookmarks/2022/04/02/models-official-projects-token-dropping-at-master-tensorflow-models1648877142
---

> Token dropping aims to accelerate the pretraining of transformer models such as BERT without degrading its performance on downstream tasks.

> A BERT model pretrained using this token dropping method is not different to a BERT model pretrained in the conventional way: a BERT checkpoint pretrained with token dropping can be viewed and used as a normal BERT checkpoint, for finetuning etc.