ravenscroftj
/
brainsteam.co.uk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add 'brainsteam/content/bookmarks/2022/04/02/models-official-projects-token-dropping-at-master-tensorflow-models1648877142.md'
continuous-integration/drone/push Build is passing Details

This commit is contained in:
ravenscroftj 2022-04-02 05:25:44 +00:00
parent c466c5e129
commit 9b4f83e13b
1 changed files with 14 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
brainsteam/content/bookmarks/2022/04/02/models-official-projects-token-dropping-at-master-tensorflow-models1648877142.md Normal file
View File

@ -0,0 +1,14 @@
---
bookmark-of: https://github.com/tensorflow/models/tree/master/official/projects/token_dropping
date: '2022-04-02T01:25:42.935585'
tags:
- nlp
title: "models/official/projects/token_dropping at master \xB7 tensorflow/models"
type: bookmark
url: /bookmarks/2022/04/02/models-official-projects-token-dropping-at-master-tensorflow-models1648877142
---
> Token dropping aims to accelerate the pretraining of transformer models such as BERT without degrading its performance on downstream tasks.
> A BERT model pretrained using this token dropping method is not different to a BERT model pretrained in the conventional way: a BERT checkpoint pretrained with token dropping can be viewed and used as a normal BERT checkpoint, for finetuning etc.