Update blog post

brainsteam/content/posts/2021-01-02-nlp-model-rationale/assets/simple_classifier.xml

@@ -1 +1 @@
-<mxfile host="app.diagrams.net" modified="2021-01-02T18:15:11.052Z" agent="5.0 (X11)" etag="sLTzhic4Rizf62juCqSs" version="14.1.7" type="device"><diagram id="F5Ue9vwdzeswKv2T4yi_" name="Page-1">zVdZc9owEP41nmkfYIyFOR45Qo9MaTpkpvRRsYWtibBcWYDpr+9Klm8TkrTTFBiQPmkP7X6rNRZa7NMPAsfhF+4TZjm2n1poaTnOYOg4lvrY/jlDps4wAwJBfbOpBDb0FzGgbdAD9UlS2yg5Z5LGddDjUUQ8WcOwEPxU37bjrG41xgFpARsPszb6nfoyzNCJMy7xj4QGYW55MJpmK3ucbzYnSULs81MFQjcWWgjOZTbapwvCVPDyuGRyqwurhWOCRPI5Ar31dH+0v63RaTW5jZJ1GvPbntFyxOxgDmw5Iwb65uFARZDRINL46OdBOTr3wBoR5RxGgfrd5nIPIodyBFxSyupgnAOfEpXSkAii7EVnGdIogOEDkdoQrOEIftbU4wyrzQuVMseercEt+zMP1eq9wEdgBbbQCmYr7KkdX3c7df5EK1ZfnqQ80kHzHonf7/efdDouMZ0/ec5JIUmqQyT3DIABDBNQqfxGS7ec3fMYgB7kBc1PIZVkEyvH0PIEtQIYPxKxY5oTIfV9ApGeC36IfKJSpqR2lLEFZ1xou2jnqrcyIAV/JJWVkX4pCR7JCp69ijOAQfD9IoEGBS2hngnfEynOsMUIjAyRTSU7QzM/lXVR7AkrNTExGDalGBSaS7bCwBD2BeR1OsjbyBSJ/Jm6BWDmAXsS6tXzlgkQv3UJXA1K5dRux6FzTBCGJT3W1XdFwli44xQMFzFHTiPozWAm/CA8YqSq1d9UNL6iSGIRENlSpBNTHPv1uXJbuVq9275v5aurTCrpwkmcXfI7mqoi0XTfGPFBPs+aCDSav0L7VuQ6eD/5l7Qf/TntSUrltjL+oe6bvmtmy9RcP3pyNpPnlgo4ojl5jQxvVlLTej5d+5UlNZy8bUmNL/Zunx6b/UwVRo0jeQNXC71El8wMNjjDOG1393OlLWaaWt2yw+ZlN7yiQ5WWkG0vFioXTeN3PKE630/70NnDay5c7eL54455yEFzdXFQeBicmYU9tGkl3tnOG627eTP9T628Rd2h03dbt9oAdVTh2H3xtQbT8jE3Y3/5ZwHd/AY=</diagram></mxfile>
+<mxfile host="Electron" modified="2021-01-03T11:17:14.237Z" agent="5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.1.5 Chrome/87.0.4280.88 Electron/11.1.1 Safari/537.36" etag="AGA9DLXnv4zJsaL713vJ" version="14.1.5" type="device"><diagram id="F5Ue9vwdzeswKv2T4yi_" name="Page-1">5VhZb+IwEP41kXYfQMHOAY8cpd2tSg8qLX1apYmTWA1x1jEQ9tevnTjkpNATrdpWxfPZHtsz3xxCgeNlck6tyL8iDgoUoDqJAicKAD0NAEX8qc42QwbQyACPYkcuKoA5/oskqEp0hR0UVxYyQgKGoypokzBENqtgFqVkU13mkqB6amR5qAHMbStoor+ww/wM7QOzwC8Q9vz85J4xyGaWVr5YviT2LYdsShA8U+CYEsKy0TIZo0AYL7dLtm+6Z3Z3MYpCdsyGzmywXKu3M7iZ9i/DeJZE5LIjtaytYCUfrAAj4PpGfk9YMMBemOLGn5W46MjmpyFayHzkic9Fvu+R5lCO8CsJZVUwyoEfsXCpjygS54Vb5uPQ48NHxNKD+JwV8o8ZtklgicVj4TKgDmf8WupP4ovZe2qtOSssBU65NLVsseLadcX741Sx+GczTMLUaPYTcrrd7rOXjgos9R/b5qRgKElNxJYBB3p8GHOV4t5wohfSPYk40OF+gaONjxmaR+JicLLhscIxskbUDVJO+NhxELf0iJJV6CDhMrHLxUEwJgGh6bnQ1cWvOIBR8oRKM0b6I3aQkJXw7Gf3hjJbctcjyp9TgiR7zhFZIka3fElSDUkZyUCT8qaIC2BIzC/FRF9ilgxFb6e5YCsfSMK+gLyghbw1T6HQGYoswCWbsyfGdtVve82CnEpeaBql9Gq95dE5RlFgMbyuZpM2S8gTbgjmN9nZHJg1o5uDrl5VEpMVtZHcV47/mioIDqpiFvUQa6hKnbN7+lH+cs3r2xA9nM3Qb9Ix7i6uHubTlmQz/bb43vBZW6iUXGbFUZboXZyIQEkpP5fbe7mcFRKg7Xzc4HmL2/dS36jbroX7/Q+ifqsp34H6KMFsURo/iJzT1aU0SWQKSoWtFFpjozV+MlYeIsOpwsrQqv7U1Zqjjg0qE9aIUff4B4cU3Fu/Hbyu1zQRGBWO5EVcTHTiNGSGfAHQoqRZ4bel0phpalTMljP3X8PeVaniJKiq47HwRf3wGxLj1N/P36G1jleucLCS5y2PbHTgSCQOzBvCoZxY8lIttreW9Fr5rmemdy7nb8ppDepqIK8HpSA0W4LQ1PfH25uymtZg8/l/UiAaxfXUBULfmxhe2NjfvU8b/7X67DdxSTMOc6nX+0wymafpNri96HZRFkq7hFhsS6UP6FJkRsqq+KFwO1U3Awfv1M1otbZI0z+3m+l/WZ7pR/LspF2zXuOZ9lqeGfoBRa/mGReLb9Sy5cX3kvDsHw==</diagram></mxfile>

brainsteam/content/posts/2021-01-02-nlp-model-rationale/assets/simple_classifier_with_generator.xml

@@ -0,0 +1 @@
+<mxfile host="Electron" modified="2021-01-03T11:17:53.284Z" agent="5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.1.5 Chrome/87.0.4280.88 Electron/11.1.1 Safari/537.36" etag="b-1z9Tfslc3BRZju0pEM" version="14.1.5" type="device"><diagram id="F5Ue9vwdzeswKv2T4yi_" name="Page-1">5VhZb+IwEP41kXYfQMHOAY8cpd2tSg8qLX1apYmTWA1x1jEQ9tevnTjkpNATrdpWxfPZHtsz3xxCgeNlck6tyL8iDgoUoDqJAicKAD0NAEX8qc42QwbQyACPYkcuKoA5/oskqEp0hR0UVxYyQgKGoypokzBENqtgFqVkU13mkqB6amR5qAHMbStoor+ww/wM7QOzwC8Q9vz85J4xyGaWVr5YviT2LYdsShA8U+CYEsKy0TIZo0AYL7dLtm+6Z3Z3MYpCdsyGzmywXKu3M7iZ9i/DeJZE5LIjtaytYCUfrAAj4PpGfk9YMMBemOLGn5W46MjmpyFayHzkic9Fvu+R5lCO8CsJZVUwyoEfsXCpjygS54Vb5uPQ48NHxNKD+JwV8o8ZtklgicVj4TKgDmf8WupP4ovZe2qtOSssBU65NLVsseLadcX741Sx+GczTMLUaPYTcrrd7rOXjgos9R/b5qRgKElNxJYBB3p8GHOV4t5wohfSPYk40OF+gaONjxmaR+JicLLhscIxskbUDVJO+NhxELf0iJJV6CDhMrHLxUEwJgGh6bnQ1cWvOIBR8oRKM0b6I3aQkJXw7Gf3hjJbctcjyp9TgiR7zhFZIka3fElSDUkZyUCT8qaIC2BIzC/FRF9ilgxFb6e5YCsfSMK+gLyghbw1T6HQGYoswCWbsyfGdtVve82CnEpeaBql9Gq95dE5RlFgMbyuZpM2S8gTbgjmN9nZHJg1o5uDrl5VEpMVtZHcV47/mioIDqpiFvUQa6hKnbN7+lH+cs3r2xA9nM3Qb9Ix7i6uHubTlmQz/bb43vBZW6iUXGbFUZboXZyIQEkpP5fbe7mcFRKg7Xzc4HmL2/dS36jbroX7/Q+ifqsp34H6KMFsURo/iJzT1aU0SWQKSoWtFFpjozV+MlYeIsOpwsrQqv7U1Zqjjg0qE9aIUff4B4cU3Fu/Hbyu1zQRGBWO5EVcTHTiNGSGfAHQoqRZ4bel0phpalTMljP3X8PeVaniJKiq47HwRf3wGxLj1N/P36G1jleucLCS5y2PbHTgSCQOzBvCoZxY8lIttreW9Fr5rmemdy7nb8ppDepqIK8HpSA0W4LQ1PfH25uymtZg8/l/UiAaxfXUBULfmxhe2NjfvU8b/7X67DdxSTMOc6nX+0wymafpNri96HZRFkq7hFhsS6UP6FJkRsqq+KFwO1U3Awfv1M1otbZI0z+3m+l/WZ7pR/LspF2zXuOZ9lqeGfoBRa/mGReLb9Sy5cX3kvDsHw==</diagram></mxfile>

brainsteam/content/posts/2021-01-02-nlp-model-rationale/index.md

@@ -18,14 +18,13 @@ tags:
   - python
   - ai
   - nlp
-  - spacy
 
 ---
 
 
 ## Introduction
 
-The ability to understand and rationalise about automated decisions is becoming particularly important as more and more businesses adopt AI into their core processes. Particularly in light of legislation like GDPR requiring subjects of automated decisions to be given the right to an explanation as to why that decision was made. There have been a number of breakthroughs in explainable models in the last few years as academic teams in the machine learning space focus their attention on the why and the how.
+The ability to understand and rationalise about automated decisions is becoming particularly important as more and more businesses adopt AI into their core processes. This requirement is particularly pertinent in light of legislation like GDPR requiring subjects of automated decisions to be given the right to an explanation as to why that decision was made. There have been a number of breakthroughs in explainable models in the last few years as academic teams in the machine learning space focus their attention on the *why* rather than the *how* of machine learning architecture.
 
 ## Recent Progress in Model Explainability
 
@@ -33,21 +32,55 @@ Significant breakthroughs in model explainability were seen in the likes of [LIM
 
 [Transformer](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html)-based models like [BERT](https://github.com/google-research/bert) which use the concept of neural attention to learn contextual relationships between  words can also be interrogated by [visualising attention patterns inside the model](https://towardsdatascience.com/deconstructing-bert-part-2-visualizing-the-inner-workings-of-attention-60a16d86b5c1)). However, these visualisations are still quite complex (especially for transformer-based models which typically have multiple parallel attention mechanisms to examine) and do not provide concise or intuitive rationalisation for model behaviour.
 
-## Rationalization of Neural Predictions
+So how can we generate a human-readable, intuitive justification or rationalization of our model's behaviour and how can we do it without further data labelling? In the remainder of this post we will explore a couple of solutions.
 
-In 2016, [Lei, Barzilay and Jaakola](https://people.csail.mit.edu/taolei/papers/emnlp16_rationale.pdf) wrote about a new architecture for rationale extraction from NLP models. The aim was to generate a new model that could extract a "short and coherent" justification for why the model made a particular prediction.
+## Solution 1: Rationalization of Neural Predictions (Lei, Barzilay and Jaakkola, 2016)
 
-{{<figure src="images/figure1.png" caption="An example of a review with ranking in two categories. The rationale for Look prediction is shown in bold from [Lei, Barzilay and Jaakola 2016](https://people.csail.mit.edu/taolei/papers/emnlp16_rationale.pdf)">}}
+In 2016, [Lei, Barzilay and Jaakkola](https://people.csail.mit.edu/taolei/papers/emnlp16_rationale.pdf) wrote about a new architecture for rationale extraction from NLP models. The aim was to generate a new model that could extract a "short and coherent" justification for why the model made a particular prediction.
 
+***NB: I've simplified the mathematical notation from the original paper here in order to communicate the concepts more easily.***
 
-The idea is actually quite simple. Firstly, let's assume we're starting with a classification problem where we want to take document **X** and train a classifier function **F(x)** to predict label **y** based on the text in the document (e.g. X is a movie review and y is positive or negative sentiment). 
+The idea is actually quite simple. Firstly, let's assume we're starting with a classification problem where we want to take document **X** and train a classifier function **F(x)** to predict label **y** based on the text in the document (e.g. imagine that X is a movie review and y is positive or negative sentiment). 
-
 
 {{<figure src="images/simple_classifier.png" caption="A simple text classifier that tries to learn to predict y based on text in X">}}
 
-What Lei, Barzilay and Jaakola propose is that we add a new step to this process.  We're going to introduce **G(X)** - a generator- which aims to generate a rationale **R** for the document. Then we're going to train our classifier **F(X)** to predict **y** not from the document representation **X** but from the rationale **R**. Our new process looks something like this:
+In order to extract rationales, Lei, Barzilay and Jaakkola propose introducing **G(X)** - a generator component - to our model. The generator aims to generate - you probably guessed it - a "short and coherent" rationale **R** for the input documents. The brilliant part is that if we plug our generator into our existing architecture in the right way, we don't need to explicitely train it. 
+
+This can be achieved by inserting the generator **G** between **X** and our original classifier **F(X)** (referred to in the paper as the encoder or **enc**) and having the classifier learn to predict **y** from the generated rationale **R**. Then we can randomly sample rationales from **G** and encourage it to generate rationales that help **F** to correctly predict **y** during back-propagation.
+
+**G** is constrained to encourage it to make short rationales (otherwise, it could propose **R=X** which wouldn't help us interpret anything). It is also discouraged from selecting distant groupings of words (this speaks to the 'coherent' part - we don't want a rationale that's just a random selection of words).
 
 
+{{<figure src="images/simple_classifier_with_generator.png" caption="An updated model that generates a rationale, R, from document X and predicts y based on R.">}}
+
+In order for this model to work, it must be possible for the generator **G** to be able to disentangle key information from the input document **X** in such a way that it's possible for **F(R)** to predict the true output label **y** from it (e.g. "is there anything better" as captured in the figure above might be a concise way to summarise a very positive movie review). 
+
+This architecture provides an excellent first pass at our rationale extraction task and Lei, Barzilay and Jaakkola show that it works well on the multi-aspect sentiment problem set out in their paper. However, there are a number of improvements that can be made that [we explore below](#solution-2-re-thinking-cooperative-rationalization-yu-chang-zhang-and-jaakola-2019)
+
+Since we are now going to focus on the improved solution, I have not provided any code for this one. However, you can find the author's [original code here](https://github.com/yala/text_nn/) and a more modern [PyTorch implementation of their work here](https://github.com/yala/text_nn/) - both are fairly well written and clear. The latter is probably more understandable these days given the former is implemented in the now-obsolete [theano](https://groups.google.com/g/theano-users/c/7Poq8BZutbY/m/rNCIfvAEAwAJ%3E%60_?pli=1) framework.
+
+
+## Solution 2: Re-thinking Cooperative Rationalization (Yu, Chang, Zhang and Jaakola, 2019)
+
+In a follow up paper in 2019, [Yu, Chang, Zhang and Jaakola](http://people.csail.mit.edu/tommi/papers/YCZJ_EMNLP2019.pdf) - Jaakola being one of the authors from the previous paper - propose an improved rationale extraction approach that offers significant performance improvements over the earlier Lei et al paper.
+
+The new framework attempts to control for a number of shortcomings:
+
+ 1. The earlier framework does not allow the generator **G(X)** any direct visibility onto the desired outcome **y&#771;** (predicted **y**) which may lead to a poor solution. 
+ 2. The encoder (**F(R) &#8594; y&#771;**) does not have any direct visibility of the text excluded from the rationale and thus may not be performing optimally.
+ 3. Whilst most classification tasks have clues that are typically short phrases, some tasks require much more complicated context-dependent combinations of words and features that cannot be adequately captured by a rationale leading again leading back to point 2.
+ 4. Due to prior focus only on **R** (point 2.), there is no control over information *not* included in the rationale which means that the generator may condense summaries down to collections of pertinent feature words (e.g. positive or negative adjectives in our sentiment analysis example) and omit contextually relevant information (we might want to know what was good or bad).
+
+The proposed solution is a re-jig of the architecture so that the encoder once again predicts **y** directly from the full input document **X**, the rationale generator **G** has access to this predicted value **y&#771;** (again where **y&#771;** denotes the *predicted* value for **y** according to our model ). This new coupling of the encoder and generator is referred to as a single component known as the *introspective generator* in the paper.
+
+
+{{<figure src="images/yu_2019_figure_1.png" caption="In this figure taken from [Yu, Chang, Zhang and Jaakola 2019](http://people.csail.mit.edu/tommi/papers/YCZJ_EMNLP2019.pdf), the original rationale extraction framework from above is represented by (a) and the new approach is represented by (c)">}}
+
+We also add a couple of new downstream predictor components:  
+ * **P(y|R)**  which is basically a clone of **F(R) &#8594; y&#771;** that's reconfigured to accept a rationale from our new *introspective* generator and predict the probability of **y** given **R**. 
+ * **P(y|R<sup>c</sup>)** which is known as the *complement* predictor and tries to predict the likelihood of **y** given **R**'s complement, **R<sup>c</sup>**, that is, all of the words that are *not* included in the rationale.
+
+The authors refer to this configuration as a three player co-operative game if you count the introspective generator and the two predictors as the players.
 
 
 <!-- which was followed up and improved upon in 2019 by [Yu, Chang, Zhang and Jaakola](http://people.csail.mit.edu/tommi/papers/YCZJ_EMNLP2019.pdf). The idea is surprisingly simple: take an existing neural classification model, tack on a new  -->