brainsteam.co.uk/brainsteam/content/posts/2015-07-15-sssplit-improvem...

7.8 KiB
Raw Blame History

title author type date url categories tags
SSSplit Improvements James post 2015-07-15T19:33:29+00:00 /2015/07/15/sssplit-improvements/
PhD
Work
demo
improvements
java
partridge
python
regex
sapienta
split
sssplit
test

Introduction

As part of my continuing work on Partridge, Ive been working on improving the sentence splitting capability of SSSplit the component used to split academic papers from PLosOne and PubMedCentral into separate sentences.

Papers arrive in our system as big blocks of text with the occasional diagram, formula or diagram and in order to apply CoreSC annotations to the sentences we need to know where each sentence starts and ends. Of course that means we also have to take into account the other stuff (listed above) floating around in the documents too. We cant just ignore formulae and citations theyre pretty important! Thats what SSSplit does. It carves up papers into sentence () elements whilst also leaving the XML structure of the rest of the document in tact.

The original SSSplit utility was written a number of years ago in Java and used Regular Expressions to parse XML (something that readers of this StackOverflow article will already know has a propensity to summon eldrich abominations from the otherworld). Due to the complex regular expressions, the old splitter was not particularly performant . Especially given the complex nature of some of the expressions (if youre interested, check out one of the simpler ones here).

Now, I can definitely see what the original authors were going for here. Regular expressions are very good for splitting sentences but not sentences inside complex XML documents. XML parsers are not particularly good for splitting sentences but are obviously good at parsing XML. I also understand that the original splitter was designed and then new bits glued on to make it suitable for new and different standards of XML leading to the gargantuan expressions like the one linked to above. I think they did a pretty good job given the information available to them at the time of writing.

I decided that the splitter needed a rewrite and went straight to my comfort zone to get it done: Python. Im very familiar with the language to the point now that I can write a fairly complicated program in it in a day if Ive had enough coffee and sugar.

Writing SSSplit 2.0

I decided that we needed to try and minimise excessive uses of regular expressions for both performance and maintainence/readability reasons.  I decided to try and do as much of the parsing of the document structure as possible using a traditional XML parser. Id heard good things about etree which is part of the standard Python library and provides an informal dom-like interface. I used etree to inspect what I dubbed P-level xml elements first. These are elements that I consider to be at a “paragraph” level. Any sentences inside these elements are completely contained they do not cross the boundaries into the next container (unless the author is a poet/fiction writer/doesnt do English very well I think its a safe bet that they wouldnt finish a paragraph mid-sentence). Within the p-level containers, I sweep for any sort of XML node were interested in text nodes but also any sort of formatting like bold () elements.

When a text node is encountered, thats when regular expressions start to kick in. We do a very simple match for punctuation just in front of a space and a capital letter and run it over the text node these are “potential” splits. We also look for full stops at the very end of the text.

pattern = re.compile('(\.|\?|\!)(?=\s*[A-Z0-9$])|\.$')

        m = pattern.search(txt)

Of course this generates lots of false positives what if weve found a decimal point inside a number? What if its an abbreviation like e.g. or i.e. or an initial like J. Ravenscroft? There is another regular expression check for decimal points and the string around the punctuation is checked against a list of common abbreviations. Theres also a list of authors both the writers of the paper in question and those who are cited in the paper too. The function checks that the full stop is not part of one of these authors names.

Theres an important factor to remember: Text node does not imply finished sentence they are interspersed with formulae and references as explained above. Therefore we cant just finish the current sentence when we reach the end of a text node only when we encounter a full stop (not part of an abbreviation or number), question mark or explanation mark. We also know that we can complete the current sentence at the end of a p-level container as I explained above.

Every time we start parsing a sentence, text nodes and other stuff deemed to be inside that sentence is accumulated into a list. Once we encounter the end of the sentence, the list is glued together and turned into an XML element.

The next step was to see how effective the new splitter was against the old splitter and also manual annotation by professional scientific literature readers.

Testing the splitter

To test the system I originally wrote a simple script that takes a set of manually annotated papers strips them of their annotations so that the new splitter doesnt get any clues runs the new routine over them and then compares the output. This was very rudimentary as I was in a rush and didnt tell me much about the success rate of my splitter. It did display the first and last words of each “detected” sentence for both manual and automatic annotation so I could at least see how well (if at all) the two lined up. I had to run the script on a paper-by-paper basis.

I managed to get the splitter working really well on a number of papers (were talking a 100% match) using this tool. However I realised that the majority of papers were still not being matched and it was becoming more and more of a chore to find which ones werent matching.

Thats why I decided to write a web-based visualisation tool for checking the splitter. The idea is that it runs on all papers giving an overall percentage of how well the automated splitter is working vs the manual splitter but also gives a per-paper figure. If you want to see which papers the system is really struggling with you can inspect them by clicking on them. This brings up a list of all the sentences and whether or not they align.

The tool is pretty useful as it gives me a clue as to which papers I need to tune the splitter with next.

Heres a quick demo video of me using the tool to find papers that dont match very well.

Next steps

A lot of tuning has been done on how this system works but theres still a long way to go yet. Ill probably post another article talking about what further changes had to be made to make the parser effective!