Evaluating Open Information Extraction on Scientific and Medical Text
Description
This dataset is the result of applying crowd sourcing to the extractions of two open information extraction tools (Open IE 4 and MinIE) linked below. Extractions were performed on both a set of random sentences from Wikipedia and randomly selected sentences from the OA-STM corpus. The aim is to evaluate the effectiveness of open information extraction tools on scientific and medical text. The initial datasets, the code for applying information, the HITS, labelling instructions, and analysis code are all included above.
Files
Steps to reproduce
The datasets used as input can be found under the data folder. Note that we use the Wiki.zip file linked below as one input. The code for applying the information extraction tools can be found above. After installing the dependencies listed in build.sbt the extractions can be generated. This will generate the HITs used for the experiment that can be uploaded to Amazon Mechanical Turk using turktools (http://turktools.net). The hits we generated are in the file hittest-20171108-publicexperiment.zip. The labelling instructions we used are provided above. After running the crowd sourcing experiment, one can preform the analytics that we did for our experiments using AnalysisAnnotationResults.ipynb.