Natalie's Multir Updates

July 25, 2014

• KBP2014:
  • Finished UWashington3 run,
  • Combined UWashington3 and UWashington2 results to create UWashington1 run
  • Next: make a new run w/extraction and sentence output

• Multilingual Multir:
  • Met with Stephen and Laurel for initiation
  • Next: create an experiment properties file to use when training a model for Laurel

• Swapping Classifiers in Multir:
  • Exchanged email with Mitchell
  • Next: identify code

• Multir Relations:
  • Received an email from Stephen
  • Next: investigate date relations, with Mitchell

August 1, 2014

out

August 8, 2014

• Findings:
  • Stanford has a 7 class (includes DATE) and a 4 class (PER,LOC,ORG,MISC) NER library. The MISC in the "4 class" does not identify DATE, numbers, or title.
  • Stanford NER sentence-level tool to see how things are tagged: http://nlp.stanford.edu:8080/ner/process
  • Mitchell's paper serves as doc for 'what training is' for KBP2014Multir
  • Mitchell's paper, Table 1, is a nice table: = #extractions by relation, including #TP and #FP

• Products:
  • Created a "model trace" for all extractions from the KBP2014 Relevant Docs. "Model Trace" contains: queryname, queryid, arg1, arg2, relation, confidence score, sentence, feature, feature weight, where the feature list is sorted by the magnitude of the feature weight.
  • Stephen wrote 2 perl scripts to change the format of the model trace.
  • All exist in: /projects/WebWare6/KBP_2014/analysis

• Plans:
  • KBP2014:
    • 1)Effect of Bolt and NG on KBP2014 Output?: Make a run with Bolt+NG+NW
    • 2)Add Dates?: Review fg code for training. How does DATE get assigned to 1938, for example, for sentence 9912?
  • SwappingClassifier+Benchmark:
    • 1)Review Mitchell's clojure code
    • 2)Review edu.washington.multir.eval.ManualEvaluation and edu.washington.multir.util.CLIUtils

August 15, 2014

• Findings:
  • KBP2014 Relevant Docs Analysis - due to the exclusion of ng and forum docs, some queries had 0 relevant docs
    • /projects/WebWare6/KBP_2014/analysis/EvalRelDocs.xlsx
  • OpenIE Demo - restarted solr instance. Found instructions on Wiki for the OpenIE Demo
  • KBP2014: Adding Dates - swapped in a different class for argument identification, so that dates would be tagged,
    • i.e. use KBP_NELAndNERArgumentIdentification.java instead of NERArgumentIdentificationPlusMISC.java
  • Effect of Bolt and NG on KBP2014Output:
    • query001, 4 hrs processing time, didn't add anything
    • query002, 1.5 hrs processing time, added two slots - one possibly correct, one not correct
  • SwappingClassifier+Benchmark - Stephen mentioned Weka (NZ) and Mallet (UMass), Mitchell mentioned Stanford (their paper after Raphael's paper)

• Products:
  • KBP2014Multir Report [1]
  • KBP2014 Relevant Docs Analysis
    • /projects/WebWare6/KBP_2014/analysis/tac2013_runOutput_EvalRelDocs
    • /projects/WebWare6/KBP_2014/analysis/tac2014_runOutput_EvalRelDocs
    • /projects/WebWare6/KBP_2014/analysis/EvalRelDocs.xlsx

• Plans:
  • M: KBP2014 - analysis of argument identification
  • T: ReadCode - ManualEvaluation, CLIUtils, EvaluationUtils - to determine cmd line inputs, and input file formats
  • W: Mtgs - Stephen, Data Integration Call, Sexual Harassment Prevention Training, LoudLab
  • Th: 1)Multi-lingual - trial run of model training using DS files provided by Laurel or report on missing pieces, 2)Read DefaultFeatureGenerator.java, Preprocess.java, and Train.java to learn inputs for multilingual project to train the model (process not quite the same as for KBP2014)
  • F: 1)Mtgs - Stephen, Multi-lingual, 2)more on multi-ling 3)argIdent

August 22, 2014

• Findings:
  • KBP2014: ArgumentIdentification - 40% of the ORG queries with no slot fills, have tagging issues; i.e. the ORG was not tagged as an ORG,
    • for example, Third Guard Division, High Point Church, Greenberg Smoked Turkey, Alessi, Zirh - this prevented expected date extractions
    • /projects/WebWare6/KBP_2014/analysis/queriesAnalysis.xlsx
  • KBP2014: Date Extractions - PER Terry Jupp had a date extraction, but the threshold was too low to be selected to fill a slot
  • Code Reading: ManualEvaluation, CLIUtils, EvalUtils - Stephen sent emails indicating input file locations, and run instructions

• Products:
  • /projects/WebWare6/KBP_2014/analysis/queriesAnalysis.xlsx

• Plans:
  • Work with Laurel on Multi-lingual project: transfer the project/set-up environment on lil, set-up bitbucket, run the scripts
  • LoudLab mtg: Expert Crowd-sourcing with Flash Teams
  • Later, Plans: argIdent design doc, manEval design, swapClassifier - explore options

August 29, 2014

• Multilingual Project: worked with Laurel during her last week -
  • joined the BitBucket
  • got set-up on the lil cluster
  • successfully ran the scripts
  • set-up a DerbyDB of the FullCorpus_First100Sentences
• Plans:
  • M: Labor Day/Holiday
  • Tu: debug/run an Experiment using a properties file which refers to the First100Sentences DerbyDB.
    • This method can be used for the multilingual project.
    • This experience can also be used to make this process easier.
  • W: LoudLab: Stephen Jonany
  • Th-F: DerbyDB/Experiment/ManualEvaluation/ArgumentIdentification

September 5, 2014

• Multilingual Project -
  • started a doc to identify the details,
    • /projects/WebWare5/multir-multilingual/DesignDocs/multilingualMultirProject.odp
  • created a derbyDB of TAC2009 FullCorpus_First100Sentences
• LoudLab: shared 590 link,
  • http://courses.cs.washington.edu/courses/cse590a/13wi/
• Plans:
  • M: ArgumentIdentification + Taggers
  • T: ArgumentIdentification + Taggers,
  • W: Stephen: discuss Multilingual Plan, LoudLab: DeepMining paper
  • Th: Bundle Multir (easier to use), studied project contents of: MultirExperiments, MultirFramework
  • F: Mtg Stephen, Mtg Sameer+Stephen, Bundle Multir

September 12, 2014

• Plans: Bundle Multir (make it easier to use)
  • M: issue- processing corpus- read John and Laurel's code
  • T: dentist + discuss PraxEng with Mitchell
  • W: meet with Stephen, conf call, LoudLab mtg
  • Th: -
  • F: meet with Stephen

September 19, 2014

out

• Findings:
  • 1)installed DerbyDB locally, used built-in function ij to examine the contents of the FullCorpus_First100Sentences DerbyDB I created,
    • /projects/WebWare5/multir-multilingual/training/testrunNH/derbyDB_First100Sentences
    • show tables;
      • There are 2 tables: DOCUMENTTABLE and SENTENCETABLE
    • select * from DOCUMENTTABLE;
      • This table has 1 column: DOCNAME; there are 10 documents in this table.
    • select * from SENTENCETABLE;
      • This table has 9 columns:
        • SENTID, DOCNAME, SENTTOKENSINFORMATION, SENTTEXTINFORMATION, SENTOFFSETINFORMATION,
        • SENTDEPENDENCYINFORMATION, TOKENNERINFORMATION, TOKENOFFSETINFORMATION, TOKENPOSINFORMATION
      • There are 100 sentences in this table.

• Plans:
  • Use Laurel's training pipeline for the multi-lingual project; explore moving code to github, write roadmap.
  • Communicate with Gagan, Mausam, and Dan re new sentences and new grad student for PraxEng project.

• • M: meet w/Stephen
  • Tu: cse ergonomics workshop
  • W: meet w/Stephen
  • F: meet w/Stephen, meet w/Stephen, Sameer and Victoria

September 26, 2014

F: Meet with Stephen, meet with Stephen and Angli, meet for Multilingual Project with Sameer, Victoria, and Stephen

Products:

• 1. /projects/WebWare5/multir-multilingual/DesignDocs/MultiLingualMultirToDoList.docx
• 2. /projects/WebWare5/multir-multilingual/DesignDocs/MultirMultilingualRoadmap.docx

Plans:

• 1) edit the multilingual code to get files to insert in a DerbyDB
• 2) give Stephen access to the bitbucket repository
• 3) help Stephen debug his DerbyDB connection issue

October 3, 2014

F: meet with Stephen

Findings:

• 1) DerbyDB: the SENTID is an INTEGER, see below

Details:

• DerbyDB - First 100 Sentences

• ij> help

• ij> describe SENTENCETABLE;
  • COLUMN_NAME |TYPE_NAME|DEC&|NUM&|COLUM&|COLUMN_DEF|CHAR_OCTE&|IS_NULL&
  • ------------------------------------------------------------------------------
  • SENTID |INTEGER |0 |10 |10 |NULL |NULL |NO
  • DOCNAME |VARCHAR |NULL|NULL|128 |NULL |256 |YES
  • SENTTOKENSINFORMATI&|VARCHAR |NULL|NULL|20000 |NULL |40000 |YES
  • SENTTEXTINFORMATION |VARCHAR |NULL|NULL|20000 |NULL |40000 |YES
  • SENTOFFSETINFORMATI&|VARCHAR |NULL|NULL|20000 |NULL |40000 |YES
  • SENTDEPENDENCYINFOR&|VARCHAR |NULL|NULL|20000 |NULL |40000 |YES
  • TOKENNERINFORMATION |VARCHAR |NULL|NULL|20000 |NULL |40000 |YES
  • TOKENOFFSETINFORMAT&|VARCHAR |NULL|NULL|20000 |NULL |40000 |YES
  • TOKENPOSINFORMATION |VARCHAR |NULL|NULL|20000 |NULL |40000 |YES

• ij> describe DOCUMENTTABLE;
  • COLUMN_NAME |TYPE_NAME|DEC&|NUM&|COLUM&|COLUMN_DEF|CHAR_OCTE&|IS_NULL&
  • ------------------------------------------------------------------------------
  • DOCNAME |VARCHAR |NULL|NULL|128 |NULL |256 |NO

• ij> select * from DOCUMENTTABLE;
• DOCNAME
• --------------------------------------------------------------------------------------------------------------------------------
• XIN__ENG__19950122.0142.LDC2007T07
• XIN__ENG__19950403.0177.LDC2007T07
• XIN__ENG__19980928.0021.LDC2007T07
• XIN__ENG__20010510.0103.LDC2007T07
• XIN__ENG__20021028.0184.LDC2007T07
• XIN__ENG__20021028.0216.LDC2007T07
• XIN__ENG__20030823.0122.LDC2007T07
• XIN__ENG__20060105.0389.LDC2007T07
• XIN__ENG__20060210.0149.LDC2007T07
• XIN__ENG__20060803.0205.LDC2007T07

• 10 rows selected

Plans:

• 1) Write code to generate the features file
• 2) Mod the training pipeline to add sentenceId to training.inst

October 10, 2014

• Tu: met with Xiao to discuss corpus processing
• W: met with Stephen
• Th: met with Victoria, discussed mods made to CoreNLPProcessor and Linker, to process docs written in Chinese
• F: meet with Stephen, meet w/Stephen and Angli, meet w/Stephen, Sameer, and Victoria

Products:

• 1) Generated file, extractions, from ExtractFromCorpus, for Angli
  • /projects/WebWare6/InfoOmnivore/Crowdsourcing
• 2) Generated file, features, as part of Standard Core Multir Pipeline
  • /projects/WebWare5/multilingual/training/testrunNH

Plans:

• 1) Preprocess features and Train model using features file generated by the Standard Core Multir (SCM) Pipeline
• 2) Get extractions from trained model
• 3) Look for Rob's code that processes the corpus documents, to address offsets in the SCM pipeline
• 4) Add DATES and other argument identification to the SCM pipeline
• 5) Address dupes in the SCM pipeline
• 6) Work with Angli
• 7) Work with Xiao to investigate deadlock issue w/the DerbyDB on rv-n14
• 8) Install maven on rv cluster, and run ssh script
• 9) Investigate sonatypes, maven central
• 10) Mitchell's MultirFramework project (= MultirFramework + MultirExperiment), compile?

October 17, 2014

This Past Week:

• M: multilingual: fixed sentences piling-up issue, further modified code adding feature generation to the pipeline
• Tu: multilingual: installed maven on rv cluster and successfully ran job, trained model and got all rel=NA extractions (using features file generated by the pipeline)
• W: re-started IARPA OpenIEdemo, exchanged thoughts with Sameer about the pipeline, met with Angli for MultirSystem orientation, talked to Xiao about DerbyDB, exploring features map issue when training model
• Th: multlilingual: fixed features map issue, trained a model, got extractions, removed duplicate sentences, added NER tagging for DATE and NUMBER, pushed updated code to bitbucket, notified Victoria
  • NOTE: the addition of DATE and NUMBER leads to a huge increase in the size of the relations.gz file, causing a MemoryError crash.
• F: met with Angli, met w/Stephen, met w/Victoria

DerbyDB:

• connect 'jdbc:derby:mydb; create=true'; (Omit create=true if you're connecting to an already existing DB.)
• connect 'jdbc:derby://rv-n14.cs.washington.edu:49152//scratch2/usr/jgilme1/FullCorpus-UIUCNotableTypes';
• connect 'jdbc:derby:/scratch2/usr/jgilme1/FullCorpus-UIUCNotableTypes';
• connect 'jdbc:derby:/homes/gws/nhawkins/KBP2014-Slotfilling-Multir/FullCorpus';

Plans:

• multilingual: need to generate new relations files for DATE and NUMBER relations, discuss with Sameer

October 24, 2014

past

• M: created new extractions for Angli
• Tu: DEFT call + minutes, met Michael, tried tagging code
• W: met w/Stephen, met w/Gene, LoudLab mtg
• Th: met w/Victoria, shared existing relations file, added Chinese changes to code,
• F:
  • 1)updated code to incorporate Victoria's changes, will commit to bitbucket, then she can work with bb directly,
  • 2)changes to code cause my pipeline to no longer run, need to debug
  • 3)added kbp relations to multilingual pipeline; grep+awk+sed'd an existing relations file to match style and format required by the multilingual pipeline

plans

• debug "chinese" code additions
• identify offset computation for kbp2014
• tagging: debug

October 31, 2014

Findings: lil faster than rv-n14 faster than porvo

past

• M:
  • 1) debugging code additions to pipeline
    • 1)missing lang in CoreNLPProcessor
    • 2)invisible /n in mvn script command, causing directory to not be found
  • 2) communicated with Victoria and Sameer, re
    • method of offset computation in kbp2014
    • parser, coref
• T:
  • 1) started kdd-nary web demo for Mausam
  • 2) pointed Angli to training files
  • 3) debugged code additions to pipeline and script
  • 4) added code to write documents.meta
  • 5) started new run
  • 6) emailed Congle, Mathias and Kenton about DEFT WA github
• W:
  • 1) pipeline run crashed due to LackOfMemory error
    • the relations.gz file being written and re-read is huge
    • increasing heap to 3GB or 6GB doesn't solve it
  • 2) communicated with Victoria, re the pipeline and running it on the Chinese corpus
  • 3) pointed Angli to Stephen and answer_key files
    • github/knowitall/MultirExtractor/manualevalutations.csv
    • /projects/WebWare6/Multir/KBP2013
  • 4) re-treating to smaller set of relations and code in bitbucket to examine memory error
  • 5) LoudLab mtg
  • 6) identified Solr corpus code
• Th:
  • emailed Jonathan Watson at BBN that no pushes need to be made to the DEFT Washington repository, so they can compact the repo
  • communicated thoughts to Victoria
  • on new run: no memory error issue when reading-in relations.gz, but memory error when writing out the 8 files
  • returned to taggers, parsing error, missing pieces
• F:
  • sent Victoria a paper comparing Chinese parsers
  • set-up jdk and maven on Dan's nlp cluster
  • exploring SUTime/NormalizedDate for Chinese [NormalizedNamedEntityTagAnnotation, NamedEntityTagAnnotation]
  • need to find tagger documentation: emailed Michael with questions, asking about examples and documentation (I have a parsing error)
  • tools edu: unix memory commands, emacs review
  • out: late afternoon, eye appt

plans

• need to get pipeline running again, with Chinese and w/o memory crashes
• tagging

November 7, 2014

past

• M: updated pipeline anew with code for Chinese, git
• T: ran pipeline on English w/o crash, attempted Chinese run which crashed, regex
• W: met with Victoria, updated pom.xml and installed a jar for Chinese run, got past first crash, experienced 2 more
• Th: explored Stanford CoreNLP DATE and NUMBER tagging in Chinese (tagged as MISC instead), explored source of training data for job titles (NEL ai)
• F: explored Stanford CoreNLP regexner for tagging (looks good), exploring docSplittler for offsets

plans

• debug setProcessing() crash with Chinese
• check exception handling for arrayOutOfBounds Stanford parsing error with Chinese
• adjust features assignment
• try regexner for tagging Chinese (Stanford CoreNLP paper indicates that regexner is available for Chinese)
• create file of tagging rules for Chinese (and English)

November 14, 2014

past

• M: regexner
• T: Veteran's Day
• W: meet with Stephen, docSplitter
• Th: regexner, cleanxml, utf8 inputstream
• F: meet with Stephen, meet with Stephen, Victoria, Sameer

plans

• multilingual pipeline

November 21, 2014

past

• M:
  • DocSplitter - 1)offsets via cleanXML 2)docDate via docID,
  • PipelineDemo - mod rawExt = ".gz" for KBP Chinese corpus
  • MultirInstance - call features function in DefaultFeatureGenerator
• T:
  • MultirInstance - call features function in DefaultFeatureGenerator
  • LuceneRelationSearcher - review
• W: meet w/Stephen, Data Integration call, LoudLab mtg
• Th: review database code
• F: meet w/Stephen, sent offset code to Victoria

plans

Solr index Chinese corpus

November 28, 2014

past

• M: meet with Dan and Angli, solr Chinese analyzers
• T: meet with Dan and Angli, install solr on rv-n16.cs.washington.edu
• W: solr concepts reading
• Th: HOLIDAY
• F: HOLIDAY

plans

Solr index Chinese corpus

December 5, 2014

Locations of solr indexed corpora:

• 1. OpenIE Extractions - old corpus
  • url: http://knowitall:knowit!@rv-n16.cs.washington.edu:9321/solr
  • data: /scratch/usr/rbart/solr

• 2. OpenIE Extractions - new corpus
  • url: http://knowitall:knowit!@rv-n16.cs.washington.edu:8123/solr
  • data: /scratch/usr/rbart/solr

• 3. TAC 2012 Corpus - old corpus
  • url: http://knowitall:knowit!@rv-n16.cs.washington.edu:9325/solr/oldCorpus
  • data: /scratch2/usr/rbart/solr

• 4. TAC 2013 Corpus - new corpus
  • url: http://knowitall:knowit!@rv-n16.cs.washington.edu:9325/solr/newCorpus
  • data: /scratch2/usr/rbart/solr

• 5. TAC 2013 Chinese corpus - newswire
  • url: http://rv-n16.cs.washington.edu:8468/solr/kbpdev
  • data: /scratch/usr/nhawkins/solr

Instructions for starting/re-starting solr instance:

• 1. ssh into rv-n16 (possibly as knowall)
• 2. cd into a dir like, /scratch/usr/rbart/solr
• 3. issue java start command for the specific port to start;
  • for example, to start the TAC 2013 Corpus, which is on port 9325, the command is: java -Djetty.port=9325 -jar start.jar

past

• M: write solrj solrpopulator
• T: out
• W: meet w/Stephen, schema.xml, run solrpopulator on test set, LoudLab mtg
• Th: Andrew's talk, Oren's talk, Victoria verified the solr finding of Lu Wei in a doc, run solrpopulator on TAC KBP 2013 Chinese corpus (newswire, 2 mil docs)
• F: meet with Stephen, completed MechTurk tutorial, reviewing RunKBP2015Chinese and related files

plans

• proceed with RunKBP2015Chinese file

December 12, 2014

past

• M: working on RunKBPChinese file
• Tu: working on FAQGuideToMultirCodeBase and RunKBPChinese files, asked Victoria for a Chinese-trained model
• W: meet with Stephen, shared chineseQuery_relDocs file w/Stephen, asked Victoria for list of cities, states or provinces, countries in Chinese
• Th: install jar of stanford chinese models, update build.sbt (use stanford core nlp 1.3.4, rather than 1.3.5, with jar of stanford chinese models)
• F: meet with Stephen, debug adding stanford chinese models

plans

chineseKBP, FAQGuide

December 19, 2014

Documentation - FAQGuideToMultirCodeBase:

• /projects/WebWare6/Multir/Documentation/FAQGuideToMultirCodeBase.docx

Findings:

• 1. new way of getting more recent Stanford Core NLP and models in build.sbt, use val stanfordNlp:
  • For KBP2014-Slotfilling-Multir's build.sbt,
  • val stanfordNlp = "edu.stanford.nlp" % "stanford-corenlp" % "3.4" artifacts (Artifact("stanford-corenlp", "models"), Artifact("stanford-corenlp"))
  • libraryDependencies ++= Seq(
  • stanfordNlp,
  • "org.apache.commons" % "commons-io" % "1.3.2",
  • "commons-lang" % "commons-lang" % "2.6",
  • "commons-cli" % "commons-cli" % "1.2",
  • // "edu.stanford.nlp" % "stanford-corenlp" % "1.3.4",
  • "com.guokr" % "stan-cn-nlp" % "0.0.4",
  • "org.codehaus.jackson" % "jackson-mapper-asl" % "1.8.5",
  • "org.apache.derby" % "derby" % "10.10.1.1",
  • "org.apache.derby" % "derbyclient" % "10.9.1.0",
  • "jp.sf.amateras.solr.scala" %% "solr-scala-client" % "0.0.7",
  • "edu.washington.cs.knowitall" % "multir-framework_2.10" % "0.3-SNAPSHOT" withSources() withJavadoc(),
  • "edu.washington.cs.knowitall.common-scala" % "common-scala_2.10" % "1.1.2",
  • "edu.washington.cs.knowitall.taggers" %% "taggers" % "0.1" excludeAll(ExclusionRule(organization = "com.googlecode.clearnlp")),
  • "edu.washington.cs.knowitall.nlptools" %% "nlptools-sentence-breeze" % "2.4.2" excludeAll(ExclusionRule(organization = "com.googlecode.clearnlp")),
  • "com.googlecode.clearnlp" % "clearnlp-threadsafe" % "1.3.0-c",
  • "edu.washington.cs.knowitall.nlptools" %% "nlptools-chunk-opennlp" % "2.4.2",
  • "edu.washington.cs.knowitall.nlptools" %% "nlptools-parse-clear" % "2.4.2" excludeAll(ExclusionRule(organization = "com.googlecode.clearnlp"))
  • // "edu.washington.cs.knowitall.stanford-corenlp" % "stanford-ner-models" % "1.3.4",
  • // "edu.washington.cs.knowitall.stanford-corenlp" % "stanford-postag-models" % "1.3.4",
  • // "edu.washington.cs.knowitall.stanford-corenlp" % "stanford-dcoref-models" % "1.3.4",
  • // "edu.washington.cs.knowitall.stanford-corenlp" % "stanford-parse-models" % "1.3.4",
  • // "edu.washington.cs.knowitall.stanford-corenlp" % "stanford-sutime-models" % "1.3.4",
  • // "edu.stanford.nlp.chinese" % "stanford-chinese-corenlp" % "1.0"
  • )
• 2. Order of dependencies matters; i.e. stanfordNlp must precede "com.guokr" % "stan-cn-nlp" % "0.0.4"
• 3. Time for Processing on-the-fly 159 Chinese documents for Lu Wei: 85 mins
• ChineseKBP
  • Parse Queries: complete
  • Get Relevant Docs via SOLR: complete
  • Plug-in model from Victoria trained on 10 yrs of XinHua News: complete
  • Overall Extract step: checking ai, sig, fg, and extract-from-sententialinstance sub-steps

• Next:
  • ChineseKBP: Extract Step
  • ChineseKBP: Filter extractions
  • ChineseKBP: Substitute KBP Relations

December 26, 2014

FAQ Guide To Multir Code Base:

• updated with run commands and pointers to scripts with run commands

Findings:

• ChineseKBP - relDoc XIN_CMN_20070605.0150 (relevant for query SF14_ENG_085, "New Republic")
  • not in test or training set from preprocessed corpus

• ChineseKBP - many of the features are not in the training set;
  • for 2 test queries (1 PER, 1 ORG) with 3 relevant docs each,
  • we get 2 extractions for the PER query (which are irrelevant extractions which would be filtered out)
  • and 0 extractions for the ORG query
  • features files on WebWare5 from Victoria for testing/debugging:
    • /projects/WebWare5/multir-multilingual/gigaword
  • Example:
    • Features: 4 (out of 4, one is in the training set)
    • Features:
      • 1) inverse_true|ORG|*LONG*|PERSON (this one is IN the training set!)
      • 2) inverse_true|的|ORG|*LONG*|PERSON|所 (NOT in the training set; chinese chars = Of and The)
        • similar ones to #2 in the training set:
        • inverse_true|的|ORG|*LONG*|PERSON|和
        • inverse_true|的|ORG|*LONG*|PERSON|,
        • inverse_true|的|ORG|*LONG*|PERSON|的
        • inverse_true|的|ORG|*LONG*|PERSON|。
        • inverse_true|的|ORG|*LONG*|PERSON|为
        • inverse_true|的|ORG|*LONG*|PERSON|主持
        • inverse_true|的|ORG|*LONG*|PERSON|随后
        • inverse_true|的|ORG|*LONG*|PERSON|、
        • inverse_true|的|ORG|*LONG*|PERSON|则
        • inverse_true|的|ORG|*LONG*|PERSON|将
        • inverse_true|的|ORG|*LONG*|PERSON|还
        • inverse_true|的|ORG|*LONG*|PERSON|得
        • inverse_true|的|ORG|*LONG*|PERSON|也
        • inverse_true|的|ORG|*LONG*|PERSON|连任
      • 3) inverse_true|举行 的|ORG|*LONG*|PERSON|所 属 (NOT in the training set)
        • similar ones to #3 in the training set:
        • inverse_true|举行 的|ORG|*LONG*|GPE|举行 的
        • inverse_true|举行 的|ORG|*LONG*|GPE|人权 纪录
        • inverse_true|举行 的|ORG|*LONG*|GPE|提出 的
        • inverse_true|举行 的|ORG|*LONG*|GPE|第33 处
      • 4) str:PERSON|<-属[nsubj]<-中间派[rcmod]<-获得[nsubj]<-显示[ccomp]->决定[dep]->选举[nsubj]->[nn]|ORG] (NOT in the training set)
        • similar ones to #4 in the training set:
        • str:PERSON|<-属[nsubj]<-称[ccomp]<-[prnmod]|GPE|->([nn]
        • str:PERSON|<-属[nsubj]<-称[ccomp]<-[prnmod]|GPE|->足球[dep]
        • str:PERSON|<-属[nsubj]<-称[ccomp]<-[prnmod]|GPE|->)[punct]
        • str:PERSON|<-属[nsubj]<-称[ccomp]<-[prnmod]|GPE

• ChineseKBP -
  • relations in KBP format -
  • PER
    • per:origin, per:employee_or_member_of, per:schools_attended
    • per:siblings, per:other_family, per:religion,

• • ORG
    • org:member_of, org:members, org:subsidiaries, org:parents, org:top_members_employees
    • org:country_of_headquarters, org:country_of_headquarters, org:country_of_headquarters

• • relations not in KBP format -
  • PER
    • people.person.place_of_birth, people.person.place_lived, people.person.parents,
    • people.person.children, people.person.spouse, people.deceased_person.place_of_death
  • ORG
    • organization.organization.founders, org:location_of_headquarters,

• W: HOLIDAY
• Th: HOLIDAY
• F: HOLIDAY

January 2, 2015

Updated FAQToMultirCodeBase

Findings:

• Features (on-fly and training file) match for: "Sentence one" in document XIN_CMN_20050201.0002,

• Training file:

• • XIN_CMN_20050201.0002.1

• • m.0d05q4 (Iraq)
  • m.07t65 (United Nations)
  • org:member_of

• • 1) inverse_true|ORG|*LONG*|GPE
  • 2) inverse_true|B_-1|ORG|*LONG*|GPE|政治
  • 3) inverse_true|B_-2 B_-1|ORG|*LONG*|GPE|政治 进程
  • 4) str:GPE|<-进程[nn]<-下[lobj]<-阶段[dep]<-关键[assmod]<-确保[top]<-指出[ccomp]->埃克哈德[nsubj]->发言人[nn]->[nn]|ORG

• On-fly:

• • 1) inverse_true|ORG|*LONG*|GPE
  • 2) inverse_true|B_-1|ORG|*LONG*|GPE|政治
  • 3) inverse_true|B_-2 B_-1|ORG|*LONG*|GPE|政治 进程
  • 4) str:GPE|<-进程[nn]<-下[lobj]<-阶段[dep]<-关键[assmod]<-确保[top]<-指出[ccomp]->埃克哈德[nsubj]->发言人[nn]->[nn]|ORG

• • Number of the 4 features in the feature set: sv.num 3, Feature id's = 24 89958 89959
  • Number of relations: 24
  • Number of features: 403682
  • Model Returns: Relation: NA Score: -Infinity 伊拉克 184 187 联合国 152 155

• One extraction found: (founders of Xinhua News Agency)
  • [info] de: Features: 11
  • [info] de: Features: [inverse_false|ORG|2月 1日 电 ( 记者|PERSON, inverse_false|B_-1|ORG|2月 1日 电 ( 记者|PERSON|), inverse_false|B_-2 B_-1|ORG|2月 1日 电 ( 记者|PERSON|) ., str:<-新华社[nn]|ORG|<-记者[nn]<-[nsubj]|PERSON|->)[punct], str:<-新华社[nn]|ORG|<-记者[nn]<-[nsubj]|PERSON, dep:<-[nn]|ORG|<-[nn]<-[nsubj]|PERSON, dir:<-|ORG|<-<-|PERSON, str:ORG|<-记者[nn]<-[nsubj]|PERSON|->)[punct], dep:ORG|<-[nn]<-[nsubj]|PERSON|->[punct], dir:ORG|<-<-|PERSON|->, str:ORG|<-记者[nn]<-[nsubj]|PERSON]
  • [info] de: arg1: 新华社 联合国109 115
  • [info] de: arg2: 刘历彬124 127
  • [info] gp: sv.num 2
  • [info] gp: model num rel: 24 403682 403682
  • [info] Features...
  • [info] 4892 6138
  • [info] Relation: organization.organization.founders Score: 5.9112424E7 新华社 联合国 109 115 刘历彬 124 127

• W: HOLIDAY
• Th: HOLIDAY
• F: HOLIDAY

January 9, 2015

Findings:

• Chinese KBP will be postponed
• Location Lists: need to be divided into 3 lists - 1)city 2)stateorprovince 3)country
• Should we get an extraction for
  • United Nations 联合国 (/m/07t65)
  • and Iraq 伊拉克 (/m/0d05q4)
  • in XIN_CMN_20050201.0002?

• • • • Sentence 1
        • 联合国秘 书长发言人埃克哈德1日在此间举行的新闻发布会上指出, 伊拉克政治进程下一阶段的关键是确保所有团体和党派以 及因各种原因没有或不愿参加此次大选者,都参与到制宪 进程中来。

• • • • UN and Iraq

• • • • • inverse_false|ORG|*LONG*|GPE
        • inverse_false|B_-1|ORG|*LONG*|GPE|政治
        • inverse_false|B_-2 B_-1|ORG|*LONG*|GPE|政治 进程
        • str:ORG|<-发言人[nn]<-埃克哈德[nn]<-指出[nsubj]->确保[ccomp]->关键[top]->阶段[assmod]->下[dep]->进程[lobj]->[nn]|GPE]
          • arg1: 联合国152 155
          • arg2: 伊拉克184 187
          • sv.num 4
          • Relation: NA Score: -Infinity 联合国 152 155 伊拉克 184 187
          • expecting: org:members

• • • • Iraq and UN

• • • • • inverse_true|ORG|*LONG*|GPE
        • inverse_true|B_-1|ORG|*LONG*|GPE|政治
        • inverse_true|B_-2 B_-1|ORG|*LONG*|GPE|政治 进程
        • str:GPE|<-进程[nn]<-下[lobj]<-阶段[dep]<-关键[assmod]<-确保[top]<-指出[ccomp]->埃克哈德[nsubj]->发言人[nn]->[nn]|ORG]
          • arg1: 伊拉克184 187
          • arg2: 联合国152 155
          • sv.num 3
          • Relation: NA Score: -Infinity 伊拉克 184 187 联合国 152 155
          • expecting: org:member_of

• • • • Sentence 2

• • • • Iraq and UN

• • • • • inverse_false|GPE|*LONG*|ORG
        • inverse_false|
        • |GPE|*LONG*|ORG|才
        • inverse_false|强调
        • |GPE|*LONG*|ORG|才 会
        • str:GPE|<-进程[nn]<-制定[dep]<-提出[conj]->派出[dep]->[nsubj]|ORG]
          • arg1: 伊拉克258 261
          • arg2: 联合国291 294
          • sv.num 2

January 16, 2015

Findings -

• Finished with Chinese KBP
• Need new group github account (no longer part of KnowItAll)
• Learning about Gene's project, ImplicitRelationExtractor
• Got set-up in Eclipse and with Gene's github for ImplicitRelationExtractor
• Met with Stephen and Gene -- finding out where they are, where they're going

Next -

ExtractionScoring program based on ExtractorTester.scala

January 23, 2015

M: MLK Jr, Holiday

Findings -

• Finished ExtractionScoring, version 1.
• Started ExtractionScoring-designDoc.docx
• Working on creating a sentences file, and then an extractions file, that goes with the answer-key
  • The existing extractions file was corrupted by Excel

Next -

• ExtractionScoring:
  • Need an extractions file to test.
  • Make two new classes for routine running of ExtractionScoring
    • TACDevelopmentRelationExtractorForScoring: make a mod to update the sequence-number-file
    • TACDevelopmentRelationExtractionAndScoring: make wrapper class
  • Append new extractions to existing answer key
  • Update ExtractionScoring-designDoc.docx

January 30, 2015

Findings -

• Finished ExtractionScoring system:
  • classes =
    • testers.ExtractionScoring,
    • tac.TACDevelopmentRelationExtractorAndScoring,
    • tac.TACDevelopmentRelationExtractorForScoring

• Created ConstrainedImplie AnswerKey
  • Need to ask specifics about it at Fri mtg for adjustments

February 6, 2015

Findings -

• Updated ExtractionScoring.scala:
  • output file (answer-key): added "untagged" to filename, so write #-answer-key-untagged.txt
  • output file (answer-key): write existing-answer-key-portion in order of sentIndex
  • output file (scoring report): print input and output file names at top of report
  • output file (scoring report): print correct and incorrect extractions in separate groups
  • output file (scoring report): print scored extractions in order by sentIndex
  • input file (extractions): ignore any lines not starting with an integer (allows #comment lines)
  • input file (answer-key): ignore any lines not starting with an integer (allows #comment lines)

• Scored Constrained version of the answer key,
  • /ImplicitRelationExtractor/tac_development/benchmark/scoring/constrained-answer-key.txt

• Created ExtractionScoringDiff System
• Created ExtractionScoringDiff-DesignDoc

• Explored question about SentIndex=21118 being added to AnswerKey as tagged in ExtractionScoring: (answer: looks ok)
  • 21118 AFP_ENG_20081215.0118.LDC2009T13 ': US official-8 nationality US-7 1 Shoe-throwing Iraqi isolated 'attention seeker': US official

• Created new Benchmark directories for:
  • diff-scoring
  • constrained

ToDo -

• Add Items To ExtractionScoringDiff report
  • Incorrect Extraction lists, in addition to the Correct Extraction lists
  • Summary Stat Tables from individual reports
• Write Script to Create New Constrained Answer Key to Tag
• Tag New Constrained Answer Key
• Identify Improvements to the System
• Identify Relevant Docs from 2009 and 2010 corpuses for 2013 and 2014 queries?

February 13, 2015

Activities -

• Finished ExtractionScoringDiff Program and Design Doc
• Created new benchmark directories for diff-scoring and constrained runs

• Discussed with Stephen his ideas for improvements to the system
  • No verb in entity
  • No "and" in entity
  • The City,State has city City cases

• Discussed with Gene his ideas for improvements to the system
  • Created a subclass of ImplicitRelationExtractor
  • Created a "filter" method by overriding the extractRelations method,
  • "filter" method calls super class's extractRelations method and filters the result
  • the question is: what should be filtered? (that isn't already?)
  • Created a sandbox ExtractorTagTester_NH program

• Discussed with Angli the data we generated for MechTurk questions
  • /projects/WebWare6/anglil/extraction_with_features
  • 10944 questions for place_of_death
  • 405774 questions for origin

Findings -

• Lists in Entity -
  • Found that entity with a long list resulting in many 'nationality' relations
  • is already eliminated by the ExpansionFunctions/conjAndAppos function
• Verb in Entity -
  • • Found that an entity with a verb, like "Chad claims rebel infiltration from Sudan."
    • which once had the extraction entity, "Chad claims", has already been eliminated
• Self-Referentials -
  • There is a function in ImplicitRelationExtractor, removeSelfModifyingRelations
    • to address the City, State has city City cases,
    • private def removeSelfModifyingRelations(relations: List[ImplicitRelation]) = {
    • relations.filter(rel => rel.head.index != rel.tag.index)
    • }
    • There is a bug with the head.index.
    • For the example, "CAIRO, Egypt 2007-09-07 01:21:47 UTC"
    • the head = CAIRO, the tag = CAIRO,
    • the head.index = 6, the tag.index = 1
• ImplicitRelation fields - (bugs)
  • ir.ners.size = 0
  • ir.explicitRelationTraces.size = 0

February 20, 2015

Findings -

• DeepDive
  • need to connect to a postgresql database
    • support is working with Stephen to use db on serenity
  • settings in application.conf
  • database tables created by DeepDive
    • sentences (sentence_id, sentence, ner tags)
    • has_spouse (person1_id, person2_id, sentence_id, description, is_true, relation_id, id)
    • people_mentions (sentence_id, startToken, length, person_id)
    • has_spouse_features
    • holdout table
  • view created by DeepDive (with prediction results)
    • has_spouse_is_true_inference (same schema as has_spouse + col for expectation)
  • calibration plots created by DeepDive

• Implie
  • checked-in first version of filterLists function
  • will further test filterLists
  • will start on filterNERTags function

• KBP2014
  • Reviewed paper, sent revision request to Stephen

February 27, 2015

Findings -

• DEFT
  • Signed-up for deft-dev mailing list, approved by Fred Choi
• Angli
  • met to discuss how to generate features for lived-in relation
  • Congle pointed him to a file for feature generation (a file not in github/knowitall), which he will probably use
• DeepDive
  • got access to postgresql on Serenity from Scott in cs-support
  • created postgresql user nhawkins and user deepdive
  • created 4 tables in the database, deepdive_spouse (data preprocessing and loading step)
  • created python scripts for the example application
  • received connection error re pg_hba.conf file
  • received read and write access for pg_hba.conf file from Stephen
  • turns out, the pg_hba.conf file owned by Stephen is a stale copy, and isn't used by postgresql
  • contacted cs-support to try to get permissions to enter a "pg_ctl reload" command, required to use the edited pg_hba.conf file
  • the real pg_hba.conf exists inside the private directory: /scratch2/data, owned by the unix user postgres
  • we don't have access to the unix postgres account, cs-support does
  • cs-support updated the real pg_hba.conf file to allow TCP connections, in addition to socket connections, and reloaded the file
  • scripts can now connect to the postgresql database
  • debugged scripts
  • completed the example application, which means we have a running deepdive system for the "has_spouse" example
  • will meet with Stephen next week to discuss details
• Implie
  • filtering out lists in the entity increased overall precision from .80 to .85
  • completed filterTagNER to address issue of PERSON being assigned to a city relation
  • identified tasks to pursue to scope out KBP evaluation
  • will meet with Stephen next week to discuss details

March 6, 2015

Findings -

• DeepDive
  • Talked to Angli about the location of his training data
  • Next: will load Angli's training data into deepdive

• Implie
  • Copied some KBP files into the Implie project
  • Setup initial RunKBPImplie file
  • Working to resolve: can't find resource error
  • Next: continue with each step leading to slotfill output

March 13, 2015

Findings -

• DeepDive
  • Writing files to upload into postgresql

• Implie
  • Completed code for a complete RunKBPImplie run
  • debugging

March 20, 2015

Findings -

• Implie -
  • removed duplicate lines in kbpReport output (now have 25 lines (1 per slot) per person)
  • switched Stanford pipeline to a "coref" pipeline
  • fixed run-time crash by updating build.sbt to include a line which explicitly downloads the model files; i.e.
    • "edu.stanford.nlp" % "stanford-corenlp" % "3.5.0" classifier "models",
  • next: need to fix Out-of-Memory Error
  • next: learn coref data structures to use this information to filter sentences and extractions
  • next: choose BestAnswers using a frequency table of extracted answers

• DeepDive -
  • wrote 3 data files to upload into postgresql (sentences.csv, lived_in.csv, lived_in_features.csv)
    • (the files are tab-delimited, will change the filenames to .tsv, for future run)
  • created database: deepdive_perloc
  • created tables: lived_in, lived_in_features, sentences
  • loaded the 3 data files into these tables (setup_database.sh, which uses schema.sql)
  • specified a "simple logistic regression" rule (in application.conf)
  • trained a model using Angli's turk supervised data for the lived_in relation (run.sh)
  • performed inference to predict lived_in
  • created calibration plot
    • [profiler] INFO --------------------------------------------------
    • [profiler] INFO Summary Report
    • [profiler] INFO --------------------------------------------------
    • [profiler] INFO inference_grounding SUCCESS [3004 ms]
    • [profiler] INFO inference SUCCESS [833 ms]
    • [profiler] INFO calibration plot written to /homes/gws/nhawkins/out/2015-03-18T125922/calibration/lived_in.is_true.png [0 ms]
    • [profiler] INFO calibration SUCCESS [232 ms]
    • [profiler] INFO --------------------------------------------------
  • next: examine, tune, calibrate the lived_in model
  • next: make further plans

• Angli -
  • talked about the multir training files
  • Note: to free-up space, cs-support moved the Multir directory from WebWare6 to WebWare8

March 27, 2015

Findings -

• Spring Break: Stephen out

• Implie -
  • MEMORY:memory error crash: modified /homes/gws/nhawkins/tools/sbt/conf/sbtopts to increase total memory allowed
    • analyzing only nw docs: get memory error again when include the others
  • OFFSETS: index out-of-bounds error for tag.interval.start and tag.interval.end: removed "cleanxml" from coref pipeline
    • with "cleanxml" the xml tags are not created as tokens, but they are included in the sentenceText
    • however, still not matching the Stanford tokens with those from the relationExtractor
  • COREF: using to filter sentences and extractions with a regular expression
  • BESTANSWERS: choosing highest frequency slotfill(s)
  • KBPReports: 2014: 5 PER, 5 ORG, 2013: PER 1
  • JobTitles: political analyst ok? (not in 2014 answer key for 001_Ahmed_Rashid), need to add: contractor, subcontractor

• DeepDive -
  • on-hold this week

April 3, 2015

Findings -

• DeepDive(DD) -
  • Precision: To use DD TagTool to compute precision, need to add processed-sentence data to sentences table,
    • and then need to generate more tables based on that
  • Dependency data: Sorted out StanfordCoreNLP to generate this
  • Creating more files to load into the database

• Implie -
  • Shared kbpreports with team, for feedback

• Angli -
  • Met to discuss NER and NEL

April 10, 2015

Findings -

• Implie -
  • noticed many cities and jobtitles missing from the lists
    • sent links to Gene for data files:
      • locations: nellCaseInsensitiveLocation data, Stanford statesandprovinces file from dcoref dir of 1.3.4 models file
      • job titles: careerplanner.com (12k job titles), Stanford type_map_clean file from regexner dir of 1.3.4 models file
    • sent Gene scripts for accessing a freebase dump (from multilingual bitbucket)
    • sent Gene links to tac2013 code for accessing the nellCaseInsensitiveLocation data file
  • created topMembers.txt classTermList
  • ran the 2013 dev set: 5 PER and 5 ORG
  • explored coref data for Andrew E. Lange - observed missing job titles
    • printed the 2 relevant docs for: Andrew Lange, to explore missing job titles, etc

• DeepDive -
  • generated 5 files of data for perloc relations: lived_in.tsv, died_in.tsv, born_in.tsv, traveled_to.tsv, nationality.tsv
  • generated 1 file of sentences data: sentences_perloc.tsv
  • generated 1 file of features data: features.tsv
  • using 4812 sentences (which come from documents with just one extracted sentence, since our sentId=docId)
  • updated: setup_database.sh and schema.sql to load these files
  • updated application.conf to train models for all 5 relations with a holdout set of 25%
  • computed precision for each relation for 3 threshold settings: .9, .95, .975
  • started exploring how to train other types of models in scala; e.g. logistic regression, random forest,
  • chatted with Angli, about logistic regression and the features he used

• OpenIE - request from Stephen and Oren
  • restarted SOLR, updated wiki instructions at:
    • https://dada.cs.washington.edu/knowitall/wiki/index.php/OpenIE_Demo
    • section: Restarting the Web Demo on Reliable

April 17, 2015

Findings -

• Implie -
  • created new directory /kbp for kbp output
  • wrote regular expressions to create query name aliases
  • using aliases (which includes query name), rather than just query name when choosing relevant docs, sentences, and extractions
    • Andrew E. Lange had 2 newswire docs with exact string match, and now has 5 newswire docs with exact string match with aliases
    • Andrew E. Lange has no extractions from his large collection of ng documents
    • Frank Baldino Jr has the same number of documents, before and after
    • Frank Baldino Jr's "Warren Buffet" extraction has been filtered out
    • China Charity Federation - needs to have 'jobtitle' assigned to top_members_employees
  • working on getting the existing NERFilterable trait which has the function, tagNERs, working with the kbp code, so that 'jobtitle' can be assigned to 'top_members"

• DeepDive -
  • planning a report summarizing findings
  • talked to Angli about logistic regression

• Chinese KBP -
  • emails, f2f's discussing status
  • have been running code to test extractions from a document

April 24, 2015

Out: on Friday, April 24 - Tues April 28

• Chinese KBP:
  • addressed Xiao's question about deathplace=Beijing case, for the version1 Chinese multir model
    • reviewed code finding that if the best relation = NA, then the score is set to -Infinity
    • examined inputs and outputs to FullInference.infer and verified that the score is the sum of the weights
    • verified that the NA relation had the highest score
      • relation, score
      • -----------------------
      • NA, 4.0672048E7
      • place_lived, 4026146
      • place_died, 4145343
      • all other relations have a negative score or score of 0.
      • The score for NA is the sum of the 5 weights:
      • > x = c(4508804,4027394,4690781,2.5924078E7,1520991)
      • > sum(x)
      • [1] 40672048
  • modified FilteredExtractions code to just check exact string match of queryName with arg1Name
  • inserted new model
  • inserted new feature generating code
  • ran model on training_001 query
    • found that the new model extracts deathplace, in addition to origin
  • working on creating location lists from files with freebase id's
  • need to incorporate location lists
  • incorporated location lists
  • ran (in batches) the system on all 32 training queries
  • one of the queries has 429 docs and took 7 hours to run
  • communicated with Xiao about time-outs/threads - the program moves on, but the long threads are not still running in the bg

May 1, 2015

Returned from vac

• Chinese KBP
  • Experimented with using Stanford's very recently released CoreNLP 3.5.2
  • Needed to install a new Java8
  • Needed to use jar's instead of Maven Central to get the libraries
  • Got an error message when parsing: No head definition (didn't resolve)
  • Reverted to former state, but apparently not quite because lost extraction for deathplace for query_001
  • Made another attempt at reverting to former state, succeeded.
  • Removed filtering by matching first char for ORG
  • Removed dupes in slots allowing multiples
  • Ignored relations ending with RE
  • Experimented with threads: t.interrupt (no effect), t.stop (some effect)

May 8, 2015

• Chinese KBP
  • running evaluation queries

• KBPImplie
  • fixing repository issue
  • re-doing set org:top_members value based on jobTitle

May 15, 2015

• Chinese KBP
  • completed the evaluation run, results were submitted

The Chinese KBP system has been awhile in coming. It is great to share our work by running a slot-filling evaluation on the existing Multir system, a monolingual Chinese system. The set of queries, 103, was a bit larger than for the English system evaluation which had 100 queries. The Chinese system runs more slowly than the English system. Some of the Chinese queries had many relevant docs (max = 2499), and a couple had no relevant docs. We submitted kbp output for all 103 queries.

• KBPImplie
  • corrected lost code to assign job title to top_employees and set proper offsets
  • plugged-in nationality mapper to convert nationality to country_of relations
  • ran system on 2013 dev set of 10, completed in 1.5 hours
  • tried to run system on 2014 dev set of 10, did not complete
  • restricted relevant sentences to those < 400 in length
  • ran system on 2014 dev set of 10, completed in 5.5 hours

• The KBPImplie has been developing with the idea of applying the system, in combination with the OpenIE system, to the next KBP Evaluation.

• KBPImplie System Notes
• ----------------------------
  • tagger: val tagger = TaggerLoader.extendedKeywordHighRecallTagger
  • ire: val relationExtractor = new ModHighRecallImplIE(tagger)
  • substitute relations PER
    • nationality -> per:origin, per:countries_of_residence
    • city -> per:cities_of_residence
    • province -> per:statesorprovinces_of_residence
    • jobtitle -> per:title
    • religion -> per:religion
    • school -> no substitution
  • substitute relations ORG
    • nationality -> org:country_of_headquarters
    • city -> org:city_of_headquarters
    • province -> org:province_of_headquarters
    • jobtitle -> org:top_members_employees, if org name in entity string, jobtitle is on topMemberJobTitles list, and there is a person in the entityString
    • religion -> org:political_religious_affiliation
    • school -> no substitution
  • relevant docs - contain exact string match to query name or to one of the aliases
    • per aliases = fmls -> fml,fl, fml -> fl, fls -> fl
  • relevant sentences
    • < 400 in string length
    • contains query name or one of the aliases (not followed by a word character) or
    • contains a coref mention name of type PROPER from a coref chain containing the query name or one its aliases
  • relevant extractions
    • arg1 contains query name or one of the aliases, or
    • arg1 contains a coref mention and at least of the query name tokens is in the entity string
  • select best answers
    • first sort the candidates by their frequency
    • case 1: take the first on the sorted list
    • case 9: take all with unique arg2 names
  • confidence score
    • need to discuss with team

The above notes have been collected by examining the code, and can be used to add detail about kbpimplie, if necessary, to the paper in preparation. The test sets for 2013 and 2014 are running and taking a lot of time. One bug needs to be fixed for the PER relations; that is, for nationality only per:countries_of_residence was set, and not per:origin, as well. That has been fixed. The output will need to be re-run or adjusted for persons who have a per:countries_of_residence slot fill.

• Next:
  • discuss kbpimplie test set run at mtg
  • coldstart work

May 22, 2015

• KBP 2015 Cold Start
  • Exploring Doc Pre-Processing
  • Exploring OpenIE

Currently, the system finds files in a directory, splits each file into documents, uses the Stanford tools to get the sentences for each document, gets ImplIE extractions, and gets openIE extractions.

• Next questions to address:
  • 1) how to filter the openIE extractions to ones which have a PER or ORG in arg1
  • 2) how to substitute the openIE extraction relation with a KBP relation
  • 3) how to extract the PER or ORG from arg1 (need to tagNERs)
  • 4) how to convert the openIE extraction arg2 to a slotfill (for example, "in Chicago" to "Chicago")
  • 5) the implIE extractions need to be converted to KBP extractions

May 29, 2015

HOLIDAY: Monday

Many of the files from the tac2013 project will need to be used to process the openie extractions on the fly. Much of the tac2013 code has been reviewed to determine how the openie extractions are converted to kbp extractions, and filtered. The conversion requires a lot of processing.

A new project tac2015 has been created (a copy of tac2013) to convert tac2013 into an on-the-fly system, rather than one that uses a solr indexed set of extractions. This deviates from last week's work of making a start from the basics.

Next: continue to add/modify files to obtain a kbp-openie-on-the-fly system.

June 5, 2015

• Out: Wednesday

• Met with Stephen to discuss Cold Start Design Doc (using coref to merge entities)

• Met with Stephen and Dan, discussed a change to the task requirement for the Cold Start

June 12, 2015

OUT: Thursday thru Sunday (6-11 - 6-14) -- Chicago

• Starting over, planning the Cold Start - Slot Filling Variant
• Examined the sample data, wrote new design doc:
  • /projects/WebWare6/KBP_2015/DesignDocs/KBP2015ColdStart_SlotFilling.docx

June 19, 2015

• met with Stephen - he gave me a DEFT Data Integration update (Congle, Kenton, Mathias - end of June)

• attended DEFT Data Integration call

• updated doc: KBP2015ColdStart_SlotFilling.docx

• working to create an end-to-end system

• • openie component
    • restricts docs to cold start corpus,
    • handles PER, ORG, GPE queries
    • handles inverse relations
    • handles filling a single slot
    • handles outputting for just a single slot
    • todo: resetting the query name, in the cases where the query name is "short"; for example, Homer rather than Homer Simpson

• • implie component
    • starting on this

• • multir
    • got notes from Angli on his training approach in python

June 26, 2015

OUT: Stephen's retirement lunch, Wed, 6/24, 12noon - 2pm, UW Club

Angli OUT: 6/23 on, internship

• KBP2015

• • multir
    • trained a model using angli's perloc feature data (18k examples)
    • made adjustments to handle the cold start needs
    • making a few more adjustments

• • implie
    • investigating SolrHelper issue when searching for reldocs, something weird has popped-up
    • made adjustments to handle the cold start needs

• • openie
    • no adjustments this week

• • scripts
    • sketched out ideas/needs for script to write round2 queries
    • sketched out ideas/needs for script to combine results from each system
    • sketched out ideas/needs for script to adjust confidence score for implie and multir

July 3, 2015

HOLIDAY: Friday, July 3

• Things have been going pretty well.
  • 1) Openie is in good shape, I think.
  • 2) Multir seems to be in pretty good shape, but has issue with on-the-fly features not matching exactly the training features, may need to generate new features and train a new model.
  • 3) Implie has been adjusted for cold start, has a serious issue with solr logging needing to be turned off. Starting testing now.

• The ToDo's are:
  • 0)handle the multir features and implie logging issues mentioned above
  • 1)test implie
  • 2)write scripts to: 1)adjust conf score for multir and implie 2)combine results from openie, implie, multir 3)write round2 queries
  • 3)there are some other things in mind

More detailed notes:

• implie:
  • Solr logging appears to have gotten turned-on somehow since running the jobs for the paper
    • This causes a big slowdown in processing documents on the fly and determining relevant docs
    • Working on figuring out how to turn it off

• multir:
  • The features generated on the fly have some similarities to Angli's, but don't match them exactly
    • Emailed Angli to learn the exact steps he took to see if those can be matched on the fly
    • If can't match those steps exactly, will need to generate a new training file using steps which can be replicated
    • For the example below, though the features differ, we do get an extraction for the relation
      • Arguments: Choi Jung-yun
      • Arguments: San Diego
      • de
      • de: Generating Features:
      • de: Features: 15
      • de: Features:

[inverse_false|PERSON|, who grew up in|LOCATION, inverse_false|interrupted|PERSON|, who grew up in|LOCATION|., inverse_false|B_-1 interrupted|PERSON|, who grew up in|LOCATION|. B_1, str:Choi[nn]->|PERSON|[rcmod]<-grew[prep]<-in[pobj]<-|LOCATION|[nn]<-San, str:interrupted[nsubj]<-|PERSON|[rcmod]<-grew[prep]<-in[pobj]<-|LOCATION| [nn]<-San, str:Choi[nn]->|PERSON|[rcmod]<-grew[prep]<-in[pobj]<-|LOCATION, dep:[nn]->|PERSON|[rcmod]<-[prep]<-[pobj]<-|LOCATION, dir:->|PERSON|<-<-<-|LOCATION, str:interrupted[nsubj]<-|PERSON|[rcmod]<-grew[prep]<-in[pobj]<-|LOCATION, dep:[nsubj]<-|PERSON|[rcmod]<-[prep]<-[pobj]<-|LOCATION, dir:<-|PERSON|<-<-<-|LOCATION, str:PERSON| [rcmod]<-grew[prep]<-in[pobj]<-|LOCATION|[nn]<-San, dep:PERSON|[rcmod]<-[prep]<-[pobj]<-|LOCATION|[nn]<-, dir:PERSON|<-<-<-|LOCATION|<-, str:PERSON|[rcmod]<-grew[prep] <-in[pobj]<-|LOCATION]

• • • • de: arg1: Choi Jung-yun12 25
      • de: arg2: San Diego42 51
      • Sentence Text: interrupted Choi Jung-yun, who grew up in San Diego.
      • Relation: /people/person/place_lived Score: 5119517.0 Choi Jung-yun 12 25 San Diego 42 51

• • • Angli's features for this example:

Choi Jung-yun 1 3 San Diego 7 9 NYT_ENG_20080426.0048.LDC2009T13 /people/person/place_lived 0 0 0 interrupt ed Choi Jung-yun, who grew up in San Diego. inverse_false|PERSON|, who grew up|O 0 inverse_false|interrupted|PERSON|, who grew up|O|Diego 0 inverse_f alse|B_-1 interrupted|PERSON|, who grew up|O|Diego . 0 str:Choi[nn]->|PERSON|[rcmod]<-grew[prep]<-in[pobj]<-Diego[nn]<-|O 0 dep:[nn]->|PERSON|[rcmod] <-[prep]<-[pobj]<-[nn]<-|O 0 dir:->|PERSON|<-<-<-<-|O 0 str:interrupted[dobj]<-|PERSON|[rcmod]<-grew[prep]<-in[pobj]<-Diego[nn]<-|O 0 d ep:[dobj]<-|PERSON|[rcmod]<-[prep]<-[pobj]<-[nn]<-|O 0 dir:<-|PERSON|<-<-<-<-|O 0 str:PERSON|[rcmod]<-grew[prep]<-in[pobj]<-Diego[nn]<-|O 0

July 10, 2015

KBP-mailing-list

• I sent an email requesting to be added to the list, don't think that has happened

Multir -

• matched tools used in system w/those used by Angli;
  • modified build.sbt to use Stanford CoreNLP 1.3.5
  • extracted the use of the Charniak-Johnson parser from the system for KBP2015
• ran a complete round1 run on 247 queries
  • it crashed on query #210, out of memory error, there is a memory leak
  • I think this is related to the threads we start as part of the time-outs approach
  • the program moves on, but does not stop operating on the thread, need to fix that
• ToDo
  • adjust the time-outs/fix memory leak
  • consider what threshold to use,
    • the current one gets 3/3 correct answers for 'gpe:residents_of_city = Springfield',
    • it gets a long list of answers for 'gpe:residents_of_city = San Diego', not sure yet how many are correct
  • check on the parseException error which occurs for some docs
  • check if there are any inputs to the Stanford tools to skip processing of long sentences,
    • an alternative would be to *potentially* process the documents differently, to process them by sentence, and then together again for coref

Implie -

• appears that the Solr issue of printing lots of info to stdout, and slowing things down a lot, has been resolved, whew!
  • openie had been added to the build.sbt, and the log4j.properties from that jar were causing the logging change
  • put a log4j.properties file in /src/main/resources which heightens the logging level
• ToDo
  • testing
  • make a full round1 run

Notes:

• multir -
  • Document Length: 20000
  • Sentence Length: <400
  • Threshold: 1,900,000
  • Sergei Lavrov
    • Sentence Text: On Tuesday, Russian Foreign Minister Sergei Lavrov said Moscow

would abandon the missile deployments if Washington scrapped its plans to install interceptor missiles in Poland and a radar in Czech Republic.

• • • 3 Extractions:
      • Extraction [arg1=Sergei Lavrov 1590 1603, arg2=Moscow 1609 1615, rel=per:origin, score=4048161.0, arg1Link=null, arg2Link=null, arg1BestMention=null, arg2BestMention=null, docName=AFP_ENG_20081111.0049.LDC2009T13, sentNum=null, arg1BestMentionSentNum=null, arg2BestMentionSentNum=null]
      • Extraction [arg1=Sergei Lavrov 1590 1603, arg2=Poland 1722 1728, rel=/people/person/place_lived, score=2712296.0, arg1Link=null, arg2Link=null, arg1BestMention=null, arg2BestMention=null, docName=AFP_ENG_20081111.0049.LDC2009T13, sentNum=null, arg1BestMentionSentNum=null, arg2BestMentionSentNum=null]
      • Extraction [arg1=Sergei Lavrov 1590 1603, arg2=Czech Republic 1744 1758, rel=/people/person/place_lived, score=3125085.0, arg1Link=null, arg2Link=null, arg1BestMention=null, arg2BestMention=null, docName=AFP_ENG_20081111.0049.LDC2009T13, sentNum=null, arg1BestMentionSentNum=null, arg2BestMentionSentNum=null]
    • Features match for: Sergei Lavrov and Moscow
      • de: Features: 16
      • de: Features: [inverse_false|PERSON|said|LOCATION, inverse_false|Minister|PERSON|said|LOCATION|would, inverse_false|Foreign Minister|PERSON|said|LOCATION|would abandon, str:Russian[nn]->|PERSON|[nsubj]->said[ccomp]<-abandon[nsubj]<-|LOCATION, dep:[nn]->|PERSON|[nsubj]->[ccomp]<-[nsubj]<-|LOCATION, dir:->|PERSON|-><-<-|LOCATION, str:Foreign[nn]->|PERSON|[nsubj]->said[ccomp]<-abandon[nsubj]<-|LOCATION, dep:[nn]->|PERSON|[nsubj]->[ccomp]<-[nsubj]<-|LOCATION, dir:->|PERSON|-><-<-|LOCATION, str:Minister[nn]->|PERSON|[nsubj]->said[ccomp]<-abandon[nsubj]<-|LOCATION, dep:[nn]->|PERSON|[nsubj]->[ccomp]<-[nsubj]<-|LOCATION, dir:->|PERSON|-><-<-|LOCATION, str:Sergei[nn]->|PERSON|[nsubj]->said[ccomp]<-abandon[nsubj]<-|LOCATION, dep:[nn]->|PERSON|[nsubj]->[ccomp]<-[nsubj]<-|LOCATION, dir:->|PERSON|-><-<-|LOCATION, str:PERSON|[nsubj]->said[ccomp]<-abandon[nsubj]<-|LOCATION]

July 17, 2015

KBP-NIST

• heard back from Hoa:
  • signed-up for the mailing list and got the .pl scripts for validating output, generating new queries, and combining round1 and round2 output

All -

• ran the KBP-Nist script CS-validateSF.pl on run output - discovered that the NIL lines (i.e. no output for the query) need to be removed
• ran the KBP-Nist script CS-GenerateQueries.pl on output - that seemed to work
• wrote NormalizeConfidenceScore.scala script, which puts scores in the range 0.2-0.9, ran that on multir and implie output
• wrote CombineOutput.scala script, which combines the round1 output from openie, implie, and multir, ran that on full runs from openie, implie, and multir

Multir -

• inserted a PERLOC sentential pair generation approach before extraction, which eliminated LOC LOC extractions
• using t.stop (Thread Death) to kill the long-running threads, this seems to free-up (some) memory
• made a full round 1 run, including forum and ng docs, runtime = 28 hours

Implie -

• made a full round 1 run, including forum and ng docs, runtime = 26 hours

All - ToDo

• Need to remove the printing of NIL lines in the output (output does not validate with those)
• Need to change slot fill type of String to STRING
• Probably filter-out queries during round1 which contain round2 slots that we don't attempt (since only round2 answers are scored)
• Find a query and run through: round1, generate round2 queries, round2, combine, and score
• test t.isAlive() to explore another potential mod to the time-outs

Multir - ToDo

• Make a full run using the PERLOC sentential instance generation and no NIL lines output
• For curiosity, check if there are any slot0=perloc or locper slot, slot1=perloc or locper slot in the 2014 set,
  • these are the only ones which would generate results in a multir-only run

Implie - ToDo

• Make a full run using no NIL lines output
• Make another pass at checking results

Openie - ToDo

• Make a full run using no NIL lines output
• The existing full run is missing output for some of the queries (i.e. not even NIL there), why is that?

Scripts - ToDo

• Make another pass at checking results
• Possibly modify the rules for combining the results
  • how many keep?
  • fill perloc slots with multir first or only?

July 24, 2015

1. Extractions - Explored Jean Biden case - why no stateorprovince of death extraction

• sentence 1: "My mother, Catherine Eugenia 'Jean' Finnegan Biden, passed away peacefully

today at our home in Wilmington, Delaware, surrounded by her children, her grandchildren, her great-grandchildren and many loved ones," Biden said in a statement.

• sentence 2: she died in Wilmington

2. Machine - checking how to run jobs on machines other than kupo (serenity is busy with deepdive)

• on *cycle machines: identified memory available, java available, and made mods to sbtopts file
• successfully ran openie, implie, and multir jobs on bicycle and tricycle
• will need 1 machine for multir, 1 machine for implie, and might want another one for openie

3. Script - post-processing

• wrote a script to extract single-system-only-query round2 output from the combined-query run2 output
• need to test it

4. Test Runs -

• running round1 jobs for openie(tricycle), implie(bicycle), and multir(kupo)
• running round2 job for openie(tricycle)
• the combined round1 output from an earlier run was 469 round2 queries

5. Single PER Names -

• explored this a bit for Knechtel and Keflezighi
• for Knechtel, the provenance doc does not have Larry Knechtel in the coref chain, or anywhere in the doc
• for Keflezighi, the coref chain has a mention span with Meb Keflezighi and several other words, the nearby text does not

July 31, 2015

Single Names

• examined 20 from the openie 2014 round1 output, applied a coref-based method to those to assign a full name
• 11/20 get a correct full name assigned
• 7/20 are dropped - 3 of those are impossible to get from the provenance doc
• 1/20 get the wrong full name assigned

Machine Plan

• running test jobs on bicycle, kupo, lil, rv, nlp, and pardosa to compare speed and make a plan
• so, far: kupo (25 hrs) >> lil|rv|nlp (36-37 hours) > bicycle (40 hrs) >> pardosa (63 hrs)

All

• inserted SingleNameResolver (coref based method) to replace single query names with full name queries for round2
• script to extract single-system-query output from all-systems-query output -- wrote it, need to test it a little

Multir

• reviewed output, don't see support for raising the threshold

Openie

• reviewed output, appears to have high precision

Implie

• want to examine results for gpe: slots
• do we want to work a little more to try to clean-up the answers, eg. St. Petersburg General, as is: ORG = St. and PERSON = Petersburg General,

gpe:HQ_in_city(St.Petersburg) = St.

Aug 7, 2015

(see post on Dec. 5 2014, for locations of existing Solr indices)

• Steps to Index Openie Extractions

1. create a core, for example, /scratch/usr/nhawkins/solr/solr/cstuples, on rv-n16

2. /scratch/usr/nhawkins/solr/solr/cstuples/conf contains two files which have been customized to this task:

schema.xml and solrconfig.xml (I copied these from rbart's directory)

3. visit the Solr admin page, http://rv-n16.cs.washington.edu:8468/solr/

click on Core Admin, and add the core, cstuples

4. Populate the Index using the SolrPopulator from the tac2013/slotfiller project, or a copy of it

The tuples (extractions) must be added separately by type; i.e. web, news, or forum

1. Run command for tac2015 (openie) SolrPopulator
2. input = a directory of xml files to process
3. output = a populated solr index

run --inputPath /projects/WebWare6/KBP_2015/corpus/LDC2015E77_TAC_KBP_2015_English_Cold_Start_Evaluation_Source_Corpus/data/mpdf --inputRaw --corpus forum http://rv-n16.cs.washington.edu:8468/solr/cstuples

run --inputPath /projects/WebWare6/KBP_2015/corpus/LDC2015E77_TAC_KBP_2015_English_Cold_Start_Evaluation_Source_Corpus/data/nw --inputRaw --corpus news http://rv-n16.cs.washington.edu:8468/solr/cstuples

• Steps to Index Doc Strings

1. create a core, for example, /scratch/usr/nhawkins/solr/solr/csdocs, on rv-n16

2. /scratch/usr/nhawkins/solr/solr/csdocs/conf contains two files which have been customized to this task:

schema.xml and solrconfig.xml (I copied these from rbart's directory)

3. visit the Solr admin page, http://rv-n16.cs.washington.edu:8468/solr/

click on Core Admin, and add the core, csdocs

4. Populate the Index using the SolrDocumentPathXMLPopulator from the tac2013/slotfiller project, or a copy of it, such as the tac2015/slotfiller project

run /projects/WebWare6/KBP_2015/corpus/LDC2015E77_TAC_KBP_2015_English_Cold_Start_Evaluation_Source_Corpus/data/all http://rv-n16.cs.washington.edu:8468/solr/csdocs

The Solr Dashboard Admin Page for the Cold Start Solr Indexes

• http://rv-n16.cs.washington.edu:8468/solr/#/~cores/csdocs

Aug 14, 2015

• Cold Start

The organizers emailed saying that they are having trouble getting their scripts to work with the evaluation queries; they're working on it. I think this means that no one has received the evaluation queries yet. Hoa plans to email me the queries on Wed (8/12) and said that we could start whenever we want. They hoped to get their scripts fixed over the weekend, but someone sent an email today asking if it would take longer for the queries. There hasn't been a response.

• Indexing the Cold Start Corpus

I spent some time on Sat and Sun locating and reviewing the Solr code and setup (on rv-n16), and downloading and unzipping the corpus of ~50k docs to WebWare6; ~80% of the cold start corpus are truncated (length = ~800) discussion forum docs. Yesterday I indexed the openie tuples (extractions), and separately the docstrings. I made this corpus available to the openie and implie systems, and removed the code that used a file of doc names to select a cold start corpus from the existing corpus. I made a first round run using openie and separately implie on the set of 247 tac2014 queries.

• Round 1 Runs

openie took 10.5 mins and returned 16 slotfills.

implie took ~17 hrs and returned 76 slotfills (this included writing a trace file)

• Things to do:

-make the new Solr index of docstrings available to the multir system, remove the code relying on a list of doc names to specify the cold start corpus, make a multir round 1 run

-modify/review the script that combines the output from multiple systems to not rely on their being output from 3 systems

-write script to group queries having the same entity name

-write script(s) to run multiple jobs / modify the runPrograms to supply input to select a subset of queries to run

-make another implie run w/o the trace file

-review openie code where docs are annotated

-serialize annotation objects?

Aug 21, 2015

Hyak Cluster

• talked with Pramod Friday, applied for e-token for Hyak

Serialized Corpus

• serialized cold start corpus with corenlp 3.5 for implie
• previously serialized cold start corpus with corenlp 1.3.5 for multir

ENG* newswire docs (the troublemakers)

• removed first line, <xml, etc. from the 1874 ENG* docs
  • used: sed -i "1d" *
• re-populated the solr csdocs index and the solr cstuples index
• re-serializing these docs for multir and for implie
• re-created the Solr Indexes for the openie tuples and for the doc strings

Aug 28, 2015

KBP 2015:

• Ran Rounds 1 and 2
• Split the provenance strings which had offset spans > 150
• Extracted the single-system-query round2 output from the multi-system-query round2 output to create data for runs 2,3,and 4
• Packaged the round1 and round2 output for each of the 4 runs
• Power outage from weekend wind storm, couldn't access login information to complete submission

Sep 4, 2015

KBP 2015:

• August 31, successfully submitted the 4 runs
• Run Submissions located in: /projects/WebWare6/KBP_2015/submit
• Registration info: /projects/WebWare6/KBP_2015/misc

UIUC Multir questions:

• investigated and responded to their questions about Multir and Mitchell's paper
• Mitchell also responded

DEFT API Repository:

• requested user account

Sep 11, 2015

Labor Day + out

Sep 18, 2015

DEFT Adept Integration:

• conference call - Illinois and Cornell reported having numerous merging conflicts (~8k) when pulling the ADEPT-API v2.1

DEFT API Repository:

• received user account
• worked with Connor to resolve all but 5 conflicts by using a list of: git checkout --theirs|--ours filename, git add filename commands, followed by a git commit
• resolved the remaining 5 conflicts
• resolved 3 of those conflicts by adding 'process' and 'processAsync' methods which replace deprecated methods
• mvn compiled the washington code
• mvn compile does not work for all universities code -- issue popping up with cmucs-compile (may need or want to pursue this later)

Sep 25, 2015

DEFT API Repository:

• mvn tested the washington code
• located/reviewed the descriptions of existing uw algorithms (multir, ollie, and reverb)
• located docs describing the new uw algorithms (newsspike, uwtime, tractable markov logic)
• met with Congle -- learned where he and Kenton are wrt integrating their code into the DEFT API repository, learned details about his code
• proceeding with the steps to add the newsspike code to the repository, create the required tests, upload the large model files to the nexus repository, compile the code, test the code, push the code to the master, mvn package to create jars, and anything and everything else required that pops-up along the way
• see if we're ready to proceed with kenton's uwtime algorithm

DEFT API Repository Issues:

• created JIRA help ticket: cannot upload large model files (nsre and figer) to nexus repository (getting "unauthorized" error)
• newsspike requires java 1.8, will need to adjust to use java1.7/1.6
• newsspike requires mosek which requires a license
• newsspike does it require jars which will need to be put in a special place?

Oct 2, 2015

Wed - 1/2Thur: out

DEFT API Repository:

• uploaded the nsre and figer large model files (2 jars) to the nexus "DEFT Model Releases" directory (note: "DEFT Releases" dir is just for BBN employees)
• moved Newspike code into the repository, the repository compiles
• removed parts of the code not needed, inclu training code which requires a library which requires a license, the repository compiles
• started work to add regression and benchmark tests,
  • created NewsSpikeExtractor and NewsSpikeSentencePredict classes, which are needed for the tests
  • repository compiles
  • wrote NewsSpikeSentencePredict.predict(String text)
    • it consists of 3 function calls:
      • List<Tuple> tuples = new ArrayList<Tuple>();
      • List<FigerParsedSentence> ps = ParseStanfordFigerReverb.process(text);
      • for(FigerParsedSentence fps : ps){tuples.addAll(Parsed2Tuple.getReverbTuples(fps));}
      • List<RelationPrediction> predictions = NewsSpikePredict.predict(model_nsre, tuples);
  • This function was created by capturing and modifying the code in the main method:
    • ParseStanfordFigerReverb.process(String text)
  • This function is a modified version of another
    • NewsSpikePredict.predict(String model_nsre, List<Tuple> tuples)
  • repository compiles
  • issue: there is an issue with moving from "Parsed2Tuple.getReverbTuples(fps)" to "NewsSpikePredict.predict(model_nsre, tuples)"
    • checking this out, a little more data is needed than is being passed in tuples, or just need to find it w/i tuples
  • next: the regression test can be tested

Newsspike Web Demo:

• there is slotfiller webapp code in tac2013
• going to read about AngularJS

Oct 9, 2015

DEFT Repository:

• identified and located the "missing" data needed for NewsSpikePredict
• several required pieces are in place for Newsspike, but 1) the jar created can't be run as an app and 2)"mvn test -pl washington -am" command results in "success", but the regression test is not creating any output (or if it is, don't know where)
  • for issue 1: exploring the DEFT Help sections: Running the Sample App, Creating a New App
  • for issues 1 and 2: planning an email to Connor, for feedback
• created 2 JIRA DEFT bugs tickets (Connor responds to these)
  • a JIRA DEFT bugs ticket for the regression test questions
  • a JIRA DEFT bugs tickets for "deliverable?" question; i.e. do they expect a jar or an "app" or ?
• no response yet to JIRA DEFT tickets
• modified deft/washington/pom.xml to create WashingtonNewsSpikeExtractorApp files via "mvn package appassembler:assemble -pl washington"
  • the files were created
  • what are they? do they do anything?
• proceeding with the NewsspikeExtractorApp approach... (i.e. rather than a jar)
  • issue: NewsSpikeSentencePredict calls ParseStanfordFigerReverb which loads a config file, figer.conf, but it cannot find this file; the file is in the main/resources dir and in the washington-adept jar with all the washington code
• WashingtonNewsSpikeExtractorApp
  • REPO: /homes/gws/nhawkins/deft/washington/target/appassembler/lib
  • BASEDIR: /homes/gws/nhawkins/deft/washington/target/appassembler

Demo Web Apps:

• read about: html, bootstrap css, javascript, node, AngularJS
• checked out:
  • http://cogcomp.cs.illinois.edu/page/demos/ (several here)
  • http://homepages.inf.ed.ac.uk/s1049478/easyccg.html (click on demo link)

DEFT Jira:

• https://www.deftbugs.bbn.com:8443/

Oct 16, 2015

DEFT Repository:

• remade the large model file jars
  • the model files need to be in the same package as the class calling them, so that they can be read as a resource from a jar file
  • removed previous file: zip -d figermodel-1.jar figer.new.model.gz
• uploaded the 2 new large model file jars to the nexus repository
  • edu.washington: figermodel and nsremodel
  • requested (via Connor) that BBN delete the files at the 'washington' level; i.e. not the 'edu.washington' level
• got the "app" approach working
• regression test does not run because it accesses files (eg. config files, types map) from different locations than the "app"
• received feedback from BBN today re the repository
  • got clarification on what we need to do, what they do
  • learned of a change that they made recently to the washington repository, need to incorporate those changes
• contacted Kenton, got a link to his UWTime code
• responded to some of BBN's comments
• got a fresh clone of the washington repository (which BBN had adjusted)
  • created a newsspike branch
  • copied newsspike files into this branch
  • 'git status' looks funny, added comment to JIRA ticket, asking if it looks on-track

• todo
  • get regression test working and verify the output
  • contact BBN for 'approval'
  • examine benchmark test
  • study Kenton's code, prepare to add it to the repository

Web Demo:

• on rv-n16.cs.washington.edu:5000
  • installed javascript, angularjs, bootstrap css
  • wrote test html files and code to start a web server
  • success: can view a test html file, which uses angularjs
• todo
  • learn how to call newsspike code from the web page

TAC 2015 Workshop:

• registered, booked hotel + flights (10/15/15)

Oct 23, 2015

DEFT Repository:

• In response to a JIRA ticket question, BBN made an adjustment (10/20/2015) to our repository - to remove some adept files which are surprisingly there
  • submitted a follow-up question
• Responded to the BBN Doodle poll re proposed Hackathons; expressed a possible interest in a Dec 8 Hackathon
• Reviewing DEFT Code Delivery Process checklist released Tues (10/20/2015) (requires Regression Test, Benchmark Test, and Profiler)

• Regression Test:

• • Regression Test passes on the set of 10 WikiNews files
    • NewsSpike does not make any predictions on these files
    • DEFT data structure, HltContentContainer, has "passages" data, but no "relations" data
    • Checking if NewsSpike makes predictions on these files using code directly from Congle's github -- answer No
    • emailed Congle for an opinion on what extractions are expected from a modified Wikinews-003.txt test file
    • Congle says he would expect (at least) one additional extraction from the test file, he suggested looking at the features
    • Congle stopped by for 5 minutes, noted that Reverb returns difference results for the 2 similar sentences we have, so we would not expect an extraction from both of them
  • they provide a set of test files, but we may need to provide our own
  • Created our own test file for the regression test; it contains sentences which have 4 known NewsSpike extractions
    • the code in the DEFT repository also returns these 4 extractions
  • On an open JIRA ticket, submitted a new question and our progress on the Regression Test
  • Regression Test passes on the test file which has 4 extractions
    • the DEFT data structure, HltContentContainer data is correct for: relation, arguments, and confidence
    • the DEFT data structure, HltContentContainer data may not be correct for: context
  • todo:
    • explore where/how HltContentContainer.context data and such are set
    • create 9 more input/test files

• Benchmark Test, starting the process, following the instructions

• • Created 10 input files with known extractions
  • Benchmark Test code creates correct output and output.sh files
  • Benchmark Test code creates correct output.score files
  • Benchmark Test passes
  • todo: probably need a larger set of input files

• Profiler Test, starting the process, following the instructions

Web Demo:

• want to demo newsspike, figer, and uwtime
• we have very simple code, less simple code, and there are other options

Oct 30, 2015

• DEFT Repository:

• • NewsSpike
    • Profiler, compile via: mvn install -pl washington -am (success)
    • Profiler, run via: mvn exec:java -pl washington -Dexec.mainClass="edu.washington.nsre.WashingtonNewsSpikeExtractorProfiler" -Dexec.classpathScope="test" -Dexec.args="/homes/gws/nhawkins/bbn/deft/washington/src/test/resources/edu/washington/nsre/input.profiler /homes/gws/nhawkins/bbn/deft/washington/src/test/resources/edu/washington/nsre/output.profiler" (success)
      • required: set MAVEN_OPTS to something bigger than 6g, else java heap out-of-memory error when loading "figermodel"
      • submitted JIRA ticket: found bug in adept.utilities.AdeptProfiler.java; if # test files is < 9, get a / zero error because int summarySize = 0
      • JIRA notes: DEFT documentation needs update for 1)MAVEN_OPTS 2)add "-pl washington" to mvn exec:java command 3)fix bug above or require 9+ test files
      • SUCCESS: profiler test completes on 10 files with success.

• • • Summary:
      • All 3 tests - Regression, Benchmark, and Profiler - have completed with success (yahoo!)
      • Need some adjustments:
        • NewsSpikePredict -> token offsets are for a sentence, need to make it for doc, if possible, then adjust HltContentContainer's contents
          • adjusted this for a doc (as best as is possible), crashes on one test doc, index out of bounds, investigating
        • Regression Test -> added 9 more test files
        • Benchmark Test -> need to create a set of 1000 test files
      • Next: create a DEFT Jenkins job

• • UWTime
    • newsspike is like multir, uwtime is different
    • todo: identify DEFT API data structure for "time" info

• Web Demo:
  • Created an overall "NLP Demos" page in space pointed out by cs-support
    • UW NLP Demos: http://knowitall.cs.washington.edu/nlp/
  • UWTime, http://lil.cs.washington.edu/uwtime/
  • NewsSpike:
    • read about Play framework
    • consulting with Kenton re set-up
    • submitted a request to cs-support, asking about the web environment, where to put things
    • consulting with CS-Support re set-up

• UW Time:
  • Kenton's Code: https://bitbucket.org/kentonl/
  • Kenton's Demo: http://lil.cs.washington.edu/uwtime/
  • maven dependency block:
    • <dependency>
    • <groupId>edu.uw.cs.lil</groupId>
    • <artifactId>uwtime-standalone</artifactId>
    • <version>1.0.1</version>
    • </dependency>
  • usage:
    • see TemporalTest.java

• Machines in 330:
  • porvo and kupo, upgraded to Fedora22
  • cs-support investigating .bashrc login issue

• ADEPT API: https://www.deftdocs.bbn.com/API%20Specifications/adept-api-v2.1/index.html

Nov 6, 2015

DEFT Repository:

• Summary
  • NewsSpike - Jenkins job completes successfully up to Profiler step, where it runs out of memory, working with Connor(BBN)
  • UWTime - waiting to hear back from Connor|David(BBN)

• NewsSpike
  • git: added, committed, pushed files to newsspike branch
    • git branch -a
    • git add ...
    • git commit -m "adding newsspike"
    • git push
  • git: pulled DEFT changes to master branch
    • git checkout master
    • git pull
  • git: merged newsspike branch into master branch
    • git merge --no-ff newsspike
  • git: pushed master branch
    • git push origin master
  • Jenkins: created and submitted a test job for NewsSpike
    • results: build error, couldn't find test file lists in Jenkins job
    • submitted a JIRA bugs ticket, Connor(BBN) responded indicating he adjusted an underlying Jenkins template, which fixed it.
    • next problem: memory error, java heap space when running the profiler, working with Connor(BBN)
• UWTime
  • looking for a DEFT data structure; maybe HltContentContainer, Chunk, List of Time Phrases
  • submitted JIRA ticket, with questions, to start the process
• Algorithms Info for Jenkins configuration:
  • NewsSpike-E2 Event Extractor
    • Algorithm #: 1.4
  • Time Temporal Semantic Parser
    • Algorithm #: 1.6
  • Tractable Markov Logic 2.0 Probabilistic Inference Engine
    • Algorithm #: 1.5

Web Demo:

• name: nlp.cs.washington.edu, already being used for the nlp cluster
• software: installed play (= good for java/scala)
• created test demo
• test demo: is working; it has two text boxes (input, output), and is using the play framework on the backend
• added newsspike code to the project, the code is not compiling due to missing dependencies
• specified dependencies in Activator (a wrapper for sbt), using build.sbt
• more code is compiling, some is still not
• todo: sort out which additional dependencies are needed

KBP2015:

• investigated score reports to assist Stephen in revealing their content

Nov 13, 2015

Nov 11: Veteran's Day

Nov 12: UW-IT Cloud Day (presentations on AMZN AWS and EC2, Google Cloud, and MSFT Azure)

Nov 13: travel to 2015 TAC Workshop

DEFT Repository:

• Newsspike:
  • Jenkins job (regression, benchmark, and profiler, ~12hrs) finished with SUCCESS
  • submitted JIRA ticket to have code promoted; i.e. merged and integrated into the ADEPT repository from the washington repository

• UWTime:
  • waiting to hear back from Connor/David(BBN) re recommendations and options for integration

WebDemo:

• Newsspike:
  • integrating newsspike code into the play framework (back-end)
    • several files (config and models) are referenced from the code; referencing and finding these files varies on the environment, so adjustments are required
    • identified the necessary dependencies to get the code to compile; how those are available and where they live also depends on the environment
    • crash: investigating: need to adjust classpath order: two files are duplicated in the code set, and are being checked in the "wrong" order
      • crashes, if start server this way: activator run
      • doesn't crash, if start server this way: activator start
        • proceeding with this for now

Nov 20, 2015

Nov 16-18: 2015 TAC Workshop + travel

TAC Workshop 2015:

• reflecting, examining materials

DEFT Repository:

• received info about adding uwtime

Web Demo:

• newsspike
  • debug: not getting tuples for sentence which should have an extraction
  • debug: reverb not extracting...

Nov 27, 2015

Nov 25-27: Thanksgiving

DEFT Repository:

• created a new branch for uwtime
  • git checkout -b uwtime
  • git push origin uwtime

• see all branches
  • git branch -a
• see local branches
  • git branch

• adding uwtime source code to uwtime branch
  • submitted a follow-up JIRA ticket regarding the DEFT data structures to use
  • they may need to modify their HltContentContainer class to hold a list of TemporalResolution's
• compile the uw code using java 1.6
  • mvn compile -pl washington -am
• compile the uw code using java 1.7
  • mvn compile -P java7 -pl washington -am
• the code is not compiling with java 1.6, the default, or java 1.7
  • the code needs java 1.8 in places where lambda expressions are used
  • submitted a JIRA ticket to clarify how to resolve this

Web Demo:

• reverb: not getting expected extraction from the test sentence

.

Dec 4, 2015

DEFT Repository:

• uwtime - waiting to hear back from BBN, re java8 and data structures
• newsspike -
  • finished the required steps, BBN has to complete their process
  • responded to a new JIRA bugs question from Connor
    • checked benchmark test results from master branch
      • mvn verify -DskipBenchmarkTest=false -pl washington -am
  • Connor replied saying that BBN has finished their process, and he will integrate newsspike into the adept master branch

Web Demo:

• we're not getting any extractions
• two test sentences which should have event extractions, do not have any reverb extractions, and so no newsspike extractions
• sent an email to Congle, showing input data leading up to a call to the reverb extractor, to see if it looks ok, and to see if he has ideas
• going through play 2.4 documentation

.

Dec 11, 2015

Out: Dec 9

DEFT Repository

• overall - David (bbn) submitted a question about ChunkedSentence.java, which appears twice in the repository
  • the two versions are not compatible
  • one occurs in compiled form in a reverb.jar
  • the other occurs in the newsspike code
  • the reverb regression test is failing, as a result
  • added a function, getRange(), to the newsspike version of ChunkedSentence to make it compatible
    • successfully ran the regression tests for newsspike, multir, ollie, and reverb
    • pushed the modified file to the deft washington repository
    • commented on the JIRA ticket to inform David (bbn)
• uwtime - waiting to hear back from BBN, re java8 and data structures

Web Demo

• ISSUE 0: newsspike extractions: we are now getting newsspike extractions!
  • SOLVED THIS ONE
  • fix: using reverb-core-1.4.1 and a couple of new resolvers in build.sbt, rather than using reverb-core-1.4.3, which produced no reverb extractions
    • we were not getting any newsspike extractions because we were not getting any reverb extractions
    • trying to track down which version of reverb was used
    • Congle pointed me to Tony's github, where he got a reverb jar
    • the reverb jar cannot be plugged in directly with the web demo; tried, it has conflicting parts (guava)
    • have been working with reverb-core 1.4.3
    • tried working with reverb-core 1.4.1 or 1.4.2, getting an error about dependency "jwnl-jwnl-1.3.3" not found
    • At the line below in ReVerbExtractorWrap, no extractions are being returned by the call to reverb.extract
      • for (ChunkedBinaryExtraction extr : reverb.extract(rvr.chunk_sent)) {
      • the input rvr.chunk_sent looks ok, and the project compiles
    • emailed Congle again, sharing info that I tried 1.4.1 and 1.4.2, and asked about an implementation of relExtr.extract

• ISSUE 1: how to remove "(implicit messages: Messages)" from being printed to index.html
  • SOLVED THIS ONE
    • removed "@()(implicit messages: Messages)" from main.scala.html
    • it was extra since index.scala.html imports main.scala.html
    • "(implicit messages: Messages)" is code for the view pages; this code is required to use play 2.4's dependency injection feature, which is new and everyone is encouraged to adopt
    • this code is necessary for the pages to work, but it also is being printed to the index/demo page, which is not intended or desired
    • investigating how to stop it from printing itself to the page, while still using it
    • reviewing instructions and code, to make sure they match

• ISSUE 2: how to start the application? there are two ways: "activator start" (deprecated, in future removed) and "activator run"
  • MAY NOT ABSOLUTELY NEED TO SOLVE THIS
  • "activator start" works, but since it's deprecated, seems we would want to use "activator run"
  • "activator run" does not work; get an error msg that "public ChunkedSentence chunkSentence(String[] tokens, String[] posTags)" method not found
    • this is probably because the project contains two copies: 1)one in reverb-core-1.4.1 and 2)a newer version in /src/main/java/edu/...
    • the newer version contains this method, the reverb-core version does not
    • in the activator console: "activator> show full-classpath" shows a classpath with an order which should provide the newer version of the class first
    • that classpath is probably used by "start" but not by "run"
    • investigating what might be able to do to get "activator run" to work

• ISSUE 3: what to present as an extraction? how to present it? what and how to offer as samples?
  • ENHANCEMENTS
  • need to do, at least a little, more work on this
  • samples - will use Congle's test sentences
  • maybe incorporate angularjs with play for front-end

• ISSUE 4: how to set the url to something other than "localhost:9000" or "porvo.cs.washington.edu:9000"?
  • LOCATION
  • will eventually put this code in the "cs" place for it
  • cs-support will give us a name, which we request
    • nlp.cs.washington.edu is already being used by the cluster

.

Dec 18, 2015

DEFT Conference Call:

• David(BBN) said:
  • at recent BBN site visit, Boyan requested that all sites show that they have used the KB api, made an attempt at inserting info into the KB
  • they will release a patch for the ADEPT v2.2 release which relates to the KB api
• Marjorie(BBN) email to all:
  • mentioned a possible KB Hackathon just after the next PI mtg; I emailed saying I might be interested, and asked the date. Sounds like maybe Feb?

DEFT Repository:

• uwtime: can move forward
  • submitted a comment to the Bugs JIRA ticket indicating that we're (still) waiting to hear back
  • BBN responded
    • issue 1) java8: it's ok to use java8 now
    • issue 2) data structures: they added a data field for TemporalResolutions to the HltContentContainer, for us to use
  • Attempted the steps to pull updates from the recent Adept tag v2.2 release
    • several conflicts need to be resolved (this is always part of the process, looks better though than in August)
    • skipping this for now
  • Pulling updates from the Adept 'staging' branch to work with the modified HltContentContainer (as suggested by Connor(BBN)
    • resolved the conflicts
    • starting to integrate
• newsspike: new things to do
  • Connor(BBN):
    • says it looks like the algorithm is working well
    • requested that we modify how we set the Type fields in Relation and Argument
    • asked about offsets for Chunk. This might be an issue with their TokenStream setting of CharOffsets
    • Investigating.

Web Demo:

• newsspike:
  • viewable at; porvo.cs.washington.edu:9000 (when I am running it, which I was, but am not right now)
• emailed cs-support asking about availability of names
  • not available: nlp.cs.washington.edu = taken by the cluster
  • could be available: ai.cs.washington.edu,
    • asked cs-support if this can be re-assigned to us, since it *might* not have much of a use right now
    • answer: ai.cs.washington.edu is being re-directed to https://www.cs.washington.edu/research/ai/, it is owned by Dan Weld and the other ai researchers, if they want to re-assign it, ok, it's hard to predict if anyone will miss its current re-direction
  • available: ai-nlp.cs.washington.edu, open-nlp.cs.washington.edu, ml-nlp.cs.washington.edu
• nslookup <domain.name>, can be used to check availability of names
  • eg. nslookup ai.cs.washington.edu returns an address, so it's currently assigned
    • Name: ai.cs.washington.edu
    • Address: 128.208.3.70
• will explore putting the demo on Amazon, rather than in /cse/www2/...
  • read about AWS Elastic Beanstalk and Docker, which uses AWS EC2
  • need to look into AWS LAMBDA

.

Dec 25, 2015

out: Dec 22-25

DEFT links:

• DEFT JIRA Bugs: https://www.deftbugs.bbn.com:8443/
• DEFT Docs: https://www.deftdocs.bbn.com/SitePages/Home.aspx
• DEFT api: https://www.deftdocs.bbn.com/API%20Specifications/adept-api-v2.1/index.html
• DEFT UW repository: http://www.deftware.bbn.com:7990/login?next=/projects/WASHINGTON/repos/deft/browse

DEFT Repository:

• uwtime
  • working with a new clone of the deft repository
  • got the staging branch, which has the HltContentContainer modified for uwtime
  • resolved the conflicts
  • created a uwtime branch
  • added the uwtime code
  • updated the deft/pom.xml to include java8
  • tried to compile with java8
    • got failure error: [WARNING] The requested profile "java8" could not be activated because it does not exist.
    • submitted a JIRA bugs ticket

.

Jan 1, 2016

out: Dec 28-Jan 1

.

Jan 8, 2016

DEFT

• newsspike
  • investigating Connor's(bbn) 2 request/comments:
    • 1) setting Relation Type and Argument Type to different values
      • previously the values were the relation and the arg names, respectively, as was done for multir
      • requested: set the values equal to the components of the relation
        • for example, for the relation, leave@/person@/organization
        • requested that: relation type = leave, arg1 type = person and arg2 type = organization
        • rather than:
        • relation type = leave@/person@/organization
        • arg1 type = 18-year-old Dalton Hayes and 13-year-old Cheyenne Phillips
        • arg2 type = a South Carolina Wal-Mart
      • RESOLVED: this change has been made in the code
    • 2)For one regression test file (not all of them), the token char offsets (begin and end) are 1 to the left, so the doc substring representing the Chunk value starts with a space and ends one char short
      • why is this the case for one file?
        • answer: looks like, because one sentence contained a period ("Fla.") followed by a line break
        • removing the line break in the input file, "solves" the problem
        • need to resolve
  • for the 2 issues: will need to report to Connor, and proceed with next steps to test and update the uw repo
• uwtime
  • debugging with bbn how to compile with java8
    • submitted new JIRA bugs ticket, the previous bbn suggestion didn't work
    • bbn responded, (appear to be) compiling with java8 now, yay!!!
  • now that using java8, need to get the added uwtime code to compile
    • turns out, the uwtime code depends on a jar, unlike the newsspike code
    • submitted JIRA request asking for an opinion on the best way to proceed
    • likely to proceed by extracting the edu/uw/cs code from the jar and adding it directly to the project, and going from there
      • the jar has other, non-uw, code -- will possibly need to and be able to specify that as dependencies
      • will have to take it a step at a time
      • notes:
        • jar xf spf-1.5.5
        • cp -r uwtime/edu uwtime-src
        • find . -name "*.class" -type f -delete
        • cp -r tiny /homes/gws/nhawkins/adept/deft/washington/src/main/java/edu/uw/cs/lil
        • cd /homes/gws/nhawkins/test/uwtime-src/edu/uw/cs
        • cp -r utils /homes/gws/nhawkins/adept/deft/washington/src/main/java/edu/uw/cs
  • then, need to progress with regression and benchmark tests
• DEFT kb
  • reviewed kb documentation/how-to-use kb api
  • emailed Marjorie(bbn) about the kb connection information; if we need more info, Dan might receive it
  • proceeding to try it out
    • first need to get a fresh copy of the uw repo, pulling the adept v2.2.1 updates, which contain the kb patch(es)
    • fetching the adept v2.2.1 updates results in merge conflicts, as usual, which need to be resolved
    • resolved conflicts with combination of: 1)git checkout --theirs <filename> + git add <filename> 2)git rm <filename> 3)git add <filename>
      • mvn compile -pl washington -am, results in BUILD SUCCESS
    • built adept-kb project with its dependencies
      • mvn install -pl :adept-kb -am
      • i.e. result = BUILD SUCCESS
    • unit tests failed
      • mvn verify -pl :adept-kb -DskipUnitTests=false
      • mvn test -pl :adept-kb -DskipUnitTests=false
      • results: No metadata URL in KBParameters.
      • need to set parameters

Web Demo

• aws lambda process
  • free for: 1 mil requests/mo and 3.2 mil seconds/mo (need to check current numbers)
  • ram: limited to 1GB?
  • fcn: what is the function?
    • Application.scala?
    • Submit: newsspike.predict?
• aws elastic beanstalk
  • play example in cookbook
• play
  • activator: start vs. run, was worrying about this
  • doc indicates, https://www.playframework.com/documentation/2.1.x/Production
    • start - use for production mode (though, when use it, warnings appear about future deprecation)
    • run - use for development mode only
    • alternative approach:
      • generate start script: play clean compile stage
      • run start script: target/start, or execute java command
  • activator start -Dhttp.port
    • Note: the HTTP port can be set by passing -Dhttp.port system variable
• domain name
  • available if willing to re-assign: ai.cs.washington.edu
  • appears available: ai.washington.edu
  • not available because it is currently being developed by Dan Boren for 14 nlp uw faculty, but should student nlp demos link from here?: nlp.washington.edu

.

Jan 15, 2016

DEFT

• kb
  • ------------------------------------------------------------------------------------------
  • end of last week,
    • pulled adept v2.2.1 updates which contain adept-kb patches
    • built adept-kb module
  • ------------------------------------------------------------------------------------------
  • David Kolas (bbn) sent metadata postgres password, yay!
  • modified KBParameters config files with info from KBInstances table and the metadata pg pw
    • deft/adept/adept-kb/src/main/resources/adept/kbapi/KBParameters.xml
    • deft/adept/adept-kb/src/test/resources/adpet/kbapi/unittests/KBUnitTestParameters.xml
  • rebuilt adept-kb module, from root dir; i.e. deft/
    • mvn install -pl :adept-kb -am
  • ---------------------------------------------------------
  • ran unit tests, which passed
  • ---------------------------------------------------------
    • mvn verify -pl :adept-kb -DskipUnitTests=false
    • Results = Tests run: 37, Failures: 0, Errors: 0, Skipped: 0
  • ---------------------------------------------------------
  • confirmed remote triple store connection
  • ---------------------------------------------------------
    • mvn clean install -pl :adept-kb -am
    • mvn exec:java -pl :adept-kb -Dexec.mainClass="adept.utilities.ConfirmRemoteTripleStoreConnection"
    • results = success
      • Testing connection to remote triple store at http://deft-kb-dmz-01.bbn.com:7606/dataset
      • log4j:WARN No appenders could be found for logger (com.hp.hpl.jena.sparql.mgt.ARQMgt).
      • log4j:WARN Please initialize the log4j system properly.
      • log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
      • Successfully inserted triple with id: 54f2ff54-2290-4385-8c89-d1acfab3ea01
      • Confirmed inference is turned on in remote triple store.
      • Successfully deleted test triple.
      • Confirmation complete.
  • --------------------------------------------------------------
  • confirmed metadata sql database connection
  • --------------------------------------------------------------
    • mvn clean install -pl :adept-kb -am
    • mvn exec:java -pl :adept-kb -Dexec.mainClass="adept.utilities.ConfirmSqlDatabaseConnection"
    • results = success
      • Testing sql database connection: jdbc:postgresql://deft-kb-dmz-01.bbn.com:7706/postgres
      • Inserting test corpus with id ac92d2eb-bb4e-43f6-a1e3-2459582b994a
      • Checking for existence of inserted corpus entry...
      • Corpus entry exists.
      • Deleting test corpus entry.
      • Successfully deleted test corpus entry.
      • Confirmation complete.
  • --------------------------------------------------------------
  • created test file: edu.washington.kb.KBTest.java
  • --------------------------------------------------------------
    • can run this file from: deft/adept/adept-kb/uw
    • cannot run this file from: deft/washington/ -> emailed Dave Kolas about that
      • answer: add dependency to deft/washington/pom.xml; i.e.
        • <dependency>
        • <groupId>adept</groupId>
        • <artifactId>adept-kb</artifactId>
        • <version>${project.parent.version}</version>
        • </dependency>
    • -------------------
    • successes
    • -------------------
      • accessed and printed KBParameters (triplestore url, metadata url)
      • executed query against the KB to determine how many "Barack Obama" entities there are
        • answer: 0
      • executed query against the KB to determine how many "born" relations there are
        • answer: 0
    • NEXT: trying to insert an entity; by creating document/token stream info from scratch and by creating an Entity by scratch, rather than from running an algorithm
      • success: created a document/token stream from scratch
        • doc = "Barack Obama was born in Honolulu, Hawaii.";
      • NEXT: dealing with the java.lang.NullPointerException from
        • //insert Barack Obama entity
        • KBEntity.InsertionBuilder insertionBuilder =
        • KBEntity.entityInsertionBuilder(entityBO, entityMentionsBO, KBOntologyMap.getTACOntologyMap());
        • first error resolved: entity CanonicalMention was null, so set it
        • next error: java.lang.IllegalArgumentException: Ontology map must have mapping for the Entity type.
          • resolved: set entity Type to Type("per")
        • next error java.lang.NullPointerException: KBTextProvenance.InsertionBuilder textProvenance = KBTextProvenance.builder(entityMention,entityMention.getConfidence(entityId));
          • resolved: set entityMention confidence
      • ----------------------------------------------------------------------------------
      • success: inserted "Barack Obama" entity into washington KB
      • ----------------------------------------------------------------------------------
        • run 1 output:
        • No BO entities in the KB.
        • Creating insertion builder
        • KBEntity Preconditions Check:
        • KBEntity Preconditions Check: entity Not Null
        • KBEntity Preconditions Check: entityMentions Not Null
        • KBEntity Preconditions Check: entityMentions Not Empty
        • KBEntity Preconditions Check: entity CanonicalMention Not Null
        • KBEntity Preconditions Check: ontologyMap Not Null
        • Inserting BO entity
        • Inserted BO entity in the KB.
        • run 2 output:
        • Number of BO entities: 1
• newsspike
  • investigating bug, raised by Connor (bbn)
    • this is looking like an issue with the adept code which processes a document, tokenizes it, and assigns character offsets to each token.
    • we have a sentence which was inadvertantly separated into two lines in the test document, so the sentence is being tokenized in two parts
      • the first half of the sentence ends in a "." followed by a space followed by a carriage return
      • the "/r" does not get tokenized, or recogized when the offset values are assigned
    • checking a few more things and will submit new comments to the JIRA ticket
    • NOTES:
    • sentence:
      • -LRB- AP Photo\\/Wal-Mart Inc. via The Grayson County , Ky. , Sheriff 's Office -RRB- In this January 2015 photo made from surveillance video and released by the Grayson County Sheriff 's Office , in Kentucky , 18-year-old Dalton Hayes and 13-year-old Cheyenne Phillips leave a South Carolina Wal-Mart .
    • token 36: -LRB-
      • In Chunk class, token 36: -LRB-
      • In Chunk class, token 36 beg char offset: 206
      • In Chunk class, token 36 end char offset: 211
    • docValue (0-1140, length = 1141)
      • docValue.substring(206,207) = a line break
      • docValue.substring(207,208) = "-"
    • ==> this means that the first token in the sentence, -LRB-, has a char start offset of 206, which is the position of the line break
    • ==> the begin offset for token 36 = -LRB- should be 207
    • ==> the begin and end char offsets for all tokens in this sentence are shifted to the left by one, which is why we see a right-hand side chop by 1 too short in the doc substring assigned to the Chunk
    • ==> these offsets are being set by the adept.utilities.DocumentMaker code
    • ==> adept.utilities.DocumentMaker calls adept.io.Reader to read the rawtext doc into lines
  • CONFIRMED: issue is in
    • adept.utilities.DocumentMaker.createDefaultDocument
      • public Document createDefaultDocument(String docId,
      • Corpus corpus,
      • String docType,
      • String uri,
      • String language,
      • String filename,
      • HltContentContainer hltContentContainer,
      • TokenizerType tokenizerType) {
      • in the section, where a raw text file is being read
      • } else {
      • System.out.println("Loading input document as raw text file");
      • This regression test file, NewsSpike-Test-003.txt, has 5 lines and so 5 "passages" in this code
      • The first line ends with a period followed by a space followed by a /r
      • The lines are tokenized individually, the last token is the period (".") after "Fla.",
      • The "." token starts at 204 and ends at 205 (the space).
      • The code sets the doc-level "start" for the next line as 205 + 1 = 206, i.e.
        • if ( pCharEnd > 0 ) {
        • docCharStart = pCharEnd+1;
      • however, 206 is the position of the /r, not the position of the first char in the next token, "-LRB-"
      • the /r is not a token
      • this starts the offsets to be off-by-1 to the left for all remaining tokens in this document
  • ----------------------------------------------------------------------------------------------------------------------------------------------------
  • RESOLUTION: submitted these findings in a JIRA ticket. There is not a bug in the newsspike portion of the code.
  • ----------------------------------------------------------------------------------------------------------------------------------------------------
• uwtime
  • • David (bbn) responded to the earlier questions re a uwtime dependency jar
      • he put a dependency jar in a deft location
    • can move forward

.

Jan 22, 2016

Jan. 18 = MLK, Jr Day

DEFT Repository

• uwtime - https://lil.cs.washington.edu/uwtime/
  • --------------------
  • compiles
  • --------------------
  • compiles, yay! (so java8 ok, new reduced uwtime spf dependency jar installed at DEFT location ok, and uwtime source code ok)
  • working on UWTimeResolver.process(Document document) method
  • -------------------------------
  • regression tests
  • -------------------------------
  • simultaneously working on Regression Tests
    • created UWTimeResolverRegressionTest.java and UWTimeResolverConfig.xml
    • added missing resource files to deft/washington/src/main/resources - /lexicon and /models
    • switched the extract method being called to: extract(text) from extract(text, "", TemporalDomain.OTHER)
    • reads the test file w/3 sentences correctly
    • successfully creates a TemporalAnnotator for getting extractions
    • extractions: getting 5 for test doc
  • working on:
    • getting token offsets correct for TimePhrases value
      • token offsets are looking good
      • the adept code is not getting correct "values" using their char offsets for the tokens, however
      • will mention this on the uwtime JIRA ticket
      • with TemporalResolutions it might not matter as much since the TimexValue stores the time mention
  • issue: for some reason, can't use new feature = HltContentContainer.setTemporalResolutions(temporalResolutions)
    • the modified HltContentContainer class is in deft/adept/adept-api/src/main/java/adept/common, but it isn't being used
    • submitted a question/comment to the uwtime JIRA ticket
      • made contact with bbn - first suggestion doesn't work, submitted new comment to JIRA ticket
      • new response, requires more communication, submitted new comment to JIRA ticket
      • will probably need to get a fresh clone of the repository
        • previously, the suggestion was to "pull staging"
        • this has transitioned to a suggestion to "pull a numbered branch"
        • all this is a result of them needing/wanting to update their HltContentContainer to contain a List<TemporalResolution>, which is an evolved idea of their's, not part of the original "time" classes they made to support UWTime - we're "trying to/needing to" get an update from them, before they have made an official "tagged" update, which makes the process more complicated
  • -------------------------------
  • benchmark tests
  • -------------------------------
  • updated washington/pom.xml to include uwtime benchmark test
  • created UWTimeResolverBenchmarkTest.java, compiles
  • created WashingtonUWTimeResolverBenchmarkTestFileList.txt
  • created WashingtonUWTimeResolverBenchmarkTestConfig.xml
  • created input files
  • updated score.py for uwtime
  • working on:
    • need to revisit UWTimeResolverBenchmarkTest.java to complete "doScore(Document document, HltContentContainer hltContentContainer)" method
  • next: try a run
  • ------------------------------
  • profiler
  • ------------------------------
  • created UWTimeResolverProfiler.java, compiles
  • created WashingtonUWTimeResolverProfilerConfig.xml
  • next: try a run
  • -----------------------------------------
  • run Jenkins Test Suite job
  • -----------------------------------------
  • next

.

Jan 29, 2016

DEFT Repository

• -----------
• uwtime
• -----------
  • repository clone
    • going to get a fresh clone (to gain access to the "temporal" changes to the adept HltContentContainer) and copy uwtime files to it
      • bbn's suggested clone step is giving an error: Permission denied
        • submitted comment to the JIRA ticket
      • bbn suggested new clone step, which worked
      • copied uwtime files to new clone
      • uwtime code compiles, can now access the HltContentContainer's List<TemporalResolution>'s, yay!
  • continuing work on
    • -------------------------
    • regression tests
    • -------------------------
      • completed run based on 1 file; process works
      • created larger set of test files
      • need to: explore the adept charOffsets for the timeMention, which are not correct for the tokens; the charOffsets come from adept, not uwtime, code
        • need to take a detailed walk through adept.common.TimePhrase to see why the string value does not match the TimexValue string value
          • resolved: line breaks in mid-sentence in the test file are causing the offsets to be off, so removed those
        • the token number is correct in the token stream, but the chars being extracted are not correct, does not match the token.getValue()
          • resolved: uwtime creates different tokens sometimes than adept.common, so using doc string char offsets instead of token numbers
      • ran the larger set of files
        • decision on timeMention charOffsets issue will impact these tests
        • issues with 2 files, as discovered when running the profiler (below)
          • issue 1: adept.common.TokenStream.findPhraseTokenOffset method does not work when phrase end word repeated in phrase
            • resolved: wrote a new method to get an adept token number from a doc string char offset
          • issue 2: uwtime computes a different number of tokens for a sentence than does adept.utilities.DocumentMaker
            • resolved: instead of using token numbers, used doc string char offset and function to find adept token number from that offset
      • need to: run full set of files after the fixes, check that results are ok, copy over the correct outputs to the reference directory, ensure all pass before creating Jenkins job
      • success: full set of test files pass
    • --------------------------
    • benchmark tests
    • --------------------------
      • completed first pass of "doScore" method
      • completed run based on 1 file; got match of output with reference
      • completed run on larger set of files
      • need to: run again, make sure works before creating Jenkins job
      • results: SUCCESS
    • ----------------
    • profiler
    • ----------------
      • compiled the profiler:
        • mvn install -pl washington -am
      • ran the profiler:
        • mvn exec:java -pl washington -Dexec.mainClass="edu.washington.uwtime.WashingtonUWTimeResolverProfiler" -Dexec.classpathScope="test" -Dexec.args="/homes/gws/nhawkins/repo/deft/washington/src/test/resources/edu/washington/uwtime/input.profiler /homes/gws/nhawkins/repo/deft/washington/src/test/resources/edu/washington/uwtime/output.profiler"
      • ran profiler on a larger set of files
      • run results: need to explore null pointer and index out of bounds
        • findings:
          • issue 1: null pointer results from adept.common.TokenStream.findPhraseTokenOffset: end token (=November) < begin token because November is in the sentence twice, need to use a different method to determine the begin and end tokens
          • issue 2: uwtime and adept.utilities.DocumentMaker compute a different number of tokens for a particular sentence
            • eg. 48-hour period is two tokens (i.e. 48-hour and period) in adept.common.TokenStream and is 3 tokens in uwtime TemporalSentence.getTokens() or .getNumTokens() (i.e. 48, hour, and period)
          • resolved both issues: profiler runs with no failures, after changes mentioned in regression test section
      • ran on full set of test files:
        • results:
        • Number of failures: 0
        • [INFO] ------------------------------------------------------------------------
        • [INFO] BUILD SUCCESS
        • [INFO] ------------------------------------------------------------------------
        • output check:
          • looks good: document_stats.txt
          • looks good: test_summary.txt
    • ---------------------------
    • Jenkins job
    • ---------------------------
    • final step: create a Jenkins job and run it
    • pre-reqs satisfied:
      • success: regression tests pass on full set of test files
      • success: benchmark tests
      • 0 failures: profiler
    • can proceed to:
      • run regression and benchmark tests for all uw algorithms, to ensure nothing broke
        • regression tests: mvn verify -DskipRegressionTest=false -pl washington -am
          • results: all pass -> uwtime, newsspike, multir, ollie, and reverb
        • benchmark tests: mvn verify -DskipBenchmarkTest=false -pl washington -am
          • results:
            • uwtime SUCCESS! (1.0/1.0)
            • newsspike SUCCESS! (1.0/0.9)
            • multir:
              • SUCCESS! Processed 1219 files in 115 seconds.
              • Benchmark Results:
              • Precision = 0.7682926829268293
              • Recall = 0.47368421052631576
      • "git add, commit, and push" new uwtime files to uwtime branch of washington repo
        • success. done.
      • review Jenkins job procedures
        • as part of review, check if need to do these steps now, or if can run Jenkins off uwtime branch of the washington repo
          • locally merge uwtime branch with master branch
          • push files to master branch of washington repo
      • -----------------------------------------------------------
      • merged uwtime branch into master branch
      • -----------------------------------------------------------
        • From repo/deft>
        • 1. git checkout master
        • 2. git pull
        • 3. git merge uwtime
        • 4. git push origin master
      • set-up Jenkins job
        • done: configured job parameters, saved job
      • ---------------------
      • run Jenkins job
      • ---------------------
        • started job... failed: compilation errors ... investigating
        • need to: discover what the errors are, fix the errors (or have bbn fix them), and re-start job
        • submitted a comment to the JIRA ticket, to check if Jenkins is using java8 and for input on discovering what the errors actually are
• ---------------
• newsspike
• ---------------
  • haven't heard back from bbn since last comment submitted to JIRA ticket
  • will submit new comment to see how they would like to proceed
  • 16/958 files resulted in exceptions in the Jenkins job; they were gracefully handled, so the algorithm can be integrated despite these; however, since I have to push a modified file for newsspike related to a suggested change by Connor(bbn), I am taking a look at this to see if I should add any changes related to these exceptions; the first exception involves converting a string to a Double

.

Feb 5, 2016

DEFT Repository

• kb
  • reviewing the documentation and my KBExample.java code, to assist with PI mtg pres
  • reviewing the adept-kb unit tests to see how they process inserting an entity
  • reviewing the Profiler for newsspike to see how documents are accessed to run against the algorithm
• newsspike
  • bbn responded
  • todo:
    • a Jenkins job is currently running; when it finishes, can check exceptions again
      • bbn is adjusting the Jenkins job, as it fails - types of failures = 1)needs java8 2)out of heap space 3)X11 graphics issue for the profiler
      • a job hasn't successfully completed yet
    • when a Jenkins job has resulted in success, can check on the 16/958 files causing exceptions
      • fixes are not required for integration, since they gracefully exit, however
      • discovered a fix that could be made to fix the first exception, will make that change
      • check the other exceptions to see if they have the same cause, or if not, see if their cause can be fixed too
    • copy the argument Type code changes to the current repo
      • done
    • run the regression and benchmark tests for all algs to make sure nothing broke
      • waiting for above
    • push the code, and
      • waiting for above
    • run a new Jenkins job
      • waiting for above
• uwtime
  • Jenkins job is now running (takes awhile)
    • the job needed a modification to: 1)use java8 2)fix X11 graphics issue for the profiler
    • this job has not yet finished with success
    • when job ends, will see if need to make any additional adjustments
• next-algorithm
  • should I add another algorithm?

Web Demo

• next: continue AWS exploration for newsspike demo
  • lambda vs elastic beanstalk
    • lambda appeared to have a memory limit of 1GB; is that true? is it a problem for newsspike?

.

Feb 12, 2016

OUT = Monday, Tuesday

DEFT Repository

• kb
  • met with Colin to give intro to DEFT ADEPT KB
  • put reference files in /projects/WebWare6/KBP_2016/kb
  • To Run Sample Code = KBExample.java (inserts "Barack Obama" entity)
    • add adept-kb dependency to washington/pom.xml
      • <dependency>
      • <groupId>adept</groupId>
      • <artifactId>adept-kb</artifactId>
      • <version>${project.parent.version}</version>
      • </dependency>
    • compile washington code: mvn install -pl washington -am
    • run KBExample: mvn exec:java -pl washington -Dexec.mainClass="edu.washington.kb.KBExample"
• uwtime
  • submitted JIRA comment asking their (bbn) thoughts on the X11 error holding up the Jenkins job
  • bbn responded, made a fix, restarted Jenkins job, which is running...
  • Jenkins job finished successfully!
  • created a new JIRA issue requesting that uwtime be "promoted" (integrated)
  • submitted a comment to existing JIRA ticket, asking bbn about a msg of "1 failure" in the Jenkins job console output; i.e. can we see any details about that?
    • if there is something there that can or needs to be fixed, will pursue it
    • bbn responded:
      • there was 1 failure out of 958 files for the deft profiler;
      • the failure was in stanford core nlp, and the mem usage was extreme
      • this does not block integration; there are no changes we need to make
  • examining document reference date -
    • should a change be made to UWTimeResolver.java?, how it handles the case of no doc date? Investigating...
• newsspike
  • submitted JIRA comment asking their (bbn) thoughts on the X11 error holding up the Jenkins job
    • re-starting job, since bbn responded, and made a fix
    • status: job is pending, waiting next available executor
    • job has started running...
      • when job finishes, will look if there are any "exceptions" that could be addressed
      • if none, will push existing changes and start a new Jenkins job
  • want to push some files, so going to run the full set of tests, then push, wait for Jenkins job first
    • mvn verify -pl washington -am -DskipRegressionTest=false
    • mvn verify -pl washington -am -DskipBenchmarkTest=false
    • Benchmark Test questions:
      • nsre - checking for carriage return in input file
        • resolved, will push new inputFile006.txt reference file
      • ollie, reverb - the output for both of these is: 0 precision and 0 recall
        • comparing input and output files from previous push and current (result: match)
        • not sure what the precision and recall were before adding newsspike, probably 0 and 0
        • the output after adding uwtime is the same as it was for newsspike
        • the .out files have fewer and different extractions than the .ref files (so not matching, resulting in the 0,0 precision,recall)
        • the regression tests pass, which means the reverb and ollie being run (version) are the same as were run when the tests were established
        • the regression test reference files do have Relations stored in the HltContentContainer, so extractions are being made
        • maybe a version change occurred for reverb and ollie before newsspike causing the 0,0 precision,recall or maybe that's how they started
        • don't seem to need to change anything now, just curious, wanted to check it; doesn't appear that newsspike or uwtime broke anything
  • modify washington.pom/xml, to just run newsspike benchmark test
    • mvn verify -pl washington -am
  • Jenkins job
    • failed in profiler - ran out of heap space memory / this needs to be 9-10gb, somehow it was set back to 8gb, set it to 12gb
    • pushed updated newsspike files to the washington repo, master branch; i.e. 1)changes to the relation and argument types being set 2)removed the carriage return from a regression test input file (file-#003)
    • restarted Jenkins job - job is in queue waiting for an executor
      • job failed in profiler, again ran out of heap space,
    • restarted Jenkins job - upped memory values to 12g to start, 16g max (from 8g start and 12g max)
      • job failed earlier (in regression test), [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (default-cli) on project washington-adept: Command execution failed. Process exited with an error: 137 (Exit value: 137) -> [Help 1]
      • submitted JIRA comment - don't know what this error means
• algorithm
  • add another one?

AWS/Web Demo

• created an AWS account for nhawkins, = 1 yr of free tier
• proceeding to check on lambda limits
  • LIMIT: Lambda function deployment package size (.zip/.jar file) 50 MB
  • newsspike depends on two models; the sum of their .zip/.jar file sizes is 198MB, suggesting that the current lambda limits are too low for newsspike

.

Feb 19, 2016

HOLIDAY - Monday

DEFT Integration Call

• took the minutes / added them to the DEFT Wiki

DEFT repository

• --------------------------
• newsspike - status
• --------------------------
  • need to get a Jenkins job to successfully complete
    • this Jenkins job is using the pushed code changes relating to changes in rel and arg types
    • interested in seeing if there are any failures/exceptions in the profiler step; if there are, any code fixes would probably be optional
    • the job has successfully passed the regression and benchmark tests
    • the job has failed in the profiler step due to "out of memory" (made config changes) and now due to mystery reason during regression tests
    • submitted a JIRA comment asking about the most recent build error msg, which is unclear
    • waiting to hear from bbn / David (bbn) is looking into it

• --------------------
• uwtime - recap
• --------------------
  • Jenkins job successfully completed
  • a JIRA issue has been created requesting that the algorithm be "promoted" (integrated)
  • todo: examining document reference date - though it "works", should a change be made to UWTimeResolver.java?, how it handles the case of no doc date?
    • answer: reviewed it, and looks like not; a thought arose, wanted to pursue it, review it, did that, an alternative "extract" call is not an improvement
    • usage: if a doc doesn't have a publication date, the user should set a desired reference date to the doc, or handle the resolved uwtime values
  • bbn will notify us, if they want to request any additional changes

• --------------------------------
• Tractable Markov Logic
• --------------------------------
  • emailed Mathias, waiting to hear from him about getting the code
  • checked github, didn't see anything
  • alchemy.cs.washington.edu has open-source code; this might serve as a code source for this algorithm

• ------------
• FIGER
• ------------
  • started a 2-pg description doc which needs to be submitted to BBN
  • cloned Xiao's github FIGER repo / ran a demo
  • Checked the Adept API: Looks like the output could be stored in these ADEPT data structures:
    • HltContentContainer
    • List<EntityMention> entityMentions or List<EntityMention> namedMentions
  • For each entityMention, can store as a string this list of predicted types
    • eg. for example sentence: "Barack Obama was born in Honolulu, Hawaii."
      • for Barack Obama, FIGER returns a list of types:
      • /person@1.5487110047134074
      • /person/artist@1.6325928807079433
      • /organization/company@1.5487988005872304
      • /person/politician@1.3571058387371622
      • /organization@1.7854870983518862
      • /person/actor@1.5642686717985461
      • /person/author@1.402615176728773
  • Reviewing FIGER code added to DEFT repo for newsspike,
    • checking: if it can be used, if any additional FIGER-specific code needs to be added
    • identified the code and resource files that need to be added
  • Reviewed the Fine-Grained Entity Recognition paper by Xiao Ling and Daniel S. Weld
    • used this information plus trial runs of the FIGER code to complete the "2 pg" algorithm description doc
  • submitted a JIRA ticket for FIGER
    • attached the "2 pg" algorithm description document
    • this starts the ball rolling with BBN for integration
  • starting to add new files

• ---------------
• General
• ---------------
  • pulled the adept tag v2.3 updates
  • successfully merged, yay!
  • pushed the newly merged files to the washington origin master
  • Regression Tests, however, are failing for multir, ollie and reverb
    • need to investigate...

AWS/Web Demo

• lambda -
  • could experiment with this, but it doesn't look like it's going to work for newsspike, due to the current lambda .jar size limit
• elastic beanstalk -
  • this is an alternative AWS solution for creating a web app
  • seems like I should try this one

.

Feb 26, 2016

Multir Questions

• responded to email questions from
  • Lihong
  • Yueyue

DEFT Repository

• KB
  • responded to question: can we connect a KBP system to it in order to populate it?

• Git
  • some things seem mysterious about our repository, so taking time to more deeply learn git
  • completed CodeSchool GettingStartedWithGit tutorial; there's a "next" tutorial too, which I plan to do
  • read through Think-Like-a-Git (this author said he spent the first 1-2 yrs using but not understanding git, so now he's helping others)
  • planning practice exercises

• General - since integrating the ADEPT v2.3 updates
  • investigate and fix the newly failing regression tests (RT) for multir, ollie, reverb
    • fixed, resolved conflicts in the pom.xml, so now the failing regression tests are for uwtime and newsspike instead
  • investigate and fix the newly failing RT's for uwtime and newsspike
    • the differences, causing the failures, appear to be superficial
    • need to verify this; if verified, need to update the RT reference files for uwtime and newsspike
    • fixed, the id's (hashes) were different, everything else the same

• FIGER
  • BBN responded saying our "2pg" description doc looks fine
  • BBN suggested implementing both process(Document document) and process(Document document, HltContentContainer hltContentContainer)
    • The latter one allows an update on an existing HltContentContainer
  • working on integration
    • Code is in the repo
    • Regression Tests pass
      • however, I want to investigate why there is an error when the input file contains > 1 passage
    • Benchmark Tests
      • input files and test have been set-up
      • working on this one
    • Profiler
      • input files and test have been set-up
      • work on this next

• Tractable Markov Logic
  • ask Pedro about Mathias? Haven't heard from Mathias.
  • Pedro said Mathias has been working on cleaning up and documenting the code
  • emailed Mathias asking if he could send the code as-is to see if I can insert something into the repo

AWS/WebDemo

• FIGER - the figer github has a simple web demo

.

Mar 4, 2016

DEFT Repository

• FIGER
  • Regression Tests
    • resolved: multiple passages are ok in an input document, as long as they are separated by a single line break with no extra spaces
    • adjusted input files to contain multiple passages
    • checking output, and copying over reference files
    • PASS: regression tests pass
  • Benchmark Tests
    • cleaned/commented score.py
    • PASS: benchmark tests pass
  • Profiler Test
    • compile Profiler Class:
      • mvn install -pl washington -am
      • make sure pom.xml's <skip>'s are set to "true"
    • run Profiler:
      • mvn exec:java -pl washington -Dexec.mainClass="edu.washington.figer.WashingtonFigerEntityRecognizerProfiler" -Dexec.classpathScope="test" -Dexec.args="/homes/gws/nhawkins/repo/deft/washington/src/test/resources/edu/washington/figer/input.profiler /homes/gws/nhawkins/repo/deft/washington/src/test/resources/edu/washington/figer/output.profiler"
    • check output: output looks good
    • PASS: profiler results ok
  • Run overall Regression Tests, Benchmark Tests in Figer Branch
    • PASS: all tests pass
  • Implement process(Document document, HltContentContainer hltContentContainer)
    • done!
  • Merge figer Branch into master Branch
    • From repo/deft>
    • 1. git checkout master
    • 2. git status / git pull if necessary
    • 3. git merge figer
    • done!
  • Run overall Regression Tests, Benchmark and Profiler Tests in Master Branch
    • PASS: all tests pass
  • Push code to remote washington repo
    • 4. git push origin master
    • done! code pushed.
  • Jenkins job
    • set-up job. done!
    • job is running...failed
      • submitting a JIRA ticket to get BBN to help diagnose and fix; the error seems like something I can't reach
      • BBN responded; they are updating the Jenkins machine, which is not complete, which is causing the error
      • BBN will let us know when the updates are complete, and will restart the FIGER Jenkins job at that time
  • request that FIGER be promoted, if/when Jenkins job successfully finishes

• Tractable Markov Logic
  • sent two emails,
    • first, asked if the code is in github, or somewhere, where I can get it (I looked and didn't find any)
    • second, at Pedro's suggestion, asked if the code can be sent to me as-is, so I can just try to add something to the DEFT repo
  • haven't heard back on either email
  • sent a 3rd email to Mathias and Pedro, mentioning I have finished the FIGER integration and can work on this one, that it would be good if we could just add something
    • Mathias replied saying he would put the code on github
    • will proceed when he notifies me it's ready; don't see anything added yet

AWS Web Demo

• created a Docker image of the newsspike Play2 Web App
  • i.e. newsspike-deploy.zip (size = 597M)
• created Elastic Beanstalk (eb) application on AWS with Platform = Docker
  • configured it to use EC2 instance of type: t2.large, which has 8GB of RAM (may not be enough)
    • anything above t2.micro is not free; this one costs $.10/hr of use
    • charges = $2.29 so far; created an alert if charges reach $10 for one month's time
    • barely used it, so not sure how reached $2.29 already, but it's not much, so not worrying about it
    • at UW Web Day, one UW presenter told a story of racking up (unintentionally) HUGE charges, which AMZN fortunately forgave
  • tried to upload newsspike-deploy.zip to the eb application
    • failed: the zip file (597M) exceeds the max allowed size of 524 288 000 (512MB)
      • stanford.core.nlp = 329M
      • figer-model = 81M
      • nsre-model = 113M
      • newsSpikeFiles = 12M
      • reverb = 12M, total 547M
      • others = 50M, total 597M
    • exploring where this max is enforced -- i.e. by eb or by the ec2 instance type?
      • Elastic Beanstalk Developer Guide says:
        • Note: Elastic Beanstalk supports only a single .war file for a Java application version and only a single .zip file for other applications. The file size limit is 512 MB.
      • Looks like it might be enforced by the eb AWS Mgmt Console Upload approach
      • So, exploring eb CLI (elastic beanstalk command-line interface) instead
      • It appears code can be stored on S3, instead of uploading as a .zip, which may avoid the 512M limit; free tier allows storage up to 5GB in S3
      • ebCLI needs to be installed; Linux instructions are for super-user installation, so emailed cs-support for their opinion, maybe they can install it
      • going to try to install ebCLI on my Windows laptop, proceed from there until I hear back from cs-support
      • uploaded the .zip to an S3 bucket called: newsspikebucket

.

Mar 11, 2016

DEFT KB

• ------------------------------------------------------------------
• KB assertions: entities, relations, events
• ------------------------------------------------------------------
  • reviewed How To Use the ADEPT KB API for inserting Relations and Events
    • answered one of my questions:
      • when inserting a relation, first insert the entities (if not already there) and then insert the relation (if not already there)
      • similarly, for events
      • precondition check = document entities that the relation contains as arguments must have already been inserted into the KB
      • this check is facilitated via: one input to the relation insert(kb) = a map of document-level arguments to an existing Adept KB id
  • reviewed KBUnitTest.java to see how to assert events; previously asserted only entities (i.e. Barack Obama)
    • events are subclasses of relations
  • each of these (entities, relations, and events) requires a check to see if already in the KB
    • i.e. entity linking, relation linking and event linking
• ------------------------------
• mongodb
• ------------------------------
  • vinculum needs a mongo db
  • mongodb
    • description:
      • MongoDB is an open-source document database that provides high performance, high availability, and automatic scaling.
      • MongoDB obviates the need for an Object Relational Mapping (ORM) to facilitate development.
      • A record in MongoDB is a document, which is a data structure composed of field and value pairs. MongoDB documents are similar to JSON objects.
      • The values of fields may include other documents, arrays, and arrays of documents.
      • MongoDB stores documents in collections. Collections are analogous to tables in relational databases.
      • Unlike a table, however, a collection does not require its documents to have the same schema.
      • In MongoDB, documents stored in a collection must have a unique _id field that acts as a primary key.
  • installed to tools/ in home dir
    • curl -O https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel70-3.2.3.tgz
    • tar -zxvf mongodb-linux-x86_64-rhel70-3.2.3.tgz
    • cp -R -n mongodb-linux-x86_64-rhel70-3.2.3/ mongodb
  • mongo dependencies appear to be installed:
    • resolved issue: was missing a dependency (libsasl2.so.2); replaced installed mongodb with one for a different/the-correct linux
  • mongo needs a data/dir
    • created a /homes/gws/nhawkins/tools/mongodb/data/db dir, and specified this when starting the mongod process
  • successfully started the mongod process (runs on port 27017 by default)
    • mongod --dbpath tools/mongodb/data/db &
      • check which port running on: lsof -i OR nmap localhost
  • successfully interacted with mongo
  • • • downloaded an example dataset, and copied it to primer-dataset.json
        • curl -O https://raw.githubusercontent.com/mongodb/docs-assets/primer-dataset/dataset.json
      • imported the example dataset into the mongodb
        • mongoimport --db test --collection restaurants --drop --file primer-dataset.json
      • started mongo shell
        • [nhawkins@porvo]$ mongo
          • mongo> help
          • switch to the test database
          • mongo> use test
          • mongo> db.restaurants.insert(
          • {
          • "address" : {
          • "street" : "2 Avenue",
          • "zipcode" : "10075",
          • "building" : "1480",
          • "coord" : [ -73.9557413, 40.7720266 ],
          • },
          • "borough" : "Manhattan",
          • "cuisine" : "Italian",
          • "grades" : [
          • {
          • "date" : ISODate("2014-10-01T00:00:00Z"),
          • "grade" : "A",
          • "score" : 11
          • },
          • {
          • "date" : ISODate("2014-01-16T00:00:00Z"),
          • "grade" : "B",
          • "score" : 17
          • }
          • ],
          • "name" : "Vella",
          • "restaurant_id" : "41704620"
          • }
          • )
          • The result set contains all documents in the restaurants collection.
          • mongo> db.restaurants.find()
          • The query condition for an equality match on a field has the following form:
          • { <field1>: <value1>, <field2>: <value2>, ... }
          • The following operation finds documents whose borough field equals "Manhattan".
          • mongo> db.restaurants.find( { "borough": "Manhattan" } )
          • mongo> db.restaurants.find( { "address.zipcode": "10075" } )
          • mongo> db.restaurants.find( { "grades.grade": "B" } )
          • mongo> db.restaurants.find( { "grades.score": { $gt: 30 } } )
          • mongo> db.restaurants.find( { "grades.score": { $lt: 10 } } )
          • mongo> db.restaurants.find( { "cuisine": "Italian", "address.zipcode": "10075" } )
          • mongo> db.restaurants.find(
          • { $or: [ { "cuisine": "Italian" }, { "address.zipcode": "10075" } ] }
          • )
          • mongo> db.restaurants.find().sort( { "borough": 1, "address.zipcode": 1 } )
          • .update, .remove, .drop Data
          • MongoDB can perform aggregation operations, such as grouping by a specified key and evaluating a total or a count for each distinct group.
          • .aggregate
          • db.restaurants.aggregate(
          • [
          • { $group: { "_id": "$borough", "count": { $sum: 1 } } }
          • ]
          • );
          • Indexes can support the efficient execution of queries. MongoDB automatically creates an index on the _id field upon the creation of a collection.
          • db.restaurants.createIndex( { "cuisine": 1 } )
          • mongo> show users
          • mongo> show dbs
          • mongo> show collections
      • more set-up needed?
        • need to create a vinculum db?
        • need to create an admin user?
• ------------------------
• entity linking?
• ------------------------
  • vinculum code needs a mongodb
    • emailed Xiao for thoughts, further comments
    • installed a mongodb in my home dir, successfully interacted with it (see above)
    • Xiao sent "getting started comments"
    • used Xiao's comments, successfully downloaded the code and ran it on some test cases:
      • input: Miami, output: link for Miami, FL
      • input: Miami, OH, output: link for Miami University
  • vinculum code
    • which class to call to process a text file?
      • answer: edu.washington.cs.vinculum.pipeline.LinkingPipeline
    • how much configuration needed?
      • vinculum.conf
        • mongo contents (for our set-up):
          • mongo{
          • // database setup
          • mongo_host:"rv-n13.cs.washington.edu",
          • mongo_port:27017,
          • mongo_db:vinculum
          • },
        • location for our setup: "unpacked" dir
        • location for Xiao's setup: /homes/gws/xiaoling/pardosa12/workspace/vinculum/src/main/resources
  • ----------------------------------
  • subroutine using vinculum?
  • ----------------------------------
    • use all of newsspike pipeline and then all of vinculum pipeline?
    • ------------------------------------------
    • possible entity-linking algorithm:
    • ------------------------------------------
      • run newsspike and vinculum on a document
      • for each newsspike entity with a link from vinculum, use link to check if entity is in KB, if not, add entity
      • for each newsspike entity with no link from vinculum, check if entity string is in KB
        • if entity string is in KB, check if types match, if not add entity
        • if entity string is not in KB, add entity
  • --------------------------------------------------------------------------------------------------------------------
  • question: how work with code in repo/deft/washington and code not in the repo,
    • adept KB classes are in the repo
    • newsspike, multir, ollie and reverb algs are in the repo
    • vinculum code is not in the deft repo
    • KBP projects are not in the deft repo
  • --------------------------------------------------------------------------------------------------------------------
    • submitted JIRA ticket asking BBN if an algorithm which depends on a mongodb can be integrated in the DEFT repo
    • trying to run the vinculum code from within the repo
      • created a symbolic link pointing to the vinculum code
      • successfully accessing code from the deft repo
      • however, the code is not compiling
        • exploring this...at a minimum, need to add dependencies to the washington pom.xml
        • reduced the compile errors from 100 to 1 (by adding dependencies)
        • error is from: multir2.MultirSentencePredict (i.e. not newsspike + vinculum)
        • this appears to be because vinculum requires stanford core nlp 1.3.4, whereas the pom.xml had 1.3.5 specified (which multir appears to need)
        • submitted a comment to the vinculum JIRA ticket, asking if this is resolvable.
      • temporarily modified MultirSentencePredict to get around the compile issue
      • running newsspike, calling vinculum from there...
        • success! when sending text "Miami" getting a link returned for "Miami, Florida"
        • failure: running out of memory when next call newsspike predict (allowing 14GB of RAM)
          • in Xiao's build.sbt, he allows a max of 24GB RAM (my machine has 16GB total)
• -------------------------------------------
• Colin and James's plan?
• -------------------------------------------
  • met with Colin on W afternoon to discuss
  • plan involves newsspike
• ------------------------------------------------------------------------------------------
• 2015 kbp system (3 parts: openie, multir, and implie) + kb?
• ------------------------------------------------------------------------------------------
  • plan? entities and relations? events?

Web Demo

• newsspike
  • switched elastic beanstalk from ec2 t2.large (8GB RAM) to ec2 t2.micro instance to stop charges
  • emailed aws billing support to verify that all settings are compatible with "free tier"
  • ec2 t2.large ~= $2.50/day + tax
  • turns out my account is ineligible for 1-yr free tier because I used that a couple years ago
  • I need to create a new account using different email to get a "free tier"

DEFT Repo

• tractable markov logic
  • Mathias replied saying he would put the code on github
  • Mathias sent new email saying he has work deadlines at the end of March and needs time to clean-up the code and write a README
  • submitted a JIRA ticket asking if they have suggestions re how this code should be added, the input is different from text from a doc
    • BBN asked if we have a sample input document?
    • asked Mathias if he can provide one of those
• figer
  • checked on this Jenkins job
  • started a new job, Connor (bbn) had said he would start a new job, but doesn't appear he had yet, previous job failed because they were upgrading their Jenkins machine
  • job finished successfully, blue dot + clouds
  • submitted a JIRA ticket asking that this algorithm be promoted (blue dot was required)
• newsspike
  • has a blue dot in Jenkins (i.e. last run was successful) and clouds (i.e. previous run probably failed)
  • things are good when we have a blue dot and sunny sun
  • this is in the hands of BBN now (that is, there are no outstanding requests for us)
• uwtime
  • has a blue dot in Jenkins (i.e. last run was successful) and clouds (i.e. previous run probably failed)
  • things are good when we have a blue dot and sunny sun
  • this is in the hands of BBN now (that is, there are no outstanding requests for us)

.

Mar 18, 2016

OUT = Monday

DEFT Data Integration Call = Wednesday

• Connor (bbn) said he's looking into whether an algorithm requiring a mongodb can be integrated; i.e. vinculum
  • we don't *need* to integrate it, just asked about it..

DEFT KB

• identified conflicts between vinculum and core nlp 1.3.5 (which is required in the deft repo)
  • Xiao provided a patch for vinculum
  • installed the patch
• can now get vinculum links and pass newsspike regression tests from NewsSpikeExtractor
  • pom.xml now only lists stanford core nlp 1.3.5 dependencies, no 1.3.4 dependencies
• integrated Colin's files to insert entities and events into the KB
  • got resource files put in the proper places
• attempting to get links and newsspike events from NewsSpikeExtractorAppKB
  • works: got link, for "Miami" test text and got newsspike prediction, "grow up in@/person@/location"
  • doesn't work: when change vinculum test text from "Miami" to the testFile.txt text, "Barack Obama was born in Honolulu, HI."
    • getting out of memory error
    • on the plus side, the program reports the total memory as: 8GB, which means the MAVEN_OPTS setting of 15GB is being ignored
    • need to explore MAVEN_OPTS and other ways to adjust the java heap space
    • Ok, appears the DEFT pom.xml ignores MAVEN_OPTS and offers this line instead:
      • <execMaxMemorySize>-Xmx14g</execMaxMemorySize>
      • The newsspike regression test is now reporting the total memory as: 14GB (rather than 8) and successfully ran newsspike after a call to vinculum with the testFile.txt
      • Will now test with the newsspike extractor kb app, which inserts into the KB
        • This test is not working; the program crashes with an out-of-memory error
        • newsspike and vinculum are both using about 6-8GB of memory, which totals 12-16GB
    • emailed Xiao asking if he knows of a machine with 24GB of memory
      • answer: rv cluster, pardosa07 and pardosa08, nlp, and lil clusters
      • comment: Xiao feels that the programs should be run separately to avoid the memory issue;
        • i.e. for a set of docs, first run vinc, and next, separately, run the newsspike extractor kb app reading in the serialized links
  • rv-n13: attempting to run the job requesting more than 20GB memory
    • had issue with: "java : Unsupported major.minor version 52.0"
      • code crashed / reviewed:
        • http://stackoverflow.com/questions/23249331/java-unsupported-major-minor-version-52-0
        • "Your code was compiled with Java Version 1.8 while it is being executed with Java Version 1.7 or below."
        • "In your case it seems that two different Java installations are used, the newer to compile and the older to execute your code."
      • solution: (and maybe a logout / login to rv-n13 was required?) (this error was very reluctant to go away..., but it did!)
        • update JAVA_HOME and JAVA_HOME_8 to a java8, in the .bashrc on rv-n13
        • source .bashrc
        • cd /homes/gws/nhawkins/repo/deft
        • mvn clean install -pl washington -am
          • success! this ran, and indicates that 20GB have been set as the mem max
        • mvn exec:java -pl washington -Dexec.mainClass="edu.washington.WashingtonNewsSpikeExtractorAppKB" -Dexec.args="-i /homes/gws/nhawkins/repo/deft/washington/src/test/resources/edu/washington/kb/input.insert/testFile.txt -o /homes/gws/nhawkins/repo/deft/washington/src/test/resources/edu/washington/kb/output.insert"
          • not success! run output: indicates total memory of <8GB, not good... program looks stuck, and crashed
        • updated pom.xml to 24GB max
        • trying: mvn verify -pl washington -am
          • success: program recognizes the max memory setting of 24GB
        • trying: the app "mvn exec:java" command again...
          • there is nothing fast about this... the sample testFile.txt is small, and there is only one...
          • maybe because it is working with 8GB only
          • maybe "the app" responds to MAVEN_OPTS, going to set that on rv-n13 and re-start the job
            • yes: "the app" does respond to MAVEN_OPTS, whereas the newsspike regression tests respond to the DEFT pom.xml
            • trying this:
            • bash-4.1$ export MAVEN_OPTS="-Xms20000m -Xmx20000m"
            • -bash-4.1$ echo $MAVEN_OPTS
            • -Xms20000m -Xmx20000m
            • ps aux shows nhawkins is using: 57.6% of rv-n13's memory (32GB * 0.576 = 18.432 GB), also shows this as the command:
              • nhawkins 24398 245 57.6 25111116 18950444 pts/2 Sl+ 16:45 5:48 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-1.b15.el6_7.x86_64/bin/java -Xms20000m -Xmx20000m -classpath /home/nhawkins/apache-maven-3.2.3/boot/plexus-classworlds-2.5.1.jar -Dclassworlds.conf=/home/nhawkins/apache-maven-3.2.3/bin/m2.conf -Dmaven.home=/home/nhawkins/apache-maven-3.2.3 org.codehaus.plexus.classworlds.launcher.Launcher exec:java -pl washington -Dexec.mainClass=edu.washington.Wa
            • success!!! omg!!!! the program completed
              • memory stats: just after call to vinculum returned
                • Total memory is: 19167
                • Used memory is megabytes: 8442
                • run garbage collector
                • Total memory is: 19167
                • Used memory is megabytes: 6069
              • memory stats: just after call to newsspike finished
                • Total memory is: 17551
                • Used memory is megabytes: 13429
                • run garbage collector
                • Total memory is: 18359
                • Used memory is megabytes: 11286

• RAM: how much memory does my machine have?
  • free -mt
    • porvo: 16GB + 16GB swap
    • rv-n13: 32GB + 32GB swap

• vinculum: if read links data from disk, and even if not,
  • need to decode JSON input
    • http://www.tutorialspoint.com/json/json_java_example.htm
    • [{"wpid":99629,"start":26,"link":"Terry Branstad","mid":"\/m\/0pf6v","end":40},{"wpid":9163,"start":65,"link":"Des Moines, Iowa","mid":"\/m\/02j3w","end":75},{"wpid":99629,"start":159,"link":"Terry Branstad","mid":"\/m\/0pf6v","end":173},{"wpid":9163,"start":205,"link":"Des Moines, Iowa","mid":"\/m\/02j3w","end":215},{"wpid":841972,"start":293,"link":"Los Ángeles","mid":"\/m\/03g4wf","end":304},{"wpid":28509884,"start":312,"link":"Austin Rivers","mid":"\/m\/0crd610","end":325},{"wpid":72892,"start":337,"link":"Los Angeles Clippers","mid":"\/m\/0jmjr","end":357},{"wpid":1642868,"start":370,"link":"Doc Rivers","mid":"\/m\/05k13g","end":380},{"wpid":22093,"start":435,"link":"National Basketball Association","mid":"\/m\/05jvx","end":438},{"wpid":18935732,"start":455,"link":"Associated Press","mid":"\/m\/0cv_2","end":457},{"wpid":33589,"start":466,"link":"Wal-Mart","mid":"\/m\/0841v","end":479},{"wpid":91575,"start":488,"link":"Grayson County, Texas","mid":"\/m\/0mrmh","end":502},{"wpid":16846,"start":509,"link":"Kentucky","mid":"\/m\/0498y","end":512},{"wpid":16846,"start":657,"link":"Kentucky","mid":"\/m\/0498y","end":665},{"wpid":16846,"start":879,"link":"Kentucky","mid":"\/m\/0498y","end":887}]

• maven: what is a maven snapshot
  • http://www.tutorialspoint.com/maven/maven_snapshots.htm

.

Mar 25, 2016

KBP2014

• pointed Chris Lin to data and code

KBP2015

• reviewed/commented on Stephen's paper

KBP2016

• reviewed/commented on Event Argument and Linking task

DEFT KB

• linking
  • vinculum - createStartOffsetLinksMap()
    • put document links in a hash map: key=start offset, value = JSON Object of link data
  • newsspike output - getWikipediaLink()
    • accessed start and end offsets for event argument (entity)
    • can use the start offset to check the "links hash map" for a wpid
  • kb insert rules - updateEntityKB()
    • created a first pass at KB entity insert or update rules
  • shared code with Colin
• kb stats
  • wrote a program to report number of entities in the washington KB (uses kb.getEntitiesByRegexMatch(regexString, false))
• entity insert vs. update
  • reviewed the methods for the InsertionBuilder and the UpdateBuilder
  • BBN changed the API for the UpdateBuilder since they wrote the: "How to use the Adept KB API" doc
  • identified entity InsertionBuilder methods to use, in addition to ones given in the "How to" KB API guide
  • identified entity UpdateBuilder methods that can be used
• corpus
  • next: check Stanford Core NLP pipelines (i.e. properties) used by vinculum and figer/newsspike
    • can we use the preprocessed cold start corpus to extract newsspike events and insert into KB? (i.e. faster)
    • if not, the raw docs could be used

DEFT Maven Commands

• mvn compile -pl washington -am
• mvn exec:java -pl washington -Dexec.mainClass="edu.washington.WashingtonNewsSpikeExtractorAppKB" -Dexec.args="-i /homes/gws/nhawkins/repo/deft/washington/src/test/resources/edu/washington/kb/input.insert/testFile.txt -o /homes/gws/nhawkins/repo/deft/washington/src/test/resources/edu/washington/kb/output.insert"

.

Apr 1, 2016

DEFT KB

• met with Colin and James
• newsspike
  • exploring Congle's Corpus of approx. 500+k articles which are in one file, in JSON format
    • wrote a file: KBConglesCorpus.java to read the file
      • currently getting an error about malformed JSON
      • looks like there an error occurred when copying the file
      • wrote 4 articles (from the 500k) as single files for testing
      • later, can extract all 500+k articles, using same file/technique
  • wrote a file, KBStats.java to check on the washington KB, for example
    • KB Stats:
    • Number of kbEntities: 21
    • Number of kbRelations: 8
    • Number of kbEvents: 14
  • working on WashingtonNewsSpikeExtractorAppKB
    • tested the 4 files from Congle's corpus
      • good news:
        • they were processed
      • not as good news:
        • 2 appear to have 0 events
        • 1 is throwing an exception in NewsSpikeExtractorKB.convert... due to a Type error, investigating
        • 1 has 2 events: "win at" and "beat" which do not map to any of the ADEPT Events
    • working on the kb insert/update entities and insert event portions (adjusting Colin's code)
      • 1) need to check if event is already in the KB
        • use sparql query
      • 2) need to update entities already inserted
        • BBN changed the API from "help" doc, so need to identify new methods for updating

.

Apr 8, 2016

DEFT KB

• newsspike
  • ------------------------------
  • BBN KB Documentation
  • ------------------------------
  • Dave Kolas (BBN) is asking someone to fix the broken examples that I have been asking about
  • -------------------------------------------------------------------------------------------
  • updated repository with the adept v2.3.1 maintenance update
  • -------------------------------------------------------------------------------------------
  • proceeded smoothly, fortunately
  • or maybe not, getting a compile error when calling kb.getEventsByArgs(arg1KBID, arg2KBID)
    • investigating with Dave Kolas (BBN)
  • ---------------------------------------
  • ran the BBN KB reports
  • ---------------------------------------
    • identified possible issues with the KB entries:
      • contains duplicate entities, relations and events
        • we have added checks for entities to prevent this
        • need to work out the check for events and relations / will need to see if we need to just check "events" or also "relations"
      • event and relation confidence scores are all 0? (that may be ok, though newsspike does report some kind of confidence > 0)
        • reviewing Colin's code
      • the event and relation justification offsets are for arg1 only (should probably be a span including arg2 too?)
        • reviewed Colin's code; created a spanChunk and changed the input from arg1Chunk to spanChunk, looks ok now
      • the event and relation justification starts off with "001" - is that supposed to be the doc name? (if so, problem, we have used at least 2 test files: "testFile.txt" and "testFile1.txt", 2 different names should lead to 2 different numbers, not just 001)
        • sent Kolas (BBN) an email about this; it's not quickly clear where we should add a doc name; it doesn't look like we're adding it anywhere.
      • event and relation args 1 and 2 both have confidence scores of 0 for entities in the KB
  • ----------------------------------------------------------------------------------------------------------------------------------------
  • working on the kb insert/update entities and insert event portions (adding to and adjusting Colin's code)
  • ----------------------------------------------------------------------------------------------------------------------------------------
    • -------------------------------------------------------------------
    • 1) need to check if event is already in the KB
    • -------------------------------------------------------------------
      • use sparql query to do that (following the "How to use the Adept KB API" doc)
        • getting error: com.hp.hpl.jena.query.QueryParseException
        • sent Dave Kolas (BBN) an email/question about forming a sparql query
        • the example in the "How to" doc seems too sparse wrt comments/explanation
          • Kolas (BBN) responded: the doc example is wrong; he suggests not using that method/approach (though the doc recommends it)
          • going to switch from sparql query approach to kb.getEventsByArg approach
      • example, "Barack Obama was born in Honolulu, HI"
        • was added several times, because no checking is in the code
        • this "event" gets automatically added as a Relation and not as an Event (this is ok, I guess; it's BBN's API)
        • it is added as "Resident", rather than "BeBorn" because the newsspike extraction is:
          • grow up in@/person@/location
          • which maps to: Resident in the ADEPT ontology
      • moving on to different method (rather than the sparql query) at BBN's suggestion
        • kb.getEventsByArgs
          • currently, not compiling, saying: error: symbol not found
          • emailed Kolas (BBN) about that. / not sure why that method is not being found
    • ------------------------------------------------------------
    • 2) need to update events already inserted
    • ------------------------------------------------------------
      • sent Kolas (BBN) an email of questions about the KBEvent.UpdateBuilder; the doc is very sparse
      • Kolas responded, also fairly sparse; said with the v2.3.1 update they will try to update the API doc online
    • ------------------------------------------------------------
    • 3) need to update entities already inserted
    • ------------------------------------------------------------
      • BBN changed the API from "help" doc, so need to identify new methods for updating
      • sent Kolas (BBN) an email mentioning that the "help" doc doesn't seem to match the code in this area, and asked if there is a more up-to-date version of the API doc available
  • ----------------------------------------
  • corpus - Congle's crawl
  • ----------------------------------------
    • need to write-out the full set of 500+k articles as individual files
    • wrote out 4 files so far, to test the method and the output text; method seems ok, and the "newsspike KB app" processed the text files ok, looks like, so far
    • wrote out approx. 546713 individual text files to use as input corpus (not all the files have valid text, I don't think, some exceptions were thrown)

.

Apr 15, 2016

DEFT KB

• site visit
• continued to populate the KB
  • wrote and ran script to extract complete set of separate files from the 500+k Congle Corpus
    • don't write files which are "binary"; test by checking for existence of 4 chars, all 4 indicate a file which can't be processed by newsspike
  • wrote script to run larger batches, i.e. batches of 1k files rather than 100 files
  • wrote .pbs files to specify 10 batches of 1k files to run for the "200k" set
  • started running the jobs
  • "Pool not open" error
    • is appearing in this set of jobs, meaning that many entities and events aren't being checked for insertion, are being skipped
    • the pool openness comes and goes
  • KB NewsSpike App fixes made:
    • offsets (had assumed every sentence has a tuple)
    • extracting confidence when more than one event predicted (code for KB, was a little different from code not for KB)

• • Friday report:
    • relations: 2503
    • events: 3251
    • entities: 14251

.

Apr 22, 2016

Wed: DEFT Data Integration Call

DEFT KB:

• wrote and ran a script to create the .pbs cluster job files for the "300k", "400k" and "500k" sets of the corpus
• the "200k" set is currently running on the nlp cluster
• ran the CreateCrawlFileLists to create batches of 100 files for the "300k", "400k", and "500k" sets
  • returning to the smaller batch size of 100 files, to hopefully eliminate or reduce the "Pool not open" error, which appeared a lot for the "200k" set which was based on batch sizes of 1k files
• examining the "200k" output,
  • to see if the "Pool not open" error is still happening
  • assessing how much time left to process
• when the "200k" set finishes, will start the "300k" set
• ran the 300k set; "Pool not open" error is under control, not a big factor
• the first part of the 400k set is currently running on the nlp cluster

• Friday counts:
  • relations: 5139
  • events: 7248
  • entities: 27443

.

Apr 29, 2016

TAC KBP 2016 Event Track

• registered for:
  • August 22-Sept 5 / Event Argument Extraction and Linking Evaluation Window
  • Sept 20-Oct 3 / Event Nugget Detection and Coreference Evaluation Window

DEFT KB

• first set of "400k" jobs finished running
• second set of "400k" jobs finished running
• remaining "400k" jobs + most of "500k" jobs finished running
• started final set of 12 "500k" jobs and first set of 8 "200k" jobs (of total 43 "200k" jobs)
• next?
  • re-running the "200k" set, since got a lot of "pool not open" errors with larger batches
• KB Notes
  • 2 of the jobs ended prematurely, not clear why (no error is in the qsub .o or .e files), looks more like it might have been killed/deleted? (though not by me)
  • sometimes entities, relations, or events are not inserted because the query times out
• currently running BBN "KB report" to select a sample of 5000 relations and 5000 events
  • this report takes a long time to run...(2 hrs to get relations, another 2+ hrs to get events, then 2 hrs to get entities)
  • 5000 is the max sample size allowed (according to Marjorie during the last Integration call)
• the breakdown of relation and event "types" is this:
  • based on 5k relations and 4998 events (for 2, query timed out)
  • the BBN report does not provide this information directly ( I am analyzing the csv file they write )
  • (E) = event, (R) = relation

• • • EmploymentMembership 2018 (R)
    • Leadership 1224 (R)
    • Resident 1342 (R)
    • SpousalRelationship 250 (R)
    • StudentAlum 166 (R)

• • • Attack 118 (E)
    • Broadcast 1034 (E)
    • Correspondence 195 (E)
    • Die 75 (E)
    • EndPosition 310 (E)
    • Marry 1 (E)
    • Meet 1607 (E)
    • MergeOrganization 126 (E)
    • Nominate 324 (E)
    • Sue 39 (E)
    • TransferMoney 21 (E)
    • TransportPerson 1148 (E)

• • • Business 0
    • Elect 0
    • PersonAge 0
    • Sentiment 0
    • StartPosition 0
    • TransferOwnership 0

• Friday counts:
  • relations: 7983
  • events: 11699
  • entities: 40750

.

May 6, 2016

KBP 2016 Registration

• edited/signed/submitted pdf files:
  • Agreement Concerning Dissemination of TAC Results (to NIST)
  • 2016 Text Analysis Conference (TAC) Knowledge Base Population Evaluation License Agreement (to LDC)
  • receipt of both was confirmed
• need to create account with LDC (?)

DEFT Repository

• Mathias's Code
  • Pedro emailed a link to a github with the beginnings of the code

DEFT KB

• finished running the last set of jobs from the "500k" set
• finished running first 2 sets of jobs from the "200k" set
• finished running the last set of jobs from the "200k" set
• finished running a set of 21 jobs for the 2015 cold start discussion forum docs (mpdf files)
  • KBCreateCrawlFileLists.java
  • KBCreatePBSFiles.java
  • repo/deft> mvn compile -pl washington -am
  • results on:
    • /projects/WebWare9/nhawkins/crawlFileLists/mpdf/100
    • /projects/WebWare9/nhawkins/cluster/mpdf/100
• emailed Congle:
  • to find location of more crawl data he collected
  • to check if there's a newsspike paper
  • haven't heard back
• working to identify and examine other possible source docs
  • English Gigaword looks good (/projects/WebWare6/EnglishGigaword/)
  • afp_eng_2008 has: 225832 docs
• running 20 jobs from the afp_eng_2008 set

Wednesday Counts

• Relations: 8894
• Events: 13167
• Entities: 45033

The ~40k discussion forum docs added a little, not a lot (=diff Wed to Fri):

Friday Counts

• Relations: 9277
• Events: 13881
• Entities: 46910

.

May 13, 2016

DEFT KBP 2016

• created an LDC account
• reviewing task document
• reviewing code resources online, mentioned in task document
  • looking for code for writing results, generating id's and validating output, the doc suggests it exists
    • WRITES RESULTS:
      • reviewing the BBNe-eal Readme; I think it has some info which might address this
    • GENERATES IDs:
      • this might be one option: bue-common-open/common-core-open/src/main/java/com/bbn/bue/common/IntIDSequence.java
    • VALIDATES OUTPUT: com.bbn.kbp.events2014.bin.ValidateSystemOutput.java
      • input = a parameter file, one parameter is the "systemOutputStore",
      • not sure what that means, in the task doc the output is described as 3 dirs of files, will need to investigate further
      • "Checks a system output store can be loaded and then dumps it in a human-readable way, resolving \n"
      • +
      • " offset spans against the original text.\n" +
      • "usage: validateSystemOutput parameterFile\n" +
      • "Parameter files are lines of key : value pairs\n" +
      • "Parameters:\n\tsystemOutputStore: the system output to be validated \n" +
      • "\tdump: whether to dump a human-readable form of the input to standard output\n" +
      • "\tdocIDMap: (only if dump is true) a list of tab-separated pairs of doc ID and path to original text.\n"
      • +
      • "\tvalidRoles: is data/2014.types.txt (for KBP 2014)\n" +
      • "\talwaysValidRoles: is 'Time, Place' (for KBP 2014)\n");
• code
  • the code used for the KB does the work of getting the docs processed via newsspike, re-use seems like a good idea
  • options: use code from within repository dir, or pull it all out

DEFT KB

• processed first set of docs from aft_eng_2008 (EnglishGigaword)
• second set is finishing
• started third set, and more
• have started 81 jobs out of 113 total for aft_eng_2008

Last Friday's Counts

• Relations: 9277
• Events: 13881
• Entities: 46910

Monday Counts (after processing 40k docs)

• Relations: 9613
• Events: 15299
• Entities: 50270

Friday Counts

• Relations: 10430
• Events: 18925
• Entities: 57909

.

May 20, 2016

UW HR

• completed self-eval

DEFT Integration Call

• Wednesday
  • BBN is still working on updates to the KB (to make it faster) and KB API
  • only 3 sites called in: Washington, Stanford, and UIUC

DEFT KB

• last jobs finished running, completed 81/103 jobs from afp_eng_2008
• copied log files to: /projects/WebWare9/nhawkins/crawlLog/afp_eng_2008
• pausing on adding more to the KB, have enough for now

Current Counts:

• Relations: 10488
• Events: 19154
• Entities: 58363

KBP 2016

• ------------------------------------
• Colin and James
• ------------------------------------
• question: how to plug-in their updated algorithms?
  • emailed them asking to meet to discuss
  • Colin = Newsspike:
    • how is the algorithm changing - just the model? or events detectable? other?
  • James = something-like-RPI's algorithm?:
    • what's the status of his code and algorithm? (we haven't been using it for the KB, haven't added it to the DEFT repo)
    • the ontology for this algorithm is different from newsspike's, I believe, so they would plug-in in different ways
• ----------------------
• BBNe-eal code:
• ----------------------
  • to use BBNe-eal code, a dependency can be included:
    • <dependency>
      • <groupId>com.bbn.kbp.eal</groupId>
      • <artifactId>tac-kbp-eal</artifactId>
      • <version> current version listed in pom.xml of the tac-kbp-eal repository </version>
        • i.e. <version>3.2.2-SNAPSHOT</version>
    • </dependency>
  • -------------------------------------
  • forked BBN-E/tac-kbp-eal
  • -------------------------------------
  • git clone https://github.com/nrhawkins/tac-kbp-eal.git
  • mvn install (this created a jar of the code)
    • the jar was installed in: /homes/gws/nhawkins/bbn/deft/com/bbn/kbp/eal/tac-kbp-eal/3.2.2-SNAPSHOT
    • copied the jar to: /homes/gws/nhawkins/.ivy2/cache/com.bbn.kbp.eal/tac-kbp-eal/jars
  • -----------------------------------------------------
  • code to generate unique ID sequence:
  • -----------------------------------------------------
    • bue-common-open/common-core-open/src/main/java/com/bbn/bue/common/IntIDSequence.java
    • added a <dependency> and successfully access this code from within our DEFT repo
• ------------------
• adept code
• ------------------
  • included in tac-kbp-eal code?
    • doesn't look like it
    • does their pom.xml list "adept" code as a dependency? (i.e. "adept" handles doc processing)
      • not clear that it does
      • will need to figure that out, or decide on a different way to handle docs
  • this code exists in our repo/deft
• -------------------------------------------------------
• added BBNe-eal code to DEFT repo
• -------------------------------------------------------
  • copied the tac-kbp-eal-3.2.2-SNAPSHOT.jar from tac-kbp-eal to the .ivy2/cache
  • added tac-kbp-eal dependency to the washington pom.xml
  • mvn compile -pl washington -am
    • successful, meaning it found the tac-kbp-eal jar
  • created an app: WashingtonNewsSpikeExtractorAppEAL.java
    • added: import com.bbn.kbp.events2014.TypeRoleFillerRealisSet;
    • successfully compiled, indicating again the tac-kbp-eal jar is found
    • added to WashingtonNewsSpikeExtractorAppEAL.java:
      • import com.bbn.bue.common.IntIDSequence;
      • IntIDSequence uniqueIntIDs = IntIDSequence.startingFrom(1);
      • //System.out.println("NSEA EAL: start/nextValue: " + uniqueIntIDs.nextValue);
      • //System.out.println("NSEA EAL: unique ids nextID: " + uniqueIntIDs.nextID());
      • //System.out.println("NSEA EAL: unique ids nextID: " + uniqueIntIDs.nextID());
      • successfully compiled
• -------------
• pipeline
• -------------
  • have a first version working inside the /repo/deft/
    • writes output for /arguments
      • needs to: write the "EventType.Subtype", now just writing "subtype"
      • needs to: write Role, now writing entity type, not entity role / need to locate if/where role is assigned
      • needs to: do entity linking
    • writes output for /linking
      • needs to: restrict events written to the KBP subset
    • does not do anything for /corpus_linking
• ----------------------
• dryrun
• ----------------------
  • downloaded the LDC dryrun corpus data
    • there are 2092 files, English
    • location: /projects/WebWare9/nhawkins/ealInput/dryrun

.

May 27, 2016

KBP 2016 - EAL

• Made a first run of our EAL system (Congle's newsspike) on the 2092 dryrun files
• Jar'd the output and sent it to Marjorie as a .gz attachment
  • BBN's email system blocked the attachment
  • Marjorie has asked Ryan and Jay at BBN to help, suggested that we might be able to ftp
  • successfully ftp'd the dryrun_Washington_v0.gz to ftp.bbn.com
• Tried to validate the system output / need to sort out errors, emailed BBN about them
  • README.md documentation does not list the missing parameters below as required
  • com.bbn.kbp.events2014.bin.ValidateSystemOutput2016
    • Missing required parameter linkableTypes
  • com.bbn.kbp.events2014.bin.ValidateSystemOutput
    • Missing required parameter alwaysValidRoles
  • BBN suggests there's a new version of com.bbn.kbp.events2014.bin.ValidateSystemOutput2016
    • true: git clone'd BBN-e's tac-kbp-eal directly (previously forked a version) so can get updates
    • mvn install'd to generate a new tac-kbp-eal-X-SNAPSHOT.jar
    • copied jar to .ivy2 so repo/deft can find it
    • current version = 3.2.5-SNAPSHOT (previously had 3.2.2-SNAPSHOT)
  • our output probably doesn't validate, not sure if it needs to yet
    • For one, need to filter on valid event roles
      • for event Life.Die, we have an event role, NONE for "age 95", which is not part of the KBP taxonomy
• In communication with BBN re questions which have arisen / they're responding; i.e.
  • output directory for "arguments" should be "arguments" not "argument" (task description mentioned both)
  • the output directories, "arguments" and "linking" should contain one file for every input file, even if it (output) is empty (this is required)
  • output directory also must contain a corpusLinking dir with a file corpusLinking, even if the file is empty
  • reponseID's:
    • Jay suggested using "ResponseID" rather than "UniqueSequenceID" from the BBN API
    • according to the task document, these need to be corpus-level unique / need to check how that works with multiple batches
• VALIDATE SYSTEM OUTPUT 2016
  • created docIdMap:
    • paste dryRunDocIdList dryRunFileList | awk '{print $1,"\t","/projects/WebWare9/nhawkins/ealInput/dryrun/"$2}' > docIdMap
  • ran: com.bbn.kbp.events2014.bin.ValidateSystemOutput2016
    • error: java.io.IOException: While reading ENG_NW_001278_20130703_F00012T56, on line 1:
    • Caused by: java.io.IOException: ID 201 202(original ID) 201 202 cannot be resolved using provided response store. ***Knownresponse IDs are [d0e5c58c0e26b468bc1bd21bc74b7950ab4f0172, c2085f0b8b05bed220e7658ca5c0abd521b745a9]transformed ***response ids are Optional.absent()
    • emailed Jay at BBN for help with this error
    • check these: they're hopefully ok, docs which have no events detected
      • these look ok; they're all files with size 0 in /arguments and /linking
      • 16/05/23 21:51:33 INFO bin.ValidateSystemOutput: For document 47cd5d4d19143f700763ebd043fca7f4 got 0 responses
      • 16/05/23 21:51:33 INFO bin.ValidateSystemOutput: For document NYT_ENG_20130708.0172 got 0 responses
      • 16/05/23 21:51:34 INFO bin.ValidateSystemOutput: For document NYT_ENG_20130929.0121 got 0 responses

• Dry Run
  • BBN is using the dryrun to debug their code as well
  • Currently their "validate system output" program is buggy
  • They're happy that we're helping test their code
• Next
  • Filter valid event roles: DONE
    • Added TIME and PLACE roles for event Life.Die
    • Added code to check that a role is valid
  • Get BBN's ValidateSystemOutput2016 working on our output
    • parameters file is ok
    • getting an error about ResponseIDs, reported it to BBN
    • BBN identified a bug in their code - id's should be separated by spaces, the code expects tabs
    • BBN posted an fix for the spaces/tab issue
    • pulled the fix, re-generated the jar, and tried to validate a system run involving just one input file
    • got a similar, but slightly different error, emailed BBN about the "new" error
    • BBN posted a fix for the id's issue / their code was "not respecting" our id's
    • pulled the fix, re-generated the jar, and tried to validate a system run involving just one input file
    • got past previous error, new error involves the justification offsets, base filler text != cas
    • investigating offsets/text-mismatch error...
      • this error is because the adept.utilities.DocumentMaker starts the offset count from the start of the text in the <TEXT> </TEXT> section, rather than from the start of the document; i.e. from the first "<" in <DOC ...
      • adept.common.Document -- does it allow access to the raw doc text?
      • if not, need to adjust how the docs are handled, need to use different classes/code
    • another issue -- all of the ENG docs have duplicate sentences, which will upset the offset code
  • Make any other adjustments necessary to get output to validate
    • curious: will it validate if the id's are not corpus level? (currently, all id's are corpus level)
    • curious: will it validate if an event role is not valid? (currently, all event roles should be valid)
  • Argument Linking
  • Event Linking

.

Jun 3, 2016

Mon = Memorial Day

KBP2016 EAL

• dryrun
  • participating in the dryrun was useful to us and to BBN, helped us both debug
  • BBN plans to post more information, maybe next week, on "steps" to validate, and maybe other steps
• doc offsets
  • the code used to handle doc processing for the dryrun took advantage of code re-use, but has offset issues
  • the adept classes worked for the KB doc processing, but is not working for the KBP processing
  • need to rework code that does doc handling to properly compute offsets
  • newsspike handles offsets at the level of a sentence
• -----------------------------------------
• realrun - KBP2016EAL
• -----------------------------------------
  • created a new project, KBP2016EAL
  • created maven pom.xml (for the project and the dependencies)
  • created classes: DocumentMaker.java, Document.java, and Reader.java to process the docs
  • created: RunKBP2016EAL.java as the starting point for running the project
    • inputs:
      • name of a file containing a list of file names to process (eg. file name: NYT_ENG_20131231.0156.xml)
      • input directory (eg. /projects/WebWare9/nhawkins/ealInput/ealTest/)
      • output directory (eg. /projects/WebWare9/nhawkins/ealOutput/ealTest/)
    • creates a Document for each file, the Document stores:
      • docId (eg. NYT_ENG_20131231.0156)
      • the doc's rawText (eg. from <DOC ... to </DOC> or from <doc ... to </doc> )
  • Raw Text Notes:
    • the discussion forum docs have <post> and <quote> sections which require special care, looking into that
      • <quote></quote>, everything in there should be ignored?
      • <post></post>, multiple pairs of these in the doc?
    • the newswire docs have <text></text> sections
  • added the newsspike, figer, vinculum, and adept code to the project
  • discovered and added associated dependencies to the pom.xml
  • specified/configured compiler in pom.xml as java 1.8 via a maven <plugin> (the default w/o this is java 1.5)
  • project compiles
  • warning displayed: about the reverb-latest.jar / may need to move it (or change something) at next step
  • communicated with BBN about their API classes
    • can only see abstract classes without implementations on the github because they use a code generation tool to generate the implementations
    • the generated classes do not appear on the github, but they exist!! There should be one for every _"AbstractClass" on the github.
    • the names of the abstract classes begin with an underscore; i.e. _"AbstractClass"
    • Classes we should (try to) use:
      • com.bbn.kbp.events2014.Response (trfr's)
      • com.bbn.kbp.events2014.SystemOutputStore2016 (write output)
      • com.bbn.kbp.events2014.DocumentSystemOutput2015 (write output as this)
      • com.bbn.kbp.events2014.ArgumentOutput (arguments)
      • com.bbn.kbp.events2014.ResponseLinking (linking)
        • with ResponseSetIDs provided to the LinkingStore as created
  • bbn jar test: successfully imported 2 bbn tac-kbp-eal classes which generate ID's
  • created KBPEALApp.java / need to add to it
  • Next:
    • "meet" the RawText and NewsSpike
      • need to decide what exactly to send newsspike as "text", how to capture that, keeping the offsets in mind
      • should insert the bbn tac-kbp-eal classes

.

Jun 10, 2016

OUT: Thu-Fri (Chicago)

DEFT repository

• newsspike
  • Dave Kolas (BBN) emailed saying that Conner said before he left BBN that newsspike passed all it's tests and can be released
  • However, Dave says that the Relations class that we're using was created for OpenIE, and not for a fixed set of relations
  • He says it's not clear that that's the case by the class name
  • After discussion, we decided that it would be good to use DocumentEvents instead of Relations
  • We use DocumentEvents for the KB work
  • need to make this change in the repository, get all the tests to pass again, and run a new Jenkins job

KBP2016 EAL

• newsspike returns sentence token offsets, not sentence char offsets, and not document char offsets
• Investigated/reviewed how offsets were computed for previous KBP's
  • Uses offsets from the sentence tokens
  • Annotation doc = Preprocessing.pipeline.annotate(docText);
  • List<CoreMap> sentences = doc.get(SentencesAnnotation.class);
  • List<CoreLabel> tokens = sentence.get(TokensAnnotation.class);
  • int sentenceStartOffset = tokens.get(0).beginPosition();
• modified: KBPEALApp.java: Sending data as sentence rather than as doc, in order to capture doc offsets
• compiles
• runtime error in: Preprocessing.initPipeline() from newsspike code
  • soln: created a separate, simple pipeline to get sentences, rather than using Preprocessing.InitPipeline()
  • this is running, but may need to change it, may be too slow...
    • Investigate modifying: newsspike/figer classes: FPS and Tuples
• runtime error: can't find reverb-latest.jar in lib dir, and can't find: edu.washington.cs.knowitall.nlp.extraction.ChunkedExtraction
  • soln: identified "reverb" dependency in the repo/deft/washington pom.xml and added it to this project's pom.xml; i.e.
    • <dependency>
      • <groupId>edu.washington.cs.knowitall.chunkedextractor</groupId>
      • <artifactId>chunkedextractor_2.10</artifactId>
      • <version>1.0.5</version>
    • </dependency>
• runs w/o any runtime errors
• test files: running without "error"; however, not returning expected events
  • excellent news -- adding these 2 dependencies results in events! Hallelujah!
  • Not sure what it was using before, but ... this is similar to problem had with the web demo (0 events to events!)
  • soln: added two dependencies to pom.xml:
    • <dependency>
      • <groupId>edu.washington.cs.knowitall</groupId>
      • <artifactId>reverb-core</artifactId>
      • <version>1.4.1</version>
    • </dependency>
    • <dependency>
      • <groupId>edu.washington.cs.knowitall </groupId>
      • <artifactId>reverb-models</artifactId>
      • <version>1.4.0</version>
    • </dependency>
• getting events from:
  • testFile4.txt,NYT_ENG_20131216.0185.xml,ba8c897bb81f1fbeb54946955e23663a.mpdf.xml,a4847d19aa3d8b2d9601ebd9e0138477.mpdf.xml
• doc offsets:
  • the current code is working for: a self-made doc, a newswire doc, and 2 discussion forum docs
  • issue: except, if the event is in the first sentence of the document (determined by Stanford CoreNLP), in that case, the arg1 contains the arg1, as well as unwanted sgml tags, need to adjust this
  • issue: as mentioned above, this sentence by sentence approach call to "ParseStanfordFigerReverb" is slow
    • maybe the code/approach within this outer loop can be moved into ParseStanfordFigerReverb" and the 2 subsequent calls
    • the slowness is related to creating and using the Stanford pipeline twice
• Next:
• use BBN tac-kbp-eal classes:
  • step 1: create a "Response"
• check mapping for: kill/person/person
  • Conflict.Attack
  • Life.Die

.

Jun 17, 2016

DEFT Conference Call

• Marjorie (BBN) recommended that Ajay (UMass) contact me about adding HLTContentContainer data to the KB
  • Ajay (UMass) contacted me
  • sent him some comments/tips
• KB
  • our KB seems to be pretty big, maybe the biggest?
  • Stanford once mentioned they would process 50k docs, and later said 1k docs, not sure of the end result
  • we processed between 600k-1mil docs (have to check the records to get exact number); i.e. a lot
  • a lot of schools haven't called in the last two calls; though more this time, than the previous call

KBP2016EAL

• dryrun
  • will follow-up with BBN on our output
    • sent a general dryrun reply and a question about the Personnel.EndPosition event below (which is not in the LDC answer key)
  • our system found events in 5 of the subset of 91 docs scored (there were 2092 docs total)
• • • AFP_ENG_20100414.0615 - this matches
    • sentence: Cipriani left Ensisheim in a police vehicle bound for an open prison near Strasbourg where the police officer said he was due to do community service including working at a food bank as part of his parole.
      • 131 AFP_ENG_20100414.0615 Movement.Transportperson Person Cipriani 731-739 731-754 731-739 NIL Actual 1.0
      • 132 AFP_ENG_20100414.0615 Movement.Transportperson Origin Ensisheim 745-754 731-754 745-754 NIL Actual 1.0
• • • c06e8bbdf69f73a69cd3d5dbb4d06a21 - should match an answer in LDC answer key but LDC is missing it,
      • not sure if the CAS's are ok with the "The" and the "coach"
    • sentence: The St. Louis Rams fired coach Steve Spagnuolo and general manager Billy Devaney, the team announced on Monday.
      • 141 c06e8bbdf69f73a69cd3d5dbb4d06a21 Personnel.EndPosition Entity The St. Louis Rams 105-123 105-151 105-123 NIL Actual 1.0
      • 142 c06e8bbdf69f73a69cd3d5dbb4d06a21 Personnel.EndPosition Person coach Steve Spagnuolo 130-151 105-151 130-151 NIL Actual 1.0
• • • NYT_ENG_20130625.0044 - no events of type Conflict.Broadcast in LDC answer key
    • question: check the newsspike mapping / is there a better one to map to?
    • sentence: Noting that Mandela’s 95th birthday is on July 18, Zuma urged South Africans to celebrate his achievements.
      • 293 NYT_ENG_20130625.0044 Conflict.Broadcast Entity Zuma 1387-1391 1387-1412 1387-1391 NIL Actual 1.0
      • 294 NYT_ENG_20130625.0044 Conflict.Broadcast Entity South Africans 1398-1412 1387-1412 1398-1412 NIL Actual 1.0
• • • NYT_ENG_20130828.0147 - looks like this matches
    • sentence: According to his lawyers, Martin-Artajo left London “on a long-planned vacation” to his native Spain without knowing of the pending charges.
      • 349 NYT_ENG_20130828.0147 Movement.Transportperson Person Martin-Artajo 2050-2063 2050-2075 2050-2063 NIL Actual 1.0
      • 350 NYT_ENG_20130828.0147 Movement.Transportperson Origin London 2069-2075 2050-2075 2069-2075 NIL Actual 1.0
• • • NYT_ENG_20131210.0203 - no events of type Conflict.Broadcast in LDC answer key

sentence: In a meeting with senior Kremlin officials in Moscow on Monday, the State Department said, Nuland “urged Russia to use its influence to press for peace, human dignity and a political solution” and stressed Ukraine’s need for “a return to economic health with the support of the International Monetary Fund.”

• • • • 491 NYT_ENG_20131210.0203 Conflict.Broadcast Entity Nuland 1533-1539 1533-1553 1533-1539 NIL Actual 1.0
      • 492 NYT_ENG_20131210.0203 Conflict.Broadcast Entity Russia 1547-1553 1533-1553 1547-1553 NIL Actual 1.0

• met with Colin and James / will meet again
  • good meeting
  • they will ask Dan and Hannah about what runs we should submit for EAL - ?
    • Combinations, Congle's Newsspike, Colin's newsspike, James's Algorithm
• offsets
  • added sentenceId to Tuple class, to assist with offsets
  • adjusted NewsSpikePredictKB.predict to use sentenceId
  • eliminated sentence-by-sentence approach (too slow)
  • now getting correct offsets for the Congle newsspike
• xml tags
  • clean with Stanford or separate program?
    • testing Stanford's cleanxml annotator
    • the Stanford cleanxml annotator is not a good solution, for the Congle newsspike case
    • will write a function to create a docString with tags replaced by spaces
• task description - reviewing
  • scoring
  • arg and event coreference
• bbn classes
  • want to create a Response, Linking, SystemOutput, etc.
    • yay!, successfully created a Response, using the Response.Builder()
  • communicated with Jay (bbn) about their recommended classes
  • bbn auto-generates code for classes which are Immutables
    • the code is not visible in the repo
    • constructors and setters are not exposed
    • a kind of Builder is used to create these objects
    • more info at: http://immutables.github.io/

.

Jun 24, 2016

DEFT Repository

• need to modify newsspike to use DocumentEvents rather than Relations (which multir used, but maybe was only meant for "OpenIE")
• putting this on hold for now

KBP2016EAL

• dryrun - newsspike system
  • found events in 211/2092 documents
  • found 5 events in the subset of 91 documents scored
    • 2 of the events match the answer key, 1 should match but is missing in the answer key, the other 2 should be mapped to a different event
• created a createCleanXMLString() to replace doc tags with spaces in the docString
  • still getting correct offsets from newsspike, and eliminated getting a <tag> and \n in an event's arg1Name
• practicing with BBN classes: Response, ArgumentOutput, ResponseLinking, DocumentSystemOutput2015, SystemOutputStore2016
  • assists with assigning a ResponseID and writing system output
  • created the string of classes needed to get to the point of writing system output, compiles
  • getting a runtime error when writing the system output
    • emailed BBN a question for help, BBN replied
  • got the BBN classes to write a test case of argument and linking output, without the runtime error
• scoring
  • arguments - for example, what if CAS is Obama rather than Barack Obama, is that correct?
  • linking - what if events are not collapsed to event hopper?
• arguments
  • how to collapse arguments
    • not focusing on this, since newsspike output was so low
• event hopper
  • how to collapse events?
    • not focusing on this, since newsspike output was so low
• Colin and James
  • met with Colin
  • emailed them about:
    • Dan's answers to questions we had raised
    • a "format" for their systems, +/- a little
    • plan to meet again next Wed
    • asked James if he could meet before next Wed to discuss running his system on the answer key
• working on finishing the end-to-end newsspike system (close)
  • created a NewsSpikeToKBPEAL2016.csv, which maps newsspike events to the Event Taxonomy given in the KBP EAL 2016 task description
  • created an KBPTaxonomy class to read and store the mapping in NewsSpikeToKBPEAL2016.csv
  • writing the code to translate between NewsSpikeRelationPrediction to KBP EAL 2016 output for arguments and linking
  • putting the BBN classes in place in the pipeline
• next:
  • run the newsspike system on the 91 answer key docs
  • validate the output
  • score the output
    • need to identify the BBN class for this
  • run James's system on the answer key docs
  • potentially, extend newsspike system to include more event types

.

Jul 1, 2016

DEFT Repository

• responded to Roger Bock's questions, as he was updating and closing JIRA tickets
• emailed Dan questions re vinculum (do we want to add it?) and markov logic (ok that it was dropped from Algorithm Readiness table?)
  • answers:
    • ok to drop markov logic
    • got answers from Xiao re vinculum
• responded back to Roger, re vinculum; he's planning to ask the stakeholders if we should include vinculum

KBP2016EAL

• shared info with the group:
  • BBN email addresses for EAL questions; i.e. Marjorie, Ryan, and Jay
  • annotation docs in the dryrun answer key packet (4 pdf's = argument filler, entities, events, relations)
  • 3 docs from the event track site 1)updated task description 2)2 annotation docs
• met with Colin and James
  • shared file system location of answer key docs
  • shared a file format for the event output (i.e. a second pass at it, after discussion today)
    • workflow - communicating with BBN about their recommendations re whether or not to use their classes
      • i.e. it *may* not provide any benefit for James and Colin to use them
      • Marjorie/BBN responded:
        • the main benefit is that the output will validate,
        • using Jython is one way to use their java classes with Python
  • planning to share core nlp doc processing on James's and Colin's systems
    • newsspike's corenlp is at least a little complicated, maybe hard to share the doc processing, need to investigate further
  • talked about processing time required to process 30k docs
    • newsspike is slow, as-is, cluster required
    • one reason for the newsspike slowness is that CoreNLP pipeline is being created for each document, ala the DEFT repo approach of processing one doc at a time and then calling the algorithm and Congle processed one file of many sentences at a time (which = 1 doc)
      • should explore newsspike code changes to create the pipeline once for a set of docs, and then annotate each doc
• end-to-end newsspike
  • an e2e system is in place which validates
  • it is slow (for 30k) docs, need to explore adjusting code re creating CoreNLP pipeline only once for a set of docs
  • confidences are all 1.0, need to incorporate the newsspike values into the BBN classes (for some reason confidence is not a field in their Response class, so that data needs to be handled differently, more complex handling required)
• newsspike validates
  • incorporated the BBN classes for the output
  • successfully writes output for: /arguments and /linking
  • validated output from akTestSix test (i.e. a test using 6 files from the answer key set)
    • mvn exec:java -pl washington -Dexec.mainClass="com.bbn.kbp.events2014.bin.ValidateSystemOutput2016" -Dexec.args="/projects/WebWare9/nhawkins/ealValidate/akTestSix"
• newsspike scoring - can't score yet
  • scoring output - using BBN's evaluate2016.sh
    • scoring class: tac-kbp-eal/tac-kbp-eal-scorer/src/main/java/com/bbn/kbp/events/ScoreKBPAgainstERE.java
    • script: evaluate2016.sh (which calls the class above, and many other tasks)
    • configured a parameter file for evaluate2016.sh, for the akTestSix output
    • error: when trying to build BBN's latest tac-kbp-eal, which contains evaluate2016.sh
      • emailed BBN...
      • BBN updated the build instructions on their github, will try again
      • resolved: cloned another BBN respository, built that, then could build tac-kbp-eal
    • error: when running evaluate2016.sh, outputs funny name for store dir name; i.e. looks like a bug in the script
      • BBN responded,
        • the output dir param should be: /projects/WebWare9/nhawkins/ealOutput,
        • not /projects/WebWare9/nhawkins/ealOutput/akTestSix
        • need to list empty file for "queries" parameter
      • it benefits us to use this script, they will use it for the evaluation
      • the script requires first processing the docs-to-score with CoreNLP 3.6, so that relaxed scoring can be done
      • downloaded the CoreNLP 3.6 zip with various jars
      • wrote a script relaxScoreCoreNLP.sh to process the docs (annotator list specified in BBN tac-kbp-eal Readme.md)
      • processed a the set of akTestSix docs with CoreNLP3.6
      • seems like everything is in place in the params file for evaluate2016.sh
        • error: getting an error about the queryFilter when running the script
        • waiting to hear back from BBN
  • set-up a github project: kbp2016eal
    • created new repository kbp2016eal on github (w/o README.md)
    • created git repo in nhawkins/kbp2016eal/; i.e.
      • echo "# kbp2016eal" >> README.md
      • git init
      • git add README.md
      • git commit -m "first commit"
      • git remote add origin https://github.com/nrhawkins/kbp2016eal.git
      • git push -u origin master
    • pushed a couple of files

.

Jul 8, 2016

Monday = July 4

DEFT KB

• received email that the KBExplorer UI and an instance of our KB will be released in early August
  • at that time, need to try it, and examine the entities and relations in the KB, may receive questions from DARPA/stakeholders
  • link to Explorer: KBExplorer (link: https://deft-demo-dmz-02.bbn.com:8443/owf/login/login.html)
  • looked at the KBExplorer using Chrome, didn't seem to work with Mozilla?
  • Interesting. Clicking on the Entity link, results in an infinitely spinning circle. Maybe they're not done yet; i.e. they mentioned early August.
  • need to review the slides Marjorie sent
• responded to questions from Ajay (umass) re the KB; he has a link to the KBExplorer
  • KBExplorer (link: https://deft-demo-dmz-02.bbn.com:8443/owf/login/login.html)

DEFT Repository

• need to merge 2.4.1 to incorporate BBN's fixes, which resolve minor issues they discovered for integrating uwtime
  • before merge
    • mvn compile -pl washington
      • success
    • mvn verify -pl washington -skipRegressionTest=false
      • success: for figer, uwtime, nsre (although testfile list = 1 file), multir, ollie
      • error: reverb, heap memory error (?)
  • merging
    • git fetch adept
    • git merge v2.4.1
    • git checkout --theirs adept/adept-kb/src/main/resources/adept/kbapi/KBParameters.xml
    • git add adept/adept-kb/src/main/resources/adept/kbapi/KBParameters.xml
    • git checkout --theirs washington/pom.xml
    • git add washington/pom.xml
    • git commit -m "merge adept 2.4.1"
    • git merge v2.4.1
    • saved the merge message that popped-up in pico editor
    • looks like merge completed
  • check merge
    • mvn compile -pl washington -am
      • success
    • mvn verify -pl washington -am -DskipRegressionTest=false
      • error: INFO] ------------------------------------------------------------------------
      • [INFO] Reactor Summary:
      • [INFO]
      • [INFO] deft .............................................. SUCCESS [0.654s]
      • [INFO] adept ............................................. SUCCESS [0.073s]
      • [INFO] adept-api ......................................... SUCCESS [45.316s]
      • [INFO] adept-kb .......................................... SUCCESS [16.064s]
      • [INFO] adept-mapreduce ................................... SUCCESS [1.196s]
      • [INFO] washington ........................................ FAILURE [5:44.603s]
      • [INFO] ------------------------------------------------------------------------
      • [INFO] BUILD FAILURE
      • [INFO] ------------------------------------------------------------------------
      • [INFO] Total time: 6:48.261s
      • [INFO] Finished at: Tue Jul 05 17:06:02 PDT 2016
      • [INFO] Final Memory: 73M/9583M
      • [INFO] ------------------------------------------------------------------------
      • [ERROR] Failed to execute goal org.apache.maven.plugins:maven-jar-plugin:2.3.2:jar (default-jar) on project washington-adept: Error ****assembling JAR: invalid entry size -> [Help 1]
    • posted Comment/Question in JIRA ticket
      • BBN responded: try "mvn install", instead of "mvn compile" and try the "./DeftRegress.sh" instead of "mvn verify"
      • specifically:
      • mvn clean install -pl :washington-adept -am -Pjava8
      • $ ./DeftRegress.sh washington > temp.txt 2> temp.err
      • worked!
• git push'd the washington repo, merged with v2.4.1
• BBN closed this "merge request" ticket

KBP2016 EAL

• evaluate2016.sh - want to score "akTestSix" output
  • BBN pushed a new version of this file, to address the "quoteFilter" bug
  • did a 'git pull' on tac-kbp-eal repo
  • tried the script, and got a new "quoteFilter" error, so emailed BBN
    • ++ /homes/gws/nhawkins/tac-kbp-eal//tac-kbp-eal-scorer/target/appassembler/bin/scoreKBPAgainstERE /projects/WebWare9/nhawkins/ealScratch/params/akTestSix/scoreKBPAgainstERE.params
    • ++ tee /projects/WebWare9/nhawkins/ealScratch/log/akTestSix/scoreKBPAgainstERE.log
    • 18:53:58.155 [main] INFO c.b.b.c.p.s.SerifStyleParameterFileLoader - Loading parameter file /projects/WebWare9/nhawkins/ealScratch/params/akTestSix/scoreKBPAgainstERE.params
    • com.google.inject.ProvisionException: Unable to provision, see the following errors:
    • 1) Error in custom provider, com.bbn.bue.common.parameters.exceptions.MissingRequiredParameter: Missing required parameter quoteFilter
    • at com.bbn.kbp.events.ScoreKBPAgainstERE$GuiceModule.getQuoteFiler(ScoreKBPAgainstERE.java:970)
    • while locating com.bbn.kbp.events2014.transformers.QuoteFilter
    • for parameter 3 at com.bbn.kbp.events.ScoreKBPAgainstERE.<init>(ScoreKBPAgainstERE.java:140)
    • while locating com.bbn.kbp.events.ScoreKBPAgainstERE
  • BBN pushed new code to the github
  • pulled the new code, re-built the BBN projects
  • ran the evaluate2016.sh again, getting error:
    • 11:19:35.584 [main] INFO c.b.b.c.p.s.SerifStyleParameterFileLoader - Loading parameter file /projects/WebWare9/nhawkins/ealScratch/params/akTestSix/scoreKBPAgainstERE.params
    • java.lang.RuntimeException: Corrupt line #1: /projects/WebWare6/KBP_2016/dryrun/LDC2016E60_TAC_KBP_2016_ English_Event_Argument_Linking_Pilot_Gold_Standard/data/ere/AFP_ENG_20100414.0615-kbp.rich_ere.xml
    • at com.bbn.bue.common.files.FileUtils.loadStringToFileMapToSink(FileUtils.java:315)
    • at com.bbn.bue.common.files.FileUtils.loadStringToFileMap(FileUtils.java:273)
    • at com.bbn.bue.common.files.FileUtils.loadSymbolToFileMap(FileUtils.java:223)
    • at com.bbn.kbp.events.ScoreKBPAgainstERE.go(ScoreKBPAgainstERE.java:151)
    • at com.bbn.kbp.events.ScoreKBPAgainstERE.trueMain(ScoreKBPAgainstERE.java:941)
    • at com.bbn.kbp.events.ScoreKBPAgainstERE.main(ScoreKBPAgainstERE.java:931)
  • submitted error to BBN, waiting to hear back... (they're in the process of developing this, so I am probably helping them debug, or helping them discover what they haven't merged into the master branch)
  • adjusted goldMap - moved files to place with shorter dir name, might have been a line break , "Corrupt line #1" dissappeared
  • getting a runtime error about ealScratch/params/params file not found
  • this error is for the last step in the script (#6), which is xcorpus, so we can ignore
  • success: got arg and linking scores for the 6 files
  • emailed BBN for interpretation of scoring in one case, the score was a TP for Ensisheim and FP for Cipriani
    • 56b69aa271eeac09a3f9613cf22b501c15a35fec AFP_ENG_20100414.0615 Movement.Transport-Person Origin Ensisheim 1053-1062 1039-1062 1053-1062 NIL Actual 1.0
    • 98330fc7806d26e508192bcea9353721ae550a52 AFP_ENG_20100414.0615 Movement.Transport-Person Person Cipriani 1039-1047 1039-1062 1039-1047 NIL Actual 1.0
    • Cipriani was scored as an alignment failure, not sure why, so it should be helpful to learn
    • AFP_ENG_20100414.0615-Movement.Transport-Person-Person([1039:1047]=Cipriani[1039-1047]; Actual; [1039-1062]; [])
• examine tuple and fps data, can rules be applied to it?
  • tuples has: arg1str, arg2str, and relstr
  • can probably work with that
  • examining tuples from the 6 test docs
• move annotation pipeline out, to apply to multiple docs and to share processed docs
  • success
• met with Colin and James
  • they ran their current systems on the 91 answer key docs
  • their systems extract event triggers, not yet event args
  • they're going to create gitlab or github projects for their code

.

Jul 15, 2016

DEFT Repo

• received a JIRA request for input on ReVerb Benchmark Test
  • what does it need to run?

KBP2016EAL

• EAL systems
  • BBN and IHMC had systems last year, made presentations at the 2015 Workshop
  • http://www.nist.gov/tac/publications/index.html
• argument scoring
  • waiting to hear back from BBN re a case where the scoring program says one of the 2 arguments fails to align, it appears it should align?
    • BBN is busy with LORELEI evals
    • don't need to hear back, I tested adjusting the offsets by subtracting one from the end offset (to remove the 1 trailing white space)
    • this solved the problem, we now have the expected 4 TP's in the small test subset
  • for the 91 answer key docs, out of 10 arguments (from 5 events), we have 4 TP's (true positives) and 6 FP's (false positives)
    • 2 FP args: answer key is missing a true event
    • 2 FP args: trigger "urged", event Contact.Broadcast, not in answer key (Zuma urged S. Africans)
    • 2 FP args: trigger "urged", event Contact.Broadcast, not in answer key (Nuland urged Russia)
    • 2 TP args: Martin-Artajo left London, scored as correct
    • 1 TP args: Cipriani in Cipriani left Ensisheim, scored as correct
    • 1 TP args: Ensisheim, is scored as correct
• estimating timing
  • newsspike: 91 answer key docs, 22 mins
  • newsspike: 2092 dryrun docs, ~9.67 hrs
  • newsspike: estimate for 30k docs, ~139 hrs, over 10 machines = 13.9 hrs
  • congle: sent an email asking his thoughts about newsspike's speed, possible speed-ups
  • corenlp: 2092 dryrun docs,
• preprocessing the docs ?
  • what is core nlp version?
    • dryrun scoring: 3.6.0
    • deft repo: 1.3.5
    • nsre2: 3.5.2
    • james: 3.4
  • what are the core nlp ner models used?
    • newsspike: 3-class
    • james: default (= 3-class, then 7-class, then 4-class)
  • what are pipeline properties?
    • newsspike:
      • Properties props = new Properties();
      • props.put("parse.maxlen", "80");
      • props.put("ner.model","edu/stanford/nlp/models/ner/english.all.3class.distsim.crf.ser.gz");
      • props.put("annotators", "tokenize, ssplit, pos, lemma, ner, parse");
      • pipeline = new StanfordCoreNLP(props);
    • scoring against dryrun 91 doc answer key:
      • -annotators tokenize,cleanxml,ssplit,parse
    • james:
      • tokenize,ssplit,pos,lemma,ner,parse,dcoref -dcoref.maxdist 1
  • how to read-in processed CoreNLP xml docs?
    • don't need to do this
    • edu.stanford.nlp.util.XMLUtils.readDocumentFromFile(String filename) creates a org.w3c.dom.Document
  • how to read-in processed CoreNLP txt docs?
    • James parses this output into a class he created
    • newsspike uses the coreNLP data from the CoreNLP Annotation class, would need to create an Annotation object
  • how to populate an Annotation object?
• arguments/=responses
  • need to set the confidences in the BBN classes; as is, they're being set to 1 (most of them are 1)
• tuples data
  • what's there? took a look at this
  • make rules?
• BBN classes - benefit = they guarantee that output will validate
  • James should be able to use them directly
  • Colin would need to use Jython, we haven't used Jython before
    • may need to write a java program to load the data, create the classes, write the output
• meeting with Colin and James
  • scoring program details:
    • If realis is wrong, the argument is scored as a False Positive (FP)
    • waiting to hear back from BBN on "FP, alignment failures" on cases which look correct
      • resolved: emailed them, end offset was the issue, needs to stop exactly at the end of the argument text, not +1 white space
  • CoreNLP sutime
    • normalized time data is populated when ner is an annotator and the 7-class ner model is used
  • Colin is meeting with Stephen to develop his event extraction plan
• ran a new dryrun of 2092 docs, findings:
  • even though I moved the corenlp pipeline out to be called once for all docs, the run time is still long, 9.67 hrs on porvo
  • some crashes from parseDouble of the newsspike confidence string (checking NewsSpikePredictKB.java, line 178)
    • appear to have fixed this one, the string was 1 too short, and for a case like 0.1345E-5, were left with 0.1345E- i.e. not a number
    • added code to select the event with the max confidence
  • need to write size 0 file, if a document crashes
    • added code to do this
  • offsets appear to be off, in some docs, certainly not off in all the docs, need to investigate
    • one example file, offsets look off in some relations, not all, printed to screen, though all look ok in relevant args printed to output
    • will investigate further
  • if the newsspike arg1 is the first words in the text of the doc, the arg1 includes text just before it, which is extra-ish
• Congle, asked him opinions about:
  • newsspike speed
  • using default ner models (i.e. 3class-7class-4class) vs. 3-class ner model

.

Jul 22, 2016

DEFT Data Integration Conference Call

• Wednesday, they very briefly mentioned the KBExplorer
• they were asking groups generally about non-English plans and their algorithms, though didn't ask us

DEFT Repository

• vinculum:
  • Roger (BBN) asked about this algorithm and MongoDB at the most recent stakeholders mtg
  • "BBN does not currently include MongoDB in their framework and don't have the resources to support it"
  • "in the Readiness Table, the algorithm is being marked as "Received (not integrable)""
  • the JIRA ticket has been closed
• figer:
  • Roger emailed about a unit test failure, re reverb, when taking steps to integrate figer
  • updated KBParameters.xml and KBParametersUnitTests.xml, required by the latest merge with Adept v2.4.1, so that the kbapi tests run, and we can reach the washington unit tests
  • the unit tests: the multir tests run, then reverb tests end in error: Out of Memory
  • submitted comment on JIRA ticket
  • adjusting MAVEN_OPTS did not cause the memory error to disappear
  • set aside the multir unit test, so just the combined ollie and reverb tests run
  • the null pointer error for the reverb and ollie tests: may be coming from the sentencer not being initialized in "activate"?
    • explored this, submitted a new comment to the JIRA ticket
      • seems that maybe a file got overwritten during a merge? so that now SimpleExtractorTest and ReVerbExtractor don't match?
      • maybe BBN can help us explore the history
• reverb:
  • made a first response to the JIRA question sent by Nick (BBN) to Stephen (commented that he's retired, and that it should be directed to me) re reverb benchmark test
  • when I run the test, I get the "import" error:
    • File "edu/washington/reverb/bin.benchmark/score", line 4, in <module>
    • from nltk.stem.porter import PorterStemmer
    • ImportError: No module named nltk.stem.porter
    • Exit value: 1
  • Nick is asking if they need to add something to the Jenkins system (maybe they need to install 'nltk')
  • placed a request with cs support to install the nltk module on: porvo, kupo and serenity
  • got nltk installed, plus self-installed necessary data (punkt and averaged_perceptron_taggers) in nltk_data
  • now, reverb benchmark test runs successfully
  • updated JIRA ticket
• newsspike:
  • offset fix mentioned below will need to be made in the DEFT repo
  • need to change Relations data structure to DocumentEvents

KBP2016EAL

• newsspike offsets, discovered an issue for some docs during last run of the 2092 dryrun docs
  • fixed: some tuples were missing a valid sentence id, so document tokens were not accumulating properly
  • make sure to -1 from the "token".endPosition() for the argument end offsets
• core nlp timing for: James pipeline and 2092 dryrun docs (5.5 hrs)
• core nlp:
  • can I create an annotation object from core nlp text output?
    • so far, so good
  • need code to read the core nlp text output
  • does the default 3-7-4-class work with newsspike?
• newsspike arguments, can be too long-ish
  • egs. "the St. Louis Rams", "the United States", other longer egs with date or "author=xxx", and maybe a \n, in them
  • what to do?
• KBP Taxonomy
  • review the 62 mappings for newsspike
    • need any revisions?
• Time
  • do we handle it?
  • normalization = 3-timex or other?
• scoring questions / can address with evaluate2016.sh
  • impact of duplicate events?
• --------------------------
• Evaluation: Docs
• --------------------------
  • LDC Joe Ellis says we will receive all 90k docs, and they will be clearly separated (by dir) for each language
• ------------------------------------
• Evaluation: Computers
• ------------------------------------
  • Luke says it's ok to use the nlp cluster during the evaluation week: Aug. 22-29,
    • estimate: CoreNLP processing of 30k docs = 76 hrs, or with 10 machines, 7.6 hrs (~5.5 hrs for 2092 docs)
    • estimate: newsspike 10 machines, 13 or so hrs (w/o doc pre-processing, 5-6 or so hrs (w/doc pre-processing) (~9.5 hrs for 2092 docs)
    • estimate: James, Colins, TBD
• other systems
  • BBN:
    • anchor (event trigger) model: logistic regression with features: surrounding words, associated dependency strucure, topic of doc, etc.
    • argument model: given event and noun phrase, predict role, logistic regression with features: anchor, candidate arg, text strings between them, their associated dependency structures
    • argument independent model: given a mention, predict the presence of an event type and role the mention plays in the event
      • targets improved recall for mentions that are highly indicative of certain classes of events (eg. the protesters->Conflict.Demonstrate, Agent, the protestors)
    • Realis: predicted
    • Document-level inference
• meeting with James and Colin
  • discussed pre-processing of docs
    • what to do before core nlp
    • the core nlp details

.

Jul 29, 2016

OUT: Mon-Fri (Chicago)

.

Aug 5, 2016

Email Questions

• responded to Chris Lin's question about needing sentences from same corpus as Angli's extractions (i.e. the derby db files)
  • /projects/WebWare8/Multir/MultirSystem/files/corpus/FullCorpus/SENTTEXTINFORMATION
    • wc -l SENTTEXTINFORMATION
    • 27353734 sentences
  • /projects/WebWare6/DEFT_corpora
• responded to Dan/Stephen question about Marjorie's email re EAL training data
  • stephen downloaded the data in March to: /projects/WebWare6/Deft-event-corpora
• responded to Dan's question about Chris working on Spanish
• responded to Nancy Johnson-Burr about upgrading to CentOS and installing nltk
  • sent install instructions for nltk, asking if they might work; i.e. pip rather than yum approach

DEFT Repository

• spending too much time on this, moving on...
• reverb benchmark test, re the python score script requires the nltk module
  • responded to JIRA conversation, the benchmark test, not the reverb algorithm, depends on nltk
  • BBN plans to continue to mark the algorithm as "releasable", but plans to make notes that nltk is required for the BM test
  • BBN is trying to get the BM test to work on their side; it works on our side
  • side note: the nltk module is not available for the new linux, CentOS, being installed on our machines, not sure if the alternative modules will work
  • BBN responded, the BM test now works on their side
  • resolved: JIRA ticket closed
• unit tests
  • "git log <filename>"
    • indicates that SimpleTestExtractor.java and ReVerbExtractor.java have not had new commits since Aug 11, 2015 (and that one was made by BBN)
    • the Aug 11, 2015 commit indicates changes were made to the adept Document and IDocumentProcessor
      • maybe additional changes have been made there, breaking ReVerbExtractor?
  • error:
  • SimpleTestExtractor and ReVerbExtractor don't match:
    • the sentencer is null in ReVerbExtractor, it is not being initialized
    • it is initialized in activate() -- did something once call activate()?
      • the error msgs:
      • testShortDocumentWithReVerbExtractor(edu.washington.SimpleExtractorTest) Time elapsed: 0.049 sec <<< ERROR!
      • java.lang.NullPointerException: null
      • at edu.washington.ReVerbExtractor.process(ReVerbExtractor.java:148)
      • at edu.washington.SimpleExtractorTest.testShortDocumentWithReVerbExtractor(SimpleExtractorTest.java:79)
      • testShortDocumentWithOllieExtractor(edu.washington.SimpleExtractorTest) Time elapsed: 0.014 sec <<< ERROR!
      • java.lang.NullPointerException: null
      • at edu.washington.OllieExtractor.process(OllieExtractor.java:139)
      • at edu.washington.SimpleExtractorTest.testShortDocumentWithOllieExtractor(SimpleExtractorTest.java:128)
  • try this again:
  • mvn verify -pl :washington-adept -am -DskipUnitTests=false
    • result:
    • java.io.FileNotFoundException: en-token.bin (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileInputStream.<init>(FileInputStream.java:93) at edu.washington.SimpleExtractorTest$ReverbNoConversionExtractor.getExtractions(SimpleExtractorTest.java:188)
  • submitted a comment to the JIRA ticket
    • seems like a different version of ReVerbExtractor once existed
    • seems like activate() needs to be called
    • activate(String configFile), actually, so we need to know a value to insert for configFile
    • see if, BBN has any insights on why the current ReVerbExtractor is the way it is
    • we haven't made any changes to ReVerbExtractor in recent history
  • status/options: BBN responded
    • bug/problem: looks like .activate needs to be called before .process / mystery = *I think* it used to work, unclear to BBN why bug now
    • we can fix the unit tests, write new unit tests, or do away with them because they are not required, just regression and benchmark tests
    • it's probably good to do away with them, and let the RT and BM tests do the checking
    • might try to add a call to .activate for curiosity sake / if it works, then *maybe* keep the unit tests / they could get in the way in future
  • Removing the (not required) unit tests from the washington repository, relying on the regression and benchmark tests, which pass
    • git rm washington/src/test/java/edu/washington/MultirExtractorTest.java
    • git rm washington/src/test/java/edu/washington/SimpleExtractorTest.java
    • mvn verify -pl :washington-adept -DskipUnitTests=false
    • mvn verify -pl :washington-adept -am -DskipUnitTests=false
    • git commit -m "removing unit tests"
    • git push

KBP2016EAL

• met with Colin and James
  • discussed preprocessing
  • goal: next week, they generate output to score
  • review/prepare BBN scripts for validating and scoring
• Core NLP sharing
  • can I create an Annotation from the CoreNLP text output?
  • yes, interestingly; but, newsspike is slow, so will need to use the cluster anyway, so going to process on-the-fly
• Create file to combine system output
  • for args: compare type, role, and offsets
• Preprocessing
  • change approach to removing XML tags from the doc string
• Cluster
  • write scripts to run batch jobs
• Submission Output: zip methods
  • zip -r test.zip .
  • or
  • jar cMf test.jar *

.

Aug 12, 2016

UW CSE Machine Upgrade to CentOS

• nltk, needed for DEFT Repo, ReVerb Benchmark tests
  • attempted to install it myself, as suggested by cs suppport
  • kupo> pip install nltk
    • error: permission denied, can't write to python packages space
  • emailed cs support, asking if they know a way around that
  • looking into virtualenv
    • https://us.pycon.org/2012/community/tutorials/199/
    • $ virtualenv nltkenv
    • $ nltkenv/bin/pip install nltk numpy
    • Successfully installed nltk numpy
    • Cleaning up...
    • how to use a virtualenv? (created: nltkenv)
      • http://docs.python-guide.org/en/latest/dev/virtualenvs/
      • virtualenv -p /usr/bin/python2.7 ntlkenv
      • source nltkenv/bin/activate
      • pip install requests
      • deactivate
      • need to test w/BM tests
        • from nhawkins home dir: source nltkenv/bin/activate
        • cd repo/deft
        • in washington pom.xml, set skipBenchMarkTest=false for ReVerb
        • mvn verify -pl washington -DskipBenchmarkTests=true
        • test passes: Benchmark Results:Precision = 0.5833333333333334, Exact Relation Recall = 0.03508771929824561
        • from nhawkins home dir: deactivate
• emailed cs support (Nancy Johnson Burr) indicating that the python virtualenv works for me on kupo, and can go ahead with upgrade on serenity and porvo

DEFT KB

• examined our KB at:
  • https://deft-demo-dmz-01.bbn.com:8443/owf/login/login.html
• TODO? (later)
  • load documents, so that the justification appears?

DEFT Repository

• Unit Tests
  • pushed changes
  • updated JIRA ticket
• Merge in new Adept
  • https://www.deftdocs.bbn.com/DEFTWiki/How%20to%20pull%20updates%20from%20master%20DEFT%20git%20repository.aspx
  • git checkout -- <filename> to discard changes in a few files
  • git fetch adept tag v2.4.2
  • git pull adept tag v2.4.2
  • saved merge note in pico editor
  • mvn clean install -pl :washington-adept -am -Pjava8
  • ./DeftRegress.sh washington > temp.txt 2> temp.err
  • checked regression tests: all passed (newsspike, uwtime, figer, multir, ollie, reverb)
  • git push
  • merge complete
• ReVerbExtractor
  • BBN is asking if we can make it thread-safe
  • A piece of code is thread-safe if it manipulates shared data structures only in a manner that guarantees safe execution by multiple threads at the same time.
  • need to explore

KBP2016EAL

• Submissions are due Sept. 5
• Cluster
  • started scripts to process the 30k docs in batches
• BBN Jars - upgraded to latest, 3.4.4
  • /homes/gws/nhawkins/.ivy2/cache/com.bbn.kbp.eal/tac-kbp-eal/jars
    • tac-kbp-eal-3.2.2-SNAPSHOT.jar
    • tac-kbp-eal-3.2.5-SNAPSHOT.jar
    • tac-kbp-eal-3.3.1-SNAPSHOT.jar
  • /homes/gws/nhawkins/bbn/deft/com/bbn/kbp/eal/tac-kbp-eal/
    • 3.2.2
    • 3.2.2-SNAPSHOT
    • 3.2.5-SNAPSHOT
    • 3.3.1-SNAPSHOT
    • 3.4.1-SNAPSHOT
    • 3.4.3-SNAPSHOT
    • 3.4.4-SNAPSHOT
• Confidence
  • the BBN Response class does not have a field for confidence
  • emailed them to ask if I selected the correct/expected method to use on "ArgumentOutput"
    • BBN says they did not include confidence in Response so that Response's could be compared irregardless of confidence value
    • Notes on adding confidence to the Responses:
      • ArgumentOutput.from(Symbol docID, Iterable<Scored<Response>> response)
      • create a scored response like this:
      • Scored.from(response, confidence)
  • modified the code to use confidence values from newsspike, rather than 1.0 for all
• Time
  • include? normalize?
  • explored CoreNLP's sutime and NormalizedNamedEntityTagAnnotation
  • experimented with scoring a time example - not sure how it's scored
    • appears that time isn't being scored?
      • emailed BBN to ask about that
      • BBN confirmed they replicated the problem with their output data, they're investigating, sounds like a bug
• Combine
  • started script to combine output from systems
  • adding to this
• Met with Colin and James
  • review with Colin technique used for determining the newsspike taxonomy for the KB and EAL
  • shared the validate and scoring programs with them
    • set-up space on WebWare9 with the 2 BBN repo's
    • created various readme's to explain the parts
    • created a java program to create docIdMaps
    • created docIdMaps for the answerKey docs

KBP2016EventNugget

• planning questions to send to organizer
  • scoring guidelines and materials, "coming soon", part of the 2014-2015 set?
  • what 15 events are included? Looks like maybe 17?
  • what exactly is the output format? - saw an example where token offsets were preceded by a "t", though task description doesn't mention this
• not sure if newsspike returns token offsets for the trigger? need to explore.

.

Aug 19, 2016

DEFT Data Integration Call

• BBN cancelled the call this month

KBP2016EAL

• located password for accessing the eval docs
• time
  • BBN fixed the time bug, pushed new code, they're working on the relaxed rules fix
  • the scoring program checks that the CAS is an exact string match with the normalized arg filler, offsets are irrelevant
  • need to adjust code, now that the scoring method is clear
• cas vs. basefiller
  • scoring program results differ from scoring rules stated on BBN tac-kbp-eal Readme; it says both CAS and BF offsets are checked, but appears only CAS offsets are being checked
  • emailed BBN about it, they're checking on it
  • BBN updated the Readme to match what the scoring program does, which is to not fall back to BaseFiller offsets
• combine
  • finishing this program
  • creating test data set
• new clean xml string adjustment
  • creates same output, which is valid and scoreable
  • the scoring is a little different/higher this go round, because the BBN code has changed/updated
• meeting with Colin and James
  • updated the WebWare9 BBN repository
  • scoring: 1)time 2)CAS vs. BaseFiller offsets
  • schedule

KBP2016EventNugget

• Retrieved new task description, which was just recently updated
  • addresses some previously unanswered questions
    • specifies the events and subtypes of interest
    • indicates the output format
• checked Tuple data
  • there are token offsets for arg1, arg2, and v
    • can use v for event nugget

.

Aug 26, 2016

Monday, Aug 22: start of the eval window

Monday, Sept 5: end of eval window

KBP2016EAL

• NIST / Hoa
  • sent an email with a link to the submission page, due date = end of day Monday Sept 5, latest time zone
• met with Colin and James
  • scored their systems
  • they're debugging
• time roles
  • normalized time for the CAS string
  • allowing that to run on both nw and df docs
• combine script - wrote a scala script to do the combining
  • combined Colin and James output
  • combined Colin, James, and Newsspike output
  • James + Colin score > James score alone
  • the score is highest when all 3 systems are combined
  • the newsspike output is very sparse, though very precise
    • for the 91 docs answer key, newsspike finds 12 arguments (in 6 events); they all appear to be true positives, only 7 args are in the answer key
    • for the docs with (~1-event) output, there aren't any/or-maybe-1-in-one-doc additional event reverb tuples, which are being missed
  • created an option in the combine script to combine James and Colin systems only when they match
    • resulting argScores is much lower than, say, James's system alone
  • may want to do a little more testing on this script to make sure it's working as expected
• cluster
  • finalized newsspike run scripts (11 jobs for df docs, and 11 jobs for nw docs)
  • shared the process with James and Colin, sample script, and readme's, in case they want to use the nlp cluster
  • Luke cleared us to use the nlp cluster during the eval window
• validate
  • set-up a sampleProject in /projects/WebWare9/KBP2016/sampleProject with a pom.xml so James and Colin could run the BBN ValidateSystemOutput2016
    • worked with BBN to identify a logger needed in the minimal pom.xml to get this to work
  • noticed on BBN tac-kbp-eal github readme that they now have a NISTValidator, which will be used in the submission web site
    • the validating code is supposed to be the same, just different front end
    • want to try that on a .zip file of sample output
  • got latest copy of bbn file specifying the event types and roles (previous version was missing Time roles for a couple events)
• repo updates
  • pull'd new changes to BBN code to the kbp2016eal project
  • pull'd new changes to the 2 bbn repo's on /projects/WebWare9/KBP2016
    • got a build failure on the tac-kbp-eal repo, emailed bbn about that
    • BBN fixed it; this code requires steps on their part different from their normal release process, so looks like I help them figure out that they need to take additional steps...
• mapping: newsspike events to kbp-taxonomy
  • fixed a typo in the resources file for one event
  • identified a few events where it appears that the role Entity should be replaced by Place
  • planning to review the newsspike events150 file with Colin, to verify that we're not leaving out relevant events
• new dryrun of the 2092 dryrun docs
  • planning to do this in a cluster fashion to double-check cluster process and to double-check that the output validates

.

Sep 2, 2016

KBP2016EAL

• kbp taxonomy
  • added some mappings in the newsspike to kbp taxonomy, no new results in the small answer key set
  • looking at dryrun output to see if any of the new ones are there, what is their quality
• time
  • added normalization of "last week" and "next week"
• dryrun cluster
  • successfully ran 20 jobs for the 2092 dryrun set on the nlp cluster
  • output successfully validated
• validation
  • there are a few warning about CAS string not matching the string obtained from the char offsets
  • this is because newsspike computes arg1name and arg2name by joining tuple strings with " "
  • when the tuple is a "," or an "'", etc. this adds an additional space, not in the original sentence
  • emailed BBN, to verify that it is just a warning, and not a failure
  • BBN verified that's it's just a warning and that the CAS strings are not used in scoring, just the offset strings (which are ok)
• argument handling
  • some args are not "pure" / investigating improving this
    • start with "the"
    • start with web link, <a href=
    • contain 1 or more newlines, /n's
• eval
  • ran newsspike on the eval corpus of 30002 documents (= 15001 df + 15001 nw)
  • zipped output
  • validated zipped output

.

Sep 9, 2016

KBP2016EAL

• eval runs
  • zipped and validated 5 runs
  • successfully submitted 5 runs:
    • Washington1.zip (James-everything + Colin-everything + NewsSpike)
    • Washington2.zip (James-everything + Colin-nounsOnly + NewsSpike)
    • Washington3.zip (James-everything)
    • Washington4.zip (Colin-everything)
    • Washington5.zip (James-everything + Colin-everything, where both systems have the event argument)
  • Hoa confirmed that the 5 runs were received

DEFT Repo

• responded to BBN's FIGER questions
  • emailed Xiao for one of them - in the given output, does FIGER believe Barack Obama more likely to be an ORG than a PERSON?
    • answer: in this particular case, yes, /org has a higher score than /per; /person/artist has the highest score
  • pointed them to ticket #535 "promote FIGER" to find expected Benchmark results
• OLLIE, is there extra data available with each extraction, or is the algorithm description incorrect?
  • BBN emailed Stephen and me, haven't seen a response from Stephen, haven't worked with OLLIE, looking into it
  • ollie code not visible in DEFT repo, examined the code on knowitall, the enabler and attribution data are part of the extraction data structure
  • running a regression test on a test file to see if these fields have data in practice
    • set Ollie Regression Test to true in pom.xml
    • mvn verify -pl washington -DskipRegressionTests=false
  • answer: yes, the enabler and attribution fields have data
  • responded to Roger's (BBN) question, saying the data fields are there and populated, not sure about the history re why the data hasn't been captured
• ReVerb, can it be made thread-safe?
  • reviewing the JIRA comments from BBN
  • possibly, they can just note that the algorithm is not thread-safe, if it can't be made thread-safe, or it's determined we shouldn't try

KBP2016EventNugget

• Eval Window = Sep 20 - Oct 3
• created a kbp2016nugget project
  • copied the necessary files and resources into the project
• created RunKBP2016Nugget, and KBPNuggetApp
• modified: Tuple, NewsSpikeRelationPrediction, and NewsSpikePredictKB to be able to extract data on the verb, rather than on arg1, arg2, and the relation
• project compiles, outputs info on the verb
• todo
  • add code to print output needed for Nugget eval
  • review sample files - what exactly is there?
  • how to validate and score the output?
    • the format this year is slightly different (char offsets, rather than tokens)
  • write a combine script to combine output from the James, Colin, and NewsSpike systems
  • will the eval docs have the same format (body and filename) as the eval docs for EAL?

.

Sep 16, 2016

DEFT Repository

• responded to question about reverb and thread-safety

KBP Nugget 2016

• emailed cmu a couple questions about the task output and the validator script
  • haven't heard back, will look for another email address
• meeting with James and Colin
  • they have an NLP retreat Sep 19-21
  • shared info with them re the validator and scoring scripts in the github EvmEval cloned at /projects/WebWare9/KBP2016/EvmEval
  • shared info with them re the acceptable eventType and realis spellings
• completed code to output verbs, rather than arguments from newsspike in format needed for kbp nugget
  • have a couple of questions about what is valid for eventType and realis (different spellings are used in different docs)
  • emailed cmu about it, haven't heard back
  • found a .tbf answer key file in the LCD2016E36 packet, which gives examples of eventType and realis, so that probably answers my questions
• wrote combine script to combine system outputs
  • testing result: ok
• examined the validator and scoring code for nugget
  • the validator script appears different from validators for other tasks; i.e. appears to only be runnable on output which has an answer key
    • emailed cmu about it, haven't heard back
  • ran newsspike on the 2015 set of 202 answer key docs
  • ns output= 23 events
  • attempted to score the output; however, the scorer script is not happy
    • the gold.tbf file has token offsets; however, this year the expectation in the system output is char offsets
    • not all docs in the system output have extractions, seems like maybe it wasn't happy about that either
    • there are no examples in the scoring demo of docs without extractions
• identified a different email to try at CMU for scoring questions (i.e. from the github for EvmEval)
  • sent questions to: hunterhector@gmail.com
    • purpose of validator
      • cmu added an option for char offset output, -c
      • git pull'd new code
    • can scorer be used with the 202 answer key docs in the LDC2016E36 EventNugget packet which have token offsets, not char offsets
    • can I score my system output?
    • cmu responded, things are ok
• validating and scoring
  • successfully validated and scored the newsspike output
    • used char based gold .tbf file from LDC2016E67 packet

.

Sep 23, 2016

DEFT Integration Call

• Wed 11am

KBP2016 Nugget

• eval window Sep 20 - Oct 3
• evaluation corpus
  • the data was posted
  • 505 documents
• meeting with James and Colin
  • plan to validate their output
  • Ning is working on a coref system, to produce the final output
• scoring - examining newsspike's score report
  • the verb phrases are too long, in some/many cases, get partial credit
  • adjusted code to just output the verb head
  • Mapped "urged" events to Contact.Contact rather than Contact.Broadcast
  • Mapped "testify" events to Contact.Meet rather than Contact.Contact
  • precision: 95+% w/o realis (i.e. 1 incorrect event, out of 23 total newsspike events)
  • precision: 85+% w/realis
    • realis: automatically assign to Actual; there are 2 Other (i.e. future) events in the set of 23
  • recall: < 1
• newsspike models
  • changed the access from DEFT repo model files to locally existing model files
  • figer model (cs.figer.util.X) + nsre model (nsre.util.DR)

.

Sep 30, 2016

I lost some blog entries starting with this one, because CS support had to re-create the wiki from a "clean" backup. I am recreating the entries here, losing some details.

KBP Nugget

• met with James and Colin
  • finalizing systems

.

Oct 7, 2016

Oct. 3 - KBP Nugget

• successfully submitted 3 combo runs

DEFT Repository

• responding to tickets

.

Oct 14, 2016

Oct 10 - Dentist

Oct 12-14: Out (dad's 70th birthday)

.

Oct 21, 2016

Oct 19 - DEFT Conference Call

• NewsSpike

KBP2016 Nugget

• retrieved and unzipped scores to: /projects/WebWare6/KBP_2016/results/nugget/

DEFT Respository

• responding to tickets
• merged latest ADEPT release into our repository
• latest ADEPT release contains NewsSpike and FIGER

.

Oct 28, 2016

DEFT Repository

• responding to tickets

.

Nov 4, 2016

DEFT Respository

• responding to tickets

Documenting

.

Nov 11, 2016

Nov 8 = election day

Nov 11 = veteran's day

.

Nov 18, 2016

BBN Repository

• NewsSpike JIRA issue marked as "resolved"
• Roger (BBN) closed the ticket

.

Nov 25, 2016

Thanksgiving

.

Dec 2, 2016

/projects/WebWare9/nhawkins

nrhawkins github:

• kbp2016nugget
• kbp2016eal
• kbFiles
• kbp2015-multir
• tac2015 = openie
• kbp2015-slotfilling-implie
• KBP2014-Slotfilling-Multir
• playWebDemo

.

Dec 9, 2016

.

Dec 16, 2016

.

Dec 23, 2016

.

Dec 30, 2016

.

Jan 6, 2017

.