Vulcan/MeetingNotes/Aug16 2013

Agenda

• Update
• System architecture
• Plan for Greg

Greg will be responsible for inference (online) components, while Niranjan will focus on the offline components (generating axioms and rules) and experimentation.

• Processing text collections (definitions, study guide etc.) using Open IE and import into Solr.
• Converting WordNet and CNC to Tuffy axiom format and import into Postgres.
• Convert scored assertions into a format that is acceptable to Vulcan's evaluation framework.

• Experiment/Evaluation plan

Update

System development ( Details on architecture and status)

1. Online inference components implemented.

• Proposition generator -- Extract tuples from input sentence and convert into a proposition.
  
• Evidence finder -- Tuple matching over Open IE Clueweb data.
  
• MLN Inference -- A wrapper around Tuffy's MLN inferencer.

2. Offline components -- axioms and rule generation -- NOT implemented.

3. Planning to use Tuffy MLN Inference system directly.

Why Tuffy and not Jena or another inference engine? Why not Alchemy?

• Inference engines such as Jena/OWLim don't directly support multiple inference paths. Community's response is to suggest Datalog/prolog implementations.
• Tuffy supports MLN capabilities in Alchemy but is orders of magnitude faster (what takes 6 hours in Alchemy takes 2 minutes in Tuffy).

Experiments and Evaluation

Not ready to do evaluation yet but here are some useful details.

1. Framework: Vulcan has a good evaluation interface setup. We will use this for starters. (Example output from the evaluation framework.)

2. Data: Training/Test splits set up by Vulcan. The questions cover 4-12th and AP exams.

Training = 474 questions.
Test = 290 questions.

Training data distribution and Vulcan's current performance:

Grade	All Questions	#Mult.Choice and Non-diag. (MC-ND)	Vulcan Performance on MC-ND
4th grade	249	108	55.09%
8th grade	476	125	55.07%
12th grade	446	160	25.83%
AP	116	81	45.68%
All	1287	474

3. Method: Input sentences that correspond to each assertion. Score assertions using our system and submit to Vulcan's web interface.