Vulcan/MeetingNotes/Aug16 2013

Update

System development ( Details on architecture and status)

1. Online inference components implemented.

• Proposition generator -- Extract tuples from input sentence and convert into a proposition.
  
• Evidence finder -- Tuple matching over Open IE Clueweb data.
  
• MLN Inference -- A wrapper around Tuffy's MLN inferencer.

2. Offline components -- axioms and rule generation -- NOT implemented.

3. Planning to use Tuffy MLN Inference system directly.

Why Tuffy and not Jena or another inference engine? Why not Alchemy?

• Inference engines such as Jena/OWLim don't directly support multiple inference paths. Community's response is to suggest Datalog/prolog implementations.
• Tuffy supports MLN capabilities in Alchemy but is orders of magnitude faster (what takes 6 hours in Alchemy takes 2 minutes in Tuffy).

Experiments and Evaluation

Not ready to do evaluation yet but here are some useful details.

1. Framework: Vulcan has a good evaluation interface setup. We will use this for starters. (Example output from the evaluation framework.)

2. Data: Training/Test splits set up by Vulcan. The questions cover 4-12th and AP exams.

Training = 474 questions.
Test = 290 questions.

Training data distribution and Vulcan's current performance:

Grade	All Questions	#Mult.Choice and Non-diag. (MC-ND)	Vulcan Performance on MC-ND
4th grade	249	108	55.09%
8th grade	476	125	55.07%
12th grade	446	160	25.83%
AP	116	81	45.68%
All	1287	474

3. Method: Input sentences that correspond to each assertion. Score assertions using our system and submit to Vulcan's web interface.

Analysis

1. Selected 10 propositions that are single Open IE tuples as starting targets.

2. Wrote down steps involved in verifying these propositions.