Difference between revisions of "Vulcan/MeetingNotes/Aug16 2013"

Revision as of 19:23, 16 August 2013

Update

System development ( Details on architecture and status)

1. Online inference components implemented.

• Proposition generator -- Extract tuples from input sentence and convert into a proposition.
  
• Evidence finder -- Tuple matching over Open IE Clueweb data.
  
• MLN Inference -- A wrapper around Tuffy's MLN inferencer.

2. Offline components -- axioms and rule generation -- NOT implemented.

3. Planning to use Tuffy MLN Inference system directly.

Why Tuffy and not Jena or another inference engine? Why not Alchemy?

• Inference engines such as Jena/OWLim don't directly support multiple inference paths. Community's response is to suggest Datalog/prolog implementations.
• Tuffy supports MLN capabilities in Alchemy but is orders of magnitude faster (what takes 6 hours in Alchemy takes 2 minutes in Tuffy).

Experiments and Evaluation

Not ready to do evaluation yet but here are some useful details.

1. Framework: Vulcan has a good evaluation interface setup. We will use this for starters. (Example output from the evaluation framework.)

2. Data: Training/Test splits set up by Vulcan. The questions cover 4-12th and AP exams.

Training = 474 questions.
Test = 290 questions.

Training data distribution and Vulcan's current performance:

Grade	All Questions	#Mult.Choice and Non-diag. (MC-ND)	Vulcan Performance on MC-ND
4th grade	249	108	55.09%
8th grade	476	125	55.07%
12th grade	446	160	25.83%
AP	116	81	45.68%
All	1287	474

3. Method: Input sentences that correspond to each assertion. Score assertions using our system and submit to Vulcan's web interface.

Agenda

Update

System architecture

Plan for Greg

Greg will be responsible for inference (online) components, while Niranjan will focus on the offline components (generating axioms and rules) and experimentation.

• Processing text collections (definitions, study guide etc.) using Open IE and import into Solr.
• Converting WordNet and CNC to Tuffy axiom format and import into Postgres.
• Convert scored assertions into a format that is acceptable to Vulcan's evaluation framework.

Experiment/Evaluation plan