Difference between revisions of "Vulcan/MeetingNotes/Aug16 2013"
From Knowitall
(→Agenda) |
|||
Line 1: | Line 1: | ||
+ | == Agenda == | ||
+ | ; Update | ||
+ | |||
+ | ; System architecture | ||
+ | <blockquote> | ||
+ | |||
+ | </blockquote> | ||
+ | |||
+ | ; Plan for Greg | ||
+ | <blockquote> | ||
+ | |||
+ | Greg will be responsible for inference (online) components, while | ||
+ | Niranjan will focus on the offline components (generating axioms and rules) and experimentation.<br/> | ||
+ | |||
+ | * Processing text collections (definitions, study guide etc.) using Open IE and import into Solr. | ||
+ | * Converting WordNet and CNC to Tuffy axiom format and import into Postgres. | ||
+ | * Convert scored assertions into a format that is acceptable to Vulcan's evaluation framework. | ||
+ | |||
+ | </blockquote> | ||
+ | |||
+ | |||
+ | ; Experiment/Evaluation plan | ||
+ | <blockquote> | ||
+ | |||
+ | </blockquote> | ||
== Update == | == Update == | ||
− | |||
; System development ([[SystemStatus | Details on architecture and status]]) | ; System development ([[SystemStatus | Details on architecture and status]]) | ||
: 1. Online inference components implemented. | : 1. Online inference components implemented. | ||
Line 63: | Line 87: | ||
</blockquote> | </blockquote> | ||
: 3. Method: Input sentences that correspond to each assertion. Score assertions using our system and submit to Vulcan's web interface.<br/> | : 3. Method: Input sentences that correspond to each assertion. Score assertions using our system and submit to Vulcan's web interface.<br/> | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− |
Revision as of 19:24, 16 August 2013
Agenda
- Update
- System architecture
- Plan for Greg
Greg will be responsible for inference (online) components, while Niranjan will focus on the offline components (generating axioms and rules) and experimentation.
- Processing text collections (definitions, study guide etc.) using Open IE and import into Solr.
- Converting WordNet and CNC to Tuffy axiom format and import into Postgres.
- Convert scored assertions into a format that is acceptable to Vulcan's evaluation framework.
- Experiment/Evaluation plan
Update
- System development ( Details on architecture and status)
- 1. Online inference components implemented.
- Proposition generator -- Extract tuples from input sentence and convert into a proposition.
- Evidence finder -- Tuple matching over Open IE Clueweb data.
- MLN Inference -- A wrapper around Tuffy's MLN inferencer.
- 2. Offline components -- axioms and rule generation -- NOT implemented.
- 3. Planning to use Tuffy MLN Inference system directly.
Why Tuffy and not Jena or another inference engine? Why not Alchemy?
- Inference engines such as Jena/OWLim don't directly support multiple inference paths. Community's response is to suggest Datalog/prolog implementations.
- Tuffy supports MLN capabilities in Alchemy but is orders of magnitude faster (what takes 6 hours in Alchemy takes 2 minutes in Tuffy).
- Experiments and Evaluation
Not ready to do evaluation yet but here are some useful details.
- 1. Framework: Vulcan has a good evaluation interface setup. We will use this for starters. (Example output from the evaluation framework.)
- 2. Data: Training/Test splits set up by Vulcan. The questions cover 4-12th and AP exams.
Training = 474 questions.
Test = 290 questions.
Training data distribution and Vulcan's current performance:
Grade All Questions #Mult.Choice and
Non-diag. (MC-ND)Vulcan Performance
on MC-ND4th grade 249 108 55.09% 8th grade 476 125 55.07% 12th grade 446 160 25.83% AP 116 81 45.68% All 1287 474
- 3. Method: Input sentences that correspond to each assertion. Score assertions using our system and submit to Vulcan's web interface.