Tue, 25 Jan 2005
Human Language Parser
In the 50-odd years that we have computers, human language parsers are the one thing people have tried to write, but failed.
Not hindered in any way by any degree of knowledge about the subject, I wonder why that is. Granted, human languages are quite complex to express in mathematical terms, but that doesn't have to mean it's entirely impossible. I've been thinking about this quite a bit lately, and it appears to me that an architecture composed of the following components should make a human language parser possible:
- A cataloguing dictionary
- Such a dictionary would catalogue any word in a sentence into a number of classes; example classes could be substantives, proper names, verbs or articles. Words could be classified into multiple classes (a word can be a substantive and a proper name); words that can be different things when spelled the same way (which happens a lot; in English, for example, many verbs in their infinite form can be used as a substantive if one drops the to) would have two entries in the database.
- A set of rules that would define the language's grammar
- This rulebook would contain rules such as "every sentence requires at least a verb and a subject", or "if the subject is changed from singular to plural, the verb has to be changed accordingly".
- An engine
- This would look at a given text, split it up into phrases, look each word up in the dictionary, and try to apply the grammar rules until a possible way to interpret the phrase has been found
Granted, this would require a huge amount of work; but I don't think it's impossible.
Now, one might wonder why it would be nice to be able to get a computer to understand human language. For one thing, it would vastly increase the quality of existing grammar checkers in word processors; but that's just the tip of the iceberg. Extend the dictionary to also include a numeric representation of each word where words that are synonyms have the same numeric value—or at least one that is very close—and where homonyms have multiple values, and you open the possibility to a number of interesting things. A command line interface like "good morning, please fire up mutt". An application that will take a text in English and output one in Dutch. Flawlessly.
</dreamstate>
This hasn't been done to date, and I can't believe I'm the first one to have come to this idea. So why hasn't it been done yet? I've seen many people try, but none of the grammar parsers I've seen thus far are actually functional to a degree that I'd rather trust the grammar checker than myself. What's the hard part? The huge amount of work? Or is there some barrier that I haven't thought about that makes the whole thing impossible?
Some mail is welcome in /dev/null. Real mail, however, is welcome at my email address
|
|
|
|