Consider the process above where the input
message can be any text message or document in any language; for
example an email message, a web page, a usenet posting, a doctorial thesis,
or a question typed into a search engine for retrieval of similar documents.
The purpose of the filter is to construct a vector with a variable number
of elements that represents the input message such that:
-
Given insignificant changes to the input,
the vector remains substantually the same.
-
The number of elements in the vector is minimal
for the task at hand.
-
Similar input messages produce similar vectors.
*****needs more work*****
-
The vectors are meaningful to humans and to
machines.
Such a filter is, of course, the hold grail
of document retrieval. My question is how close can we come today
with easily available technology. Here is my proposal:
-
Break the input message into words ignoring
markups (simple parser)
-
Spelling check preserving the most likely
(most likely might need some work)
-
implement some algorithm to determine proper
names
-
implemtnt some algorithm+knowledge to determine
compound references
-
transform the words into their stemms (stem
dicationary is available)
-
transform the stemms into their synanoms (choice
of most characteristic synomym - use wordnet)
-
break the stemms into syllabals (available
technology)
-
sort the words by the number of syllabals.
Assumption(1): the level of sophistocation of a word is directly
proportional to its number of syllabels.
-
alphabetizize the stem words with the same
number of syllabals
the context vector (CV) would be this
orderdd set words. The length of the vector is arbitrary. The
more words that are included the higher the resolution of the vector because
of Assumption(1).
Now what if we replaced Assumption(1) with
a detailed knowledge of the level of sophistication of a word. Say
the knowledge from dictionaries of what words can be defined in terme of
what other words. ??
How can this technique be enhansed by the
usage of clustering given by ?
-
citations
-
hyperlinks
-
interested accesser's profiles
How fast can such a filter be constructed
to perform?
What is the most practical context in which
to deploy this technique?
--- random thoughts---
Parse out compound terms like 'white house'.
Should each of these should be treated like a multiple syllabic word?
* So every document has a vector
* The vectors cluster
* form a user community for each cluster
* the user community annotes and weeds
theri vectors
* uer communities are indexed and published
? what is the structure of the overlaping
of user communities
|
