NUTCH

I developed interest in search engines recently. I had been
hunting a lot for gaining information on search engines so that
I can start developing my own search engine.

In my hunt, I found a very good and developing search engine which is called NUTCH.

You can find more information at the NUTCH site.

Semantic Web Developments

I have been reading stuff on Sematic Web, Web 2.0 & 3.0 and their
technologies and such kinds of stuff.
It's been really amazing to unearth the intricacies of the
upcoming technologies. I was almost mesmerised by the RDF standards
which have been set for storing information for the semantic web.

In my upcoming posts, I will be writing about semantic web
also. To start with, I will want you people to have a look
at this project started by MIT.
SIMILE PROJECT (http://simile.mit.edu)

Inside this one, don't miss - WELKIN, a graph
based RDF visualiser(http://simile.mit.edu/welkin/).

Checking for valid password

If you want to check for valid password in the Unix/Linux Environment in C++.

• Include crack.h
• Use the function FascistCheck.
• Pass the password and the path to the password dictionary as the arguments.
• If the password is correct, a NULL string is returned.
• If it is maligned, then a pointer is returned to the string which explains the problem.
• Note that you must link the program compilation with -lcrack.

Anand Bora

An analogy based on my experience

Ever wondered what is the difference between a corporate company
and a small shop by your roadside.

The difference is huge obviously when you consider the massiveness
of the corporate company and the work culture in such companies.
A small shop has a very hectic culture throughout with small
number of people working in it.

I have worked in a small shop, a big shop and a big store,
both as the owner of the place and as a worker at that place
too. Currently I am working as an employee in a corporate company.

After getting through all these experiences, I thought to find
out the analogy and similarity between the small shop and a large
corporate company.

Though one might not agree, but I believe both the entities work on common norms
and protocols.
My justification is as follows:

A corporate level company has large number of employees
with the less experienced people working at the low level
of the organization. These are controlled by people who are
more experienced and who have knowledge of the work and work
environment.
A shop has 5-6 employees at a time. The owner is the head
and he maintains the finance and management of the shop.
One or two experienced people get higher wage and are given
previleges to manage other people. They also act as substitute
for maintaining finance when the owner is not at place.
The others work as normal workers.

When a fresher enters an organization, he his given ample rest.
He gets training of the technologies he is going to work on.
In a shop, a new person is first made aware of the things of the
shop, their selling prices and their placements in the shop.
In 6 months time, that person is perfectly trained to work in the shop.

A company provides a lot of incentives to the employees.
In a small shop, the shopkeeper gives small incentives
at special occassions to the employees.

There are other similarities too.
But at the end of the day, you find that you are a small
entity in an organization. You have lost your importance.
You dont have a name in this massive mob of 6 billion people.

This analogy is just to show that if you are an employee,
you have rested your fate in the hands of other people.
Just like the person working in a small shop in a small village.
You dont have your own importance.
Only your lifestyle is better and you think that you are well placed
in life.

Do think!!

Which media player is the best?

I suggest mplayer. I tried a lot of players during
my undergraduate study. But finally I stuck on to
mplayer. Though it is command line oriented and it
does not have a cool interface, still it has the
capability to run any video file.

I think that's what it is important.
Purpose should be important.

I have did many experiments with mencoder.
There are lots of licence software available
and most of the people cannot buy licence software.
So it is better to opt for freewares like mplayer
and VLC. mencoder is very very handy if you are a
power user. The compression utilities are marvellous
and you dont need an expertise to use it.

Just go through the docs, follow the examples
and get going.

Go for mplayer and mencoder.
An alternative is VLC.

I will post some of my mencoder experiments in the
near future.

More info on mplayer

Anand

Fiddling with GCC

This is my second post on technical part..

I enjoy working on GCC...
Just try these out..
It would be fun..

gcc -E filename

gcc -c -save-temps filename

The second one would produce a .s file which
contains something which I expect the reader would
be aware of...

Do try this readers..
It is fun!!

Nandi

NLP & Telecom

Imagine a world in which a system could answer your calls.

Imagine a world in which you have 24 hours uninterrupted

commercial services on your phone. A world in which humans dont have to incessantly keep on answering calls at the call centers. A simplified world in which your SMSs would be automatically interpreted by the systems and appropriate responses would be generated.

This is being made possible. Intensive Research is going on in the field of Natural Language Processing. The applications of NLP are immense and with time this will be embedded in our day to day life.

WHAT IS NLP?

Natural Language Processing or NLP is a sub field of Artificial Intelligence. It is a vast field which deals with processing and understanding of textual data. To simplify the explanation, consider a conversation of two people. These two understand what they say to each other. Now replace one of the persons with a terminal which has a NLP mechanism to analyze and interpret things. This system would be talking to the person in the same manner as the first person. Hence an accomplished NLP system is capable of interpreting like a normal human being. The biggest advantage of NLP is that a NLP can be designed for different colloquial languages.

HOW NLP is designed?

NLP is an extensive area of research. I will try to explain the whole thing in as small text as possible. A standard NLP consists of different phases which are very similar to a compiler. These have been explained subsequently in a brief manner.

1. Lexical Analyzer

The given word is fed into the lexical analyzer for validation in that particular language. A separate dictionary is maintained which is being used in the process. It is similar to the lexical analyzer in a compiler.

2. Morphogical Analyzer

A morphological analyzer is used for extracting the root word for a word. The remaining part is called a morpheme. For instance, consider a word 'boys'. Here 'boy' is the root word and 's' is the morpheme.

3. Syntactic Analyzer

This is similar to the parser of the compiler which we are all aware of. The only difference lies in the implementation part. Grammars are based on the grammar of the colloquial language being implemented. Different syntactic analyzers are:

Top Down parser

Bottom Up parser

Recursive Transition Networks

Augmented Transition Networks

There are others too and they are in the development phase.

4. Semantic Analyzer

This is the most important phase. This phase does the main understanding part. Here we use predicate calculus for analyzing a part. To exemplify, consider the following statement:

The Ball is Red.

This statement has two important entities - Ball and Red.

We can show the statement as Red(Ball).

Consider another statement - Ram and Rohit.

This can be explained mathematically as Ram^Rohit.

After building this table of PC, we go for machine translation which conforms to analyzing and understanding

on the machine level.

5. Discourse

Discourse is a part of human talk. Nothing much has

been done in this field.

Different adaptive systems are being built on the basis of NLP. Learning has become a major part. Statistical analysis and neural network techniques are being used these days which have provided impetus to the research in this field.

Other phases can be implemented after this phase. We can build different systems like a QA system by using the above structure.

NLP & TELECOM

NLP has tremendous applications in the field of telecommunications. NLP engine can be used for automatically correcting grammatical errors in a SMS application. NLP can be used for parsing different

languages and analyzing them. Thus an integrated system can be provided which will automatically parse texts of any language. Problems on the customer side would be automatically debugged just by an SMS from the customer.

A popular example of the application of NLP in telecommunications is the 'AMITIES' project in Europe. It facilitates easier billing system for the customers. Also the telecom customers directly interact with the system.

More dimensions of exploration in the field are coming up day in and day out. With massive applications, NLP would be a driving technology to reckon with in coming days. Telecommunication would be revolutionized with its power.

Interesting Links

http://domino.watson.ibm.com/comm/research.nsf/pages/r.nlp.html

http://research.microsoft.com/nlp/

http://www.lti.cs.cmu.edu/

http://nlpers.blogspot.com/

http://www.dcs.shef.ac.uk/nlp/amities/

http://www.iiit.net/ltrc/Anusaaraka/anu_home.html