I recently worked on a project titled "Recommending Similar defects on Apache Hadoop" .Its a recommendation system that predicts similar defects and then predicts the effort estimate for each defect.
Steps:
1) Extract XML/Excel data from Apache Hadoop Issue Tracker.
https://issues.apache.org/jira/browse/HADOOP
2)Convert the extracted data into CSV for persistent storage.
3)Extract required Column
Python COde :
import csv
import re
def col_selector(table, column_key):
return [row[column_key] for row in table]
with open("Data/next.csv","r") as csvfile:
reader = csv.DictReader(csvfile, delimiter=",")
table = [row for row in reader]
foo_col = col_selector(table, "Summary")
bar_col = col_selector(table, "Description")
The above example extract two columns from Apache Hadoop Issue Tracker CSV file. Your program must include python library called csv.py
http://docs.python.org/2/library/csv.html
4)From these Column we will generate a set of words specific to Hadoop.
We will apply various NLP to generate various words from the summary and description.
5)There are 5 Steps in Natural Language Processing
1. Tokenizing
2. Stemming
3. Stop Word Removal
4. Vector Space Representation
5. Similarity Measures
2. Stemming
3. Stop Word Removal
4. Vector Space Representation
5. Similarity Measures
Step 1 : Tokenizing :
The tokenization process involves breaking a stream of characters of text up into words or phrases, symbols or other meaningful elements called tokens. Before indexing, we Fillter out all common English stopwords.I obtained a list of around 800 stopwords online.
K. Bounge. Stop Word List.
https://sites.google.com/site/kevinbouge/stopwords-lists
https://sites.google.com/site/kevinbouge/stopwords-lists
The list contained articles, pronouns, verbs etc. I filtered out all those words from our extracted text. After reviewing the list, we felt stopwords list for a Hadoop Database has to be built separately, as numbers and sym-
bols are also to be filtered out.
Step 2:
Stemming is used to try to identify a ground form for each word in the text. Some words that carry the same information can be used in different grammatical ways, depending on how the creator of the report wrote it down. This phase will remove a xes and other components from each token
that resulted from tokenization so that only the stem of each word remains. For stemming, we used a python library called PortorStemmer. We passed to it stream of extracted words. Words like caller, called, calling whose stem was call were Filtered and only 1 word, call, was kept in the nal list.I Filtered around 1200 words this way.
Step 3:
Stop Word Removal
Synonyms removal and replace by 1 common word.I used wordnet NLTK to perform this.
Second Phase : Spell checking: List compared with list of misspelled words.
Step 4:
Vector Space representation.
After the first 3 steps I had around 5500 words. These words were used to identify tags.Each defect with tags was then represented into a Vector space model.Used general method used by scikit.
Step 4: Similarity Measure.
Calculated the cosine similarity between the two defect vectors.
bols are also to be filtered out.
Step 2:
Stemming is used to try to identify a ground form for each word in the text. Some words that carry the same information can be used in different grammatical ways, depending on how the creator of the report wrote it down. This phase will remove a xes and other components from each token
that resulted from tokenization so that only the stem of each word remains. For stemming, we used a python library called PortorStemmer. We passed to it stream of extracted words. Words like caller, called, calling whose stem was call were Filtered and only 1 word, call, was kept in the nal list.I Filtered around 1200 words this way.
Step 3:
Stop Word Removal
Synonyms removal and replace by 1 common word.I used wordnet NLTK to perform this.
Second Phase : Spell checking: List compared with list of misspelled words.
Step 4:
Vector Space representation.
After the first 3 steps I had around 5500 words. These words were used to identify tags.Each defect with tags was then represented into a Vector space model.Used general method used by scikit.
Step 4: Similarity Measure.
Calculated the cosine similarity between the two defect vectors.
