Week #9

8/4/2017

   This week was focused on two main objectives​: presenting what I had learned over the past 9 weeks in a poster/keynote speech and submitting the final report of what the project was about. Constructing the presentation file was the first component to this stage. The second, and arguably harder, component was practicing how the presentation would go and getting enough run throughs in to be confident in what I would present. When the day came for the final presentations, I went up and gave my 6 minute talk about I had done over the summer with full confidence in my accomplishments.
   The second objective was submitting the final report. Doing so entailed combing through past updates and saved files to compile the culmination of my work for the past 9 weeks. Formulating the paper was mildly challenging, especially concerning format, but was not impossible. In the end, I was able to have my mentor(s) peer review my paper and upload the final form to the website, as shown below.
   In addition to the final presentations, there was a workshop on Wednesday. This was a panel about applying to and funding graduate school. Personally, I really enjoyed the panel because it was a candid discussion about the logistics behind getting into, working in and eventually leaving graduate school. Moreover, I believe it really tied together the message of DREU, which was preparation for graduate level work in an environment that serves as a sneak preview of what to expect at this collegiate level. Ultimately, I know that this program has provided with the right tools to succeed in undergraduate and beyond.

dreu_final_report.pdf
File Size:	809 kb
File Type:	pdf

Download File

---

Week #8

7/28/2017

   This week was focused on finishing the final piece of the project. That is, completing the Naive Bayes analysis of the scraped sites. The classifier I constructed through the Scikit-learn model would be the ultimate test of each site's text content. From the results of the text from the Scrapy spider file, over 60% of the content matched that of Ponzi schemes, while the remaining content, which accounted for 30-40% of the content, did not match that of Ponzi schemes. 
   Another facet of this week was preparation for the final week. The first portion was finishing the poster for new week. Doing so involved extensive proofreading with my mentors and advisor about layout structure, phrasing of certain concepts, as well as spelling and grammar. Really, the poster is about presenting the work I've done on the project more for display purposes.  
   In conjunction with the poster, I have also began working on the Powerpoint/Keynote that I will present next week. This will be an allotted 5 minutes for each DREU student  before the poster session. Being a brief overview of the work I've accomplished over the past 9 weeks, I find it important to illustrate the highlights of the research to the audience. Ultimately, I am excited to showcase all that I have done over the past 2 months and demonstrate how great of an opportunity this research experience through the DREU this was!

Week #7

​7/21/2017

   At the beginning of the week, I was working on completing the Naive Bayes Classifier through the Scikit-Learn package in Python. Using this classifier allows for a baseline benchmark of how accurate the predictor may be. Fortunately, I was able to complete the classifier for the collected ponzi scheme and non-ponzi scheme website data files, which compared a training set to a test set, run them both through a pipeline that utilized a CountVectorizer, TfidfTransformer, and MultinomialNB (NB = Naive Bayes), and then use various strings and text blocks to see how it performs. Thankfully, the prediction accuracy for the model is 91.6%, based on the trained set of 80% of the data and 20% allotted for the test data.
   In discussing the next steps for the project with my advisor, he proposed to using nslookup to test out the various domain names, based on a list of countries in the world  for the collected sites and configuring a way to automate this. That is, running a Python script that iterates over a list of possible URLs for the MMM websites, particularly because of their uniformity.  These will be called in Python from an Excel worksheet and be iterated over to be input to the nslookup function in Python. Should the output have integer results, it receives a value of 1 in a separate Excel worksheet. If the output does not yield results, because it may not exist, then it receives a value of 0 in the aforementioned new sheet.
   Additionally, a portion of the week has been dedicated to looking ahead to week 8. Moreover, getting ready for and showcasing the final stages of project and the project as a whole, respectively. By next week, I intend on completing the last few steps of the project and start developing the poster and Keynote (PowerPoint equivalent) presentation for the project. With this, the last component would be practicing the presentation in its entirety.

Week #6

7/14/2017

   At the beginning of the week, I started testing out the Naive Bayes Classifier on the Ponzi scheme and non-Ponzi scheme HTML pages collected. Using Jupyter Notebook as a Python environment and referencing multiple pages of documentation for this classifier, I was able to successfully test out the classifier. With the help of one of the graduate students in the lab, I was also able to better understand each step of the algorithm. A useful tip I gained was to comment in what each step of the algorithm was doing so that I could implement the code, as well as easily determine where a problem was if debugging of an error was necessary.
   In conjunction with this classifier, I also tested out the classifiers within the Sci-kit learn module. Some examples include the GaussianNB classifier, Logistic Regression classifier, Random Forest classifier, and Linear SVC classifier. For the purposes of the text files, I focused on  using the GaussianNB and Logistic Regression classifiers. With these two in mind, the aim is to study how the train and test data compare and how accurate each classifier turns out to be.
   Additionally, I have had contact with my advisor, updating him on my progress. Apart from the work with the classifiers, I have also been looking ahead to the future of the project, which includes web scraping tools through Python. As such, when searching for tools that used elements of the Beautiful Soup parsing methodology, I was able to find a couple of web crawlers that employ web scraping. One was Scrapy, which used web spiders to perform the aforementioned scraping and could also work in conjunction with Selenium, which dealt with sites that heavily incorporate Javascript in their site's content. A second web crawler was Grub Next Generation, which was available through C, C#, and Python. A third option was to create a simple web crawler manually through Python. By next week, I hope to have investigated more into these web crawlers and classifiers.

Week #5

7/7/2017

   Now in the Analysis phase of the project, I have been focusing my efforts on text mining. That is, testing out different forms of stemming and tokenizing text content of local files. To that end, I have been testing out various functions in this library with sample '.txt' files. The goal with this is to eventually tokenize specific text in an HTML file sentence by sentence.
   Following the weekly meeting with my mentor, I was able to gain more direction towards the aforementioned goal. That is, to focus more on training the Naive Bayes Classifier on a certain portion of the collected Ponzi scheme and non-Ponzi scheme websites. Then, using non-trained collected websites, test against the trained set to determine whether or not the classifier learned to positively identify Ponzi scheme sites (based on the given trained set) or not. What's important is that the trained set uses sites that do not show up again in the set, for which there would be extreme bias in the results. 
  Coinciding with the analysis utilizing this classifier, I am also working on other NLTK functions, like word frequency. Being able to call a corpus, which houses many '.txt' or, in this case, '.html' files, to perform a count of how many times the word occurs in the text content is a helpful tool. Additionally, the distribution of those frequencies both within and between these files will prove interesting to see. Eventually, I wish to experiment with other classifiers in the NLTK library to explore the results of these functions.

Week #4

6/30/2017

​   Last week was very productive in moving through and past the Parsing phase of the project. Using my progress from this as a basis for this week's work, I was able to further tweak the parsing function. With the help of one of my mentors, Pankaj Chand, I was able to use the Beautiful Soup 4 library in Python to pull data from each HTML page. Working in tandem with the lxml parser I have installed, I was able to call for multiple, specific tags that have the text content the parser in scraping the HTML page for.
   In the Skype meeting I had with my advisor, we discussed finishing up the Parsing phase of the project so we could move onto the Analysis and Classification phases. Another point during the conversation my two mentors, my advisor and I had was that the Analysis phase was next, specifically the NLTK Sentiment Analysis. How this would work is that I would configure a method so as to feed the text content from the HTML into the Sentiment Analysis sentence by sentence. Furthermore, once in the Classification phase, machine learning will become a large part of how the algorithm will web scrape for Ponzi scheme websites, having the hub of already collected websites as a foundation. 
   The DREU workshop this week was also provided a helpful insight in project management as a whole. Titled "Getting Things Done: Organization and Productivity for Computer Scientists", the talk was based off a best selling book, Getting Things Done: The Art of Stress Free Productivity. The lecture portion centered on debunking some common misconceptions about work efficiency, like when to check emails, proper allocation of tasks, how to think about project management form the ground up, etc. The discussion portion was more about integrating our own pre-existing understanding about organization of work and the guidelines the presentation and book gave. Participating in this event was a useful reminder of how to keep in pace with any project.

Week #3

6/23/2017

   At the start of the week, I continued learning more about NLTK in Python and the sentiment analysis available. Within the Python libraries I explored via cursory research, I was able to narrow down the functions, particularly classifiers, and the proper syntax as steps towards this goal. Additionally, I have been able to find more Ponzi scheme websites to use as a foundation for the characteristics of such sites, set against non-Ponzi scheme websites.
   During the weekly meeting with my advisor and my mentors, we discussed the conceptual and technical steps in the project. The first step was retrieving the URLs of the various Ponzi scheme websites, which was manually aggregated. The next step was parsing the HTML pages so as to divide them into different arrays of objects, text, metadata, etc. The third step was analyzing the textual content and possibly the object content, which leads into the next step involving classifying the collected data into meaningful and distinctive identifiers.
   For this week specifically, I focused on understanding the parsing function for HTML, seeing as how I have retrieved a bevy of Ponzi scheme website URLs. Using Asana as a guideline/check list and Lynda.com to learn more Python fundamentals, I was able to judiciously determine which methods with the HTMLParser class to use. Organizing this section into subtasks was  quite helpful in sequentially completing the retrieval step of the project. Fortunately, I was able to successfully make code that requests a URL address and parses the page as an HTML document for specified tags. By next week, I anticipate to have successfully parsed the HTML pages of the sites into readable documents.

Week #2

6/16/2017

     I have been able to learn more about the clustering function in the Python-based ecosystem called SciPy. Installing this through the Terminal on macOS was a learning experience in how pip, a package manager, and conda, the package manager for R and Python, function and are structured. In exploring more about clustering, parsing also became a topic conversation. Specifically, HTML Parsing through the Jupyter Notebook web application for Python. With the assistance of a couple of graduate students in the lab, I was able to successfully parse the URL for the HTML for the Ponzi scheme websites I have been examining and gain one more step in the project. 
    SSL certificates are another component of the examination of these Ponzi scheme websites. The lifetime for these sites' certificates ranged from 3 months to about 6 months. In a handful of websites, the connection to the main origin was not secure, therefore deeming the page not possessing a valid HTTPS and not showing the digital server certificate. Investigating this aspect of the sites gave a helpful insight as to when these online Ponzi schemes are built and what measuring that against the site's content means.
   Additionally, in the weekly meeting with my mentors, we detailed the progression of the project and where to go next. Part of the discussion entailed logistic regression, models of statistical analysis, and monitoring for missing words and/or phrases among the Ponzi scheme websites. All in all, this week served as a great push in the right direction. I feel very confident about the remainder of this project. ​

Week #1

6/9/2017

   After arriving on Friday, June 2, I began my first full week at University of Florida on Monday, June 5. During the week, I researched various literature through Google Scholar on papers concerning Ponzi schemes and botnets created for the purpose administering spam through avenues such as email. Additionally, I collected and compared the various well-known Ponzi scheme websites in countries like Nigeria and India to note similarities and distinct characteristics in their structure and appearance. Using information acquired from intrusion detection techniques for network security, I transposed those standards to how they would function for detecting Ponzi scheme websites based on a set profile of characteristics given to an A.I. or algorithm designed for that specific purpose. 
    The results I found were that the Ponzi scheme sites I was given shared major text-based and design-based similarities. That is, analysis on whether or not a site is a Ponzi scheme would predominantly reside on the content and layout of the website. A technique for performing such an analysis is supervised machine learning, which would utilize classifiers on what constitutes a fraudulent website based on criteria manually entered into an algorithm for the machine learning. Fortunately, there are people within the FICS department who are at least familiar with and have used such a program beforehand. ​