Car Brand Analysis

Web Scraping, Data mining, Text Analysis

Objective

To gather comments on forums to find out which brands of cars are similar to each other in terms of attributes such as equipment, aesthetics, performance, cost and quality and to provide recommendations based on the analysis.

Methodology

This project consists of several parts :

Web scraping : A Web crawler/scraper was developed using Selenium to fetch messages posted in Edmunds.com discussion forums URL that was scraped :edmunds.com The code for web scraping is included as one of the code snippets in the repository.

Plotting MDS map : The association among different car brands were calculated using a metric called Lift and an MDS plot was generated using libraries in python. For more details please view the code snippet attached in the Github repository. For more details on MDS plots, please visit the MDS Wiki For more information on Lift please visit Details about Lift

Recommendations : On analysing different correlations among brands, several recommendations were provided to the top management. Please view the document attached in the Github project for more details.

Please refer to the github link to know more about the project

Github

• Category: Web Scraping, Data mining, Text Analysis, Lift ratios, MDS Plots, Presentation
• Tools: Python packages like BeautifulSoup,selenium,sklearn, nltk

Close Project

Implementation of Client side GIT Hooks

Version Control ,GIT

Hooks are programs you can place in a hooks directory to trigger actions at certain points in git’s execution. Hooks that don’t have the executable bit set are ignored.

The types of GIT hooks implemented in this project are :

pre-commit

1. We can auto-format python code using Black formatter

commit-msg

1. Check if commit message follows standard practises
2. We can provide meaningful instructions on how to prepare a commit message.

pre-rebase

1. can be used to abort rebase command on any commits

Please refer to the github link to know more about the project

Github

• Category: Version Control ,GIT
• Tools: GIT

Close Project

Legalisation of Marijuana in Canada

Natural Language Processing, Web Scraping, Sentiment Analysis

Objective

Analysis of tweets related to the legalisation of marijuana in Canada. This projects analyses the sentiments across various provinces and also provides recommendations as to how the process of legalization could have been carried out in a better way and the optimal way to legalize controversial policies across Canada

Methodology

1. Use TwitterScraper API to scrape tweets

2. Perform webscraping to obtain the number of followers for each user

3. Text pre-processing

4. Data Analysis consisting of Unweighted sentiment analysis (without considering the number of followers in the province) and Weighted sentiment analysis (Taking into consideration the number of followers in the province)

5. Recommendations

For more information please visit the Github link :

Github

• Category: Natural Language Processing, Web Scraping, Sentiment Analysis
• Tools: TwitterScraper, Selenium, Vader, Spacy, nltk

Close Project

Increasing engagement on National Geographic’s Instagram

Text Mining, Web Scraping, LDA topic modelling

Objective

This project is aimed at increasing engagement on National Geographic’s Instagram page (Instagram: natgeo). The final outcome is to advise Natgeo regarding the type of content they should post more and less of – i.e., what types of images increase engagement? What types decrease engagement?

Methodology

1. Scrape 500-1000 images from the natgeo Instagram page using scrapers available online. Along with the images, scrape captions, the number of likes and the number of comments for each post.

2. Image labels for 1500 pictures were collected using Google Cloud vision API.

3. Create a metric (score) for engagement by using a weighted sum of #_likes and #_comments. Be sure to normalize #_likes and #_comments. Now create an engagement score = 0.4*#_likes (normalized) + 0.6*#_comments (normalized).

4. Build 3 different models to predict engagement with :

Image labels (text) as predictors.

using captions to predict the same?

Using both image labels and Captions as predictors

5. Perform topic modeling (LDA) on the original image labels and check engagement score per topic

6. Recommendations to NatGeo

Please refer to the code and document attached in the repository for more details

Github

• Category: NLP, Metric analysis
• Tools: Python

Close Project

SFTP Transfer automation

Python automation

Objective

This project is an initiative to automatically transfer files onto an SFTP server using python script and to schedule the script to run at a specific time during the day using windows scheduler.

Methodology

The following analyses were performed in this project :

Development of python code to perform the following : 1.Connect to impala to fetch data and convert into CSV file

2.Connect to SFTP server, check the directory structure, and place the CSV file into the required directory

3.Creation of .bat file to enable running of the python code on shell

4.Scheduling the .bat file using windows scheduler. Use the below link to schedule a .bat file using windows task scheduler : Scheduling a .bat file Scheduling a .bat file

Please refer to the github link to know more about the project

Github

• Category: Python automation
• Tools: pysftp, requests

Close Project

Salary Prediction from job descriptions

Text Classification

Objective

The objective is to develop a classification model to predict salary from job description.The idea here is to test the predictive power of text and compare it with that of numeric data

Data Description

Please use the following link to download the data : Job descriptions data set. Please use training dataset train_rev1

Methodology

Salary information was bucketized based on percentiles (high (75th percentile and above) or low (below 75th percentile) salary from the text contained in the job descriptions). The job descriptions were vectorized using TF-IDF.

There are 3 classification models that were built for predicting the range of salaries : 1. Using only Text predictors : Two kinds of Naive Bayes models were built for this purpose , Bernoulli Classifier and Multinomial Text Classifier

2. Only using numerical predictors

3. Hybrid model (using both numerical and text predictors)

The above models were then compared using accuracy score.

Please refer to the github link to know more about the project

Github

• Category: Text Classification
• Tools: Python NLTK

Close Project

Network Analysis of Cancellation of Airpower

Social Media Analytics,Graphs,Networks

Objective

The objective of the project is to analyse the event of the cancellation of Airpower on 29th March, 2019 to understand the influence of networks on the sentiments of people.(Background : Apple cancelled the release of Airpower, a wireless charging mat on 29th March 2019 after much anticipation)

Methodology

The following analyses were performed in this project :

1. Data Cleaning

2. WordCloud of Prominent words before and After Cancellation

3. Emoji Analysis before and after cancellation

4. Who are the top influencers?

5. Are the top influencers more positive or negative in general?

6. Where are the top influencers located? (DOES INFLUENCERS HAVE IMPACT ON A PARTICULAR LOCATION?)

7. Were the top influencers spreading Positive or negative sentiments in each location?

8. Who have the tendency to connect diverse people together? Where are they located? (Highest Betweenness centrality)

9. IS IT A LOCAL IMPACT OR GLOBAL IMPACT? —are influencers of the same location influencing the sentiments on social media?

10. Recommendations

Please refer to the github link to know more about the project

Github

• Category: Social Media Analytics,Graphs,Networks
• Tools: networkx, Gephi, nltk, TwitterScraper

Close Project