by ·0 Comments

English version and some details below the image keep scrolling!

Inne spojrzenie na moją skromną kolekcje twitów z okresu 06-08 2017.

Wizualizacja przygotowana przy użyciu analizy grafów. Każdy punkt reprezentuje użytkownika twittera. Sięć połączeń jest wygenrowania na podstawie hashtagów. Użytkownik, który publikował twitta zawierającego danego hashtaga jest połączony z każdym użytkownikiem, który również publikował z użyciem tego hashtaga.

W rezultacie, użytkownicy publikujący przy użyciu wielu hashtagów będą łączeni z wieloma użytkownikami. Na wyżej przedstawionej ilustracji, ci użytkownicy znajdują się w środkowej części grafu, gdzie bardzo ciężko wyróżnić odizolwane klastry punktów.

Po drugiej stronie tego spektrum, są użytkownicy którzy  używają tylko ograniczonej ilości hashtagów w swoich twittach. W tym przypadku powstaną odizolowane grupy użytkowników, którzy komunikją się tylko wewnątrz danego hashtaga. Na zamieszczonej ilustracji, przykładem takich grup są klastry: #wtylewizji i #woronicza17.

 

Wykorzystane narzędzie :

  • Python, Numpy, Pandas,
  • Gephi, www.gephi.org

 

English version

The second look at my twitter data. This time I used the graph network analysis to see connections between twitter users. Each node in the graph represents a twitter user, the edge between users is created if two users publish something using the same hashtag.

If someone uses many hashtags on regular basis it will be connected to many others twitter users. This is the case for node (users) in the middle of presented graph, which are packed together and it is hard to isolated them from others.

On the other side, there are users who publish usually for one or small number of hashtags. These users will create sort of clusters, as the communication happen only between them. See #woronicza17 and #wtylewizji group as an example.

 

Tool used to get the results :

  • Python, Numpy, Pandas,
  • Gephi, www.gephi.org

by ·0 Comments

English version and some details below the image keep scrolling!

Lista tagów zawartych w tweetach zebranych w okresie wakacji (Czerwiec – Sierpień) w roku 2017. Tweety były filtrowane słowem ‘polityka’.

Clustering twitter data is not easy. I know that now :).  A single tweet is mostly a noise. So at the end you will get a sparse matrix of corpus with density far below 1%.  Simply if all corpus contains around 50 000 words (dimensions) but a single tweet has only 10 words (and probably only one or two has some meaning). Lots of noise and sparsity. And finally clustering like k-means put all tweets to one big cluster.

So I decided to change approach and try get something different from collected data.

On my first attempt was to try make words cloud with hashtags.

To do it I used nice python module:

  • website: http://amueller.github.io/word_cloud/
  • blog: http://peekaboo-vision.blogspot.co.uk/2012/11/a-wordcloud-in-python.html

This image was done on twitter data collected during June, July and August in 2017. The data was filter by polish word ‘polityka’.

by ·0 Comments

In short: This post describes the process of a data collection and gives a bit insight about my text processing algorithm.

 

A bit of time has elapsed since my last post about twitter data mining. The reason is quite simply… recently I spent most of my free time working on a prototype of my new game. I am quite excited about it and hopefully in the near future I will announce more. Anyway back to Twitter Data Mining.

Recently I have been collecting a polish tweets which are contain the word ‘politics’ or some declination of this this word. Namely : ‘polityka‘, ‘polityce’, ‘polityk’, ‘politycy’, ‘polityke’, ‘polityczna’, ‘politologia’.

To access the Twitter API and get tweets to my computer I use tweepy Python library (http://www.tweepy.org/).

Then collected tweets are processed by my Tweet Processor into pandas data frame. (http://pandas.pydata.org/)

I am improving the Tweet Processor gradually as soon I learn something new about my data set. At this moment Processor is quite simple and can perform really basic things, mainly:

  • soft clean of a tweet text – for example removing https links, html tags, user tags etc.
  • politics groups name extraction – this maps politics group name declinations into one name,
  • stop words removal – this is still in progress and my list of stop words is probably too small right now. I will post someday about my polish language stop words,
  • stemming  – this is the weakest part of my Text Processor so far. At this moment I map just a few words with its declinations. Polish language is not easy for this task and there is plenty of room to improvement.

As most of this it is just at the begging of development the output data are quite noisy. However instead of trying to make it perfect in first go. I decided to work in a loop:

This will allow me to focus on the most important issues and I will learn massively about text analysis.

Next time I will describe more about my clustering approach.

Thank for you time.

by ·0 Comments

English version is a bit lower keep scrolling!

Pierwsze i bardzo podstawowe rezultaty analizy tweetów na temat polityki. Analiza obejmuje tweety zebrane wciągu jednego miesiąca. Na temat metody analizy i dalszych planów będę pisał więcej w najbliższych postach.

 

English version!

My first and really basic results of a twitter data mining project. The data set includes an one month collection of tweets. The tweets were filter by polish word ‘politics’ and declination of it. In the next couple posts I will talk more about a technique to get this data and the future goal for this project.

Page_001_EN