Total Pageviews

Sunday, March 17, 2013

3 days to create a data visualizer tool from scratch


In this blog I will like to pen down my 3days experience in learning a technology called NodeXL from scratch and creating an app out of that which can be downloaded for usage at http://netme.cloudapp.net/Downloads.aspx.
This could have been done in a shorter span, but considering my laziness, the delay was bound to happen J
Following is the story line best to my knowledge.

March 11, 2013 6:00pm


One of the member of the Microsoft’s Most Valued Professional group,  saw the Data Visualization section in NetMe and asked me about my experience in NodeXL.
NodeXL is a free open source set of libraries which helps you create a graph out of a GraphML or GEXF file format. You can also create your custom graphs using the libraries. This is an easy to learn feature and very effective in Data Visualization.
 The objective was that NodeXL comes with an Microsoft Office Excel dependency, he was looking for an approach how we can remove this dependency and create a windows form app which will fetch the twitter related nodes and its children and show it in a graphical format.
NodeXL was totally new to me and creating an app on that seems to be challenging. I thought to give a try and see how it goes.

March 11, 2013 9:00 pm


I got to know the basic features of what NodeXL do. Downloaded the source code of the same, but diving into 18 projects in the source code repository was not only time consuming but also not feasible for me. So I posted the discussion in NodeXl community forum at http://nodexl.codeplex.com/discussions/436250
To my query, I was really satisfied by the quick response given by the moderator. At around 10 pm IST I received a response from him that it was possible and there was a library set that we can download and start using.
 

March 13, 2013


Till now I was confident enough about the usage in NodeXL. I prepared a small POC wherein I created a graph using a sample graphml file. GraphML file format are used to create directed, undirected and mixed graphs. It is based on XML based file format. A sample GraphML format could be created as likewise
<?xml version="1.0" encoding="UTF-8"?>
<graphml xmlns="http://graphml.graphdrawing.org/xmlns" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns
     http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd">
  <graph id="G" edgedefault="undirected">
    <node id="n0"/>
    <node id="n1"/>
    <node id="n2"/>
    <node id="n3"/>
    <node id="n4"/>
    <node id="n5"/>
    <node id="n6"/>
    <node id="n7"/>
    <node id="n8"/>
    <node id="n9"/>
    <node id="n10"/>
    <edge source="n0" target="n2"/>
    <edge source="n0" target="n3"/>
    <edge source="n1" target="n2"/>
    <edge source="n2" target="n3"/>
    <edge source="n3" target="n5"/>
    <edge source="n3" target="n4"/>
    <edge source="n4" target="n6"/>
    <edge source="n6" target="n5"/>
    <edge source="n5" target="n7"/>
    <edge source="n6" target="n8"/>
    <edge source="n8" target="n7"/>
    <edge source="n8" target="n9"/>
    <edge source="n8" target="n10"/>
  </graph>
</graphml>
 

Previous NodeXL community discussions helped me a lot in this. I am not highlighting the code as that can be easily searched in the NodeXL community discussion forum. Rather let me focus on the major problem statement of how to interact NodeXL with Twitter API.
At first I thought to take help of community member on this so posted the second thread on the forum

March 14, 2013


To this I got the whole step by step approach by the moderator on how we can use the NodeXL libraries to fetch twitter data. This was indeed of great help, but at this moment the Azure Mobile script which I wrote for NetMe flashed up. I thought to reuse the code.
The code was to fetch top 18-20 tweets with their username for any particular searched keyword.
Following is the AMS Node.JS code that is fulfilling the requirement
function read(query, user, request) {
    console.log(request.parameters.type);
    readExpert(request);
}
function readExpert(request)
{    console.log(request.parameters.topic);
    var res="<xml>";  
   var req = require('request');     
    req.get({            uri:"http://search.twitter.com/search.json?q="+request.parameters.topic+"&result_type=mixed",            headers: {'content-type': 'application/x-www-form-urlencoded'},      
},

    function(error,response,body){
        if (!error && response.statusCode == 200) { 
               var resp = JSON.parse(body);
               for(var attribute in resp)
               {                  
                   if(attribute == "results")
                   {
                       console.log(resp[attribute]);
                       var len = resp["results"].length;                      
                       for(var loop=0; loop< resp[attribute].length; loop++)
                        {
                                res += "<Tweet>"
                                res += "<UserName>" + resp[attribute][loop].from_user_name +"</UserName>";
                                res += "<UserStatus>" + resp[attribute][loop].text+ "</UserStatus>"
                                res += "</Tweet>";                         
                        }
                   }
               }
               res += "</xml>";
               request.respond(200, res);
                                }   
                                else   
                                {
            console.log(error);
            request.respond(statusCodes.BAD_REQUEST, error);
                                }
    });
}

Now the next step was to use this data and convert it into graph using NodeXL libraries. Here let me make a note for all the libraries I used
1.       Smrf.NodeXL.Adapters
2.       Smrf.NodeXL.Control.WPF
3.       Smrf.NodeXL.Core
4.       Smrf.NodeXL.Layouts
5.       Smrf.NodeXL.Visualization.Wpf
The code for creating a graph is as follows

XmlDocument graphML = new XmlDocument();
 
//Helper class is having the logic of sending the request to Azure Mobile Service which has been mentioned in my previous blog

                HelperClass _helperClass = new HelperClass("https://datavisualizer.azure-mobile.net/tables/datavisualizer", <<Azure Mobile Service Key>>, Guid.NewGuid().ToString());
                string response = _helperClass.ReadExperts(topic);
                string tweet = String.Empty;
                XmlDocument doc = new XmlDocument(); 

//Define NodeXL control
                NodeXLControl nodexl_Graph = new NodeXLControl();
if (response != "")
                {
                    doc.LoadXml(response);
                    IVertex headVertex = nodexl_Graph.Graph.Vertices.Add();
                    headVertex.SetValue(ReservedMetadataKeys.PerVertexShape, VertexShape.Circle);
                    headVertex.SetValue(ReservedMetadataKeys.PerVertexLabel, topic);
                    headVertex.SetValue(ReservedMetadataKeys.PerVertexLabelFillColor, System.Drawing.Color.Blue);
                    foreach (XmlNode node in doc.SelectNodes("//xml//Tweet"))
                    {
                        IVertex vertex = nodexl_Graph.Graph.Vertices.Add();
                        foreach (System.Xml.XmlElement item in node)
                        {
                            if (item.Name == "UserName")
                            {
                                vertex.SetValue(ReservedMetadataKeys.PerVertexShape, VertexShape.Circle);
                                vertex.SetValue(ReservedMetadataKeys.PerVertexLabel, item.InnerText);
                            }
                            if (item.Name == "UserImage")
                            {
                                var webClient = new WebClient();
                                byte[] imageBytes = webClient.DownloadData(item.InnerText);
                                BitmapImage imageSource = new BitmapImage();
                                using (MemoryStream stream = new MemoryStream(imageBytes))
                                {
                                    stream.Seek(0, SeekOrigin.Begin);
                                    imageSource.BeginInit();
                                    imageSource.StreamSource = stream;
                                    imageSource.CacheOption = BitmapCacheOption.OnLoad;
                                    imageSource.EndInit();
                                }
                                vertex.SetValue(ReservedMetadataKeys.PerVertexImage, imageSource);
                                vertex.SetValue(ReservedMetadataKeys.PerVertexLabelFillColor, Colors.Transparent);

                            }
                            if (item.Name == "UserStatus")
                            {
                                IVertex childvertex = nodexl_Graph.Graph.Vertices.Add();
                                childvertex.SetValue(ReservedMetadataKeys.PerVertexShape, VertexShape.Square);
                                childvertex.SetValue(ReservedMetadataKeys.PerVertexLabel, item.InnerText);
                                childvertex.SetValue(ReservedMetadataKeys.PerColor, System.Drawing.Color.Green);
                                IEdge childEdge = nodexl_Graph.Graph.Edges.Add(vertex, childvertex);
                                childEdge.SetValue(ReservedMetadataKeys.PerColor, System.Drawing.Color.Red);
                            }
                            IEdge edge = nodexl_Graph.Graph.Edges.Add(headVertex, vertex);
                        }
                    }
                }
                nodexl_Graph.DrawGraph(true);
 
// NodeXL is a WPF control. If you need to host this control in Windows form you need to create an element host control and assign the control to that.
                elementHost_Graph.Child = nodexl_Graph;

POC

 
This app has been uploaded under the downloads page in http://netme.cloudapp.net/ .  To try this app you need to download the zip folder, extract it and run the msi file. It will get installed in your machine and you will get to see the NetMe shortcut in your desktop. As this is a POC, I have not focused much on the look and feel of the app.
Once the app gets installed you will be able to search for any keyword and see the corresponding graph. You can shift the nodes accordingly.
The major intent of this blog was to highlight how fast and easy it is to code using NodeXL libraries and create your own data visualizer tool within a short time span. Hope you will like it  J

 

Monday, March 11, 2013

Visualize you favorite Facebook Pages


How this idea came?

This feature is pretty interesting for me, and I liked connecting the dots. The story started when I saw a tweet from Avkash Chauhan wherein he mentioned the use of an open source tool Gephi for Data Visualization. He posted some videos related to how he was using Gephi to visualize the data at http://cloudcelebrity.wordpress.com/2013/02/20/visualizing-social-network-data-using-gephi/
It was pretty interesting and felt inline to what NetMe aims to achieve. We saw a potential to give more insights to user discussion and post related to the track keyword using a comprehensive analytics like Gephi. Gephi is an open source tool and was quick to learn and explore.

Approach

The dots were discrete. The approach that we thought was

1.       We will fetch all the tweets and FB posts from the corresponding social media platforms and then convert them into GEXF (Graph Exchange XML format) or GDF format which can be read by Gephi

2.       We will rank them by degree and assign colors according to their rank clearly differentiating which post are popular and which are not

3.       We will apply a clustering algorithm on the fetched GEXF or GDF. For our use we implemented Fruchterman Reingold algorithm as the aperture between the nodes or distance was optimal and helped us to avoid any kind of incorrigible/meaningless text.

4.       We will convert this preview to Web template and publish the same in NetMe

Pre-requisite


1.       Download Gephi 0.8.2 beta

2.       Download the Sigma.JS Template addon for Gephi which will help us to convert the data visualization in web template

Steps


 The dots were clear. Now it was the time to connect them. I faced few challenges doing this which I will mention as the blog goes.

1.       Fetch the GEXF or GDF format xml. GEXF or GDF is simply an xml file with some parameters which helps it to get exchanged into a Graph. You will find many APIs for this which are easy to use. Our aim was to fetch the Facebook posts related to liked pages. For this we have used the Facebook Netvizz API. This is an Oauth supported API which will need your authentication token, after authorizing the app you will be able to see all your pages and groups. Then it allows you to fetch post by page admin or post by page admin and other users.
2.         Once fetched, Open Gephi Tool and import this GDF data

 
3.       Once the data is imported Click on the Color tab in the top left menu bar and then select Degree as a category. Click on Apply
 
4.       Click on Size/Weight icon by the side of color bar and again click on degree apply the sizes and click Apply
Here I am mentioning the maximum size to be 50.5. This denotes that the 1st ranked post or the most popular post will have a weight of 50.5 and size 1 then as per weight the size and weight will decrease. 
5.       Choose Fruchterman Reingold algorithm from the Layout section and click Run
 
To see the text of the corresponding nodes you can click on “T” literal in the bottom menu bar. After doing this go to Preview Section by clicking on the button in the top menu bar.
6.       Click on File->Export-> Click on Sigma.JS template
 
After filling some details this addon will help you with a website template which will have 2 json files
a.       Config.JSON which will have the report related details
b.      Data.Json which will have the raw data which is required to build the graph.
7.       Clicking on any nodes will give more information about that node and also the users who have participated in that discussion. More insights can be seen such as how many discussion the user has participated, popular posts, popular user etc.
 
We are tracking 5 Facebook pages corresponding to Azure, Hadoop, Cloud, Microsoft and Windows8. Details at http://netme.cloudapp.net/DataVisualization.aspx

Sunday, March 10, 2013

NetMe Integration with Facebook


How NetMe is interacting with Facebook


 NetMe a BigData analytics tool is based on analysis of tweets posted in Twitter and posts published in Facebook. NetMe was created with an intention to give one single platform to the viewer to view all latest discussion corresponding to a keyword. In the current version of NetMe we are giving the ability of searching a particular keyword in Twitter or Facebook. We are enabling user to see all the discussion pertaining to his LinkedIn profile, and most importantly, out of thousands of daily news we are recommending user to read most popular news which are getting discussed in the social platforms.

Why Facebook interaction is needed? 


Facebook is getting adopted crazily. Its exponential popularity is making it a favorite platform to interact personal and professional related messages. I feel Facebook did a paradigm shift after introducing the page feature in its platform. Now companies are including this feature in their marketing checklist. As NetMe is tracking cutting edge technologies, it became inevitable for us to consider the interaction with Facebook.
NetMe interaction with Facebook was focused on Search and Fetch Data features. We also have an offline service to publish post automatically to Facebook. We did it for http://www.facebook.com/CloudBazaar . After a few customizations, NetMe will have this service under its subscription plan.
NetMe Facebook Search had added the value to the website. We are recording users tracking movies, politicians etc. through the search feature. The posts are fetched not pertaining to one’s individual account instead it does a global search for that keyword.

What APIs NetMe is using for this? 


NetMe is using Facebook Graph API to fetch and post data from Facebook platform. Facebook Graph API is a REST based OAUTH authentication API which enables developer to query different apis on a https secure channel.

How NetMe is using Facebook Graph API? 


NetMe is hosting 5 Azure Mobile Services for its features. One of the Azure Mobile Service has been allocated to data fetched from Facebook.

Step by Step approach to implement a Facebook Graph API implementation through Node.JS in Azure Mobile Service

1.       Create an app for Facebook. Go to https://developers.facebook.com/apps


After successful creation of the app, the administrator will get an App ID and an App Secret. We will have to make a note of these 2 keys which will help us to get the access token. 

2.       Click on Tools from the Top Menu Bar. Click on Graph API Explorer link

 
3.       In the Graph API Explorer, change the Application name to the previously created app and click on Get Access Token.




4.       On clicking the Get Access Token a popup will appear which will ask you to give permissions. For our use NetMe requires

a.       publish_actions under “User Data Permissions”,

b.      status_update, manage_pages, share_item, publish_stream under “Extended Permissions”

Click “Get Access Token” button



5.       Challenge: I faced my first challenge after this. On clicking Get Access Token, I got an Access Token which I thought can be directly implemented in my http request to get the desired response. Access_Token are required with every API request because this token decides what permission should be given to the app. I was successful in getting the response in first one hour, but after 1 hour it started throwing error. This was annoying because prior one hour everything was fine. Then I figured out that this access_token given is a short term access token.  

Earlier Facebook use to give a never die access token, which developers used as static variables in their API request, but they have stopped that service. After few googling I figured out that there was a way to get a long term access token. This long term access token is valid till 60 days, after which the developer has to again renew the access token.
 
Node.JS Script to request a long term access token

var req = require('request');             

        req.get({

                 method: 'GET',

                 url: 'https://graph.facebook.com/oauth/access_token?client_id=<<app_id>>&client_secret=<<app_secret>&grant_type=fb_exchange_token&fb_exchange_token=<<short term access token>>,

                 },function(err_access, resp_access, body_access){

                     req.get({

                           method: 'GET',

                            url: 'https://graph.facebook.com/search?q='+request.parameters.topic+'&type=post&'+body_access,

                           },function(err_getFeed, resp_getFeed, body_getFeed){                               

                               var body = JSON.parse(body_getFeed);

                               console.log(body);                              

                              request.respond(200,body);

                 });
                 });   

Above is a Node.JS script implemented in Azure Mobile Service which queries for a long term access token with help of app id, app secret and short term access token and uses the same to do a Facebook Graph API Search query.

Note next time if this script is run with the same short term access token, it will give the same long term access token. The life span of this will continue till 60 days. 

6.       Following is the AMS script snapshot that is running for Facebook Search query


7.       To call this script from client side I wrote 2 lines of code in the aspx page

HelperClass _helperClass = new HelperClass("https://facebookpost.azure-mobile.net/tables/facebookcompute", <<Azure Mobile Service Key>>, Guid.NewGuid().ToString());

                string response = _helperClass.ReadFacebookPost(topic);

Note in the above call with the service url you also need to send the key associated with the service. This key you will get at Azure Mobile Service portal by clicking on the service and then clicking on manage keys option.




 
8.       I have clubbed all the Netme API request modules in a class called Helper Class
Search Facebook code in the Helper class
public string ReadFacebookPost(string topic)

        {

            string key = String.Empty;

            string fullUrl = _baseurl+"?topic="+topic ;

            string tweet = String.Empty;

            char[] arr = new char[] { '\"', ',', ' ' };

            List<string> tweets = new List<string>();

            try

            {

                HttpWebRequest _azureRequest = WebRequest.Create(fullUrl) as HttpWebRequest;

                _azureRequest.AllowWriteStreamBuffering = false;

                _azureRequest.Method = "GET";

               

                _azureRequest.Headers = new WebHeaderCollection{

                    {"X-ZUMO-APPLICATION", Guid.NewGuid().ToString()},

                    {"X-ZUMO-MASTER", _appKey},

                };

                using (HttpWebResponse response = (HttpWebResponse)_azureRequest.GetResponse())

                {

                    Stream stream = response.GetResponseStream();

                    using (StreamReader readStream = new StreamReader(stream, Encoding.UTF8))

                    {

                        key = readStream.ReadToEnd();

                        key = key.TrimStart(arr);

                        key = key.TrimEnd(arr);

                    }

                }

            }

            catch (WebException ex)

            {

                using (WebResponse response = ex.Response)

                {

                    HttpWebResponse httpResponse = (HttpWebResponse)response;

                    using (Stream data = response.GetResponseStream())

                    {

                        string text = new StreamReader(data).ReadToEnd();

                    }

                }

            }

            return key;

        }

 

Here I am creating the API request using the service url and sending the search topic as a GET parameter. One important thing here is you need to mention X-ZUMO-APPLICATION and X-ZUMO-MASTER.

 

X-ZUMO-APPLICATION could be any GUID but X-ZUMO-MASTER will have to be the Azure Mobile Service key.

Challenge: Challenges never cease do they? J

Here comes the second challenge. Once all this is done and I executed the code. I was getting nothing when querying for request.parameters.topic at the server side. Ideally this should have the search parameter with it. Now came the second glitch. Apart from X-ZUMO APPLICATION and X-ZUMO-MASTER there is another parameter that has to be provided in the header. It is X-ZUMO-APPKEY. Now this APP KEY is different from Azure Mobile Service Key, and it is complex to retrieve. One can use a proxy such as Burp or Paros to retrieve this APP key.

An easy way to overcome this challenge was go to the portal and click on Azure mobile Service. Go to the Data Tab and click on Permissions. Edit the permission to “Everyone” where you need to execute the script without any App Key.



That’s it.
Our Facebook Graph integration is done. The response is in a JSON format which we are parsing and showing it on our page.
Try searching for some text in NetMe. One of my favorite category is movie and often I search for movie reviews with request such as this http://netme.cloudapp.net/SearchFBPost.aspx?topic=diehard5

     

J J