Total Pageviews

Wednesday, July 29, 2015

Somebody clean my log container, it is growing exponentially

Who has the time to keep a check of log container size. Nowadays most organization rely on cloud  services for storage and maintenance. One of the most common artifact which gets uploaded in these storage account is Logs.

Let me give an example from my experience. I have seen teams using Azure blob storage for storing logs files, dumps, manifest files etc. During investigation these files are downloaded and rendered at user's box. Similarly alert traces go to Azure Table storage.

Now who has the time and effort to keep a check on these storage capacity. I once wrote a simple powershell command which helped me to delete all artifacts older than 60 days as of that day.

Cool! but after 2 months, here comes a mail from one team member saying he is not able to see the containers from CloudXplorer or Azure Explorer.

What happened?? I forgot to rerun my script again and the container grew exponentially again.

OK so now I added a task scheduler which will invoke my script on daily basis.

After another 2-3 months later I got to see a similar mail from another team member. What happened this time?

My machine got formatted because I wanted to deploy a new OS, and everything got deleted. Again I thought I need to write the script and enable the task scheduler.

But Wait!! What if my account stops working? What if again I format my machine??

It will not only be a re-work but also loss of time and effort.

In a midst of confusion one of my team member told me why don't you write a stateless azure fabric service for this and deploy it on one of the azure cluster and forget about it?

Cool! why did not I think about this....

So I started writing this service. Problem statement was delete azure container and azure table which are 60 days older, so Ideally the service will trigger itself once in a day and find out all the container and table older than 60 days and delete it.

Now comes designing of this app. There are 2 actions performed by this service DeleteAzure container and Delete Azure Table. Both depends on time limit of 60 days. The user can also delete a Azure Table or Azure Container by its name.

As per me this resembled a Strategy design pattern. Following is a simple conceptual diagram


Now I hope I will never ever get any issues for cloud storage clean :)

Saturday, July 11, 2015

Does my organization need AzureServiceFabric?


Azure Service Fabric is a technology to provide high reliable stateful/stateless/stateful persisted services. In this blog I want to relate few of the solution which helped me envisage answers for simple day to day problem at work. Microsoft is trying to promote this service currently for both private and public clouds. In Open networking summit of 2015 Azure chief Technology officer Mark Russinovich presented the roadmap of this service which encompasses future release support for Linux and Java containers.

Support for Java can open horizon of comparison between existing technologies prevailing in distributed computing field. Based on same Leader Election and Replication concept, technologies like Hadoop, HDFS, Zoo Keeper already dominate the space. Deep Diving into Azure Service Fabric pops out many similar concepts.

Similarity with existing technologies 


1.      Data locality in Hadoop = Hot data in Azure Service Fabric (ASA)


Hadoop is having this concept of Data Locality where it tries to co-locate the data with the compute nodes, so data access is faster. This is the reason for it good performance and which makes it at the heart of data computing.

Azure Service Fabric term the local data as hot data (ref: link). The intent of this is same high performance. These hot data features are provided by Reliable actors or Reliable services. Service Fabric replicates the data to other instances for data reliability.

This concept of hot data is also debatable as some say it is good to have this for higher performance and in some guidelines it is not preferred. Let me point to the link at Cloud foundry. Cloud Foundry avoids writing to local file system primarily because the disk might be taken. It always suggest you to go with Cold Data which can be stored in external storage. Again it depends on the architecture and design of replication implemented. Discussion on this might be another blog of comparison.

Organization which are new to big data analysis, which is thinking around the terms of hot and cold data might want to leverage Azure service Fabric for their requirement. Here Cold data is termed as that set of data that can be stored into external storage like database, or storage containers for future reference. Here going with Hadoop or small micro-service on ASA is a point of debate. It depends on what is the business need. If you are a c# guy your best bet is to try ASA. 

2.      Reliable Application development using Zookeeper in Hadoop or Reliable Service/Actors in ASA


Zookeeper is about building general distributed applications. It is also termed as Hadoop’ distributed coordination system. Zookeeper deals with some common terminologies in distributed computing world- Distributed Queues, Distributed Locks and leader election. At Yahoo, Zookeeper has been benchmarked at over 10000 operation per seconds.

Azure Service Fabric comes up with a similar feature which it calls as Reliable Service and Reliable Actors. Reliable Services/Actors offers reliable data structures like Reliable dictionaries and Reliable Queues. These are based on same replication concepts. One can create an application and define x number of partition, each partition may have n number of replicas which fulfil the purpose. Hot data in Service Fabric Cluster get replicated across other instances for high reliability.

Here Zookeeper is having an advantage over ServiceFabric and that is termed as HBase. Hbase which is based on ZooKeeper has proved its ability to geo replicate. Microsoft on other hand has still not mentioned anything about geo replication in Service Fabric.

Organization can really leverage the reliable data structures provided by Service Fabric. One simple implementation that I envisage is machine management where in organizations are having huge number of machines and bucketing them in different scenarios like health, test failures, suspicious activities can easily be done using reliable dictionaries. Reliable Queues on other hand can provide a good implementation of priority queues.

Conclusion


Still Service Fabric is in its infancy and this blog needs a lot of meat, but this is a start. Compared to Hadoop a huge opportunity lies in terms of feature development in ASA. Once Microsoft decides to release this in Java and Linux containers it will open more opportunities to explore. As reported this is also a back bone for some of upcoming technologies Microsoft plan to release like Cortana along with existing technologies like Skype, Intune and Azure SQL, so this will encourage developers to create real time, high reliable applications. The intent is to provide a one button solution for all the needs.

Wednesday, June 24, 2015

3 lines of Node.JS code to run a Web Server on ServiceFabric

Since Service Fabric got publically released we saw many enthusiasts trying creating their apps with this. One of the proven strong case study is having a website hosted on Service Fabric.

If one goes through the documentation of Service Fabric he/she will find the tutorial on OWIN hosted webapp at https://azure.microsoft.com/en-us/documentation/articles/service-fabric-reliable-services-communication-webapi/. This is pretty descriptive and explains how quickly you can host a website on Service fabric.

I started this blog taking reference of this tutorial. The intent was to reduce the code and try to come up with a story around this. This was the time when I thought to see if we can figure out something with Node.JS. My early experience with Node.Js was pretty exciting wherein we created a Azure Mobile Service pinger to Service fabric Cluster. http://nabaruns.blogspot.com/2015/05/azure-mobile-service-service-fabric.html. The only challenge was how to call Node.JS script from C#.

With a little bit of research I found that this issue has been solved by Tomasz Janczuk. Acknowledging Tomasz let me now touch a little bit of Edge.JS.

Edge.JS GIT: https://github.com/tjanczuk/edge

Pre-requisite of this exercise is you have Node.JS and Service Fabric installed in your machine.

Step By Step : How to create a web host from Service Fabric

1. In my example I have created a Stateful Service Fabric Service. Stateful is just by choice as I plan to explore some more stuff with Node.JS and at a later stage have a stateful story around that. You can start with stateless or actor services too


2. Install the Nuget package of Edge.js from NPM (Tools -> Nuget Package Manager)
 
 
3. Now let's write our 3 line code for creating the webserver

 public static void Main(string[] args)
{
            try
            {
                using (FabricRuntime fabricRuntime = FabricRuntime.Create())
                {
                    // This is the name of the ServiceType that is registered with FabricRuntime.
                    // This name must match the name defined in the ServiceManifest. If you change
                    // this name, please change the name of the ServiceType in the ServiceManifest.

                    fabricRuntime.RegisterServiceType("WebHostStaefulSFType", typeof(WebHostStaefulSF));
                    ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id, typeof(WebHostStaefulSF).Name);
                    CallWebServer("Thanks for creating this server").Wait();
                    Thread.Sleep(Timeout.Infinite);
                }
            }
            catch (Exception e)
            {
                ServiceEventSource.Current.ServiceHostInitializationFailed(e);
                throw;
            }
}
private async static Task CallWebServer(string greeting)
{
            const string jSFunction = "return function createServer (data, callback){ var http = require(\"http\");  var server = http.createServer(function(request, response) { response.write(\"Hello World!\"); response.end();});server.listen(8081);}";

            var func = Edge.Func(jSFunction);
            WriteLine(await func(greeting));
}


That's it. That is what we need to spawn a http web server using node.js

As per my understanding, from Service Fabric point of view it does not care how the web server is hosted. Here it just considers the Primary replica is ready or not. Once ready it executes the code.

It will be great to know other views on this.

 

Thursday, May 14, 2015

Azure Mobile Service + Service Fabric = Easy Monitoring

I want to thank all my readers who appreciated my previous blogs on Service Fabric. It is indeed a motivation which drives an tech-enthusiast like me. While driving home, I was thinking of my next blog, when all of a sudden I remembered of a POC which involved Service Fabric Rest API and Azure Mobile Service.

Intent of the POC

The intent of POC was to create a quick efficient way of monitoring Service Fabric cluster without writing tons of code and making things complex. The POC ask was from a client who wanted a feasible solution to monitor his Service Fabric Cluster. The ETA for this was decided to be 7 days, but believe me the first POC was ready by end of second day.

Why Azure Mobile Service?

Azure Mobile Service is an awesome tool for developers who want to write lightweight services, which runs continuously without incurring much effort and cost. I was aware of AMS as I wrote couple of scripts interacting with facebook api and Linkedin in past. (If interested, you can take a look into my netme blog). I thought it will be efficient if the same scripts are used for Service Fabric monitoring. As Service Fabric was exposing REST APIs , it was easy to perform a call from AMS and output the result into Azure Table Storage. A MVC app was created to read the Azure Table storage and show the output in a graphical chart.

Acknowledgement

I want to acknowledge Balaji who mentored me through out this process and made the simple blatant script turn out to be a generic utility.

Let's get our hands dirty

Before starting the problem statement was broken down to major 2 chunks
1. Do Rest API calls to Service Fabric
2. Store the output in table storage

Performing above 2 operations were pretty simple as AMS provide 2 Node.JS packages
a. azure
b. request

If you are new to Node.Js no issues, following script section will help one understand what is the usage for them

Pre-requisite

Ensure that the Http gateway is enabled in Service Fabric Cluster. This will be a section in Cluster Manifest

   <Section Name="HttpGateway">
        <Parameter Name="IsEnabled" Value="true"/>
    </Section>

Also in your ServiceDefinition.csdef under Endpoints, we should have an endpoint for HttpGateway

      <InputEndpoint name="HttpGatewayListenAddress" protocol="http" port="19007" />

In my case the endpoint port is 19007

Do Rest API calls to Service Fabric

Service Fabric provides multiple REST Apis which are documented at https://msdn.microsoft.com/library/azure/dn707692.aspx

For this POC we will discuss few of them

1. In order to start with the POC first we need to create a AMS service



2. Once the mobile service is created one should go to the Scheduler section and create a new job


3. Once the scheduler job is created one can click the AMS scheduler job and go to the Scheduler section, where we will get an area to write our script

4.  Once the job is created one will see an empty function with the job name. Consider this to be the first entry point to the script. As my job name is smurfjob , I got a function name with the same name

function smurfjob() {
      console.log("Starting My App Monitor");
      console.log("This will monitor Node, Application, App Services , Sestem Services, Partition and Replica");
    var dataTypes = [
        "Node"
        , "Application"
        , "AppServices"
        , "SystemServices"
        , "Partition"
        , "Replica"
        ];
    dataTypes.forEach(getData);
}

As mentioned in the above log statements we are going to monitor Service Fabric Node, Application, Application Services, System Services, Partition and Replica

5. Our next module will be saying what is the input and what should be the output. For instance if input is AppServices, it should go to MyAppCurrTable storage table

function getData(dataType, index, array) {
    if (dataType == "Node") {
        getNodesData();
    } else if (dataType == "Application") {
        getApplicationData();
    } else if (dataType == "SystemServices") {
        fetchAndUpdateServicesDataForAppName("System");
    } else if (dataType == "AppServices") {
        getChildDataForParent("MyAppCurrTable", fetchAndUpdateServicesData);
    } else if (dataType == "Partition") {
        getChildDataForParent("MySvcCurrTable", fetchAndUpdatePartitionData);
    } else if (dataType == "Replica") {
        getChildDataForParent("MyPrtnCurrTable", fetchAndUpdateReplicaData);
    }
}

6. Now let's have the modules for different datatypes

Nodes

Let me now take a single module and explain what we want

function getNodesData() {
    //Import the request node.js and azure node.js module
    var req = require('request');
    var azure = require('azure');
 
    // Url for fetching the Node Data 
    var nodeUrlToMonitor = "http://serviceFabricClusterName.cloudapp.net:19007/Nodes?api-version=1.0";
 
    console.log("Url to Monitor" + urlToMonitor);
 
    // fetch Node data
    req.get({
    uri:urlToMonitor,
    timeout : 200000,
    headers:{'content-type': 'application/x-www-form-urlencoded'},
   },
   function (error, response, body) {

   //This is where we fetch the response body
       if (!error && response.statusCode == 200) {
       
           //Parse the json body
           var items = JSON.parse(body);
         
           //Now create the azure storage Table 
           var tableService = azure.createTableService("azure storage account name", "azure storage account key");

           tableService.createTableIfNotExists(TableName, function (error) {
               if (!error) {

                      //Parse the items in json and insert them in Azure Table Storage
                      tableService.insertEntity( parse relevant info)
               }
               else {
                   console.log("CreateTableIfNotExists for " + currentTableName + " returned error: " + error);
               }
            })
       }
       else {
           console.log("ERROR: GET " + urlToMonitor + " returned error: " + error + "and statusCode " + response.statusCode);
       }
   })

}

Hope in the above section we got the actual intent of the idea mentioned earlier. Following is a snapshot of tabular structure of Application service

PartitionKey RowKey ServiceKind Name TypeName ManifestVersion HasPersistedState HealthState ServiceStatus IsServiceGroup AppName
System_ClusterManagerService 5VbGMVNM2EZ7NFYu81gAfO Stateful fabric:/System/ClusterManagerService ClusterManagerServiceType 4.0.99.9490 TRUE Ok 1 FALSE System
System_FailoverManagerService GSc38KxoFFXFThEJbP+vQ0 Stateful fabric:/System/FailoverManagerService FMServiceType 4.0.99.9490 TRUE Ok 1 FALSE System
System_ImageStoreService YcPTmeiqsR0Vh33CpC+7nC Stateful fabric:/System/ImageStoreService FileStoreServiceType 4.0.99.9490 TRUE Ok 1 FALSE System
System_InfrastructureService uEvwax1amAIoal0LvoRL0y Stateful fabric:/System/InfrastructureService InfrastructureServiceType 4.0.99.9490 FALSE Ok 1 FALSE System
System_NamingService x+vT3v9YgNundefinedYvZJPgbTKvo Stateful fabric:/System/NamingService NamingStoreService 4.0.99.9490 TRUE Ok 1 FALSE System
System_RepairManagerService 1bj5zeS2ReeKwbPVSQ1dOE Stateful fabric:/System/RepairManagerService RepairManagerServiceType 4.0.99.9490 TRUE Ok 1 FALSE System

Now one can easily create a MVC app or use PowerBI to read this data , and create a simple dashboard

7. Once all this is done you can save the script and enable it. AMS will act as a background task manager script, and will keep uploading the results as required

Here is the full Azure Mobile Script

function smurfjob() {
      console.log("Starting My App Monitor");
      console.log("This will monitor Node, Application, App Services , Sestem Services, Partition and Replica");
    var dataTypes = [
        "Node"
        , "Application"
        , "AppServices"
        , "SystemServices"
        , "Partition"
        , "Replica"
        ];
    dataTypes.forEach(getData);
}


function getData(dataType, index, array) {
    if (dataType == "Node") {
        getNodesData();
    } else if (dataType == "Application") {
        getApplicationData();
    } else if (dataType == "SystemServices") {
        fetchAndUpdateServicesDataForAppName("System");
    } else if (dataType == "AppServices") {
        getChildDataForParent("MyAppCurrTable", fetchAndUpdateServicesData);
    } else if (dataType == "Partition") {
        getChildDataForParent("MySvcCurrTable", fetchAndUpdatePartitionData);
    } else if (dataType == "Replica") {
        getChildDataForParent("MyPrtnCurrTable", fetchAndUpdateReplicaData);
    }
}

function getNodesData() {
    var req = require('request');
    var azure = require('azure');
    var nodeUrlToMonitor = "http://servicefabricname.cloudapp.net:19007/Nodes?api-version=1.0";
    var nodeHistoryTableName = "NodeHistoryTable";
    var nodeCurrentTableName = "NodeCurrentTable";
    var nodeEventsTableName = "NodeEventsTable";
    var nodePrimaryKeyName = "Name";
    
    fetchAndUpdate("Node", nodeUrlToMonitor, nodePrimaryKeyName, nodeCurrentTableName, nodeEventsTableName, nodeHistoryTableName, 15, null);
}

function getApplicationData() {
    var req = require('request');
    var azure = require('azure');
    var appCurrentTableName = "MyAppCurrTable";
    var appEventsTableName = "MyAppAppEventsTable";
    var appHistoryTableName = "MyAppAppHistoryTable";
    var applicationURL = "http://servicefabricname.cloudapp.net:19007/Applications?api-version=1.0";
    var appPrimaryKeyName = "Id";
    
    fetchAndUpdate("Application", applicationURL, appPrimaryKeyName, appCurrentTableName, appEventsTableName, appHistoryTableName, 15, null);
}

function getChildDataForParent(parentTableName, fetchAndUpdateChildData) {
    var azure = require('azure');
    var tableService = azure.createTableService("storageaccountname", "storageaccountkey");
    var getMatchingEntities = azure.TableQuery
    .select()
    .from(parentTableName);
    tableService.queryEntities(getMatchingEntities, function (error, entities) {
        if (!error) {
            entities.forEach(fetchAndUpdateChildData);
        }
        else {
            console.log("ERROR: queryEntities in " + parentTableName + " returned error " + error);
        }
    });
}

function fetchAndUpdateServicesData(item, index, array) {
    fetchAndUpdateServicesDataForAppName(item.PartitionKey);
}

function fetchAndUpdateServicesDataForAppName(appName) {
    var svcCurrentTableName = "MySvcCurrTable";
    var svcEventsTableName = "MyAppSvcEventsTable";
    var svcHistoryTableName = "MyAppSvcHistoryTable";
    var serviceURL = "http://servicefabricname.cloudapp.net:19007/Applications/" + appName + "/$/GetServices?api-version=1.0";
    var svcPrimaryKeyName = "Id";
    var additionalData = {"AppName": appName};
    console.log("Fetching svc data for app " + appName);
    fetchAndUpdate("Service", serviceURL, svcPrimaryKeyName, svcCurrentTableName, svcEventsTableName, svcHistoryTableName, 15, additionalData);
}

function fetchAndUpdatePartitionData(item, index, array) {
    var svcName = item.PartitionKey.toString();
    var appName = item.AppName;
    fetchAndUpdatePartitionDataForServiceName(appName, svcName);
}

function fetchAndUpdatePartitionDataForServiceName(appName, svcName) {
    var prtnCurrentTableName = "MyPrtnCurrTable";
    var prtnEventsTableName = "MyAppPrtnEventsTable";
    var prtnHistoryTableName = "MyAppPrtnHistoryTable";
    var updatedSvcName = svcName.toString().replace('_', '/');
    console.log("Fetching partition data for app " + appName + " and svc " + updatedSvcName);
    var partitionURL = "http://servicefabricname.cloudapp.net:19007/Applications/" + appName + "/$/GetServices/" + updatedSvcName + "/$/GetPartitions?api-version=1.0";
    var prtnPrimaryKeyName = "PartitionId";
    var additionalData = {"AppName": appName, "ServiceName": updatedSvcName};
    fetchAndUpdate("Partition", partitionURL, prtnPrimaryKeyName, prtnCurrentTableName, prtnEventsTableName, prtnHistoryTableName, 15, additionalData);
}

function fetchAndUpdateReplicaData(item, index, array) {
    var svcName = item.ServiceName.toString().replace("_", "/");
    fetchAndUpdateReplicaDataForPartitionId(item.AppName, svcName, item.PartitionKey, item.ServiceKind);
}

function fetchAndUpdateReplicaDataForPartitionId(appName, svcName, partitionId, serviceKind) {
    var replCurrentTableName = "MyAppReplCurrTable";
    var replEventsTableName = "MyAppReplEventsTable";
    var replHistoryTableName = "MyAppReplHistoryTable";
    var updatedSvcName = svcName.toString().replace('_', '/');
    console.log("Fetching replica data for app " + appName + " and svc " + updatedSvcName + " and prtnId " + partitionId);
    var replicaURL = "http://servicefabricname.cloudapp.net:19007/Applications/" + appName + "/$/GetServices/" + svcName + "/$/GetPartitions/" + partitionId + "/$/GetReplicas?api-version=1.0";
    var replicaPrimaryKeyName = "NodeName";
    /*
    if (serviceKind == "Stateless") {
        replicaPrimaryKeyName = "InstanceId";
    }
    */
    var additionalData = {"AppName": appName, "ServiceName": svcName, "PartitionId": partitionId};
    fetchAndUpdate("Replica", replicaURL, replicaPrimaryKeyName, replCurrentTableName, replEventsTableName, replHistoryTableName, 15, additionalData);
}

function fetchAndUpdate(dataType, urlToMonitor, primaryKeyName, currentTableName, eventsTableName, historyTableName, frequency, additionalData)
{
    var date = new Date(); 
    var current_min = date.getMinutes();
    var fUpdate = current_min % frequency;

    var req = require('request');
    var azure = require('azure');

     console.log("fetchAndUpdate " + urlToMonitor);
    // fetch data
    req.get({
    uri:urlToMonitor,
    timeout : 200000,
    headers:{'content-type': 'application/x-www-form-urlencoded'},
   }, 
   function (error, response, body) {
       if (!error && response.statusCode == 200) {
           var items = JSON.parse(body);
           // console.log(items);
           var tableService = azure.createTableService("storageaccountname", "storageaccountkey");
           tableService.createTableIfNotExists(currentTableName, function (error) {
               if (!error) {
                   // for each row
                   items.forEach(updateTables.bind(null, dataType, primaryKeyName, urlToMonitor, currentTableName, eventsTableName, historyTableName, fUpdate, additionalData));
                   // items.forEach(getHealth.bind(null, dataType, primaryKeyName, urlToMonitor, additionalData));
               }
               else {
                   console.log("CreateTableIfNotExists for " + currentTableName + " returned error: " + error);
               }
            })
       }
       else {
           console.log("ERROR: GET " + urlToMonitor + " returned error: " + error + "and statusCode " + response.statusCode);
       }
   })
}

function getHealth(dataType, primaryKeyName, primaryKeyValueToUse, urlToMonitor, additionalData, item
// , index, array
) {
    // Modify url to add data and GetHealth query
    // console.log("getHealth " + dataType + " item " + item[primaryKeyName]);
    var primaryKeyValue = item[primaryKeyName];
    if (dataType == "Partition") {
        primaryKeyValue = primaryKeyValueToUse;
    }
    
    /*
    var primaryKeyValue = item[primaryKeyName];
    if (dataType == "Partition") {
        var partitionInformation = item["PartitionInformation"];
        primaryKeyValue = partitionInformation["Id"];
    }
    */
    var healthUrl = urlToMonitor.replace("?api-version=1.0", "/" + primaryKeyValue +"/$/GetHealth?api-version=1.0");
    var req = require('request');

    req.get({
    uri:healthUrl,
    timeout : 200000,
    headers:{'content-type': 'application/x-www-form-urlencoded'},
   }, 
   function (error, response, body) {
       if (!error && response.statusCode == 200) {
           var items = JSON.parse(body);
           // console.log(items);
            var entityName = primaryKeyValue;
            if (dataType == "Service") {
                entityName = primaryKeyValue.toString().replace('/', '_');
            }
            else if (dataType == "Replica") {
                entityName = additionalData["PartitionId"] + "_" + primaryKeyValue;
            }
           items["HealthEvents"].forEach(updateHealthEventsTable.bind(null, dataType, entityName));
       }
       else {
           console.log("ERROR: GET health for " + healthUrl + " returned error: " + error + "and statusCode " + response.statusCode);
       }
   })
}

function updateHealthEventsTable(dataType, entityName, item, index, array) {
    var healthEventsTable = "MyAppHealthEventsTable";
    var azure = require('azure');
    var tableService = azure.createTableService("storageaccountname", "storageaccountkey");
    tableService.createTableIfNotExists(healthEventsTable, function (error) {
       if (!error) {
           item["PartitionKey"] = entityName;
           item["RowKey"] = item["SequenceNumber"];
           item["DataType"] = dataType;
           item["HealthState"] = mapEnum("HealthState", item["HealthState"]);
           tableService.insertOrReplaceEntity(healthEventsTable, item, function(error) {
                    if (!error) {
                        console.log(item.PartitionKey + " Health event updated");
                    }
                    else {
                        console.log("ERROR while updating health event for " + item.PartitionKey + " : " + error);
                    }
           }) 
       }
       else {
           console.log("ERROR: CreateTableIfNotExists for " + healthEventsTable + " returned error: " + error);
       }
    })
}

function updateTables(dataType, primaryKeyName, urlToMonitor, currentTableName, eventsTableName, historyTableName, fUpdate, additionalData, item, index, array) {
    var azure = require('azure');
    var primaryKeyValue = item[primaryKeyName];
    if (dataType == "Service") {
        primaryKeyValue = item[primaryKeyName].toString().replace('/', '_');
    }
    else if (dataType == "Partition") {
        var partitionInformation = item["PartitionInformation"];
        primaryKeyValue = partitionInformation["Id"];
        item["PartitionInformation"] = null;
    }
    else if (dataType == "Replica") {
        primaryKeyValue = additionalData["PartitionId"] + "_" + item[primaryKeyName];
    }
    
    for (var prop in additionalData) {
        item[prop] = additionalData[prop];
    }
    
    var getMatchingEntities = azure.TableQuery
    .select()
    .from(currentTableName).where('PartitionKey eq ?', primaryKeyValue);
    var tableService = azure.createTableService("storageaccountname", "storageaccountkey");
    tableService.queryEntities(getMatchingEntities, function (error, entities) {
        if (!error) {
            if (entities.length > 0) {
                for (var prop in item) {
                    // console.log("item[" + prop + "] = " + item[prop] + " ; eProp " + entities[0][prop]);
                    if (prop != "id" && prop != "Id" && prop != "link" && prop != "updated" && prop != "etag" && prop != "PartitionKey" && prop != "RowKey" && prop != "Timestamp") {
                        // if different from currenttable, then update events table
                        if (dataType == "Application" && prop == "Parameters") {
                            item[prop] = JSON.stringify(item[prop]);
                        }
                        item[prop] = mapEnum(prop, item[prop]);
                        var iProp = (item[prop] == null) ? null : item[prop].toString();
                        var eProp = (entities[0][prop] == null) ? null : entities[0][prop].toString();
                        if (iProp != eProp) {
                            writeToEventsTable(eventsTableName, primaryKeyValue, prop, item[prop]);
                            entities[0][prop] = item[prop];
                        }
                    }
                }
                // update row of existing entity
                item["PartitionKey"] = primaryKeyValue;
                item["RowKey"] = entities[0].RowKey;
                tableService.updateEntity(currentTableName, item, function (error) {
                    if (!error) {
                        // console.log(item.PartitionKey + " Entity updated");
                    }
                    else {
                        console.log("ERROR while updating entity " + item.PartitionKey + " : " + error);
                    }
                });
            }
            else {
               item["PartitionKey"] = primaryKeyValue;
               item["RowKey"] = randomString(128);
               for (var prop in item) {
                    item[prop] = mapEnum(prop, item[prop]);
                    if (dataType == "Application" && prop == "Parameters") {
                        var jsonValue = JSON.stringify(item[prop]);
                        console.log("App Parameters = " + jsonValue);
                        item[prop] = jsonValue;
                    }
               }
               tableService.insertEntity(currentTableName, item, function (error) {
                   if (!error) {
                       console.log(item.PartitionKey + " " + dataType + " Entity inserted in current table");
                   }
                   else {
                       console.log("Error while inserting entity " + item.PartitionKey + " in current table : " + error);
                   }
               });
            }
            // Update history table if frequency is met
            if (fUpdate == 0) {
               tableService.createTableIfNotExists(historyTableName, function (error) {
                   if (!error) {
                       item["RowKey"] = randomString(128);
                       tableService.insertEntity(historyTableName, item, function (error) {
                           if (!error) {
                               // console.log(item.PartitionKey + " Entity inserted in history table");
                           }
                           else {
                               console.log("ERROR while inserting entity " + item.PartitionKey + " in history table : " + error);
                           }
                       });
                   }
               });
            }
        }
        else {
            console.log("queryEntities returned error: " + error);
        }
    });
    getHealth(dataType, primaryKeyName, primaryKeyValue, urlToMonitor, additionalData, item);
}

function mapEnum(prop, value) {
    var retVal = value;
    var NodeStatus = {
        "0": "Invalid"
        , "1": "Up"
        , "2": "Down"
        , "3": "Enabling"
        , "4": "Disabling"
        , "5": "Disabled"
    }
    var HealthState = {
        "0": "Invalid"
        , "1": "Ok"
        , "2": "Warning"
        , "3": "Error"
        , "65535": "Unknown"
    }
    var ReplicaRole = {
        "0": "Invalid"
        , "1": "None"
        , "2": "Primary"
        , "3": "IdleSecondary"
        , "4": "ActiveSecondary"
    }
    var ReplicaStatus = {
        "0": "Invalid"
        , "1": "InBuild"
        , "2": "Standby"
        , "3": "Ready"
        , "4": "Down"
        , "5": "Dropped"
    }
    var ServiceKind = {
        "0": "Invalid"
        , "1": "Stateless"
        , "2": "Stateful"
    }
    switch (prop) {
        case "NodeStatus":
            retVal = NodeStatus[value];
            break;
        case "HealthState":
            retVal = HealthState[value];
            break;
        case "ReplicaRole":
            retVal = ReplicaRole[value];
            break;
        case "ReplicaStatus":
            retVal = ReplicaStatus[value];
            break;
        case "ServiceKind":
            retVal = ServiceKind[value];
            break;
        default:
            retVal = value;
            break;
    }
    return retVal;
}

function writeToEventsTable(eventsTableName, partitionKey, propertyName, propertyValue)
{
    var azure = require('azure');
    var tableService = azure.createTableService("storageaccountname", "storageaccountkey");
    tableService.createTableIfNotExists(eventsTableName, function (error) {
        if (!error) {
            var rowKey = randomString(128);
            var task = {
                PartitionKey: partitionKey
                            , RowKey: rowKey
                            , NotifyStateName: propertyName
                            , NotifyStateValue: propertyValue
            }
            tableService.insertEntity(eventsTableName, task, function (error) {
                if (!error) {
                    // console.log('Event table updated for ' + partitionKey + ' property ' + propertyName);
                }
                else {
                    console.log('ERROR while Event Table update: ' + error + ";pk = " + partitionKey + "; rk = " + rowKey + "; pname = " + propertyName + "; pval = " + propertyValue);
                }
            });
        }
    });
}

function randomString(bits){var chars,rand,i,ret
          chars='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+'
          ret=''
          // in v8, Math.random() yields 32 pseudo-random bits (in spidermonkey it gives 53)
          while(bits > 0){
              rand=Math.floor(Math.random()*0x100000000) // 32-bit integer
            // base 64 means 6 bits per character, so we use the top 30 bits from rand to give 30/6=5 characters.
            for(i=26; i>0 && bits>0; i-=6, bits-=6) ret+=chars[0x3F & rand >>> i]}
              return ret
}



Hope this blog will give readers confidence of creating a service fabric monitor with less cost and effort. Any suggestion or views are always welcome.

Thursday, May 7, 2015

ServiceFabric + ASA = Crime Detection, Part2

This blog is a continuation of http://nabaruns.blogspot.com/2015/05/servicefabric-asa-crime-detection-smurf.html

In this blog we want to highlight the PowerMap Analysis and the Database schema which we already discussed.

Thanks to one of my colleague, who quickly got the PowerMap skills, I am keeping this short by showing the analytics fetched by him.





The data is getting simulated while I am writing this blog. It is fetching the results from Database as follows

The latitude and longitude data are helping us to point it in PowerMap, thus giving a real time tracking analysis of the stolen car, hence proving the POC.

 

ServiceFabric + ASA = Crime Detection, Smurf Part2

This blog captures a story of an idea which was in my mind from a long time, but did not prove to be feasible in past. With Service Fabric and Azure Stream Analytics this has now become possible for us to perceive the potential of technology to prevent crime.

Scenario

Here in this story we have tried to emulate a scenario wherein there are multiple traffic cameras capturing the vehicle number plates and sending it to ASA(Azure Stream Analytics) service. Suppose there is a vehicle that got stolen and has been reported just now. Law agencies need to add this vehicle in lookout table. Now the stolen vehicle happens to pass from Camera3. Camera3 which is continuously emitting data to Event Hub which in turn goes to ASA, captures the number plate data. ASA queries the input against the lookout table and filters the vehicle data as soon as it is scanned by the ASA query into the database. PowerMap sheet provides a analysis based on the database content and shows a pointer at Camera3 location, thus providing the law agencies a quick mechanism to track the vehicle

Problem Breakdown 

Broadly for our POC we need to see these 2 paths
1. Simulate the traffic cam scenario which is sending continuous data to event hubs
2. Input is read by ASA and stolen vehicle details with the Camera details are sent to output DB
 

Technical details

Input

In order to have input. We created a Stateless Service Fabric Actor project. As Service Fabric Actor has a compute and a state , this is feasible to simulate a traffic camera.
 
Broadly we are looking for following data structures. StolenCarModel is a datastructure that is holding the Car details such as SerialId (Any identification, we are considering this as unique id, and this will help us to create the clustered index in our db), Vehicle Number(Captured from image), CameraSerial(Identification Id of Camera), CaptureTime(Timestamp when it was captured, which will help in tracking investigation)
 
 public class StolenCarModel
    {
        public int SerialId { get; set; }
        public string VehicleNumber { get; set; }
        public string CameraSerial { get; set; }
        public string CaptureTime { get; set; }
    }
 
Another datastructure that needs to be maintained is CameraDetails , which will have the CameraID, Latitude and Longitude where Camera is positioned (This will help us to point in powermap), and Area which will have the name of the area
 
    public class CameraDetails
    {
        public string CamID { get; set; }
        public string Latitude { get; set; }
        public string Longitude { get; set; }
        public string Area { get; set; }
    }
 

Service Fabric Details 

First and foremost is the interface Actor for our service
 
public interface IStolenCarStatelessActor : IActor
    {
        Task<bool> StartSendingCarDetails(List<StolenCarModel> stolenCarList);
    }
 
Corresponding to the interface the Actor Class should have the implementation
 
public async Task<bool> StartSendingCarDetails(List<StolenCarModel> cameraDetails)
        {
            ActorEventSource.Current.ActorMessage(this, "Start Sending Data");
            EventHubLayer hub = new EventHubLayer();
            var sendDetails = await hub.SendEventsToEventHub(cameraDetails);
            return sendDetails;
        }
 

Smurf Details 

We are using Smurf core libraries in order to send data to event Hubs. Please refer to my previous blog on Smurf Core api at http://nabaruns.blogspot.com/2015/03/smurfan-asa-based-tool-in-making-part-1.html
 
In order to populate the StolenCarDetails class, we are using this function
 
/// <summary>
        /// Get List of sample StolenCarModel details which can be sent later to the event hub
        /// </summary>
        /// <returns>Returns the list of stolen car model</returns>

        private static List<StolenCarModel> GetMetricDataList(string camera)
        {
            Random rand = new Random();
            var metricDataList = new List<StolenCarModel>();
 
            Console.WriteLine();
            Console.WriteLine("#####################{0}####################", camera);
            for (var loop = 0; loop < 5; loop++)
            {
                var date = DateTime.Now;
                var serialId = rand.Next(0, 50);
                var nowTime = date.Ticks.ToString();
               
                Console.Write("Camera:{0} SerialId:{1} TimeStamp:{2} Vehicle:{3}", camera, serialId, nowTime, "ADC01" + serialId);
                Console.WriteLine();
                var info = new StolenCarModel() { CameraSerial = camera, SerialId = serialId, CaptureTime = nowTime, VehicleNumber = "ADC01" + serialId };
                metricDataList.Add(info);
            }
            Console.WriteLine("##############################################");
            Console.WriteLine();
            return metricDataList;
        }
 
In our POC at time t=0, we are assuming there are 5 events getting generated by each camera. We are currently simulating the camera details as mentioned in the marked line
 
This is giving us the following input
 
 
 
Now moving ahead we need to simulate multiple actors as cameras.
 
var camera1 = ActorProxy.Create<IStolenCarStatelessActor>(ActorId.NewId(), "fabric:/StolenCarStatelessActorApplication");
            var camera2 = ActorProxy.Create<IStolenCarStatelessActor>(ActorId.NewId(), "fabric:/StolenCarStatelessActorApplication");
         ...
            var camera10 = ActorProxy.Create<IStolenCarStatelessActor>(ActorId.NewId(), "fabric:/StolenCarStatelessActorApplication");
            int loop = 0;
            do
            {
                var tollboothList = GetMetricDataList("Camera1");
                var sendSuccess = camera1.StartSendingCarDetails(tollboothList);
                Console.WriteLine("Sending Camera1 Details {0}: {1}", camera1.GetActorId(), sendSuccess.Result);
                Thread.Sleep(5000);
                tollboothList = GetMetricDataList("Camera2");
                sendSuccess = camera2.StartSendingCarDetails(tollboothList);
                Console.WriteLine("Sending Camera2 details {0}: {1}", camera2.GetActorId(), sendSuccess.Result);
                Thread.Sleep(5000);
                ...
                tollboothList = GetMetricDataList("Camera10");
                sendSuccess = camera10.StartSendingCarDetails(tollboothList);
                Console.WriteLine("Sending Camera10 details {0}: {1}", camera10.GetActorId(), sendSuccess.Result);
                Thread.Sleep(5000);
                loop++;
            } while (loop < 10);
 
In above code we are creating 10 actor objects calling the Stolen Car service and then sending random data under them
 
Now having done the input part next is ASA query which will help us populate the data to db
 

ASA Query

In order to have a POC for the idea, our ASA query is very simple
 
SELECT SerialId, VehicleNumber, CameraSerial, CaptureTime FROM CallStream WHERE VehicleNumber='ADC0116';
 
In here we are saying that the stolen vehicle is ADC0116, if I get any data related to that , populate the data in my database.
 

SQL Database

In order to create a database which will keep a track of the stolen vehicles we need to create a table similar to the StolenCar Schema
 
Keeping it very simple I create the table as follows
 
Create table StolenCardTable (SerialId varchar(255), VehicleNumber varchar(255), CameraSerial varchar(255), CaptureTime varchar(255));
 
CREATE CLUSTERED INDEX StolenCardTable
    ON StolenCardTable(SerialId ASC);
GO 

Action Time

Till now I have all pieces in place to verify my app. Here is a screenshot which verifies the stateless service actor app deployed on my localhost cluster
 
 
Now after starting the client first point of verification is event hub dashboard and logs which should start receiving the events
 
 
Second verification will be ASA dashboard and logs
 
 
If till now you have seen everything is fine most probably you will get the data in database
 
16 ADC0116 Camera7 NULL
16 ADC0116 Camera9 NULL
16 ADC0116 Camera7 NULL
16 ADC0116 Camera10 NULL
16 ADC0116 Camera3 NULL
16 ADC0116 Camera2 NULL
16 ADC0116 Camera2 NULL
16 ADC0116 Camera9 NULL
16 ADC0116 Camera2 NULL
I have got this sample output in my db which says ADC0116 was first seen in Camera7 then 9 then again 7 and so on , eventually ending in 2. This can be now easily highlighted in PowerMap based on camera details and location which will be my next blog.
 
I think by now I am having a story to share with you all. The intent of this blog is to utilize the potent technologies and try to make this world a better place. Hope you will like it. Please feel free to comment your views or suggestions.

Continuation blog is at http://nabaruns.blogspot.com/2015/05/servicefabric-asa-crime-detection-part2.html