Why F# Type Provider?

F# type provider is a very nice feature. I was asked several times why I need type provider? Just because of the intellisense? I believe there should be something else. In the current team, I found people "hate" type, they prefer to use C# object all the times. Now I realize why type is so important and why type can increase code quality.

I want to use a simple approval process as a sample. For my understanding, a type defines the meaning of a area of memory and certain operations which can be executed on this type.

The sample is simple. You can have something to be approved by lead, manager, and director. If a lead approves, it returns a manager type. If manager disapproves, the process will stop and director won't know it. After director's approval, this process stops. The type provider can have three types, director's approve method returns nothing while other type's approve method return higher level type. Lead's approve method returns manager, and manager's approve method returns director.

I'd bet people can say I can use a single type to do the same job. A general class can be used to create three instances. The director instance's approve method can return NULL or any special value to stop the approval process. It can solve this problem provides you know director's approval will stop this process beforehand. A runtime error is not far away if this information is not known beforehand. The type provider uses type to provide the information. After you invoke director's approve method, which returns NULL, user will know immediately that this process is terminated.

I would say in this case, type is a way to move runtime check/error to compile time. The attached is the demo project for this approval process.

Tweet to fsharp

Use F# to Write PowerShell Snapin & Cmdlet

When I was in Minneapolis, I was thinking to use the F# to write PowerShell snapin and cmdlet. I got the feeling that this can speed our development speed.

Because I use Visual Studio 2012 and it generates the .NET 4 binary. So I need to install PowerShell 3.0 in order to use .NET 4.0. You can use $psversiontable to check the CLR runtime version, make sure it is a number equal or greater than 4.

 namespace Log4NetPsSnapIn  
 open System  
 open System.Management.Automation  
 open System.ComponentModel  
 [<RunInstaller(true)>]  
 type Log4NetSnapIn() =  
   inherit PSSnapIn()  
   override this.Name with get() = "aa"  
   override this.Vendor with get() = "bb"  
   override this.Description with get() = "dd"  
 [<Cmdlet(VerbsCommunications.Write, "Hi")>]  
 type WriteHelp() =   
   inherit Cmdlet()  
   override this.ProcessRecord() =   
     base.WriteObject("help");  

After compiling the above code, a DLL is generated. You have to use installutil.exe to add the snapin. The PowerShell snapin has Name = "aa", so you can use Add-PsSnapIn aa to load the DLL. You can also use installutil.exe /u to uninstall the snapin from your system. Make sure you open the cmd window or powershell window with administrator privilege.

• If you are running 64-bit version machine, make sure you use installUtil.exe under C:\Windows\Microsoft.net\Framework64\. 
• if your OS is 32-bit, you can use installUtil.exe under C:\Windows\Microsoft.net\Framework\

I like the way F# write PowerShell snapin. The code is concise and easier to understand.

Tweet to fsharp

Web Service using F#

this is a code to read file from local disk:

make sure it runs in the console application.

 open System  
 open System.IO   
 open System.ServiceModel  
 open System.ServiceModel.Web  
 let siteRoot = @"c:\myCode\"  
 [<ServiceContract>]  
 type MyContract() =  
   [<OperationContract>]  
   [<WebGet(UriTemplate="{file}")>]  
   member this.Get(file:string) : Stream =  
     printfn "Requested : '%s'" file  
     let path = Path.Combine(siteRoot, file)  
     let fileExists = File.Exists(path)  
     if fileExists then  
       WebOperationContext.Current.OutgoingResponse.ContentType <- "text/html"  
       let bytes = File.ReadAllBytes(path)  
       upcast new MemoryStream(bytes)  
     else  
       printfn "cannot find %s" path  
       upcast new MemoryStream()  
 let startAt address =  
   let host = new WebServiceHost(typeof<MyContract>, new Uri(address))  
   host.AddServiceEndpoint(typeof<MyContract>, new WebHttpBinding(), "")  
    |> ignore  
   host.Open()  
   host  
 let server = startAt "http://localhost:8081/"  
 printfn "started service..."  
 ignore <| System.Console.Read()  
 printfn "closing service..."  
 server.Close()  
 printfn "closed"  

Tweet to fsharp

SelfNote: ListBox scroll bar size

 <ScrollViewer.Resources>  
                                        <sys:Double x:Key="{x:Static SystemParameters.VerticalScrollBarWidthKey}">5</sys:Double>  
                                              <sys:Double x:Key="{x:Static SystemParameters.VerticalScrollBarButtonHeightKey}">5</sys:Double>  
                                              <sys:Double x:Key="{x:Static SystemParameters.HorizontalScrollBarHeightKey}">5</sys:Double>  
                                              <sys:Double x:Key="{x:Static SystemParameters.HorizontalScrollBarButtonWidthKey}">3</sys:Double>  
                                      </ScrollViewer.Resources>  

SelfNote: PowerShell on SQL Jobs

You know what. The lack of "Start" button on Win8 makes me really learn PowerShell. I feel I am more like a Unix admin than an average user. :-D

 # disable backup job on a server  
 function Disable-BackupJob($serverName)  
 {  
   invoke-command -computerName $serverName -ScriptBlock { `  
     param($serverName); `  
     [System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.SMO") | Out-Null; `  
     $srv = New-Object Microsoft.SqlServer.Management.SMO.Server($serverName); `  
     $jobs = $srv.JobServer.Jobs | Where-Object {$_.IsEnabled -eq $TRUE} | Where-Object { $_.Name -like '*backup*' }; `  
     foreach ($job in $jobs) `  
     { `  
       write-host "$serverName.$job is to FALSE"; `  
       $job.IsEnabled = $false; `  
       $jobs.Alter(); `  
     }; `  
     $jobs = $srv.JobServer.Jobs | Where-Object {$_.IsEnabled -eq $TRUE} | Where-Object { $_.Name -like '*backup*' }; `  
     if ($jobs.Count -eq 0) { write-host "$serverName Done." } else { write-host "$serverName failed!" }; `  
   }`  
   -ArgumentList $serverName  
 }  

SelfNote: PowerShell scripts

the following three functions are use PowerShell to set password, move cluster, and test connection to a database server.

 #set account username and password  
 function Set-Password($computerName, $serviceName, $serviceAccount, $password)  
 {    
   invoke-command -computerName $computerName -ScriptBlock `  
   { param($computerName, $serviceName, $serviceAccount, $password); `  
     write-host "on computer " $env:ComputerName "working on " $serviceName; `  
     $filter = "Name='" + $serviceName + "' "; `  
     $sqlservice=Get-WMIObject win32_service -filter $filter;`  
     $result = $sqlservice.change($null,$null,$null,$null,$null,$null, $serviceaccount,$password,$null,$null,$null);`  
     if ($result.ReturnValue -eq 0) { write-host $computerName " done!"; } else { write-host $computerName " failed!"; } `  
   } `  
   -ArgumentList $computerName,$serviceName,$serviceAccount,$password `  
 }  
 # move cluster  
 function Move-Cluster($computerName)  
 {  
   invoke-command -computerName $computerName -ScriptBlock `  
   { `  
     import-module failoverclusters;`  
     $result = Move-ClusterGroup sqlgroup;`  
     write-host "owner node = " $result.OwnerNode; `  
     $result = Move-ClusterGroup sqlgroup;`  
     write-host "owner node = " $result.OwnerNode; `  
   }  
 }  
 #test connection  
 function Test-Connection($servername)  
 {  
   $SqlConnection = New-Object System.Data.SqlClient.SqlConnection;  
   $SqlConnection.ConnectionString = "Server=$servername;Database=master;Integrated Security=True";  
   try  
   {  
     $SqlConnection.Open();  
     write-host "$servername connection OK."      
   }  
   catch  
   {  
     write-host "$servername connection failed"  
     write-host $error[0]  
   }  
   finally  
   {  
     $SqlConnection.Close();  
   }  
 }  

Hadoop Day 3

I got my first map/reduce job running on the HDInsight Services For Windows. The material provided by HDInsight team is very good. :-)

The Hadoop gives a new way to store and process data. We generate huge amount of data every day and the data schema changes constantly. Every times we design a database, we assume the data does not change that often, but this is not true. The possible solution is to dump all the data with low cost. If the portion of data suddenly becomes "valuable", a HIVE external table can be created and give a schema to these data. By using the ODBC driver, the data can be copied to SQL Server which is a high cost solution storage data.  Traditional SQL Server will not be replaced by Hadoop,  it only serves for the "valuable" data. For some "useless" data, let them stay in Hadoop.

I can't publicly discuss or blog the HDInsight feature right now. So this post is to summarize the public materials.

Personally I am more interested in finding a good solution to store the data and later process it as fast as possible. I know Hadoop is ready to for "big data", so I am more interested in

1. How to link with other technologies, such as SQL server and SQL Azure
2. Move data between different data storage
3. perform my own map/reduce work

The article about SSIS catches my eye. I covers question 1 and 2. The ASV protocol mentioned in the article is the way to access Azure blob from Hadoop. If you link to my previous post about Azure blob, you can tell where that post is from. The HIVE can point to a ASV folder by using the Create External Table statement (in section 8.3). You might want to use this link to check all HIVE data types.

Once the data is organized as a table, it can be accessed by using HIVE ODBC. The ODBC enables all kinds of connections and our existing tools and skills are connected. You can NOT write data by using ODBC.

The map/reduce program is very simple, it is basically a clone of C# sample. The only problem I found is debug. The execution log from UI is not that helpful. My trick is to export debug information to the output file. Once the algorithm is correct, the debug information can be eliminated.