Docker Image build clean up using PowerShell on local PC

When I run the docker image to do the debug on PC, I was struggling to keep the hard disk space consumption low and running the recent binary in "Docker on Windows Desktop". This development stage, so I won't consider the Azure cloud for hosting.

I need a script to reclaim hard disk, remove the currently running container, build and run the new image in docker on windows desktop. 

The following code first removes unused images and then gets the image running on the port 5555. I will have to play the string replace tricks using PowerShell. Please check "-replace" part for details. 

#prune unused image

docker image prune -f

#remove existing running docker image

$a = docker container ls --filter expose=5555

$a2 = $a | foreach-object {$_ -replace '\s{3,}', '  '} | foreach-object {$_ -replace '  ', "`t"}

$obj = ConvertFrom-Csv -InputObject $a2 -Delimiter "`t"

if ($obj -ne $null)

{

    docker rm -f  $obj.names

}

else

{

    Write-Verbose "cannot find running container image so won't remove"

}

#build flask app and set its version to 1

docker build -t flaskapp:1 .

docker run -d -p 5555:5555 flaskapp:1

#test running instance

sleep -s 4

invoke-restmethod "http://localhost:5555/api/health"

Standford NLP Quick Setup on Win10 with WSL

After setting up the Stanford NLP on my PC, I was struggling with how to run it faster. Then I ran into Windows Subsystem for Linux (WSL). I realize that the modification of Stanford NLP is not an option for me. I can use WSL to quickly start an NLP web service and start my work.

I ordered a more powerful VM from Azure and use PowerShell to set up the NLP environment. 

1. Enable WSL
   
   Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
   
   It will trigger a reboot if WSL is not enabled
2. Add Ubuntu disco to WSL
   
   # download ubuntu 18.04 as save it as Ubuntu.appx at local directory
   Invoke-WebRequest -Uri https://aka.ms/wsl-ubuntu-1804 -OutFile Ubuntu.appx -UseBasicParsing
   
   # add Ubuntu.appx to WSL
   Add-package Ubuntu.appx
3. Download Standard NLP zip file and unzip it to the current folder
   
   # download the Stanford NLP and save the zip file locally as "corenlp.zip"
   Invoke-WebRequest -uri http://nlp.stanford.edu/software/stanford-corenlp-full-2018-10-05.zip -outfile corenlp.zip -UseBasicParsing
   
   # unzip the corenlp.zip
   Expand-Archive corenlp.zip -DestinationPath .\CoreNlp\
4. install Java in WSL. Since Stanford NLP does not require Oracle Java, so I use Open Java to make the command shorter
   
   wsl sudo apt-get update
   wsl sudo apt-get install default-jdk

I go into WSL from PowerShell and launch Stanford NLP from WSL. 

• go to WSL from Powershell by using "wsl"
• go to the folder which stores the unzipped Stanford NLP files in step 3
• run java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer

Open Edge and go to http://localhost:9000/, it will show similar UI like http://corenlp.run.

The PowerShell script used to access the localhost 9000 ports are listed below:

$data = "The quick brown fox jumped over the lazy dog."
$url2 = 'http://localhost:9000/?properties={"annotators":"tokenize,ssplit,pos,lemma,ner, entitymentions,depparse,parse,relation,openie,dcoref,kbp","outputFormat":"json"}'
$r = Invoke-RestMethod -Uri $url2 -Method post -Body $data

The annotators are listed here in case you need it.

Solve Automation & Biz Flow Project Resource Problem with PowerShell

When I take over the DevOps project to do the automation work. My project is not juts to streamline the software development process, it involves in large number of internal systems. For a company with many legacy systems and different business process. It is literally re-engineer the business flow.

My challenge is to have the people who know the business flow and the technology and make them work together. I do not want to hire BA because I have enough people who know business knowledge. The pressing task to have more person to code the logic. Thanks to my language background, the solution I found is to use different languages to solve the problem.

The first decision I made is to have no-GUI at all. The fancy GUI is nice to a new user; however, it will block the future automation for good. When you have a meeting with your financial planner, you will notice how many times s/he do copy/paste from one system to the other.

To fast code the solution, I need a scripting language or a language support scripting feature. That language should be good for folks with less coding experience. I do not need them to understand what is singleton pattern or difference between NTLM and Basic Authentication. I want my dev team to focus on the technical challenges and provide API's. Biz-background people can use those API's to streamline those processes in his/her area.

The initial setup is like the diagram following. Part1 is finished by biz people by invoke API's provided by dev's working on Part2.

I landed on the PowerShell language for Part1. PowerShell (PS) is designed for support/op folks. It has large number of samples. The most compelling reason for PS is that it can move the interface line from top to very bottom in the above diagram. PS can go deeper. Almost everything C#/F#/VB.net can do, PS can do. At the project initial stage, Part2 is much bigger than part1. I want a language has the potential to go deeper to the binary code in case my dev does not have the bandwidth to provide API. Other .Net languages can invoke PowerShell as well. As a result, the coding effort has little waste. A perfect eco-system where every bit of coding effort can be used and reused.

The current result is my biz people can code PowerShell and slowly push the "interface line" down to the ultimate binary format. The dev side does not have to deal with those "boring" business work. And the dev workload is getting less. By using PowerShell scripting language,

• Efficiency: one line PS code will do 10+ lines of .net language code. The overall progress will be faster when using a scripting language. 
• cost saving, I have the potential to expand my team to accommodate more people with various programming level
• Solid and quick solution: both parties can focus on the area in which s/he has deep expertise. The implementation speed is good.
• Flexible solution: since the interface line can move up and down based on the resource / expertise. 
• Vendor-independent: my solution is general enough so won't be locked in any vendor's customized solution. 

When the automation happens, one of the push back is the job lose. Having a person in the domain is the deciding factor for project success. The PowerShell provide a comfortable environment for biz people to start and grow, which makes my biz part team stable. 

To use a widely adopted language also stabilize the dev side. I witness many effort to have an in-house language which is only used in that team. Many vendor solution provides a programming interface which few people know how to use. The customized solution literally blocks the people from mainstream technology and few people wants their career to be in this situation. If you search for PowerShell, you can still find job openings requires that skill. This option really helps the adoption of the solution. With .net core moves to Linux platform, PowerShell 6 and all other .net languages will flood into Linux world. So the future is really bright. 

So far I am very happy about the result!

BTW: if you want to use F# to write PS module, here is the link.

Use F# to Write PowerShell Snapin & Cmdlet

When I was in Minneapolis, I was thinking to use the F# to write PowerShell snapin and cmdlet. I got the feeling that this can speed our development speed.

Because I use Visual Studio 2012 and it generates the .NET 4 binary. So I need to install PowerShell 3.0 in order to use .NET 4.0. You can use $psversiontable to check the CLR runtime version, make sure it is a number equal or greater than 4.

 namespace Log4NetPsSnapIn  
 open System  
 open System.Management.Automation  
 open System.ComponentModel  
 [<RunInstaller(true)>]  
 type Log4NetSnapIn() =  
   inherit PSSnapIn()  
   override this.Name with get() = "aa"  
   override this.Vendor with get() = "bb"  
   override this.Description with get() = "dd"  
 [<Cmdlet(VerbsCommunications.Write, "Hi")>]  
 type WriteHelp() =   
   inherit Cmdlet()  
   override this.ProcessRecord() =   
     base.WriteObject("help");  

After compiling the above code, a DLL is generated. You have to use installutil.exe to add the snapin. The PowerShell snapin has Name = "aa", so you can use Add-PsSnapIn aa to load the DLL. You can also use installutil.exe /u to uninstall the snapin from your system. Make sure you open the cmd window or powershell window with administrator privilege.

• If you are running 64-bit version machine, make sure you use installUtil.exe under C:\Windows\Microsoft.net\Framework64\. 
• if your OS is 32-bit, you can use installUtil.exe under C:\Windows\Microsoft.net\Framework\

I like the way F# write PowerShell snapin. The code is concise and easier to understand.

Tweet to fsharp

SelfNote: PowerShell on SQL Jobs

You know what. The lack of "Start" button on Win8 makes me really learn PowerShell. I feel I am more like a Unix admin than an average user. :-D

 # disable backup job on a server  
 function Disable-BackupJob($serverName)  
 {  
   invoke-command -computerName $serverName -ScriptBlock { `  
     param($serverName); `  
     [System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.SMO") | Out-Null; `  
     $srv = New-Object Microsoft.SqlServer.Management.SMO.Server($serverName); `  
     $jobs = $srv.JobServer.Jobs | Where-Object {$_.IsEnabled -eq $TRUE} | Where-Object { $_.Name -like '*backup*' }; `  
     foreach ($job in $jobs) `  
     { `  
       write-host "$serverName.$job is to FALSE"; `  
       $job.IsEnabled = $false; `  
       $jobs.Alter(); `  
     }; `  
     $jobs = $srv.JobServer.Jobs | Where-Object {$_.IsEnabled -eq $TRUE} | Where-Object { $_.Name -like '*backup*' }; `  
     if ($jobs.Count -eq 0) { write-host "$serverName Done." } else { write-host "$serverName failed!" }; `  
   }`  
   -ArgumentList $serverName  
 }  

SelfNote: PowerShell scripts

the following three functions are use PowerShell to set password, move cluster, and test connection to a database server.

 #set account username and password  
 function Set-Password($computerName, $serviceName, $serviceAccount, $password)  
 {    
   invoke-command -computerName $computerName -ScriptBlock `  
   { param($computerName, $serviceName, $serviceAccount, $password); `  
     write-host "on computer " $env:ComputerName "working on " $serviceName; `  
     $filter = "Name='" + $serviceName + "' "; `  
     $sqlservice=Get-WMIObject win32_service -filter $filter;`  
     $result = $sqlservice.change($null,$null,$null,$null,$null,$null, $serviceaccount,$password,$null,$null,$null);`  
     if ($result.ReturnValue -eq 0) { write-host $computerName " done!"; } else { write-host $computerName " failed!"; } `  
   } `  
   -ArgumentList $computerName,$serviceName,$serviceAccount,$password `  
 }  
 # move cluster  
 function Move-Cluster($computerName)  
 {  
   invoke-command -computerName $computerName -ScriptBlock `  
   { `  
     import-module failoverclusters;`  
     $result = Move-ClusterGroup sqlgroup;`  
     write-host "owner node = " $result.OwnerNode; `  
     $result = Move-ClusterGroup sqlgroup;`  
     write-host "owner node = " $result.OwnerNode; `  
   }  
 }  
 #test connection  
 function Test-Connection($servername)  
 {  
   $SqlConnection = New-Object System.Data.SqlClient.SqlConnection;  
   $SqlConnection.ConnectionString = "Server=$servername;Database=master;Integrated Security=True";  
   try  
   {  
     $SqlConnection.Open();  
     write-host "$servername connection OK."      
   }  
   catch  
   {  
     write-host "$servername connection failed"  
     write-host $error[0]  
   }  
   finally  
   {  
     $SqlConnection.Close();  
   }  
 }