Wednesday, February 19, 2020

SelfNote: VSCode setup

Recently I started to use the Azure cloud machine to do my coding work. I can still do my coding in my laptop using full Visual Studio but an Azure VM with VSCode will be much convenient. Low tech but works. :)

The following VSCode setup script helps me set up a new machine:

  1. open PowerShell 
  2. copy extension names to the clipboard (yes, Ctrl+C)
  3. run get-clipboard | % { code --install-extension $_ }



DavidAnson.vscode-markdownlint
eamodio.gitlens
ms-azuretools.vscode-docker
ms-python.python
ms-vscode-remote.remote-wsl
ms-vscode.cpptools
ms-vscode.powershell
npclaudiu.vscode-gn
yzhang.markdown-all-in-one

Monday, February 3, 2020

Docker Image build clean up using PowerShell on local PC

When I run the docker image to do the debug on PC, I was struggling to keep the hard disk space consumption low and running the recent binary in "Docker on Windows Desktop". This development stage, so I won't consider the Azure cloud for hosting.

I need a script to reclaim hard disk, remove the currently running container, build and run the new image in docker on windows desktop. 

The following code first removes unused images and then gets the image running on the port 5555. I will have to play the string replace tricks using PowerShell. Please check "-replace" part for details. 

#prune unused image
docker image prune -f

#remove existing running docker image
$a = docker container ls --filter expose=5555
$a2 = $a | foreach-object {$_ -replace '\s{3,}', '  '} | foreach-object {$_ -replace '  ', "`t"}
$obj = ConvertFrom-Csv -InputObject $a2 -Delimiter "`t"
if ($obj -ne $null)
{
    docker rm -f  $obj.names
}
else
{
    Write-Verbose "cannot find running container image so won't remove"
}

#build flask app and set its version to 1
docker build -t flaskapp:1 .
docker run -d -p 5555:5555 flaskapp:1

#test running instance
sleep -s 4
invoke-restmethod "http://localhost:5555/api/health"

Sunday, January 19, 2020

SelfNote: Setup Spacy Module in Visual Studio

I guess not so many people use Visual Studio doing NLP with Python. I have been searching for a solution on how to add a Spacy module in Visual Studio. I kept getting cannot find en_core_web_sm module error and the only solution I could find is to run python -m spacy download en_core_web_sm. I was struggling to find where in the UI to run the Python command. And finally, the solution came.

you noticed the "Admin" icon? yes, that is the key to solve the problem. With admin permission, you can open PowerShell and run python -m spacy download en_core_web_sm.

I installed en_core_web_sm, but it shows a warning message. And the en_core_web_lg module can eliminate the warning.




Monday, January 6, 2020

Standford NLP Quick Setup on Win10 with WSL

After setting up the Stanford NLP on my PC, I was struggling with how to run it faster. Then I ran into Windows Subsystem for Linux (WSL). I realize that the modification of Stanford NLP is not an option for me. I can use WSL to quickly start an NLP web service and start my work.

I ordered a more powerful VM from Azure and use PowerShell to set up the NLP environment. 

  1. Enable WSL

    Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

    It will trigger a reboot if WSL is not enabled
  2. Add Ubuntu disco to WSL

    # download ubuntu 18.04 as save it as Ubuntu.appx at local directory
    Invoke-WebRequest -Uri https://aka.ms/wsl-ubuntu-1804 -OutFile Ubuntu.appx -UseBasicParsing

    # add Ubuntu.appx to WSL
    Add-package Ubuntu.appx
  3. Download Standard NLP zip file and unzip it to the current folder

    # download the Stanford NLP and save the zip file locally as "corenlp.zip"
    Invoke-WebRequest -uri http://nlp.stanford.edu/software/stanford-corenlp-full-2018-10-05.zip -outfile corenlp.zip -UseBasicParsing

    # unzip the corenlp.zip
    Expand-Archive corenlp.zip -DestinationPath .\CoreNlp\
  4. install Java in WSL. Since Stanford NLP does not require Oracle Java, so I use Open Java to make the command shorter

    wsl sudo apt-get update
    wsl sudo apt-get install default-jdk
I go into WSL from PowerShell and launch Stanford NLP from WSL. 

  • go to WSL from Powershell by using "wsl"
  • go to the folder which stores the unzipped Stanford NLP files in step 3
  • run java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer
Open Edge and go to http://localhost:9000/, it will show similar UI like http://corenlp.run.

The PowerShell script used to access the localhost 9000 ports are listed below:

$data = "The quick brown fox jumped over the lazy dog."
$url2 = 'http://localhost:9000/?properties={"annotators":"tokenize,ssplit,pos,lemma,ner, entitymentions,depparse,parse,relation,openie,dcoref,kbp","outputFormat":"json"}'
$r = Invoke-RestMethod -Uri $url2 -Method post -Body $data

The annotators are listed here in case you need it.


Monday, September 2, 2019

6-Year-Old Rule?

The story begins recently I am indulging in the quantum computing programming whose foundation involves a lot of linear algebra. The direct consequence is I want to solve almost all problems using the complex number, vector, and matrix. The Mars Rover problem got my attention.

The rover on the Mars surface only takes L, R, and M instruction. L means turn left. R means turn right. M means move 1 step forward. If the initial position is (0, 0) and facing North, what will be the location and facing direction after receiving couples of instructions? 

This is a perfect opportunity to practice my math, I thought. The direction and location are all complex numbers. I can manipulate numbers to make the code concise! I convert the L, R, and M instruction to complex number op:


  • turn left L is complex (0, 1) where the real part is 0 and the imaginary part is 1
  • turn right R is complex (0, -1)
  • move forward M is complex number (1, 0)
The new location and direction can be computed like the following:

  • new location += op.Real * Direction  //L and R's real part is 0, so it does not move location
  • new direction *= op  //M is 1, so it has no effect on the direction
It looks cool and code is concise. I avoided those unpleasant switch statements. 

Everything looks satisfying until I realize those expensive multiply operations might be a performance problem. I wrote a simple solution using those switch statements. The simple solution doubled the code. However, the performance is 30+% faster than my "elegant" one! 

I told my wife what was going on when I encountered her in the kitchen, with a little bit of frustration. She smiled and pointed my 6-year-old son. "If the solution can be understand by him, it must be faster." she said, "It requires a smaller brain so it runs faster."

Well, both algorithms are O(n). Maybe next time, I should think on the big O level. 

Sunday, March 3, 2019

Old stuff: my first IEEE paper on Computational Intelligence

Before I finished my college, I managed to publish a paper on evolutionary computation (EC). The paper is on the encoding part. My next task is to use quantum to rewrite my algorithm. Well, there is a quantum genetic algorithm (QGA) there. However, I think with the latest decision on Q# and quantum algorithms. The paper link is here.

Saturday, January 19, 2019

Microsoft Q# Day 1: Qubit Collapsed

After going through the quantum linear algebra, I am going to see some quantum concept by using Q#.

The first one I want to prove is measurement will collapse the qubit. I will use the AssertProb to verify the value with probability. The pseudo code is listed below:

  1. use H gate to set the qubit to superposition
  2. check the probability to 1 equals 0.5 (50%)
  3. measure the qubit
  4. check the qubit equals the measured value equals to 1 (100%). The qubit collapsed. 
the code listed below:


1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
namespace Quantum.QSharpApplication2
{
    open Microsoft.Quantum.Primitive;
    open Microsoft.Quantum.Canon;
 open Microsoft.Quantum.Extensions.Convert;

    operation Operation1 () : Unit
    {
        body(...)
        {
            using (ancilla = Qubit())
   {
    H(ancilla);

    AssertProb([PauliZ], [ancilla], Zero, 0.5, "error", 1e-5);
    AssertProb([PauliZ], [ancilla], One, 0.5, "error", 1e-5);

    mutable gate = PauliZ;
    let mX = Measure([gate], [ancilla]);
    AssertProb([gate], [ancilla], mX, 1.0, "error", 1e-5);

    Message(ToStringD(mX == Zero ? 0. | 1.));

    Reset(ancilla);
   }
        }
    }
}