tag:blogger.com,1999:blog-41000915906946284722024-03-05T06:34:44.544-05:00Apollo 13 - Tao Liu's blogFailure is not an option.Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.comBlogger244125tag:blogger.com,1999:blog-4100091590694628472.post-89961695044597111182020-10-18T10:40:00.005-04:002020-10-18T10:54:51.436-04:00SelfNote: Enable Python in Visual Studio Interactive<div>This note is to document how to configure Python interactive window in Visual Studio (not VS Code)</div><div><br /></div><h3 style="text-align: left;">Reference a local library</h3><div>When I added a new .py file in the project, I cannot add it to the python interact window. The reason is the default path for python interact is under "program files". The following command is to change the current working folder to a new folder:</div><div><br /></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div><div>import os</div></div><div><div>os.chdir(r"c:\mycode\xyz\")</div></div></blockquote><p>Make sure the string is started with "r". </p><h3 style="text-align: left;">Reload a library</h3><div style="text-align: left;">After change a library, the new content is not loaded to the python interactive session by default. The following command is to reload the library into the session.</div><div style="text-align: left;"><br /></div><div style="text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div style="text-align: left;">import myModule <span style="color: #38761d;">#import my module</span></div></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div style="text-align: left;"> </div></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div style="text-align: left;"><span style="color: #6aa84f;">#the following is to reload myModule</span></div></blockquote></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div style="text-align: left;"><p style="text-align: left;">import importlib</p></div></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div style="text-align: left;"><p style="text-align: left;">importlib.reload(myModule )</p></div></blockquote><p>Please note: myModule does not need quote or double quote </p><div style="text-align: left;"><div style="text-align: left;"> </div></div>
<div>
<script>
!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");
</script></div>Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-91822101362751429152020-08-07T14:54:00.001-04:002020-08-07T14:54:07.397-04:00Matrix Multiplication & Graph Reachable<div>I have been struggling to find a good way to represent graph structure. There are number of graph library where Node and Edge are defined. I found that matrix can be used to represent a graph. </div><div><br /></div><div>The row and column of the matrix represent the node in a graph. The element in the matrix is connection between two nodes. In a graph, this connection is called Edge. The following diagram represents a graph with 3 nodes and 3 edges. </div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmIO0Mp1aBwtTQe2TFCGnTzDEzVLCcOXZ6YAhWY-yDVhH0h1xIyh-n4L23ECtB5FC9MD1aNajZxFEkGwyPTnG3TGUgmVa9hUAdzEsyhK2MAnl0YZjqBDX-NsGCJ4xaUqH4ZE1Mfjt1LlQ/s396/Graph.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="300" data-original-width="396" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmIO0Mp1aBwtTQe2TFCGnTzDEzVLCcOXZ6YAhWY-yDVhH0h1xIyh-n4L23ECtB5FC9MD1aNajZxFEkGwyPTnG3TGUgmVa9hUAdzEsyhK2MAnl0YZjqBDX-NsGCJ4xaUqH4ZE1Mfjt1LlQ/s0/Graph.png" /></a></div> </div><div>In a matrix representation, it can be written as a matrix. Row and column are named A, B, and C. If there is an edge between two nodes, the corresponding cell is set to 1. For example, edge AB shows 1 in cell (A, B) in the grid. We can denote this matrix as M</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiArns20V0flVrkXpfCWCXa9h6QMEGgYZysytrbaRlLW-DEVgNCcAqhGl5IFjOfdsuADZSQhjuBDiKTpV_L5w4bSWP7YA_IMMRjADFSGuaCYTBpAaru7YXGAyEW8fikmZxuolgz1kq4IaY/s261/Matrix.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="174" data-original-width="261" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiArns20V0flVrkXpfCWCXa9h6QMEGgYZysytrbaRlLW-DEVgNCcAqhGl5IFjOfdsuADZSQhjuBDiKTpV_L5w4bSWP7YA_IMMRjADFSGuaCYTBpAaru7YXGAyEW8fikmZxuolgz1kq4IaY/s0/Matrix.png" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">Matrix "multiplication" to itself can reveal if a certain node can reach the other nodes in two steps. The "multiplication" operation is different from the linear algebra matrix multiplication. </div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><blockquote>K = M ⊗ M</blockquote><p> Each element b in N can be calculated by the following formula, where N is the number of the nodes (or the column number of the matrix M).</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMgmWPhTb_gvwdr4VJDixaFvRn_eK2XhCniTxY8FFYmhQ4b0zAbx6vyiKOxARCaatGqg8yGBpzcGZrg17QZIQ6DpWi1U7XsF1sK8YBwUQYnnirsr3UTVjRUP-R1jRcEXnmHBJw7UIgU8Q/s257/MatrixOp.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="145" data-original-width="257" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMgmWPhTb_gvwdr4VJDixaFvRn_eK2XhCniTxY8FFYmhQ4b0zAbx6vyiKOxARCaatGqg8yGBpzcGZrg17QZIQ6DpWi1U7XsF1sK8YBwUQYnnirsr3UTVjRUP-R1jRcEXnmHBJw7UIgU8Q/s0/MatrixOp.png" /></a></div><div class="separator" style="clear: both; text-align: left;">Two nodes i and j. If there j is reachable from i, either of the following two conditions are met:</div><div class="separator" style="clear: both; text-align: left;"><ul style="text-align: left;"><li>there is a direct edge between i and j </li><li>there is a node p which is reachable from i and p can reach j</li></ul><div>Written as math language in the matrix M</div><div><ul style="text-align: left;"><li>M(i, j) = 1</li><li>彐p, M(i, p) = 1 and M(p, j) = 1</li></ul><div>Repeat the "multiplication" operation <i>logN</i> times, it can calculate any node in a graph can be reachable from other nodes. </div></div><div><br /></div><div>Finally, I can move away from a complicated node/edge definition into a formal math world. Hopefully the existing linear algebra can help me down the road. </div></div><p><br /></p></div><div><br /></div><div><br /></div><div><br /></div><div><br /></div>
<div>
<script>
!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");
</script></div>Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-16330025267019517982020-02-19T18:11:00.001-05:002020-02-19T18:11:46.268-05:00SelfNote: VSCode setup Recently I started to use the Azure cloud machine to do my coding work. I can still do my coding in my laptop using full Visual Studio but an Azure VM with VSCode will be much convenient. Low tech but works. :)<div>
<br /></div>
<div>
The following VSCode setup script helps me set up a new machine:</div>
<div>
<br /></div>
<div>
<ol>
<li>open PowerShell </li>
<li>copy extension names to the clipboard (yes, Ctrl+C)</li>
<li>run <b>get-clipboard | % { code --install-extension $_ }</b></li>
</ol>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<div>
DavidAnson.vscode-markdownlint</div>
<div>
eamodio.gitlens</div>
<div>
ms-azuretools.vscode-docker</div>
<div>
ms-python.python</div>
<div>
ms-vscode-remote.remote-wsl</div>
<div>
ms-vscode.cpptools</div>
<div>
ms-vscode.powershell</div>
<div>
npclaudiu.vscode-gn</div>
<div>
yzhang.markdown-all-in-one</div>
</div>
</div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com1tag:blogger.com,1999:blog-4100091590694628472.post-14564812367926951922020-02-03T22:56:00.000-05:002020-02-03T22:56:01.103-05:00Docker Image build clean up using PowerShell on local PCWhen I run the docker image to do the debug on PC, I was struggling to keep the hard disk space consumption low and running the recent binary in "Docker on Windows Desktop". This development stage, so I won't consider the Azure cloud for hosting.<br />
<div>
<br /></div>
<div>
I need a script to reclaim hard disk, remove the currently running container, build and run the new image in docker on windows desktop. </div>
<div>
<br /></div>
<div>
The following code first removes unused images and then gets the image running on the port 5555. I will have to play the string replace tricks using PowerShell. Please check "-replace" part for details. </div>
<div>
<br /></div>
<div>
<div>
#prune unused image</div>
<div>
docker image prune -f</div>
<div>
<br /></div>
<div>
#remove existing running docker image</div>
<div>
$a = docker container ls --filter expose=5555</div>
<div>
$a2 = $a | foreach-object {$_ -replace '\s{3,}', ' '} | foreach-object {$_ -replace ' ', "`t"}</div>
<div>
$obj = ConvertFrom-Csv -InputObject $a2 -Delimiter "`t"</div>
<div>
if ($obj -ne $null)</div>
<div>
{</div>
<div>
docker rm -f $obj.names</div>
<div>
}</div>
<div>
else</div>
<div>
{</div>
<div>
Write-Verbose "cannot find running container image so won't remove"</div>
<div>
}</div>
<div>
<br /></div>
<div>
#build flask app and set its version to 1</div>
<div>
docker build -t flaskapp:1 .</div>
<div>
docker run -d -p 5555:5555 flaskapp:1</div>
<div>
<br /></div>
<div>
#test running instance</div>
<div>
sleep -s 4</div>
<div>
invoke-restmethod "http://localhost:5555/api/health"</div>
</div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-33358527862394525172020-01-19T10:26:00.002-05:002020-01-26T12:09:27.749-05:00SelfNote: Setup Spacy Module in Visual Studio<div>
I guess not so many people use Visual Studio doing NLP with Python. I have been searching for a solution on how to add a Spacy module in Visual Studio. I kept getting cannot find en_core_web_sm module error and the only solution I could find is to run python -m spacy download en_core_web_sm. I was struggling to find where in the UI to run the Python command. And finally, the solution came.<br />
<br />
you noticed the "Admin" icon? yes, that is the key to solve the problem. With admin permission, you can open PowerShell and run <b>python -m spacy download en_core_web_sm</b>.<br />
<br />
I installed en_core_web_sm, but it shows a warning message. And the <b>en_core_web_lg</b> module can eliminate the warning.<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDx08E9k9AqxJwB2m30Ihbn68NRIizXzJaBgwcmIXd-v_weCGKE3QCGyxMJhIWzoWhGu8SV91BQx1_CIygYqV37NmaDGbgq9uIdkznC38Mh-fLNNIV-wk2HHfDoN2KICxPdTtOvrYQ4P0/s1600/PythonVSModule.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="463" data-original-width="690" height="214" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDx08E9k9AqxJwB2m30Ihbn68NRIizXzJaBgwcmIXd-v_weCGKE3QCGyxMJhIWzoWhGu8SV91BQx1_CIygYqV37NmaDGbgq9uIdkznC38Mh-fLNNIV-wk2HHfDoN2KICxPdTtOvrYQ4P0/s320/PythonVSModule.PNG" width="320" /></a></div>
<br /></div>
<div>
<script>
!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");
</script></div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-16053912980233650832020-01-06T14:53:00.001-05:002020-01-06T18:52:53.963-05:00Standford NLP Quick Setup on Win10 with WSLAfter setting up the <a href="https://apollo13cn.blogspot.com/2018/11/move-nlp-ikvm-to-f.html">Stanford NLP</a> on my PC, I was struggling with how to run it faster. Then I ran into Windows Subsystem for Linux (WSL). I realize that the modification of Stanford NLP is not an option for me. I can use WSL to quickly start an NLP web service and start my work.<br />
<div>
<br /></div>
<div>
I ordered a more powerful VM from Azure and use PowerShell to set up the NLP environment. </div>
<div>
<br /></div>
<div>
<ol>
<li>Enable WSL<br /><br />Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux<br /><br />It will trigger a reboot if WSL is not enabled</li>
<li>Add Ubuntu disco to WSL<br /><br /># download ubuntu 18.04 as save it as Ubuntu.appx at local directory<br />Invoke-WebRequest -Uri https://aka.ms/wsl-ubuntu-1804 -OutFile Ubuntu.appx -UseBasicParsing<br /><br /># add Ubuntu.appx to WSL<br />Add-package Ubuntu.appx</li>
<li>Download Standard NLP zip file and unzip it to the current folder<br /><br /># download the Stanford NLP and save the zip file locally as "corenlp.zip"<br />Invoke-WebRequest -uri http://nlp.stanford.edu/software/stanford-corenlp-full-2018-10-05.zip -outfile corenlp.zip -UseBasicParsing<br /><br /># unzip the corenlp.zip<br />Expand-Archive corenlp.zip -DestinationPath .\CoreNlp\</li>
<li>install Java in WSL. Since Stanford NLP does not require Oracle Java, so I use Open Java to make the command shorter<br /><br />wsl sudo apt-get update<br />wsl sudo apt-get install default-jdk</li>
</ol>
<div>
I go into WSL from PowerShell and launch Stanford NLP from WSL. </div>
</div>
<div>
<br /></div>
<div>
<ul>
<li>go to WSL from Powershell by using "wsl"</li>
<li>go to the folder which stores the unzipped Stanford NLP files in step 3</li>
<li>run java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer</li>
</ul>
<div>
Open Edge and go to http://localhost:9000/, it will show similar UI like http://corenlp.run.<br />
<br />
The PowerShell script used to access the localhost 9000 ports are listed below:<br />
<br />
$data = "The quick brown fox jumped over the lazy dog."<br />
$url2 = 'http://localhost:9000/?properties={"annotators":"tokenize,ssplit,pos,lemma,ner, entitymentions,depparse,parse,relation,openie,dcoref,kbp","outputFormat":"json"}'<br />
$r = Invoke-RestMethod -Uri $url2 -Method post -Body $data<br />
<br />
The annotators are listed <a href="https://stanfordnlp.github.io/CoreNLP/annotators.html">here </a>in case you need it.</div>
</div>
<div>
<br /></div>
<div>
<br /></div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-64484687463206903382019-09-02T15:48:00.000-04:002019-09-02T15:48:14.567-04:006-Year-Old Rule?The story begins recently I am indulging in the quantum computing programming whose foundation involves a lot of linear algebra. The direct consequence is I want to solve almost all problems using the complex number, vector, and matrix. The Mars Rover problem got my attention.<br />
<br />
<blockquote class="tr_bq">
The rover on the Mars surface only takes L, R, and M instruction. L means turn left. R means turn right. M means move 1 step forward. If the initial position is (0, 0) and facing North, what will be the location and facing direction after receiving couples of instructions? </blockquote>
<div>
<br /></div>
<div>
This is a perfect opportunity to practice my math, I thought. The direction and location are all complex numbers. I can manipulate numbers to make the code concise! I convert the L, R, and M instruction to complex number <i>op</i>:<br />
<br />
<br />
<ul>
<li>turn left L is complex (0, 1) where the real part is 0 and the imaginary part is 1</li>
<li>turn right R is complex (0, -1)</li>
<li>move forward M is complex number (1, 0)</li>
</ul>
<div>
The new location and direction can be computed like the following:</div>
<div>
<br /></div>
<div>
<ul>
<li>new location += op.Real * Direction //L and R's real part is 0, so it does not move location</li>
<li>new direction *= op //M is 1, so it has no effect on the direction</li>
</ul>
<div>
It looks cool and code is concise. I avoided those unpleasant <i>switch </i>statements. </div>
</div>
<div>
<br /></div>
<div>
Everything looks satisfying until I realize those expensive <i>multiply </i>operations might be a performance problem. I wrote a simple solution using those <i>switch </i>statements. The simple solution doubled the code. However, the performance is 30+% faster than my "elegant" one! </div>
<div>
<br /></div>
<div>
I told my wife what was going on when I encountered her in the kitchen, with a little bit of frustration. She smiled and pointed my 6-year-old son. "If the solution can be understand by him, it must be faster." she said, "It requires a smaller brain so it runs faster."</div>
<div>
<br /></div>
<div>
Well, both algorithms are <i>O(n)</i>. Maybe next time, I should think on the big O level. </div>
</div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-87818030839305267972019-03-03T11:27:00.000-05:002019-03-03T11:27:00.339-05:00Old stuff: my first IEEE paper on Computational IntelligenceBefore I finished my college, I managed to publish a paper on evolutionary computation (EC). The paper is on the encoding part. My next task is to use quantum to rewrite my algorithm. Well, there is a quantum genetic algorithm (QGA) there. However, I think with the latest decision on <a href="https://apollo13cn.blogspot.com/2018/12/microsoft-q-is-my-choice.html">Q#</a> and <a href="https://apollo13cn.blogspot.com/2018/12/selfnotes-quantum-algorithms.html">quantum algorithms</a>. The paper link is <a href="https://drive.google.com/file/d/1p3kRkEsVU2T_L_FVvxJ6y6J4O1Xa1kjI/view?usp=sharing">here</a>.Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-26271126122421322372019-01-19T16:19:00.000-05:002019-01-19T16:19:14.848-05:00Microsoft Q# Day 1: Qubit CollapsedAfter going through the <a href="https://apollo13cn.blogspot.com/2018/12/selfnotes-quantum-algorithms.html">quantum linear algebra</a>, I am going to see some quantum concept by using Q#.<br />
<div>
<br /></div>
<div>
The first one I want to prove is measurement will collapse the qubit. I will use the AssertProb to verify the value with probability. The pseudo code is listed below:</div>
<div>
<br /></div>
<div>
<ol>
<li>use H gate to set the qubit to superposition</li>
<li>check the probability to 1 equals 0.5 (50%)</li>
<li>measure the qubit</li>
<li>check the qubit equals the measured value equals to 1 (100%). The qubit collapsed. </li>
</ol>
<div>
the code listed below:</div>
<div>
<br /></div>
<div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #008800; font-weight: bold;">namespace</span> <span style="color: #0e84b5; font-weight: bold;">Quantum</span>.<span style="color: #bb0066; font-weight: bold;">QSharpApplication2</span>
<span style="color: #333333;">{</span>
<span style="color: #008800; font-weight: bold;">open</span> <span style="color: #0e84b5; font-weight: bold;">Microsoft</span>.<span style="color: #0e84b5; font-weight: bold;">Quantum</span>.<span style="color: #bb0066; font-weight: bold;">Primitive</span><span style="color: #333333;">;</span>
<span style="color: #008800; font-weight: bold;">open</span> <span style="color: #0e84b5; font-weight: bold;">Microsoft</span>.<span style="color: #0e84b5; font-weight: bold;">Quantum</span>.<span style="color: #bb0066; font-weight: bold;">Canon</span><span style="color: #333333;">;</span>
<span style="color: #008800; font-weight: bold;">open</span> <span style="color: #0e84b5; font-weight: bold;">Microsoft</span>.<span style="color: #0e84b5; font-weight: bold;">Quantum</span>.<span style="color: #0e84b5; font-weight: bold;">Extensions</span>.<span style="color: #bb0066; font-weight: bold;">Convert</span><span style="color: #333333;">;</span>
operation <span style="color: #bb0066; font-weight: bold;">Operation1</span> <span style="color: #007020;">()</span> <span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">Unit</span>
<span style="color: #333333;">{</span>
body<span style="color: #333333;">(...)</span>
<span style="color: #333333;">{</span>
using <span style="color: #333333;">(</span>ancilla <span style="color: #333333;">=</span> <span style="color: #bb0066; font-weight: bold;">Qubit</span><span style="color: #007020;">()</span><span style="color: #333333;">)</span>
<span style="color: #333333;">{</span>
<span style="color: #bb0066; font-weight: bold;">H</span><span style="color: #333333;">(</span>ancilla<span style="color: #333333;">);</span>
<span style="color: #bb0066; font-weight: bold;">AssertProb</span><span style="color: #333333;">([</span><span style="color: #bb0066; font-weight: bold;">PauliZ</span><span style="color: #333333;">],</span> <span style="color: #333333;">[</span>ancilla<span style="color: #333333;">],</span> <span style="color: #bb0066; font-weight: bold;">Zero</span><span style="color: #333333;">,</span> <span style="color: #0000dd; font-weight: bold;">0</span><span style="color: #333333;">.</span><span style="color: #0000dd; font-weight: bold;">5</span><span style="color: #333333;">,</span> <span style="background-color: #fff0f0;">"error"</span><span style="color: #333333;">,</span> <span style="color: #0000dd; font-weight: bold;">1</span>e<span style="color: #333333;">-</span><span style="color: #0000dd; font-weight: bold;">5</span><span style="color: #333333;">);</span>
<span style="color: #bb0066; font-weight: bold;">AssertProb</span><span style="color: #333333;">([</span><span style="color: #bb0066; font-weight: bold;">PauliZ</span><span style="color: #333333;">],</span> <span style="color: #333333;">[</span>ancilla<span style="color: #333333;">],</span> <span style="color: #bb0066; font-weight: bold;">One</span><span style="color: #333333;">,</span> <span style="color: #0000dd; font-weight: bold;">0</span><span style="color: #333333;">.</span><span style="color: #0000dd; font-weight: bold;">5</span><span style="color: #333333;">,</span> <span style="background-color: #fff0f0;">"error"</span><span style="color: #333333;">,</span> <span style="color: #0000dd; font-weight: bold;">1</span>e<span style="color: #333333;">-</span><span style="color: #0000dd; font-weight: bold;">5</span><span style="color: #333333;">);</span>
<span style="color: #008800; font-weight: bold;">mutable</span> gate <span style="color: #333333;">=</span> <span style="color: #bb0066; font-weight: bold;">PauliZ</span><span style="color: #333333;">;</span>
<span style="color: #008800; font-weight: bold;">let</span> mX <span style="color: #333333;">=</span> <span style="color: #bb0066; font-weight: bold;">Measure</span><span style="color: #333333;">([</span>gate<span style="color: #333333;">],</span> <span style="color: #333333;">[</span>ancilla<span style="color: #333333;">]);</span>
<span style="color: #bb0066; font-weight: bold;">AssertProb</span><span style="color: #333333;">([</span>gate<span style="color: #333333;">],</span> <span style="color: #333333;">[</span>ancilla<span style="color: #333333;">],</span> mX<span style="color: #333333;">,</span> <span style="color: #0000dd; font-weight: bold;">1</span><span style="color: #333333;">.</span><span style="color: #0000dd; font-weight: bold;">0</span><span style="color: #333333;">,</span> <span style="background-color: #fff0f0;">"error"</span><span style="color: #333333;">,</span> <span style="color: #0000dd; font-weight: bold;">1</span>e<span style="color: #333333;">-</span><span style="color: #0000dd; font-weight: bold;">5</span><span style="color: #333333;">);</span>
<span style="color: #bb0066; font-weight: bold;">Message</span><span style="color: #333333;">(</span><span style="color: #bb0066; font-weight: bold;">ToStringD</span><span style="color: #333333;">(</span>mX <span style="color: #333333;">==</span> <span style="color: #bb0066; font-weight: bold;">Zero</span> <span style="color: #333333;">?</span> <span style="color: #0000dd; font-weight: bold;">0</span><span style="color: #333333;">.</span> <span style="color: #333333;">|</span> <span style="color: #0000dd; font-weight: bold;">1</span><span style="color: #333333;">.));</span>
<span style="color: #bb0066; font-weight: bold;">Reset</span><span style="color: #333333;">(</span>ancilla<span style="color: #333333;">);</span>
<span style="color: #333333;">}</span>
<span style="color: #333333;">}</span>
<span style="color: #333333;">}</span>
<span style="color: #333333;">}</span>
</pre>
</td></tr>
</tbody></table>
</div>
<br /></div>
</div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-36353075201148907482018-12-26T21:59:00.000-05:002018-12-26T21:59:11.369-05:00Microsoft Q# is my choiceToday I am thinking to dig into the real quantum programming after learning the <a href="https://apollo13cn.blogspot.com/2018/12/selfnotes-quantum-algorithms.html">foundation </a>part. There are three big players I was looking at on quantum programming framework/language. I hate to waste my time learning the framework or language itself. I want to focus on so I can focus on solving the problem. I have three choices:<div>
<ul>
<li>IBM, IBM Q</li>
<li>Google Cirq</li>
<li>Microsoft Q#</li>
</ul>
<div>
It seems IBM Q and Cirq are reusing the Python to provide a framework while Q# is a brand new language. It makes sense IBM and Google use Python as base as it has a good math library. However, I doubt this choice is the best choice. Like the GPU, quantum is special hardware. Cuda is a like C language but it has its own features. For quantum computing, it has a totally different way to compute. Therefore, I think the best way to use a new language, maybe it is just a DSL. C++ can be used to write a functional program, but the best way is still to use a functional programming language. </div>
<div>
<br /></div>
<div>
Also, the examples from those three companies show a huge difference. Q# provides the best sample pack so far. It has a quantum foundation, matrix computation, and programming language. It will help me a lot to adopt this new stuff. </div>
<div>
<br /></div>
<div>
The setup and debugging is the last thought. I had tried to step away from Microsoft stack; however, the debugging and the setup needs more time I expected. I was spoiled by double click and just working. </div>
<div>
<br /></div>
<div>
Anyway, the current Q# shows good progress and the watch, star, and pull request number is bigger than other players. I will continue to focus on the quantum basics, like matrix and math part, anyway. </div>
<div>
<br /></div>
</div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-69654590952210973162018-12-23T21:27:00.000-05:002019-01-02T20:40:47.055-05:00SelfNotes: Quantum AlgorithmsI believe the quantum computing is the key to enable real AI. No matter how complicated the AI algorithm can be, the fundamental computing power will make a real difference.<br />
<div>
<br /></div>
<div>
The huge difference between a classical computer and a quantum computer is so intriguing! It is the first time that math can come back to computer science and guide its evolution. Also, the quantum algorithm is the first time it introduces the internal parallelism. Those existing algorithms barely considered parallelism when it is designed. Now the quantum can change this situation!<br />
<div>
<br /></div>
<div>
The following are some notes about quantum algorithms. They are not related to any language implementation, it is just pure math. The only tool I want to use is "that piece of meat between my ears". :)</div>
</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZqUF4Lh7VMyg8chHuEnMmg9VUl9pam2Sl58zVnlMenDhqB0Y0gUH_hyphenhyphen1EHs1Y4k5sdD2OeF15rCU_AG92wfUX-H6UhLXa-cE_I402ZVXe0F6usOs5L62Fmlk7_ogeAkQw4pKvoOgHb80/s1600/IMG_6394.HEIC" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZqUF4Lh7VMyg8chHuEnMmg9VUl9pam2Sl58zVnlMenDhqB0Y0gUH_hyphenhyphen1EHs1Y4k5sdD2OeF15rCU_AG92wfUX-H6UhLXa-cE_I402ZVXe0F6usOs5L62Fmlk7_ogeAkQw4pKvoOgHb80/s320/IMG_6394.HEIC" width="240" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjwCwqX9f8FpErgKBN8SEUe3R-LRGIlhpfLoXdh0_RlPyAWLUPqsKUPWKhh1ifzk0VvXX__rw7LVK-s77m-U7qrLtlHRHYej5ODdjwc1S4y2yyXQezbT-6fsTBbhGqrXaH1fQcTRabesM/s1600/IMG_6364.HEIC" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjwCwqX9f8FpErgKBN8SEUe3R-LRGIlhpfLoXdh0_RlPyAWLUPqsKUPWKhh1ifzk0VvXX__rw7LVK-s77m-U7qrLtlHRHYej5ODdjwc1S4y2yyXQezbT-6fsTBbhGqrXaH1fQcTRabesM/s320/IMG_6364.HEIC" width="240" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh72Y6WwLsOada-qcqTu1VcrCJh7KkJ1i4LF5oMI0ykbSPhhDhwntddKuTRz9mSigKJNsp6ddJ5Wn9-7PdJriIIHc-VXTMiqeU_N8zPO2oZqNaSKPMZf8V37bjAXPYettviyF-akckAe60/s1600/IMG_6344.HEIC" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh72Y6WwLsOada-qcqTu1VcrCJh7KkJ1i4LF5oMI0ykbSPhhDhwntddKuTRz9mSigKJNsp6ddJ5Wn9-7PdJriIIHc-VXTMiqeU_N8zPO2oZqNaSKPMZf8V37bjAXPYettviyF-akckAe60/s320/IMG_6344.HEIC" width="240" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9hzoatEQ6F8sOEqtYCz_nVa-bHRQcCd0HKa0exMnCpVOT3JOYy0DV4QaXdWCfkOSUPMSNhZQMTwD90WG0ezC6dstlaIOJ_MLsXlTMN-Roh7QtY2w6VdftZawDCbRVPnAoRE2PDzENHJQ/s1600/IMG_6182.HEIC" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9hzoatEQ6F8sOEqtYCz_nVa-bHRQcCd0HKa0exMnCpVOT3JOYy0DV4QaXdWCfkOSUPMSNhZQMTwD90WG0ezC6dstlaIOJ_MLsXlTMN-Roh7QtY2w6VdftZawDCbRVPnAoRE2PDzENHJQ/s320/IMG_6182.HEIC" width="240" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinbo2tJVPYu6QmX5B_Lz8w4gVmeyVxP2mz-SXW_Xzo4ePTAGaiLSoRoG8QVvvxuVCgPql7zElNhGhdK4Q8soc2YrxiFYDS_zhewI5yajOATJ5QIJhecxAr2XLaiueP_hXrKPz_7X1ZxmA/s1600/IMG_6411.HEIC" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1600" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinbo2tJVPYu6QmX5B_Lz8w4gVmeyVxP2mz-SXW_Xzo4ePTAGaiLSoRoG8QVvvxuVCgPql7zElNhGhdK4Q8soc2YrxiFYDS_zhewI5yajOATJ5QIJhecxAr2XLaiueP_hXrKPz_7X1ZxmA/s320/IMG_6411.HEIC" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgH9DfUlHS-LaSCfxiDSnVEQe_YamZnu7RYYEkFvo8lB8z_-GWU4pGFnIFG_9jOc3s9rtNuYAiNnIQNrjraGnQ1MIX_WXyKVxmgSFwN8_ocpy1rMX-9Fil91x5kYgs49Qq_dDmOZYxS2Fk/s1600/IMG_6410.HEIC" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgH9DfUlHS-LaSCfxiDSnVEQe_YamZnu7RYYEkFvo8lB8z_-GWU4pGFnIFG_9jOc3s9rtNuYAiNnIQNrjraGnQ1MIX_WXyKVxmgSFwN8_ocpy1rMX-9Fil91x5kYgs49Qq_dDmOZYxS2Fk/s320/IMG_6410.HEIC" width="240" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<br /></div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-44319196054480573842018-12-08T13:58:00.000-05:002018-12-08T14:45:54.393-05:00Automation with NodeJS-Puppeteer & Visual Studio CodeI saw many attempts to automate the processing using commercial software. However, I do not see this is necessary if this task is not that huge. When more and more application is moving to the web-based. Automation of the Chrome browser will automate most of the scenarios. Therefore, I feel 90% of those cases won't need to pay and still get the job done.<br />
<div>
<br /></div>
<div>
First, we need to set up IDE. There are three files needs to be updated and placed in the place. Please make sure you put launch.json and tasks.json are under .vscode folder. </div>
<div>
<br /></div>
<div>
<ul>
<li>launch.json: it is used to launch the application with predefined tasks such as Typescript build. </li>
<li>tsconfig.json is to used to build the application using Puppeteer</li>
<li>tasks.json is to define the Typescript build process</li>
</ul>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijiwxBwrwdqMlI114BraX9c0h4t1CsCEWo1MM8LkE8uELI97SSm5pAGFhhPXkVW9htg27AWO3uVP7BlHjeCt99VtDWxx-JiRClMXtNTS8yPWr_YI1E16pPp5NQSSJMlkVEwItEjwBTWoE/s1600/VSCodeIDE.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="281" data-original-width="183" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijiwxBwrwdqMlI114BraX9c0h4t1CsCEWo1MM8LkE8uELI97SSm5pAGFhhPXkVW9htg27AWO3uVP7BlHjeCt99VtDWxx-JiRClMXtNTS8yPWr_YI1E16pPp5NQSSJMlkVEwItEjwBTWoE/s1600/VSCodeIDE.PNG" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The content of these three files</div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<ul>
<li>launch.json</li>
</ul>
<br />
<div class="separator" style="clear: both; text-align: left;">
<!-- HTML generated using hilite.me --></div>
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #333333;">{</span>
<span style="color: #888888;">// Use IntelliSense to learn about possible attributes.</span>
<span style="color: #888888;">// Hover to view descriptions of existing attributes.</span>
<span style="color: #888888;">// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387</span>
<span style="background-color: #fff0f0;">"version"</span><span style="color: #333333;">:</span> <span style="background-color: #fff0f0;">"0.2.0"</span><span style="color: #333333;">,</span>
<span style="background-color: #fff0f0;">"configurations"</span><span style="color: #333333;">:</span> <span style="color: #333333;">[</span>
<span style="color: #333333;">{</span>
<span style="background-color: #fff0f0;">"type"</span><span style="color: #333333;">:</span> <span style="background-color: #fff0f0;">"node"</span><span style="color: #333333;">,</span>
<span style="background-color: #fff0f0;">"request"</span><span style="color: #333333;">:</span> <span style="background-color: #fff0f0;">"launch"</span><span style="color: #333333;">,</span>
<span style="background-color: #fff0f0;">"name"</span><span style="color: #333333;">:</span> <span style="background-color: #fff0f0;">"Launch Program"</span><span style="color: #333333;">,</span>
<span style="background-color: #fff0f0;">"sourceMaps"</span><span style="color: #333333;">:</span> <span style="color: #007020;">true</span><span style="color: #333333;">,</span>
<span style="background-color: #fff0f0;">"preLaunchTask"</span><span style="color: #333333;">:</span> <span style="background-color: #fff0f0;">"TSC"</span><span style="color: #333333;">,</span>
<span style="background-color: #fff0f0;">"program"</span><span style="color: #333333;">:</span> <span style="background-color: #fff0f0;">"${workspaceFolder}/app.ts"</span><span style="color: #333333;">,</span>
<span style="background-color: #fff0f0;">"outFiles"</span><span style="color: #333333;">:</span> <span style="color: #333333;">[</span>
<span style="background-color: #fff0f0;">"${workspaceFolder}/**/*.js"</span>
<span style="color: #333333;">]</span>
<span style="color: #333333;">}</span>
<span style="color: #333333;">]</span>
<span style="color: #333333;">}</span><span style="color: #008800; font-weight: bold;">'</span>
</pre>
</td></tr>
</tbody></table>
</div>
<br />
<br />
<ul>
<li>tasks.json</li>
</ul>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #333333;">{</span>
<span style="color: #888888;">// See https://go.microsoft.com/fwlink/?LinkId=733558</span>
<span style="color: #888888;">// for the documentation about the tasks.json format</span>
<span style="background-color: #fff0f0;">"version"</span><span style="color: #333333;">:</span> <span style="background-color: #fff0f0;">"2.0.0"</span><span style="color: #333333;">,</span>
<span style="background-color: #fff0f0;">"tasks"</span><span style="color: #333333;">:</span> <span style="color: #333333;">[</span>
<span style="color: #333333;">{</span>
<span style="background-color: #fff0f0;">"label"</span><span style="color: #333333;">:</span> <span style="background-color: #fff0f0;">"TSC"</span><span style="color: #333333;">,</span>
<span style="background-color: #fff0f0;">"type"</span><span style="color: #333333;">:</span> <span style="background-color: #fff0f0;">"typescript"</span><span style="color: #333333;">,</span>
<span style="background-color: #fff0f0;">"tsconfig"</span><span style="color: #333333;">:</span> <span style="background-color: #fff0f0;">"tsconfig.json"</span><span style="color: #333333;">,</span>
<span style="background-color: #fff0f0;">"problemMatcher"</span><span style="color: #333333;">:</span> <span style="color: #333333;">[</span>
<span style="background-color: #fff0f0;">"$tsc"</span>
<span style="color: #333333;">],</span>
<span style="background-color: #fff0f0;">"group"</span><span style="color: #333333;">:</span> <span style="color: #333333;">{</span>
<span style="background-color: #fff0f0;">"kind"</span><span style="color: #333333;">:</span> <span style="background-color: #fff0f0;">"build"</span><span style="color: #333333;">,</span>
<span style="background-color: #fff0f0;">"isDefault"</span><span style="color: #333333;">:</span> <span style="color: #007020;">true</span>
<span style="color: #333333;">}</span>
<span style="color: #333333;">}</span>
<span style="color: #333333;">]</span>
<span style="color: #333333;">}</span>
</pre>
</td></tr>
</tbody></table>
</div>
<br />
<br />
<ul>
<li>tsconfig.json</li>
</ul>
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1
2
3
4
5
6
7
8
9
10
11
12</pre>
</td><td><pre style="line-height: 125%; margin: 0;">{
<span style="background-color: #fff0f0;">"compilerOptions"</span><span style="color: #333333;">:</span> {
<span style="background-color: #fff0f0;">"target"</span><span style="color: #333333;">:</span> <span style="background-color: #fff0f0;">"es5"</span>,
<span style="background-color: #fff0f0;">"module"</span><span style="color: #333333;">:</span> <span style="background-color: #fff0f0;">"commonjs"</span>,
<span style="background-color: #fff0f0;">"lib"</span><span style="color: #333333;">:</span> [<span style="background-color: #fff0f0;">"es2015"</span>, <span style="background-color: #fff0f0;">"dom"</span>],
<span style="background-color: #fff0f0;">"sourceMap"</span><span style="color: #333333;">:</span> <span style="color: #008800; font-weight: bold;">true</span>
},
<span style="background-color: #fff0f0;">"include"</span><span style="color: #333333;">:</span> [
<span style="background-color: #fff0f0;">"src"</span>,
<span style="background-color: #fff0f0;">"node_modules/@types/puppeteer/index.d.ts"</span>
]
}
</pre>
</td></tr>
</tbody></table>
</div>
<br />
<br />
The <b>node_modules</b> folder is created from the npm command provided by the NodeJS framework. The command to run is listed the following. The first line is to install TypeScript compiler which is not installed by default with VS Code. The second line is to install Puppeteer. The last line is to install the library for Typescript to work with Puppeteer.<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1
2
3</pre>
</td><td><pre style="line-height: 125%; margin: 0;">npm i -g typescript
npm i --save puppeteer
npm i --save-dev <span style="color: #996633;">@types</span>/puppeteer
</pre>
</td></tr>
</tbody></table>
</div>
<br />
The last file is the Puppeteer file:<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1
2
3
4
5
6
7
8
9
10</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #008800; font-weight: bold;">import</span> <span style="color: #333333;">*</span> <span style="color: #008800; font-weight: bold;">as</span> puppeteer from <span style="background-color: #fff0f0;">'puppeteer'</span>
(async () <span style="color: #333333;">=></span> {
<span style="color: #008800; font-weight: bold;">const</span> browser <span style="color: #333333;">=</span> await puppeteer.launch();
<span style="color: #008800; font-weight: bold;">const</span> page <span style="color: #333333;">=</span> await browser.newPage();
await page.<span style="color: #008800; font-weight: bold;">goto</span>(<span style="background-color: #fff0f0;">'http://www.google.com'</span>);
await page.screenshot({path<span style="color: #333333;">:</span> <span style="background-color: #fff0f0;">'example.png'</span>});
await browser.close();
})();
</pre>
</td></tr>
</tbody></table>
</div>
<br />
<br />
<div>
</div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-88001553064073960182018-11-11T23:39:00.000-05:002018-11-11T23:39:49.623-05:00Move NLP IKVM to F# .NET-FriendlySince the <a href="https://apollo13cn.blogspot.com/2018/10/f-stanford-nlp-is-running.html">last post</a>, I read Sergey's code. Then I decided to work on refactoring the code to store the data into the .NET and F# format. Stanford NLP does provide a <a href="https://stanfordnlp.github.io/CoreNLP/corenlp-server.html">server</a>. I still want to make it .Net friendly and also get myself familiar with the NLP core.<br />
<div>
<br /></div>
<div>
I prefer the project-based solution than the interactive solution is because it can have Visual Studio's watch, immediate windows, and debug visualizer. Once I got the information into a comfortable environment, it will be easy to move forward. </div>
<div>
<br /></div>
<div>
The 200-line code is to build up a structure like the following:</div>
<div>
<br /></div>
<div>
I call multiple sentences a story. A story contains (1) sentences and (2) cross-references. The sentence structure shows NLP info about a sentence. The sentence includes a token list, tree, and dependency graph. The cross-references maintain the relationship among elements from different sentences. The first file is the main file. It shows how to invoke the underlying functions and show the structures. </div>
<div>
<br /></div>
<div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">// Learn more about F# at http://fsharp.org</span>
<span style="color: #888888;">// See the 'F# Tutorial' project for more help.</span>
<span style="color: #008800; font-weight: bold;">open</span> <span style="color: #bb0066; font-weight: bold;">System</span>
<span style="color: #008800; font-weight: bold;">open</span> <span style="color: #0e84b5; font-weight: bold;">System</span>.<span style="color: #bb0066; font-weight: bold;">IO</span>
<span style="color: #008800; font-weight: bold;">open</span> java<span style="color: #333333;">.</span>util
<span style="color: #008800; font-weight: bold;">open</span> java<span style="color: #333333;">.</span>io
<span style="color: #008800; font-weight: bold;">open</span> edu<span style="color: #333333;">.</span>stanford<span style="color: #333333;">.</span>nlp<span style="color: #333333;">.</span>pipeline
<span style="color: #008800; font-weight: bold;">open</span> edu<span style="color: #333333;">.</span>stanford<span style="color: #333333;">.</span>nlp<span style="color: #333333;">.</span>ling
<span style="color: #008800; font-weight: bold;">open</span> <span style="color: #0e84b5; font-weight: bold;">Utils</span>.<span style="color: #bb0066; font-weight: bold;">NLPUtils</span>
<span style="color: #008800; font-weight: bold;">open</span> <span style="color: #0e84b5; font-weight: bold;">Utils</span>.<span style="color: #bb0066; font-weight: bold;">NLPExtensions</span>
<span style="color: #008800; font-weight: bold;">open</span> edu<span style="color: #333333;">.</span>stanford<span style="color: #333333;">.</span>nlp<span style="color: #333333;">.</span>util
<span style="color: #008800; font-weight: bold;">open</span> edu<span style="color: #333333;">.</span>stanford<span style="color: #333333;">.</span>nlp<span style="color: #333333;">.</span>trees
<span style="color: #008800; font-weight: bold;">open</span> edu<span style="color: #333333;">.</span>stanford<span style="color: #333333;">.</span>nlp<span style="color: #333333;">.</span>semgraph
<span style="color: #008800; font-weight: bold;">open</span> <span style="color: #0e84b5; font-weight: bold;">Utils</span>.<span style="color: #bb0066; font-weight: bold;">NLPStructures</span>
<span style="color: #008800; font-weight: bold;">open</span> edu<span style="color: #333333;">.</span>stanford<span style="color: #333333;">.</span>nlp<span style="color: #333333;">.</span>coref
<span style="color: #333333;">[<</span><span style="color: #bb0066; font-weight: bold;">EntryPoint</span><span style="color: #333333;">>]</span>
<span style="color: #008800; font-weight: bold;">let</span> main argv <span style="color: #333333;">=</span>
<span style="color: #008800; font-weight: bold;">let</span> text <span style="color: #333333;">=</span> <span style="background-color: #fff0f0;">"Kosgi Santosh sent an email to Stanford University. He didn't get a reply email."</span><span style="color: #333333;">;</span>
<span style="color: #888888;">// Annotation pipeline configuration</span>
<span style="color: #008800; font-weight: bold;">let</span> props <span style="color: #333333;">=</span> <span style="color: #bb0066; font-weight: bold;">Properties</span><span style="color: #007020;">()</span>
props<span style="color: #333333;">.</span>setProperty<span style="color: #333333;">(</span><span style="background-color: #fff0f0;">"annotators"</span><span style="color: #333333;">,</span><span style="background-color: #fff0f0;">"tokenize, ssplit, pos, lemma, ner, parse, dcoref"</span><span style="color: #333333;">)</span> <span style="color: #333333;">|></span> ignore
props<span style="color: #333333;">.</span>setProperty<span style="color: #333333;">(</span><span style="background-color: #fff0f0;">"ner.useSUTime"</span><span style="color: #333333;">,</span><span style="background-color: #fff0f0;">"0"</span><span style="color: #333333;">)</span> <span style="color: #333333;">|></span> ignore
<span style="color: #008800; font-weight: bold;">let</span> pipeline <span style="color: #333333;">=</span> <span style="color: #bb0066; font-weight: bold;">StanfordCoreNLP</span><span style="color: #333333;">(</span>props<span style="color: #333333;">)</span>
<span style="color: #888888;">// Annotation</span>
<span style="color: #008800; font-weight: bold;">let</span> annotation <span style="color: #333333;">=</span> <span style="color: #bb0066; font-weight: bold;">Annotation</span><span style="color: #333333;">(</span>text<span style="color: #333333;">)</span>
pipeline<span style="color: #333333;">.</span>annotate<span style="color: #333333;">(</span>annotation<span style="color: #333333;">)</span>
<span style="color: #888888;">//get annotation info</span>
<span style="color: #008800; font-weight: bold;">let</span> keys <span style="color: #333333;">=</span> annotation<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">GetToken</span><span style="color: #333333;"><</span><span style="color: #bb0066; font-weight: bold;">HashMap</span><span style="color: #333333;">>(</span>typeof<span style="color: #333333;"><</span><span style="color: #0e84b5; font-weight: bold;">CorefCoreAnnotations</span>.<span style="color: #bb0066; font-weight: bold;">CorefChainAnnotation</span><span style="color: #333333;">>)</span>
<span style="color: #008800; font-weight: bold;">let</span> mentions <span style="color: #333333;">=</span> keys <span style="color: #333333;">|></span> <span style="color: #0e84b5; font-weight: bold;">Seq</span>.exactlyOne <span style="color: #333333;">|></span> getMentions
<span style="color: #008800; font-weight: bold;">let</span> sentences <span style="color: #333333;">=</span>
<span style="color: #333333;">[</span>
<span style="color: #008800; font-weight: bold;">let</span> sentences <span style="color: #333333;">=</span> annotation<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">GetToken</span><span style="color: #333333;"><</span><span style="color: #bb0066; font-weight: bold;">CoreMap</span><span style="color: #333333;">>(</span>typeof<span style="color: #333333;"><</span><span style="color: #0e84b5; font-weight: bold;">CoreAnnotations</span>.<span style="color: #bb0066; font-weight: bold;">SentencesAnnotation</span><span style="color: #333333;">>)</span>
<span style="color: #008800; font-weight: bold;">for</span> s <span style="color: #008800; font-weight: bold;">in</span> sentences <span style="color: #008800; font-weight: bold;">do</span>
<span style="color: #008800; font-weight: bold;">let</span> tokens <span style="color: #333333;">=</span> s<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">GetToken</span><span style="color: #333333;"><</span><span style="color: #bb0066; font-weight: bold;">CoreLabel</span><span style="color: #333333;">>(</span>typeof<span style="color: #333333;"><</span><span style="color: #0e84b5; font-weight: bold;">CoreAnnotations</span>.<span style="color: #bb0066; font-weight: bold;">TokensAnnotation</span><span style="color: #333333;">>)</span>
<span style="color: #008800; font-weight: bold;">let</span> words <span style="color: #333333;">=</span> getWords tokens
<span style="color: #008800; font-weight: bold;">let</span> t <span style="color: #333333;">=</span> s<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">GetToken</span><span style="color: #333333;"><</span><span style="color: #bb0066; font-weight: bold;">Tree</span><span style="color: #333333;">>(</span>typeof<span style="color: #333333;"><</span><span style="color: #0e84b5; font-weight: bold;">TreeCoreAnnotations</span>.<span style="color: #bb0066; font-weight: bold;">TreeAnnotation</span><span style="color: #333333;">>)</span>
<span style="color: #008800; font-weight: bold;">let</span> tree <span style="color: #333333;">=</span> t <span style="color: #333333;">|></span> <span style="color: #0e84b5; font-weight: bold;">Seq</span>.exactlyOne <span style="color: #333333;">|></span> buildTree words
<span style="color: #008800; font-weight: bold;">let</span> deps <span style="color: #333333;">=</span> s<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">GetToken</span><span style="color: #333333;"><</span><span style="color: #bb0066; font-weight: bold;">SemanticGraph</span><span style="color: #333333;">>(</span>typeof<span style="color: #333333;"><</span><span style="color: #0e84b5; font-weight: bold;">SemanticGraphCoreAnnotations</span>.<span style="color: #bb0066; font-weight: bold;">CollapsedDependenciesAnnotation</span><span style="color: #333333;">>)</span>
<span style="color: #008800; font-weight: bold;">let</span> relationships <span style="color: #333333;">=</span> deps <span style="color: #333333;">|></span> <span style="color: #0e84b5; font-weight: bold;">Seq</span>.exactlyOne <span style="color: #333333;">|></span> getDependencyGraph words
<span style="color: #008800; font-weight: bold;">let</span> sentence <span style="color: #333333;">=</span> <span style="color: #333333;">{</span> <span style="color: #bb0066; font-weight: bold;">Words</span> <span style="color: #333333;">=</span> words<span style="color: #333333;">;</span> <span style="color: #bb0066; font-weight: bold;">Dependency</span> <span style="color: #333333;">=</span> relationships<span style="color: #333333;">;</span> <span style="color: #bb0066; font-weight: bold;">Tree</span> <span style="color: #333333;">=</span> tree<span style="color: #333333;">;</span> <span style="color: #333333;">}</span>
<span style="color: #008800; font-weight: bold;">yield</span> sentence
<span style="color: #333333;">]</span>
<span style="color: #008800; font-weight: bold;">let</span> story <span style="color: #333333;">=</span>
<span style="color: #333333;">{</span>
<span style="color: #bb0066; font-weight: bold;">CrossLinks</span> <span style="color: #333333;">=</span> mentions<span style="color: #333333;">;</span>
<span style="color: #bb0066; font-weight: bold;">Sentences</span> <span style="color: #333333;">=</span> sentences<span style="color: #333333;">;</span>
<span style="color: #333333;">}</span>
printfn <span style="background-color: #fff0f0;">"%O"</span> story
<span style="color: #0000dd; font-weight: bold;">0</span> <span style="color: #888888;">// return an integer exit code</span>
</pre>
</td></tr>
</tbody></table>
</div>
<br /></div>
<div>
<br /></div>
<div>
The second file is the library file. I do not think the structure will stay same after two weeks. I might decide to add more fields. But currently the foundation is there.<br />
<br />
<br /></div>
<div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #008800; font-weight: bold;">namespace</span> <span style="color: #bb0066; font-weight: bold;">Utils</span>
<span style="color: #008800; font-weight: bold;">module</span> <span style="color: #bb0066; font-weight: bold;">NLPStructures</span> <span style="color: #333333;">=</span>
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">Word</span> <span style="color: #333333;">=</span> <span style="color: #333399; font-weight: bold;">string</span>
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">Ner</span> <span style="color: #333333;">=</span> <span style="color: #333399; font-weight: bold;">string</span>
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">POS</span> <span style="color: #333333;">=</span> <span style="color: #333399; font-weight: bold;">string</span>
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">Index</span> <span style="color: #333333;">=</span> <span style="color: #333399; font-weight: bold;">int</span>
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">Relationship</span> <span style="color: #333333;">=</span> <span style="color: #333399; font-weight: bold;">string</span>
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">Span</span> <span style="color: #333333;">=</span> <span style="color: #333399; font-weight: bold;">int</span> <span style="color: #333333;">*</span> <span style="color: #333399; font-weight: bold;">int</span>
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">Head</span> <span style="color: #333333;">=</span> <span style="color: #333399; font-weight: bold;">int</span> <span style="color: #333333;">*</span> <span style="color: #333399; font-weight: bold;">string</span>
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">SentenceIndex</span> <span style="color: #333333;">=</span> <span style="color: #333399; font-weight: bold;">int</span>
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">WordType</span> <span style="color: #333333;">=</span>
<span style="color: #333333;">{</span>
<span style="color: #bb0066; font-weight: bold;">Word</span> <span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">Word</span>
<span style="color: #bb0066; font-weight: bold;">Ner</span><span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">Ner</span>
<span style="color: #bb0066; font-weight: bold;">POS</span><span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">POS</span>
<span style="color: #bb0066; font-weight: bold;">Index</span><span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">Index</span>
<span style="color: #333333;">}</span>
<span style="color: #008800; font-weight: bold;">member</span> this<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">IsSame</span> word <span style="color: #333333;">=</span>
<span style="color: #008800; font-weight: bold;">match</span> this <span style="color: #008800; font-weight: bold;">with</span>
<span style="color: #333333;">|</span> <span style="color: #333333;">{</span> <span style="color: #bb0066; font-weight: bold;">Word</span> <span style="color: #333333;">=</span> w<span style="color: #333333;">;</span> <span style="color: #333333;">}</span> <span style="color: #333333;">-></span> w <span style="color: #333333;">=</span> word
<span style="color: #008800; font-weight: bold;">member</span> this<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">IsSame</span> index <span style="color: #333333;">=</span>
<span style="color: #008800; font-weight: bold;">match</span> this <span style="color: #008800; font-weight: bold;">with</span>
<span style="color: #333333;">|</span> <span style="color: #333333;">{</span> <span style="color: #bb0066; font-weight: bold;">Index</span> <span style="color: #333333;">=</span> i <span style="color: #333333;">}</span> <span style="color: #333333;">-></span> i <span style="color: #333333;">=</span> index
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">MentionEntity</span> <span style="color: #333333;">=</span>
<span style="color: #333333;">{</span>
<span style="color: #bb0066; font-weight: bold;">Index</span><span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">Index</span>
<span style="color: #bb0066; font-weight: bold;">Relationship</span><span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">Relationship</span>
<span style="color: #bb0066; font-weight: bold;">Span</span> <span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">Span</span>
<span style="color: #bb0066; font-weight: bold;">Head</span> <span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">Head</span>
<span style="color: #bb0066; font-weight: bold;">SentenceIndex</span> <span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">SentenceIndex</span>
<span style="color: #333333;">}</span>
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">RepresentiveMention</span> <span style="color: #333333;">=</span> <span style="color: #bb0066; font-weight: bold;">MentionEntity</span>
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">CrossLinkType</span> <span style="color: #333333;">=</span>
<span style="color: #333333;">{</span>
<span style="color: #bb0066; font-weight: bold;">Index</span><span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">Index</span>
<span style="color: #bb0066; font-weight: bold;">RepresentiveMention</span> <span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">RepresentiveMention</span>
<span style="color: #bb0066; font-weight: bold;">Mentions</span> <span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">MentionEntity</span> <span style="color: #333399; font-weight: bold;">list</span>
<span style="color: #333333;">}</span>
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">DependencyGraph</span> <span style="color: #333333;">=</span>
<span style="color: #333333;">|</span> <span style="color: #bb0066; font-weight: bold;">Link</span> <span style="color: #008800; font-weight: bold;">of</span> <span style="color: #bb0066; font-weight: bold;">Relationship</span> <span style="color: #333333;">*</span> <span style="color: #bb0066; font-weight: bold;">WordType</span> <span style="color: #333333;">*</span> <span style="color: #bb0066; font-weight: bold;">WordType</span>
<span style="color: #333333;">|</span> <span style="color: #bb0066; font-weight: bold;">CrossLink</span> <span style="color: #008800; font-weight: bold;">of</span> <span style="color: #bb0066; font-weight: bold;">CrossLinkType</span>
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">TreeNode</span> <span style="color: #333333;">=</span>
<span style="color: #333333;">|</span> <span style="color: #bb0066; font-weight: bold;">Node</span> <span style="color: #008800; font-weight: bold;">of</span> <span style="color: #333399; font-weight: bold;">string</span> <span style="color: #333333;">*</span> <span style="color: #bb0066; font-weight: bold;">WordType</span>
<span style="color: #333333;">|</span> <span style="color: #bb0066; font-weight: bold;">SubNodes</span> <span style="color: #008800; font-weight: bold;">of</span> <span style="color: #bb0066; font-weight: bold;">POS</span> <span style="color: #333333;">*</span> <span style="color: #bb0066; font-weight: bold;">TreeNode</span> <span style="color: #333399; font-weight: bold;">list</span>
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">SentenceType</span> <span style="color: #333333;">=</span>
<span style="color: #333333;">{</span>
<span style="color: #bb0066; font-weight: bold;">Words</span><span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">WordType</span> <span style="color: #333399; font-weight: bold;">list</span>
<span style="color: #bb0066; font-weight: bold;">Dependency</span><span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">DependencyGraph</span> <span style="color: #333399; font-weight: bold;">list</span>
<span style="color: #bb0066; font-weight: bold;">Tree</span><span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">TreeNode</span>
<span style="color: #333333;">}</span>
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">StoryType</span> <span style="color: #333333;">=</span>
<span style="color: #333333;">{</span>
<span style="color: #bb0066; font-weight: bold;">CrossLinks</span> <span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">CrossLinkType</span> <span style="color: #333399; font-weight: bold;">list</span>
<span style="color: #bb0066; font-weight: bold;">Sentences</span> <span style="color: #333333;">:</span> <span style="color: #bb0066; font-weight: bold;">SentenceType</span> <span style="color: #333399; font-weight: bold;">list</span>
<span style="color: #333333;">}</span>
<span style="color: #008800; font-weight: bold;">module</span> <span style="color: #bb0066; font-weight: bold;">NLPUtils</span> <span style="color: #333333;">=</span>
<span style="color: #008800; font-weight: bold;">open</span> edu<span style="color: #333333;">.</span>stanford<span style="color: #333333;">.</span>nlp<span style="color: #333333;">.</span>trees
<span style="color: #008800; font-weight: bold;">open</span> edu<span style="color: #333333;">.</span>stanford<span style="color: #333333;">.</span>nlp<span style="color: #333333;">.</span>ling
<span style="color: #008800; font-weight: bold;">open</span> edu<span style="color: #333333;">.</span>stanford<span style="color: #333333;">.</span>nlp<span style="color: #333333;">.</span>semgraph
<span style="color: #008800; font-weight: bold;">open</span> <span style="color: #bb0066; font-weight: bold;">NLPStructures</span>
<span style="color: #008800; font-weight: bold;">let</span> toEnumerable<span style="color: #333333;"><</span><span style="color: #008800; font-weight: bold;">'</span><span style="color: #bb0066; font-weight: bold;">T</span><span style="color: #333333;">></span> <span style="color: #333333;">(</span>obj<span style="color: #333333;">:</span>obj<span style="color: #333333;">)</span> <span style="color: #333333;">=</span>
<span style="color: #008800; font-weight: bold;">let</span> l <span style="color: #333333;">=</span> obj <span style="color: #333333;">:?></span> java<span style="color: #333333;">.</span>util<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">ArrayList</span>
l <span style="color: #333333;">|></span> <span style="color: #0e84b5; font-weight: bold;">Seq</span>.cast<span style="color: #333333;"><</span><span style="color: #008800; font-weight: bold;">'</span><span style="color: #bb0066; font-weight: bold;">T</span><span style="color: #333333;">></span>
<span style="color: #008800; font-weight: bold;">let</span> toJavaClass <span style="color: #333333;">(</span>t<span style="color: #333333;">:</span><span style="color: #0e84b5; font-weight: bold;">System</span>.<span style="color: #bb0066; font-weight: bold;">Type</span><span style="color: #333333;">)</span> <span style="color: #333333;">=</span> java<span style="color: #333333;">.</span>lang<span style="color: #333333;">.</span><span style="color: #0e84b5; font-weight: bold;">Class</span>.op_Implicit<span style="color: #333333;">(</span>t<span style="color: #333333;">)</span>
<span style="color: #008800; font-weight: bold;">let</span> findWord <span style="color: #333333;">(</span>words<span style="color: #333333;">:</span><span style="color: #bb0066; font-weight: bold;">WordType</span> <span style="color: #333399; font-weight: bold;">list</span><span style="color: #333333;">)</span> <span style="color: #333333;">(</span>word<span style="color: #333333;">:</span><span style="color: #bb0066; font-weight: bold;">Word</span><span style="color: #333333;">)</span> <span style="color: #333333;">=</span>
words <span style="color: #333333;">|></span> <span style="color: #0e84b5; font-weight: bold;">Seq</span>.find <span style="color: #333333;">(</span><span style="color: #008800; font-weight: bold;">fun</span> n <span style="color: #333333;">-></span> n<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">IsSame</span><span style="color: #333333;">(</span>word<span style="color: #333333;">))</span>
<span style="color: #008800; font-weight: bold;">let</span> findIndex <span style="color: #333333;">(</span>words<span style="color: #333333;">:</span><span style="color: #bb0066; font-weight: bold;">WordType</span> <span style="color: #333399; font-weight: bold;">list</span><span style="color: #333333;">)</span> <span style="color: #333333;">(</span>i<span style="color: #333333;">:</span><span style="color: #bb0066; font-weight: bold;">Index</span><span style="color: #333333;">)</span> <span style="color: #333333;">=</span>
words <span style="color: #333333;">|></span> <span style="color: #0e84b5; font-weight: bold;">Seq</span>.find <span style="color: #333333;">(</span><span style="color: #008800; font-weight: bold;">fun</span> n <span style="color: #333333;">-></span> n<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">IsSame</span><span style="color: #333333;">(</span>i<span style="color: #333333;">))</span>
<span style="color: #008800; font-weight: bold;">let</span> <span style="color: #008800; font-weight: bold;">inline</span> getObjFromMap <span style="color: #333333;">(</span>x<span style="color: #333333;">:^</span><span style="color: #bb0066; font-weight: bold;">T</span><span style="color: #333333;">)</span> t <span style="color: #333333;">=</span>
<span style="color: #008800; font-weight: bold;">let</span> key <span style="color: #333333;">=</span> t <span style="color: #333333;">|></span> toJavaClass
<span style="color: #333333;">(^</span><span style="color: #bb0066; font-weight: bold;">T</span> <span style="color: #333333;">:</span> <span style="color: #333333;">(</span><span style="color: #008800; font-weight: bold;">member</span> get <span style="color: #333333;">:</span> java<span style="color: #333333;">.</span>lang<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">Class</span> <span style="color: #333333;">-></span> obj<span style="color: #333333;">)</span> <span style="color: #333333;">(</span>x<span style="color: #333333;">,</span> key<span style="color: #333333;">)</span> <span style="color: #333333;">)</span>
<span style="color: #008800; font-weight: bold;">let</span> getWords <span style="color: #333333;">(</span>tokens<span style="color: #333333;">:</span>seq<span style="color: #333333;"><</span><span style="color: #bb0066; font-weight: bold;">CoreLabel</span><span style="color: #333333;">>)</span> <span style="color: #333333;">=</span>
<span style="color: #333333;">[</span>
<span style="color: #008800; font-weight: bold;">for</span> token <span style="color: #008800; font-weight: bold;">in</span> tokens <span style="color: #008800; font-weight: bold;">do</span>
<span style="color: #008800; font-weight: bold;">let</span> word <span style="color: #333333;">=</span> typeof<span style="color: #333333;"><</span><span style="color: #0e84b5; font-weight: bold;">CoreAnnotations</span>.<span style="color: #bb0066; font-weight: bold;">TextAnnotation</span><span style="color: #333333;">></span> <span style="color: #333333;">|></span> getObjFromMap token <span style="color: #333333;">:?></span> <span style="color: #bb0066; font-weight: bold;">Word</span>
<span style="color: #008800; font-weight: bold;">let</span> pos <span style="color: #333333;">=</span> typeof<span style="color: #333333;"><</span><span style="color: #0e84b5; font-weight: bold;">CoreAnnotations</span>.<span style="color: #bb0066; font-weight: bold;">PartOfSpeechAnnotation</span><span style="color: #333333;">></span> <span style="color: #333333;">|></span> getObjFromMap token <span style="color: #333333;">:?></span> <span style="color: #bb0066; font-weight: bold;">POS</span>
<span style="color: #008800; font-weight: bold;">let</span> ner <span style="color: #333333;">=</span> typeof<span style="color: #333333;"><</span><span style="color: #0e84b5; font-weight: bold;">CoreAnnotations</span>.<span style="color: #bb0066; font-weight: bold;">NamedEntityTagAnnotation</span><span style="color: #333333;">></span> <span style="color: #333333;">|></span> getObjFromMap token <span style="color: #333333;">:?></span> <span style="color: #bb0066; font-weight: bold;">Ner</span>
<span style="color: #008800; font-weight: bold;">let</span> index <span style="color: #333333;">=</span> token<span style="color: #333333;">.</span>index<span style="color: #007020;">()</span>
<span style="color: #008800; font-weight: bold;">let</span> word <span style="color: #333333;">=</span> <span style="color: #333333;">{</span> <span style="color: #bb0066; font-weight: bold;">Word</span> <span style="color: #333333;">=</span> word<span style="color: #333333;">;</span> <span style="color: #bb0066; font-weight: bold;">Ner</span> <span style="color: #333333;">=</span> ner<span style="color: #333333;">;</span> <span style="color: #bb0066; font-weight: bold;">POS</span> <span style="color: #333333;">=</span> pos<span style="color: #333333;">;</span> <span style="color: #bb0066; font-weight: bold;">Index</span> <span style="color: #333333;">=</span> index <span style="color: #333333;">}</span>
<span style="color: #008800; font-weight: bold;">yield</span> word
<span style="color: #333333;">]</span>
<span style="color: #008800; font-weight: bold;">let</span> getDependencyGraph words <span style="color: #333333;">(</span>deps<span style="color: #333333;">:</span><span style="color: #bb0066; font-weight: bold;">SemanticGraph</span><span style="color: #333333;">)</span> <span style="color: #333333;">=</span>
<span style="color: #333333;">[</span>
<span style="color: #008800; font-weight: bold;">for</span> edge <span style="color: #008800; font-weight: bold;">in</span> deps<span style="color: #333333;">.</span>edgeListSorted<span style="color: #007020;">()</span><span style="color: #333333;">.</span>toArray<span style="color: #007020;">()</span> <span style="color: #333333;">|></span> <span style="color: #0e84b5; font-weight: bold;">Seq</span>.cast<span style="color: #333333;"><</span><span style="color: #bb0066; font-weight: bold;">SemanticGraphEdge</span><span style="color: #333333;">></span> <span style="color: #008800; font-weight: bold;">do</span>
<span style="color: #008800; font-weight: bold;">let</span> gov <span style="color: #333333;">=</span> edge<span style="color: #333333;">.</span>getGovernor<span style="color: #007020;">()</span>
<span style="color: #008800; font-weight: bold;">let</span> dep <span style="color: #333333;">=</span> edge<span style="color: #333333;">.</span>getDependent<span style="color: #007020;">()</span>
<span style="color: #008800; font-weight: bold;">let</span> govEntity <span style="color: #333333;">=</span> findIndex words <span style="color: #333333;">(</span>gov<span style="color: #333333;">.</span>index<span style="color: #007020;">()</span><span style="color: #333333;">)</span>
<span style="color: #008800; font-weight: bold;">let</span> depEntity <span style="color: #333333;">=</span> findIndex words <span style="color: #333333;">(</span>dep<span style="color: #333333;">.</span>index<span style="color: #007020;">()</span><span style="color: #333333;">)</span>
<span style="color: #008800; font-weight: bold;">let</span> e <span style="color: #333333;">=</span> <span style="color: #bb0066; font-weight: bold;">Link</span><span style="color: #333333;">(</span>edge<span style="color: #333333;">.</span>getRelation<span style="color: #007020;">()</span><span style="color: #333333;">.</span>getLongName<span style="color: #007020;">()</span><span style="color: #333333;">,</span> govEntity<span style="color: #333333;">,</span> depEntity<span style="color: #333333;">)</span>
<span style="color: #008800; font-weight: bold;">yield</span> e
<span style="color: #333333;">]</span>
<span style="color: #008800; font-weight: bold;">let</span> <span style="color: #008800; font-weight: bold;">rec</span> buildTree words <span style="color: #333333;">(</span>tree<span style="color: #333333;">:</span><span style="color: #bb0066; font-weight: bold;">Tree</span><span style="color: #333333;">)</span> <span style="color: #333333;">=</span>
<span style="color: #008800; font-weight: bold;">let</span> label <span style="color: #333333;">=</span> tree<span style="color: #333333;">.</span>value<span style="color: #007020;">()</span>
<span style="color: #008800; font-weight: bold;">let</span> children <span style="color: #333333;">=</span> tree<span style="color: #333333;">.</span>children<span style="color: #007020;">()</span>
<span style="color: #008800; font-weight: bold;">if</span> children<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">Length</span> <span style="color: #333333;">=</span> <span style="color: #0000dd; font-weight: bold;">0</span> <span style="color: #008800; font-weight: bold;">then</span>
<span style="color: #008800; font-weight: bold;">let</span> x <span style="color: #333333;">=</span> tree<span style="color: #333333;">.</span>label<span style="color: #007020;">()</span> <span style="color: #333333;">:?></span> <span style="color: #bb0066; font-weight: bold;">CoreLabel</span>
<span style="color: #008800; font-weight: bold;">let</span> i <span style="color: #333333;">=</span> x<span style="color: #333333;">.</span>index<span style="color: #007020;">()</span>
<span style="color: #008800; font-weight: bold;">let</span> entity <span style="color: #333333;">=</span> findIndex words i
<span style="color: #bb0066; font-weight: bold;">Node</span><span style="color: #333333;">(</span>label<span style="color: #333333;">,</span> entity<span style="color: #333333;">)</span>
<span style="color: #008800; font-weight: bold;">else</span>
<span style="color: #008800; font-weight: bold;">let</span> nodes <span style="color: #333333;">=</span> children <span style="color: #333333;">|></span> <span style="color: #0e84b5; font-weight: bold;">Seq</span>.map <span style="color: #333333;">(</span><span style="color: #008800; font-weight: bold;">fun</span> tree <span style="color: #333333;">-></span> buildTree words tree<span style="color: #333333;">)</span> <span style="color: #333333;">|></span> <span style="color: #0e84b5; font-weight: bold;">Seq</span>.toList
<span style="color: #bb0066; font-weight: bold;">SubNodes</span><span style="color: #333333;">(</span>label<span style="color: #333333;">,</span> nodes<span style="color: #333333;">)</span>
<span style="color: #008800; font-weight: bold;">let</span> getMention <span style="color: #333333;">(</span>mention<span style="color: #333333;">:</span>edu<span style="color: #333333;">.</span>stanford<span style="color: #333333;">.</span>nlp<span style="color: #333333;">.</span>coref<span style="color: #333333;">.</span>data<span style="color: #333333;">.</span><span style="color: #0e84b5; font-weight: bold;">CorefChain</span>.<span style="color: #bb0066; font-weight: bold;">CorefMention</span><span style="color: #333333;">)</span> <span style="color: #333333;">=</span>
<span style="color: #008800; font-weight: bold;">let</span> mentionId <span style="color: #333333;">=</span> mention<span style="color: #333333;">.</span>mentionID
<span style="color: #008800; font-weight: bold;">let</span> span <span style="color: #333333;">=</span> <span style="color: #333333;">(</span>mention<span style="color: #333333;">.</span>startIndex<span style="color: #333333;">,</span> mention<span style="color: #333333;">.</span>endIndex<span style="color: #333333;">)</span>
<span style="color: #008800; font-weight: bold;">let</span> relation <span style="color: #333333;">=</span> mention<span style="color: #333333;">.</span>animacy<span style="color: #333333;">.</span>name<span style="color: #007020;">()</span>
<span style="color: #008800; font-weight: bold;">let</span> head <span style="color: #333333;">=</span> <span style="color: #333333;">(</span>mention<span style="color: #333333;">.</span>headIndex<span style="color: #333333;">,</span> mention<span style="color: #333333;">.</span>mentionSpan<span style="color: #333333;">)</span>
<span style="color: #008800; font-weight: bold;">let</span> sentenceIndex <span style="color: #333333;">=</span> mention<span style="color: #333333;">.</span>sentNum
<span style="color: #008800; font-weight: bold;">let</span> m <span style="color: #333333;">=</span> <span style="color: #333333;">{</span> <span style="color: #bb0066; font-weight: bold;">Index</span> <span style="color: #333333;">=</span> mentionId<span style="color: #333333;">;</span> <span style="color: #bb0066; font-weight: bold;">Relationship</span> <span style="color: #333333;">=</span> relation<span style="color: #333333;">;</span> <span style="color: #bb0066; font-weight: bold;">Span</span> <span style="color: #333333;">=</span> span<span style="color: #333333;">;</span> <span style="color: #bb0066; font-weight: bold;">Head</span> <span style="color: #333333;">=</span> head<span style="color: #333333;">;</span> <span style="color: #bb0066; font-weight: bold;">SentenceIndex</span> <span style="color: #333333;">=</span> sentenceIndex<span style="color: #333333;">;</span> <span style="color: #333333;">}</span>
m
<span style="color: #008800; font-weight: bold;">let</span> getMentions <span style="color: #333333;">(</span>keys<span style="color: #333333;">:</span>java<span style="color: #333333;">.</span>util<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">HashMap</span><span style="color: #333333;">)</span> <span style="color: #333333;">=</span>
<span style="color: #333333;">[</span>
<span style="color: #008800; font-weight: bold;">for</span> key <span style="color: #008800; font-weight: bold;">in</span> keys<span style="color: #333333;">.</span>keySet<span style="color: #007020;">()</span><span style="color: #333333;">.</span>toArray<span style="color: #007020;">()</span> <span style="color: #008800; font-weight: bold;">do</span>
<span style="color: #008800; font-weight: bold;">let</span> v <span style="color: #333333;">=</span> keys<span style="color: #333333;">.</span>get<span style="color: #333333;">(</span>key<span style="color: #333333;">)</span> <span style="color: #333333;">:?></span> edu<span style="color: #333333;">.</span>stanford<span style="color: #333333;">.</span>nlp<span style="color: #333333;">.</span>coref<span style="color: #333333;">.</span>data<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">CorefChain</span>
<span style="color: #008800; font-weight: bold;">let</span> representiveMention <span style="color: #333333;">=</span> v<span style="color: #333333;">.</span>getRepresentativeMention<span style="color: #007020;">()</span>
<span style="color: #008800; font-weight: bold;">let</span> m <span style="color: #333333;">=</span> getMention<span style="color: #333333;">(</span>representiveMention<span style="color: #333333;">)</span>
<span style="color: #008800; font-weight: bold;">let</span> index <span style="color: #333333;">=</span> v<span style="color: #333333;">.</span>getChainID<span style="color: #007020;">()</span>
<span style="color: #008800; font-weight: bold;">let</span> mentions <span style="color: #333333;">=</span> v<span style="color: #333333;">.</span>getMentionsInTextualOrder<span style="color: #007020;">()</span><span style="color: #333333;">.</span>toArray<span style="color: #007020;">()</span>
<span style="color: #008800; font-weight: bold;">let</span> ms <span style="color: #333333;">=</span> mentions
<span style="color: #333333;">|></span> <span style="color: #0e84b5; font-weight: bold;">Seq</span>.cast<span style="color: #333333;"><</span>edu<span style="color: #333333;">.</span>stanford<span style="color: #333333;">.</span>nlp<span style="color: #333333;">.</span>coref<span style="color: #333333;">.</span>data<span style="color: #333333;">.</span><span style="color: #0e84b5; font-weight: bold;">CorefChain</span>.<span style="color: #bb0066; font-weight: bold;">CorefMention</span><span style="color: #333333;">></span>
<span style="color: #333333;">|></span> <span style="color: #0e84b5; font-weight: bold;">Seq</span>.map getMention
<span style="color: #333333;">|></span> <span style="color: #0e84b5; font-weight: bold;">Seq</span>.toList
<span style="color: #008800; font-weight: bold;">let</span> r <span style="color: #333333;">=</span> <span style="color: #333333;">{</span> <span style="color: #bb0066; font-weight: bold;">Index</span> <span style="color: #333333;">=</span> index<span style="color: #333333;">;</span> <span style="color: #bb0066; font-weight: bold;">RepresentiveMention</span> <span style="color: #333333;">=</span> m<span style="color: #333333;">;</span> <span style="color: #bb0066; font-weight: bold;">Mentions</span> <span style="color: #333333;">=</span> ms<span style="color: #333333;">;</span> <span style="color: #333333;">}</span>
<span style="color: #008800; font-weight: bold;">yield</span> r
<span style="color: #333333;">]</span>
<span style="color: #008800; font-weight: bold;">let</span> returnSeq<span style="color: #333333;"><</span><span style="color: #008800; font-weight: bold;">'</span><span style="color: #bb0066; font-weight: bold;">T</span><span style="color: #333333;">></span> <span style="color: #333333;">(</span>x<span style="color: #333333;">:</span>obj<span style="color: #333333;">)</span> <span style="color: #333333;">=</span>
<span style="color: #008800; font-weight: bold;">if</span> x <span style="color: #333333;">:?</span> java<span style="color: #333333;">.</span>util<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">ArrayList</span> <span style="color: #008800; font-weight: bold;">then</span>
toEnumerable<span style="color: #333333;"><</span><span style="color: #008800; font-weight: bold;">'</span><span style="color: #bb0066; font-weight: bold;">T</span><span style="color: #333333;">></span> x
<span style="color: #008800; font-weight: bold;">else</span>
<span style="color: #0e84b5; font-weight: bold;">Seq</span>.singleton <span style="color: #333333;">(</span>x <span style="color: #333333;">:?></span> <span style="color: #008800; font-weight: bold;">'</span><span style="color: #bb0066; font-weight: bold;">T</span><span style="color: #333333;">)</span>
<span style="color: #008800; font-weight: bold;">module</span> <span style="color: #bb0066; font-weight: bold;">NLPExtensions</span> <span style="color: #333333;">=</span>
<span style="color: #008800; font-weight: bold;">open</span> <span style="color: #bb0066; font-weight: bold;">NLPUtils</span>
<span style="color: #008800; font-weight: bold;">open</span> edu<span style="color: #333333;">.</span>stanford<span style="color: #333333;">.</span>nlp<span style="color: #333333;">.</span>util
<span style="color: #008800; font-weight: bold;">type</span> <span style="color: #bb0066; font-weight: bold;">CoreMap</span> <span style="color: #008800; font-weight: bold;">with</span>
<span style="color: #008800; font-weight: bold;">member</span> this<span style="color: #333333;">.</span><span style="color: #bb0066; font-weight: bold;">GetToken</span><span style="color: #333333;"><</span><span style="color: #008800; font-weight: bold;">'</span><span style="color: #bb0066; font-weight: bold;">T</span><span style="color: #333333;">></span> <span style="color: #333333;">(</span>t<span style="color: #333333;">:</span><span style="color: #0e84b5; font-weight: bold;">System</span>.<span style="color: #bb0066; font-weight: bold;">Type</span><span style="color: #333333;">)</span> <span style="color: #333333;">=</span>
t <span style="color: #333333;">|></span> getObjFromMap this <span style="color: #333333;">|></span> returnSeq<span style="color: #333333;"><</span><span style="color: #008800; font-weight: bold;">'</span><span style="color: #bb0066; font-weight: bold;">T</span><span style="color: #333333;">></span>
</pre>
</td></tr>
</tbody></table>
</div>
<br /></div>
<div>
The execution result shows below:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3fFuwk4cL9jNoG7VuEBgOLtewoyTXkovh41RANfux8bNUBgknn2Fwz-4q1ClpuregGEuv7Dq2gpsoSwr25gF0iljejUoUFYB1xW-TfiiD1nIjURuifmeix52Xol1OM8pqVSIKQSkmNL4/s1600/NLPResult0.PNG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="481" data-original-width="819" height="372" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3fFuwk4cL9jNoG7VuEBgOLtewoyTXkovh41RANfux8bNUBgknn2Fwz-4q1ClpuregGEuv7Dq2gpsoSwr25gF0iljejUoUFYB1xW-TfiiD1nIjURuifmeix52Xol1OM8pqVSIKQSkmNL4/s640/NLPResult0.PNG" width="640" /></a></div>
<br />
<br />
<br /></div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-16982170495428852972018-11-03T20:11:00.002-04:002018-11-26T21:35:24.973-05:00SelfNote: Stanford NLP<div>
I create this page as the master page for using F# on Stanford NLP.<br />
<br />
<ul>
<li>Get the <a href="https://apollo13cn.blogspot.com/2018/10/f-stanford-nlp-is-running.html">F# project running with Stanford NLP</a></li>
<li>POS tag explanation</li>
<ul>
<li><a href="https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html">Penn Treebank P.O.S tag</a></li>
<li>Stack Overflow <a href="https://stackoverflow.com/questions/1833252/java-stanford-nlp-part-of-speech-labels/22864092">Q&A</a> about POS tag</li>
<li>A <a href="http://erwinkomen.ruhosting.nl/eng/2014_Longdale-Labels.htm">list </a>with an explanation. </li>
<li>UN dependency tag is <a href="http://universaldependencies.org/en/dep/index.html">here</a></li>
<li>the difference between tags are listed at this <a href="https://sites.google.com/site/partofspeechhelp/home/nn_nnp">google site</a></li>
</ul>
<li>Dependency Tree</li>
<ul>
<li>Tree relation manual is <a href="https://nlp.stanford.edu/software/dependencies_manual.pdf">here</a></li>
</ul>
<li><a href="https://apollo13cn.blogspot.com/2018/11/move-nlp-ikvm-to-f.html">Move the IKVM version to be more .NET friendly</a> with some helper functions and my internal data structure</li>
<li>Stemming is performed in Stanford NLPCore by Lemma tag. It will get the stem of the word no matter the word is a noun or a verb. </li>
<ul>
</ul>
</ul>
</div>
<div>
<script>
!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");
</script></div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-74456381263874570212018-10-29T22:38:00.000-04:002018-10-29T22:38:06.285-04:00F# Stanford NLP is runningAfter some configuration, I can successfully run the first NLP project with F#. Special thanks to Sergey's <a href="https://sergey-tihon.github.io/Stanford.NLP.NET/StanfordCoreNLP.html">post</a>! The post is very informative. His solution is based on the F# interactive while I prefer to use the project-based solution.<br />
<div>
<br /></div>
<div>
Sergey points out that one of the common problems to setup is the path problem. His claim is so true. I had stuck in this problem for days. Here is the process I followed.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<ul>
<li>Open Visual Studio 2017 and create an F# console application. </li>
<ul>
<li>I tried .net core app; it does not work as the IKVM has the dependency on the .NET framework</li>
</ul>
<li>compile the F# console application and remember the debug folder location</li>
<li>Open NuGet and retrieve Stanford NLP CoreNLP. The current version is 3.9.1</li>
<ul>
<li>Current Stanford NLP is 3.9.2. I suggest you download 3.9.1 version</li>
</ul>
<li>download the Standard NLP 3.9.1 zip file</li>
<li>unzip the 3.9.1 file to the F# console app debug folder</li>
<li>go the unzipped folder and find the model JAR file</li>
</ul>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiRVgTKh-CvO3Om9DzPyX_3-eGeCNT3x7pk40uwZXatB0FBlDAU0YlzxkQLu47_3jkQmFs-d4EkzVecBX6rpITO5QU3HDyxW8JsQ_pZMa7_ZCArd7A0r98OYvt83cr9tsf4LcxPgShrn4/s1600/ModelFiles.PNG" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="107" data-original-width="621" height="108" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiRVgTKh-CvO3Om9DzPyX_3-eGeCNT3x7pk40uwZXatB0FBlDAU0YlzxkQLu47_3jkQmFs-d4EkzVecBX6rpITO5QU3HDyxW8JsQ_pZMa7_ZCArd7A0r98OYvt83cr9tsf4LcxPgShrn4/s640/ModelFiles.PNG" width="640" /></a><br />
<ul>
<li>download WINRAR to unzip the JAR file to a folder, this folder should contain a folder called "EDU"</li>
<li>copy the "EDU" folder up to debug folder, so the structure in the "DEBUG" folder is like the following.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4nq8jVCOOe7xpQ7DO4ygoF5GNtUtwIXaiMVsyGcCJTWIc26d1t2tZv0aOxvMC1HNBgHdZTXPohsRgar-hkIydQEDlBiBW77d60UdovO015Wd9pHpYlBUVIbB1Di2KNhmqVtU03zNrX5w/s1600/StanfordNLPFSharp.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="288" data-original-width="673" height="171" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4nq8jVCOOe7xpQ7DO4ygoF5GNtUtwIXaiMVsyGcCJTWIc26d1t2tZv0aOxvMC1HNBgHdZTXPohsRgar-hkIydQEDlBiBW77d60UdovO015Wd9pHpYlBUVIbB1Di2KNhmqVtU03zNrX5w/s400/StanfordNLPFSharp.PNG" width="400" /></a> </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The F# file I was using is listed below. Set the "EDU" folder to the debug folder can save you from setting the CurrentDirectory. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">// Learn more about F# at http://fsharp.org</span>
<span style="color: #888888;">// See the 'F# Tutorial' project for more help.</span>
<span style="color: #008800; font-weight: bold;">open</span> <span style="color: #bb0066; font-weight: bold;">System</span>
<span style="color: #008800; font-weight: bold;">open</span> <span style="color: #0e84b5; font-weight: bold;">System</span>.<span style="color: #bb0066; font-weight: bold;">IO</span>
<span style="color: #008800; font-weight: bold;">open</span> java<span style="color: #333333;">.</span>util
<span style="color: #008800; font-weight: bold;">open</span> java<span style="color: #333333;">.</span>io
<span style="color: #008800; font-weight: bold;">open</span> edu<span style="color: #333333;">.</span>stanford<span style="color: #333333;">.</span>nlp<span style="color: #333333;">.</span>pipeline
[<<span style="color: #bb0066; font-weight: bold;">EntryPoint></span><span style="color: #333333;">]</span>
<span style="color: #008800; font-weight: bold;">let</span> main argv <span style="color: #333333;">=</span>
<span style="color: #008800; font-weight: bold;">let</span> text <span style="color: #333333;">=</span> <span style="background-color: #fff0f0;">"Kosgi Santosh sent an email to Stanford University. He didn't get a reply."</span><span style="color: #333333;">;</span>
<span style="color: #888888;">// Annotation pipeline configuration</span>
<span style="color: #008800; font-weight: bold;">let</span> props <span style="color: #333333;">=</span> <span style="color: #bb0066; font-weight: bold;">Properties</span><span style="color: #007020;">()</span>
props<span style="color: #333333;">.</span>setProperty<span style="color: #333333;">(</span><span style="background-color: #fff0f0;">"annotators"</span><span style="color: #333333;">,</span><span style="background-color: #fff0f0;">"tokenize, ssplit, pos, lemma, ner, parse, dcoref"</span><span style="color: #333333;">)</span> <span style="color: #333333;">|></span> ignore
props<span style="color: #333333;">.</span>setProperty<span style="color: #333333;">(</span><span style="background-color: #fff0f0;">"ner.useSUTime"</span><span style="color: #333333;">,</span><span style="background-color: #fff0f0;">"0"</span><span style="color: #333333;">)</span> <span style="color: #333333;">|></span> ignore
<span style="color: #008800; font-weight: bold;">let</span> pipeline <span style="color: #333333;">=</span> <span style="color: #bb0066; font-weight: bold;">StanfordCoreNLP</span><span style="color: #333333;">(</span>props<span style="color: #333333;">)</span>
<span style="color: #888888;">// Annotation</span>
<span style="color: #008800; font-weight: bold;">let</span> annotation <span style="color: #333333;">=</span> <span style="color: #bb0066; font-weight: bold;">Annotation</span><span style="color: #333333;">(</span>text<span style="color: #333333;">)</span>
pipeline<span style="color: #333333;">.</span>annotate<span style="color: #333333;">(</span>annotation<span style="color: #333333;">)</span>
<span style="color: #888888;">// Result - Pretty Print</span>
<span style="color: #008800; font-weight: bold;">let</span> stream <span style="color: #333333;">=</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #bb0066; font-weight: bold;">ByteArrayOutputStream</span><span style="color: #007020;">()</span>
pipeline<span style="color: #333333;">.</span>prettyPrint<span style="color: #333333;">(</span>annotation<span style="color: #333333;">,</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #bb0066; font-weight: bold;">PrintWriter</span><span style="color: #333333;">(</span>stream<span style="color: #333333;">))</span>
printfn <span style="background-color: #fff0f0;">"%O"</span> <span style="color: #333333;"><|</span> stream<span style="color: #333333;">.</span>toString<span style="color: #007020;">()</span>
stream<span style="color: #333333;">.</span>close<span style="color: #007020;">()</span>
printfn <span style="background-color: #fff0f0;">"%A"</span> argv
<span style="color: #0000dd; font-weight: bold;">0</span> <span style="color: #888888;">// return an integer exit code</span>
</pre>
</td></tr>
</tbody></table>
</div>
<br />
Executing the NLP program seems taking a lot of memory. My program uses 2G memory and takes a while to show the result. Hopefully, your computer is faster enough. :)<br />
<ul><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4nq8jVCOOe7xpQ7DO4ygoF5GNtUtwIXaiMVsyGcCJTWIc26d1t2tZv0aOxvMC1HNBgHdZTXPohsRgar-hkIydQEDlBiBW77d60UdovO015Wd9pHpYlBUVIbB1Di2KNhmqVtU03zNrX5w/s1600/StanfordNLPFSharp.PNG" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"> </a></ul>
</div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-80355047036088716982018-10-25T11:40:00.000-04:002018-10-25T11:40:54.091-04:00F# Enum usage II<div>
I want to take a quick note on the F# enum usage again. The supporting of the space and unique character in F# language and editor is a great feature can make your development work much more comfortable. I am in the Web API front these days.<br />
<br />
One of the requirement is to provide options to end users. If an application only takes "excellent choice", "good option", "ok choice", and "bad and never go there" as options, I'd like to offload these values check to the type system instead of handling the error in my code.<br />
<br />
If I can declare the enum like the following<br />
<br />
type enum MyEnum =<br />
``excellent choice`` = 0<br />
| ``good option`` = 1<br />
....<br />
<br />
After declaring the enum-as-string attribute on the attribute <span style="background-color: #eff0f1; color: #242729; font-family: "consolas" , "menlo" , "monaco" , "lucida console" , "liberation mono" , "dejavu sans mono" , "bitstream vera sans mono" , "courier new" , monospace , sans-serif; font-size: 13px; white-space: pre-wrap;">[JsonConverter(typeof(StringEnumConverter))]</span>, the output and input validation is solved in one shoot.</div>
<div>
</div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-70382113336614694062018-09-24T22:20:00.000-04:002018-09-24T22:24:41.083-04:00F# Enum's Usage<div>
I have a web service project and I found my team constantly needs to convert enum to a string. The only conversion is adding space. It wastes lots of time and involves in reflection which slows down the run time performance.<br />
<br />
Enum type is one of the favorite types in F#. C#'s enum type does not have the ability to define a value with space. Having an enum value with space is very important because this feature can save me lots of time to output enum value as string.<br />
<br />
For example, if the enum value can be "Post Release", the ToString function can output a nice string and won't have to use attribute and reflection to do the job. C# does have the ability to have space in enum but it will need to use reflect and emit to generate the type. If you can use F#, the problem can be easily solved. Please check the following F# code:<br />
<br />
namespace ClassLibrary1<br />
<br />
type public EnumEng =<br />
Registration = 0<br />
| ``Under Review``=1<br />
| Approval = 2<br />
| Release = 3<br />
| ``Post Release`` = 4<br />
<br />
type public EnumChn =<br />
注册 = 0<br />
| 审批 = 1<br />
| 批准 = 2<br />
| 发布 = 3<br />
| 发布后 = 4<br />
<br />
type public EnumIndex =<br />
Reg = 0<br />
| Review = 1<br />
| Approval = 2<br />
| Release = 3<br />
| PostRelase = 4<br />
<br />
<br />
module Test =<br />
let a = EnumEng.``Under Review``<br />
let b = EnumChn.审批<br />
<br />
In the C# side, the intellisense won't display the enum value if it contains space. However, it will be displayed in debug mode. You can execute the following code and stop at the end of the function.<br />
<br />
static void Main(string[] args)<br />
{<br />
//get all string from enum0<br />
var strs = Enumerable.Range(0, 5)<br />
.Select(n => (ClassLibrary1.EnumEng)n)<br />
.Select(n => n.ToString())<br />
.ToList();<br />
<br />
// get all string from enum2<br />
var i = Enumerable.Range(0, 5)<br />
.Select(n => (ClassLibrary1.EnumChn)n)<br />
.Select(n => n.ToString())<br />
.ToList();<br />
<br />
// parse string to enum<br />
var v = strs.Select(n => Enum.Parse<classlibrary1 .enumeng="">(n))</classlibrary1><br />
.ToList();<br />
<br />
var x = ClassLibrary1.EnumIndex.PostRelase;<br />
<br />
var str = (ClassLibrary1.EnumChn)x;<br />
}<br />
<br />
Both # and C# support non-English variable name, it will provide a way to localize the output as well. From the sample above, the EnumIndex is the type can used in C#/F# code. Once the value needs to be output as string, it can be convert to EnumEnglish (English string) or EnumChinese(Chinese string).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvY9O0pEvZSxdW6wG24ZwO3-3MBregn5dN9UgEJfHERXq2Jys0_h4OJZ2pj6YTXbpv6In8nKkWB7Zgn68V1-7RXghmIqIxTLfkYYJlJBhJQM1dWWf9erkIM7Xy4-OicQZz3-FHng5g0mE/s1600/FSharpEnum.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="728" data-original-width="720" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvY9O0pEvZSxdW6wG24ZwO3-3MBregn5dN9UgEJfHERXq2Jys0_h4OJZ2pj6YTXbpv6In8nKkWB7Zgn68V1-7RXghmIqIxTLfkYYJlJBhJQM1dWWf9erkIM7Xy4-OicQZz3-FHng5g0mE/s320/FSharpEnum.PNG" width="316" /></a></div>
<br />
<br />
Is that convenient?<br />
<br />
<a class="twitter-mention-button" data-lang="en" data-related="ttliu2000" href="https://twitter.com/intent/tweet?screen_name=fsharp&text=fsharp%20programming" style="background-color: white;">Tweet to fsharp</a></div>
<div>
<script>
!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");
</script></div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com2tag:blogger.com,1999:blog-4100091590694628472.post-11437356384273773902018-09-13T14:13:00.001-04:002018-09-13T14:32:01.756-04:00SelfNote: WebAssembly With Blazor & .NET Core UpgradeAfter set up the HTML5/Typescript roadmap for my group, I started to move my interest to other UI/visualization technology. With the Blazor is in the starting phase, I feel this is an opportunity to get real-time web rendering and .net core in one shoot. The get start part in Blazor is very helpful. Only one small bug for the project creation if you upgrade .NET core.<br />
<div>
<br /></div>
<div>
From PowerShell window, you can find the .Net core version by using "dotnet --info". The Blazor service generates the global.json file, which sits beside the solution file. This file is required to load the .Net core. The default value in the file is 2.1.300. </div>
<div>
<br /></div>
<div>
<blockquote class="tr_bq">
{<br />
"sdk": {<br />
"version": "<b>2.1.300</b>"<br />
}<br />
}</blockquote>
My computer is new and I directly installed 2.1.402 version. Now I got the error complains about "cannot import package". After change that version to 402, I can now manually add those created projects (Blazor Server and Client) to the solution. </div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-40121209998940643322018-09-09T20:23:00.000-04:002018-09-09T20:23:59.664-04:00A Tech Manager's View on Full Stack Developer<div>
<div style="background-color: white; color: #454545; font-family: ".SF UI Text"; font-size: 17px; font-stretch: normal; line-height: normal;">
<span style="font-family: ".SFUIText"; font-size: 17pt;">As a hiring manager, I was asked by agent about full stack developer. I am not a big fan of so-called full stack developers. </span></div>
<div style="background-color: white; color: #454545; font-family: ".SF UI Text"; font-size: 17px; font-stretch: normal; line-height: normal; min-height: 20.3px;">
<span style="font-family: ".SFUIText"; font-size: 17pt;"></span><br /></div>
<div style="background-color: white; font-stretch: normal; line-height: normal;">
<span style="color: #454545; font-family: .SFUIText;"><span style="font-size: 17pt;">No one can have unlimited memory and time to be expert in all area. I am more interested in knowing what the depth one can go. Those average stuff can be searched from internet. Those experience does not demonstrate the critical thinking skills. Those solution, with high probability, introduces high </span><span style="font-size: 22.6667px;">maintenance</span><span style="font-size: 17pt;"> cost down the road.</span></span></div>
<div style="background-color: white; color: #454545; font-family: ".SF UI Text"; font-size: 17px; font-stretch: normal; line-height: normal;">
<span style="font-family: ".SFUIText"; font-size: 17pt;"><br /></span></div>
<div style="background-color: white; color: #454545; font-family: ".SF UI Text"; font-size: 17px; font-stretch: normal; line-height: normal;">
<span style="font-family: ".SFUIText"; font-size: 17pt;">Being deep in one area requires perseverance and talent, those attributes will provide a foundation to long lasting solution. </span></div>
<div style="background-color: white; color: #454545; font-family: ".SF UI Text"; font-size: 17px; font-stretch: normal; line-height: normal;">
<span style="font-family: ".SFUIText"; font-size: 17pt;"><br /></span></div>
<div style="background-color: white; color: #454545; font-stretch: normal; line-height: normal;">
<span style="font-size: 22.6667px;">Being a technology manager, maintaining different technology stack has a huge cost factor and segment team into small silos. The team effort will be lost. The tech plan and road map is missing. </span></div>
<div style="background-color: white; color: #454545; font-family: ".SF UI Text"; font-size: 17px; font-stretch: normal; line-height: normal;">
<span style="font-family: ".SFUIText"; font-size: 17pt;"><br /></span></div>
<div style="background-color: white; color: #454545; font-family: ".SF UI Text"; font-size: 17px; font-stretch: normal; line-height: normal;">
<span style="font-family: ".SFUIText"; font-size: 17pt;">I have read some job description that requires tens of unrelated technology. This kind of position really makes me thinking how effective the tech management is and what the team looks like.</span></div>
</div>
<div>
<script>
!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");
</script></div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-67329469216290772322018-08-31T01:58:00.001-04:002018-08-31T01:58:27.843-04:00Technology from a Tech Manager's View - Measure Productivity<div>
It is always an intriguing topic about software development productivity. There are some attempts to measure productivity. From those numbers, there are some conclusions and team chaos and management challenge are knocking at the door.<br />
<br />
<br />
First of all, I do not think this productivity is measurable. If a computer program can understand how much effort to solve a problem, this program will be smart enough to solve the problem. Then we don’t need software developers at all.<br />
<br />
The worst thing about this measurement is let team members know about it. The assumption is measurement will give insight into the problem and eventually leads to a solution. Not mention the trust between employee and management is broken, the assumption itself has a fatal error. You can measure stone's size and weight and those numbers won’t change. However, the subject of the measurement is human who will change the behavior if s/he knows someone is watching.<br />
<br />
Naturally the subject (human) changes behavior to make the number better. Will this change really improve the productivity? Most likely not. The first thing human subject can do is to game the system. Will this effort result in high quality software? No. When you know you are not trusted, are you willing to contribute more?<br />
<br />
Einstein's noble prize paper is only a few pages. Essentially the attempt to measure is put equivalence between software development activities and operational labor work.<br />
<br />
<br /></div>
<div>
<script>
!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");
</script></div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-86146853813882651302018-08-25T01:45:00.000-04:002018-08-25T01:45:29.090-04:00Technology from a Tech Manager's view - Coding StandardI have been in this industry for more than 10 years. Growing from a junior dev to a senior manager, I associated my name with many projects. I witnessed some interesting practices during the software development, especially in a team environment. This post is the first post I want to share my thoughts as a tech manager.<div>
<br /></div>
<div>
Coding standard or best practice? I have witnessed so many attempts to set up some coding standards in a team and failed. There are still coding standard proposed every day and I bet there are some emails flowing around now. I can't say all of them, but most of them, will fail miserably. I have seen quant code, dev's code, system admin's code, etc. It seems that everyone has its own coding standard. Every industry has its own way to practice the software development. Is there a general coding standard, or should a coding standard exist at all? </div>
<div>
<br /></div>
<div>
Now days, I feel those standard or effort to push such standard does not make much sense. Programming language is a way human tell a computer how s/he understand and solve the problem. It is like human language. Is there a standard way or "best practice" to say something? Do you have the "best" way to describe how solar system works for a 10-year-old boy? Is the "best way" still the best when facing a 20-year-old? </div>
<div>
<br /></div>
<div>
For me, I think the best way to set up the "best practice" to understand your team. if my team has quant or physics background people. They want to use single-character variable name, it should be fine. Take a look at those mathematics paper they read every day, it is not difficult to understand why single-character variable name is the best choice for them. </div>
<div>
<br /></div>
<div>
If I had a team whose members are all system admins, should I ask them to define interface or introduce complicated inheritance? Definitely not. Does this mean my project will fail? No. </div>
<div>
<br /></div>
<div>
My job is to have my team to tell the computer how to solve a business problem. My team will use a programming language + framework to describe the solution. As a tech manager, my job is to choose the right technology for my team. That technology should be the most efficient and convenient way to </div>
<div>
<ul>
<li>understand and communicate among team members</li>
<li>communicate between a human and a computer</li>
</ul>
<div>
The diversity of technology is the outcome from innovative person and team to serve different people in different business environment. If you are pushing for innovation and still enforce a rigid coding standard, think twice.</div>
</div>
<div>
<br /></div>
<div>
<br /></div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-20641845224428932792018-08-12T16:09:00.001-04:002018-08-12T16:20:38.289-04:00Solve Automation & Biz Flow Project Resource Problem with PowerShellWhen I take over the <a href="http://apollo13cn.blogspot.com/2018/06/my-devop-project.html">DevOps project</a> to do the automation work. My project is not juts to streamline the software development process, it involves in large number of internal systems. For a company with many legacy systems and different business process. It is literally re-engineer the business flow.<br />
<br />
My challenge is to have the people who know the business flow and the technology and make them work together. I do not want to hire BA because I have enough people who know business knowledge. The pressing task to have more person to code the logic. Thanks to my language background, the solution I found is to use different languages to solve the problem.<br />
<br />
The first decision I made is to have <b>no-GUI</b> at all. The fancy GUI is nice to a new user; however, it will block the future automation for good. When you have a meeting with your financial planner, you will notice how many times s/he do copy/paste from one system to the other.<br />
<br />
To fast code the solution, I need a <b>scripting language</b> or a language support scripting feature. That language should be good for folks with less coding experience. I do not need them to understand what is singleton pattern or difference between NTLM and Basic Authentication. I want my dev team to focus on the technical challenges and provide API's. Biz-background people can use those API's to streamline those processes in his/her area.<br />
<br />
The initial setup is like the diagram following. Part1 is finished by biz people by invoke API's provided by dev's working on Part2.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgISgvCtPXyd8aN8Ry0Ip7YVSgodNiMnM3EXKrBunUqXq-NJGSg5ufm6zjbfFbSXaGfGikvFoV6luygr2XCSxlP1Vh3XLpxRaD0Lipaz6O8xviRAXKZaW7bzEEvdeho7Kx7sVlCh0D5-mY/s1600/PSLangTaskDivision.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="648" data-original-width="1152" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgISgvCtPXyd8aN8Ry0Ip7YVSgodNiMnM3EXKrBunUqXq-NJGSg5ufm6zjbfFbSXaGfGikvFoV6luygr2XCSxlP1Vh3XLpxRaD0Lipaz6O8xviRAXKZaW7bzEEvdeho7Kx7sVlCh0D5-mY/s1600/PSLangTaskDivision.png" /></a></div>
<br />
<br />
<br />
I landed on the <b>PowerShell</b> language for Part1. PowerShell (PS) is designed for support/op folks. It has large number of samples. The most compelling reason for PS is that it can move the interface line from top to very bottom in the above diagram. <b>PS can go deeper</b>. Almost everything C#/F#/VB.net can do, PS can do. At the project initial stage, Part2 is much bigger than part1. I want a language has the potential to go deeper to the binary code in case my dev does not have the bandwidth to provide API. Other .Net languages can invoke PowerShell as well. As a result, the coding effort has little waste. A perfect eco-system where every bit of coding effort can be used and reused.<br />
<br />
The current result is my biz people can code PowerShell and slowly push the "interface line" down to the ultimate binary format. The dev side does not have to deal with those "boring" business work. And the dev workload is getting less. By using PowerShell scripting language,<br />
<br />
<ul>
<li>Efficiency: one line PS code will do 10+ lines of .net language code. The overall progress will be faster when using a scripting language. </li>
<li>cost saving, I have the potential to expand my team to accommodate more people with various programming level</li>
<li>Solid and quick solution: both parties can focus on the area in which s/he has deep expertise. The implementation speed is good.</li>
<li>Flexible solution: since the interface line can move up and down based on the resource / expertise. </li>
<li>Vendor-independent: my solution is general enough so won't be locked in any vendor's customized solution. </li>
</ul>
<div>
When the automation happens, one of the push back is the job lose. Having a person in the domain is the deciding factor for project success. The PowerShell provide a comfortable environment for biz people to start and grow, which makes my biz part team stable. </div>
<div>
<br /></div>
<div>
To use a widely adopted language also stabilize the dev side. I witness many effort to have an in-house language which is only used in that team. Many vendor solution provides a programming interface which few people know how to use. The customized solution literally blocks the people from mainstream technology and few people wants their career to be in this situation. If you search for PowerShell, you can still find job openings requires that skill. This option really helps the adoption of the solution. With .net core moves to Linux platform, PowerShell 6 and all other .net languages will flood into Linux world. So the future is really bright. </div>
<div>
<br /></div>
<br />
So far I am very happy about the result!<br />
<br />
BTW: if you want to use F# to write PS module, here is the <a href="http://apollo13cn.blogspot.com/2013/11/use-f-to-write-powershell-snapin-cmdlet.html">link</a>.<br />
<br />
<br />Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-29642700850123778542018-06-12T10:36:00.002-04:002018-06-12T10:57:56.735-04:00My DevOp Project<div>
I am into devOps for a week. The basic building block is to automate those systems. I have to say the PowerShell is a good tool to do this kind of work from a Win32 box. My first task is to call some services in a web site. I can find those services URL from the web page but need to go through some authentication processes.<br />
<br />
The Invoke-WebRequest cmdlet + Chrome's network monitoring turns out to be the perfect solution to do this work.<br />
<br />
First step is to understand how the web page performs.<br />
<br />
I open the Chrome network monitoring tool by right click the web page and select "inspect". It shows the current DOM structure of displaying page. Click the "network" tab and select 'preserve log". The "preserve log" feature will keep all the logs even the page changes.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkxCGqG8LbLnK8uh4HzU2fNkXyKCzlYGYEt2drH3ai7GVDN0vHKOSCNABTRkMn6MuEqgL1JJMXZn-eVpY8mLNAWIH7Mv3ALx7BZVgL1ohDhPfnLVNIcv9hSsRyikm3LU16aVLmatZX7-A/s1600/NetworkMonitoring.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="279" data-original-width="524" height="170" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkxCGqG8LbLnK8uh4HzU2fNkXyKCzlYGYEt2drH3ai7GVDN0vHKOSCNABTRkMn6MuEqgL1JJMXZn-eVpY8mLNAWIH7Mv3ALx7BZVgL1ohDhPfnLVNIcv9hSsRyikm3LU16aVLmatZX7-A/s320/NetworkMonitoring.PNG" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Now the target web page address can be pasted to Chrome and lots of log will be generated. I can see many page direction and form submission during this process (sorry, I cannot paste screenshot. 😉)</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Step 2: Write down all the pages "network" tab shows after the initial web page is loaded. Please write down those web addresses. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Now the simulation part starts. There are two variables needed for Invoke-WebRequest cmdlet:</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<ul>
<li><b>SessionVariable</b>: Browser maintains a session between client and server. So "-sessionVariable" parameter is a must. It will create a mutable variable to store session information between client and server, e.g. cookies. </li>
<li><b>MaximumRedirection</b>: PowerShell can automatically redirect to next page if you set the value to this variable > 0. I recommend to use 0 as initial test so you can know what is going on exactly.</li>
</ul>
<div>
Now let's the fun begin. </div>
<div>
<br /></div>
<div>
Step 3: open PowerShell </div>
<div>
Step 4: run $r0 = invoke-webrequest -uri <br />
<address>
-SessionVariable sessionVarialbe0 -MaximumRedirection 0</address>
</div>
<div>
<br /></div>
<div>
This will create the session variable "sessionVariable0". There is an error shows the maximum direction step is smaller. Please ignore this error. </div>
<div>
<br /></div>
<div>
The cmdlet returns a $r0 variable, please check the $r0 content and you will find the web page address and redirection address from header field. Make sure the redirection address equals the one you get from Chrome network monitoring page (in step 2)</div>
<div>
<br /></div>
<div>
Step5: $r1 = inovke-webrequest -uri <address2> -webSession $sessionVariable0 -MaximumRedirection 0</address2></div>
<div>
<br /></div>
<div>
Repeat the check performed on $r0.</div>
<div>
<br /></div>
<div>
if you cannot find the redirection address, please refer to "Form" variable in the $r1. You can invoke the address in the Form's action field and submit the fields in the HTTP request body part. For example:</div>
<div>
<br /></div>
<div>
$addressInForm = $r1.Forms[0].Action</div>
<div>
$r2 = invoke-webrequest -uri $addressInForm -SessionVariable sessionVarialbe0 -MaximumRedirection 0 <b>-Method Post </b>-UseDefaultCredentials <b>-Body $r1.Forms[0].Fields</b></div>
<div>
<b><br /></b></div>
<div>
you might find the default browser opens, that's because PowerShell uses default browser to parse. The browser might show some error message, please ignore. </div>
<div>
<br /></div>
<div>
You can repeat the above steps until you get the final destination. And the sessionVariable0 is the treasure from this journey, it contains all the information you need for future automation work. </div>
<div>
<br /></div>
<div>
Happy <strike>hacking </strike>automation. ;)</div>
<div>
<br /></div>
<div>
</div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<br /></div>
<div>
<script>
!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");
</script></div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-6416402426584649692014-07-29T23:04:00.001-04:002014-07-29T23:04:15.801-04:00SelfNote: Credit Risk Model<div>
<u>My first paper on Credit Risk model.</u><br />
<u><br /></u>
<u>www.fonsrisksolutions.com/Documents/Using%20Default%20Rates%20FAJ.pdf </u>is a good good paper. The unusual pattern "crisis-at-maturity" model is quite interesting to me.</div>
<div>
<script>
!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");
</script></div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0tag:blogger.com,1999:blog-4100091590694628472.post-18541525872426906472014-03-02T22:02:00.003-05:002014-07-29T07:37:17.655-04:00Setup Azure cloud on Windows Server<div>
<div style="background-color: white; color: #222222; font-family: 'Times New Roman', serif; font-size: 12pt; line-height: 14.65pt; margin: 0in 0in 0.0001pt; vertical-align: baseline;">
<span style="color: #3d3d3d; font-family: Helvetica, sans-serif; font-size: 10pt;">Service Bus is Azure Cloud solution, but it can be installed on your own hardware which must run Windows Server 2012.<u></u><u></u></span></div>
<div style="background-color: white; color: #222222; font-family: 'Times New Roman', serif; font-size: 12pt; line-height: 14.65pt; margin: 0in 0in 0.0001pt; min-height: 8pt; vertical-align: baseline;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: 'Times New Roman', serif; font-size: 12pt; line-height: 14.65pt; margin: 0in 0in 0.0001pt; vertical-align: baseline;">
<span style="border: 1pt none windowtext; color: #1f497d; font-family: inherit, serif; font-size: 10pt; padding: 0in;">Here are the links I think you need:</span><span style="color: #3d3d3d; font-family: Helvetica, sans-serif; font-size: 10pt;"><u></u><u></u></span></div>
<div style="background-color: white; color: #222222; font-family: 'Times New Roman', serif; font-size: 12pt; line-height: 14.65pt; margin: 0in 0in 0.0001pt; vertical-align: baseline;">
<span style="border: 1pt none windowtext; color: #1f497d; font-family: inherit, serif; font-size: 10pt; padding: 0in;">Documentations: <a href="http://msdn.microsoft.com/en-us/library/windowsazure/dn282144.aspx" style="color: #1155cc;" target="_blank"><span style="color: #3778c7;">http://msdn.<wbr></wbr>microsoft.com/en-us/library/<wbr></wbr>windowsazure/dn282144.aspx</span></a></span><span style="color: #3d3d3d; font-family: Helvetica, sans-serif; font-size: 10pt;"><u></u><u></u></span></div>
<div style="background-color: white; color: #222222; font-family: 'Times New Roman', serif; font-size: 12pt; line-height: 14.65pt; margin: 0in 0in 0.0001pt; vertical-align: baseline;">
<span style="color: #3d3d3d; font-family: Helvetica, sans-serif; font-size: 10pt;">Installations and configuration instructions: <a href="http://msdn.microsoft.com/en-us/library/dn441394.aspx" style="color: #1155cc;" target="_blank"><span style="border: 1pt none windowtext; color: #3778c7; font-family: inherit, serif; padding: 0in;">http://msdn.<wbr></wbr>microsoft.com/en-us/library/<wbr></wbr>dn441394.aspx</span></a><u></u><u></u></span></div>
<div style="background-color: white; color: #222222; font-family: 'Times New Roman', serif; font-size: 12pt; line-height: 14.65pt; margin: 0in 0in 0.0001pt; vertical-align: baseline;">
<span style="color: #3d3d3d; font-family: Helvetica, sans-serif; font-size: 10pt;">To get the software: <a href="http://msdn.microsoft.com/en-us/library/dn441412.aspx" style="color: #1155cc;" target="_blank"><span style="border: 1pt none windowtext; color: #3778c7; font-family: inherit, serif; padding: 0in;">http://msdn.<wbr></wbr>microsoft.com/en-us/library/<wbr></wbr>dn441412.aspx</span></a><u></u><u></u></span></div>
<div style="background-color: white; color: #222222; font-family: 'Times New Roman', serif; font-size: 12pt; line-height: 14.65pt; margin: 0in 0in 0.0001pt; min-height: 8pt; vertical-align: baseline;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: 'Times New Roman', serif; font-size: 12pt; line-height: 14.65pt; margin: 0in 0in 0.0001pt; vertical-align: baseline;">
<span style="color: #3d3d3d; font-family: Helvetica, sans-serif; font-size: 10pt;">the WebPI folder download folder is %LocalAppData%\Microsoft\Web Platform Installer\installers. You can get the WebPI installation file there.<u></u><u></u></span></div>
<div style="background-color: white; color: #222222; font-family: 'Times New Roman', serif; font-size: 12pt; line-height: 14.65pt; margin: 0in 0in 0.0001pt; min-height: 8pt; vertical-align: baseline;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: 'Times New Roman', serif; font-size: 12pt; line-height: 14.65pt; margin: 0in 0in 0.0001pt; vertical-align: baseline;">
<span style="color: #3d3d3d; font-family: Helvetica, sans-serif; font-size: 10pt; line-height: 14.65pt;">the service bus connection string can be retrieved by using Get</span><span style="border: 1pt none windowtext; color: grey; font-size: 10pt; line-height: 14.65pt; padding: 0in;">-</span><span style="color: #3d3d3d; font-family: Helvetica, sans-serif; font-size: 10pt; line-height: 14.65pt;">SBClientConfiguration. Please make sure you import the certificate from the server to your local computer. Make sure use </span><span style="font-family: Calibri, sans-serif; font-size: 15px; line-height: normal;">Get-SBAuthorizationRule to get the connection string with shared key.</span></div>
</div>
<div>
<script>
!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");
</script></div>
Taohttp://www.blogger.com/profile/10181624105046973883noreply@blogger.com0