Http client libraries, Threads, Google finance, Google charts and Factor (Part 1)

(Small disclaimer, this entry won't introduce the actual cool code that performs a request against the google chart or google finance api. I was kind of ham-stringed because of my web proxy; so I will have to work on that later).

As I mentioned in one of my earlier entries, I generally tend to gravitate towards writing simple, throw-away web client tests. Just simple stuff like testing if a server or URI is available, what is the response time. What do the returned server headers look like. How many concurrent connections can the server handle. Here is some example code that does just that. In my next entry, I want to build a simple wrapper for the google finance web api and google charts. Google charts as what you might expect. It is a simple web api for generating chart images. You provide the X and Y axis data. Google chart generates the chart.

The developer api would have to generate a URL something along the following:

http://chart.apis.google.com/chart?cht=lc&chs=200x125&chd=s:BFeKZ&chxt=y&chxr=0,0,10

Google finance has an api for gathering historical data on historical stock information.

http://finance.google.com/finance/historical?cid=694653&startdate=Jan+10%2C+2006&output=csv

Threads

This isn't really related to google charts or finance but I was kind of curious how the factor libraries handled threading. The threading library is easy to jump into. You provide a quotation code block and call the "in-thread" word definition and the block of code runs parallel with other threads that are running. If you want to see example usage see the 3D graphic examples (e.g boids), the http server or the other web applications.

Here is the in-thread help documentation from the factor help system (\ in-thread help)

"The new thread inherits the current data stack and name stack. The call stack initially contains the new quotation only, so when the quotation returns the thread stops. The catch stack contains a default handler which logs errors to the stdio stream."

A worker thread that concurrently performs requests against a web server comes out like this:

USING: threads

: worker ( -- )
max-worker-delay random dup sleep
"Worker sleep done at t=" write pprint " ms" print
[ main-request-url http-get pprint nl ] time
"Done " print ;

: run-manager-daemon ( -- )
max-daemon-delay random dup
sleep "Manager thread done at t=" write pprint " ms" print ;

: init-botchart ( -- )
loop on
"INFO: launching loop" print
max-worker-threads [ [ worker ] in-thread ] times
run-manager-daemon ;

The init word definition does the bulk of the work by launching 10000 worker threads. The worker thread is composed of operations that do a HTTP GET request against the URL and also the time word determines how long the operation executes.

Download source for example:

http://haskellnotebook.googlecode.com/svn/trunk/factor/botchart/

Reviving the factor code and a discussion of the Factor based Hobby Operating System

First, I would like to apologize for using blogger as a blog host. Some of the past factor code on this blog is highly unreadable and probably won't compile with the most recent version of Factor. I have moved a couple of the simple examples to a google code subversion repository and I will try to stay on top of keeping the code in sync with the most recent Factor builds and images.

Use the most recent code

In getting back to Factor, I was able to compile Factor from the git repository pretty easily under cygwin.

git clone http://factorcode.org/git/factor.git

(git output)
got 9e702a358399c7200e456d3097bb024eda0c2cbc
got a059b9bd38d03225223ed17b855676da30f7889b
got 75782c7e4db9563ffeb7bf9a92377f7305ad0dd2
Checking 2465fills out....
100% (2465/2465) done

I added the no-cygwin flag to the gcc compile flags and simply typed make
-mno-cygwin

(As discussed in Doug Coleman's blog)
http://code-factor.blogspot.com/2007/04/building-factor-in-cygwi-now-suppported.html

I got the most recent bootstrap image (win32 x86) from factorcode.org and then compiled the words into a factor-nt.image. This is all described in the documentation, but I just wanted to mention it to confirm that it does actually work. (For example, trying to compile the most recent version of X11under cygwin might be more of an issue. Compiling factor for windows was pretty straight-forward).

Code under google-code

The simple examples are just that, but they can be run like you would run a script from any other open source scripting language (e.g. python)

• http://haskellnotebook.googlecode.com/svn/trunk/factor/examples1/hello1/

I copied the hello world example straight from the repository but added a couple of more comments to remind you about the word definitions and how to look them up through the help system. You will also notice that the example is structured for running in a shell environment.

hello.sh is a simple (a lazy/easy approach) bash script that invokes the factor binary and image and also launches the myhelloworld vocab. The vocab should have a MAIN: word definition that calls your particular entry point code.

time $FACTOR_BIN -i=$FACTOR_IMG -e="\"$VOCAB_NAME\" run" -quiet -run=none

Startup times comparred with Java Example

I also included a similar Java source example just to see how long it would take the java virtual machine to startup.

$ time java HelloWorld
Hello World
real 0m0.108s
usr 0m0.015s
sys 0m0.000s

$ ./hello.sh
Compiling 1 words...
Hello wrld real
l 0m0.145s
user 0m0.015s
sys 0m0.015s

Note: the directory and file structure is important
Note: the directory and file structure is important.

hello.sh - shell script that invokes the factor executable to load the myhelloworld module.
factor_bin_conf.sh - simple include script with factor binary paths

myhelloworld/ - myhelloworld module and subdirectory
* authors.txt - simple text file, list of author names
* myhelloworld.factor - source code with hello word definition
* tags.txt - whitespace separated list of tags to classify the vocabulary
* summary.txt - one line description of the module

• http://haskellnotebook.googlecode.com/svn/trunk/factor/examples1/csvparser1/

The majority of example code in the csvparser example comes from Phil Dawes. He wrote a parser and included source that parses different common separated formats from file. In my client csv parser application, I load all the data from a simple repository of URLs and then count then number of URLs. Over the next couple of more posts I plan to add a bit more functionality like map-reduce words, and some URL analytical tools. This particular Botlist URL data drop contains over 100,000 URLs and associated titles and keywords.

http://www.phildawes.net/blog/2007/10/02/baby-steps-with-factor-a-csvparser/

wget http://openbotlist.googlecodmp.tar.gles/botlist_datadump.tar.gz

When running the factor version against a similar Java version, I get the following results:
$ ./parse.sh
Compiling 10 words...
Running simple csv parser example
Opening file, testcsv.txt
Number of lines in file:
85364
Done.

real 0m9.885s
user 0m0.015s
sys 0m0.000s

(see the java directory)
Running the java Application and 174,000 lines of URL data:

$ time java AnalyzeURLFiles botlist_datadump.dat
Analyzing Files
Reading file=botlist_datadump.dat
Total Lines:174008
Completed in=1859 ms

real 0m1.936s
user 0m0.031s
sys 0m0.000s

The java version probably performed a little bit better over all because IBM's JVM is optimized for an Intel dual core machine. Also, I believe I didn't compile factor win32 correctly. But, my goal was only to compare and contrast run times without any tweaking or performance enhancements.

• http://haskellnotebook.googlecode.com/svn/trunk/factor/examples1/httptest1/

The last example opens a socket connection to a HTTP web server and sends a basic HTTP/1.0 GET request and retrieves the response. I have written several of these examples in the past and I will develop a more robust Factor oriented http load testing library as I do a lot of analysis of HTTP traffic.

Full Download

http://haskellnotebook.googlecode.com/files/factor_examples1.tar.gz

Part 2: A Factor Operating System (maybe)

I have been developing small hobby operating systems off and on for a while now (more off since I have been doing web development over the last 6 years or so). When I was in college, I leaned more toward systems engineering than pure computer science (hence the Electrical Engineering degree). My interest originally stemmed from religiously studying the Minix (a small microkernel operating system) source code and then early versions of Linux. Over time I created simple 100 line systems. Some in nasm assembly, some in gcc assembly. Some on Motorola chipsets, some of old intel cpus. It was fun, none of the code was useful but still some interesting projects.

In 2002, I created an abandonware operating system project for the intel x86 (I worked on the project for a month or two and abandoned it). The premise was simple; initialize the operating system kernel. Boot from the floppy disk, initialize the interrupts, timers, keyboard driver, VGA driver. I never made it past the hard drive driver, filesystem, scheduling, and application loading. That is what I hope to do with in recreating OrangeLinux/AkitaOS as this new project, OctaneOS.

Basically, OctaneOS is a simple x86 hobby (currently only has a couple thousand lines of C/Asm code) operating system. I plan to finish the floppy and hard drive driver, file-system and then hopefully boot a customized Factor image. There is a lot of refactoring work that needs to be done on the old code and there are many driver pieces still missing but I was happy that it actually runs in the bochs emulator and compiles on a modern host system (Ubuntu 7.10).


http://code.google.com/p/octaneos/wiki/ProjectDiary
http://octaneos.googlecode.com/svn/trunk/octaneos/

(cross post) Factor screencast, basic usage.

(Image of the Factor UI, edited by Elie)

(Images taken from Elie's blog at http://fun-factor.blogspot.com/ he created them, I didn't)

The following screen cast was developed to show basic factor usage including analysis of the data stack, how one would go about looking up help on words, etc.

Also, working with the stack takes a little getting used to, that is why I went it over in the screencast. Elie has some great images that I want to show here of some basic operations. I go over a couple of them in the screencast.

"Let's take our rot example from above and practice it with the help of the illustration. Write 1 2 3 4 in your input area and hit Enter (the Return key) on your keyboard. You should see the start stack displayed in your stack area. Notice the position of the top and bottom of the stack in your stack display area. Now write "rot" (without the quotation marks) in your input area and hit Enter. You should see the rot stack displayed as above because the "rot" word caused the 3rd stack item to be moved to the top while the 1st and 2nd stack item were pushed down." -- quote from Elie's blog on stack manipulation

In the second video (part two), I demonstrate the new vocabulary modules for structuring your factor application and a short demo on what your code would look like in emacs.

Screencast (1)





Resources

• http://factorcode.org/
• http://www.bluishcoder.co.nz/
• http://fun-factor.blogspot.com/
• http://useless-factor.blogspot.com/

Screencast

This screen cast was created gtk-recordMyDesktop. And then encoded with mencoder (ffmpeg could also be used).

• bbrown@houston:~/tmp/capture$ mencoder factorcaptureA.ogg.3 -o f4_part2.avi -ovc lavc -ss 09:58
• bbrown@houston:~/tmp/capture$ ffmpeg -i factorcaptureA.ogg.3 -t 00:09:58 factorscreenlin1.avi

Factor for novices updates

First, some comments on blogger.com. This blog (obviously) is hosted on blogger(blogspot).com. It is a pitiful blog service. I was convinced I could create a better personal blog in a couple of days but why waste my time when I already have so many entries on blogger. I think it should be up to blogger.com to improve their service. It would take a day to describe all of the issues, but there are three main issues that I have a problem with; doesn't work well with any of the opensource browsers (FF3, FF2, Opera). Right now, I am writing this without a focus caret and just hoping the text inserts where it is supposed to. This is on Firefox3 alpha. Not only that, it is amazingly buggy to the point I may have to resort to restarting my machine because the Firefox browser will lockup. Somehow blogger figured out a way to crash a linux operating system through a web service. I have to give them credit for that one.

Number Two: the wysiwyg tools are seriously lacking. Can't do highlight background. Can't really insert code or blocked text. Pasting whitespace like newlines will get removed.

Number Three: The templates just don't look that good. It would be nice to have a simple black/gray on white template but no. That is too simple for blogger so we are given overdone, over-trendy crap. Even jroller.com has improved in that department. Which is kind of disappointing, I didn't think I could ever give the jroller blog service a compliment.

With that being said, a lot of the reason you haven't seen many updates is because I took a break from blogging because it become too cumbersome to find a good service. I thought I would be at home on blogger but I was more than disappointed. Sure, I could go to another service or write my own but it seems like google (blogger was recently bought out by google) tends to put out great products. I was waiting for them to overhaul blogger. The strange part, googledocs allows me to post a document to blogger. Googledocs is 30 times better than the blogger service. And I have done that in the past, posted from googledocs to my blogger blog. But it just got too complicated.

On Factor

In the future I will probably post factor related posts to my new blog. I may cross-post to this blog if it isn't too much trouble. But, I would like to focus on one or two blogs as opposed to the eight that I have across the web.

You will find more updates on this blog (berlinbrowndev). The focus topics won't be entirely about Factor but I hope to include the progress of my factor related projects as often as I can.

http://berlinbrowndev.blogspot.com/

Thanks and tell blogger to get their act together.

More updates coming soon!!!!

I have been caught up in a project that I want clean up. More factor language for novice updates are coming. I am especially looking to update the mysql binding and show some examples.

Practical Factor, building a simple web test system

Overview:

You have built your application, put everything in place, all the pages are available; went through all of the pages with your browser. It sure would be nice to run some kind of script that could just hit all those pages and get some of the internal details like the headers that are output and the status code, maybe even the response time of accessing the time. I would say, for every web application I put together I always want to build such a tool. I could use some of the java or python oriented web test frameworks out there but they are either too much, too complicated, or too little.

Once again, it was easy to decide upon Factor for this task. The goal is to build a simple http client that connects to a webserver from a list of URLs in a file and print the status code, content, any 500 or 404 errors and keep a tally of the results.

Key vocabs:

USING: arrays compiler io kernel math tools xml xml-utils threads hashtables
prettyprint namespaces sequences sequences-contrib strings test http http-client ;


Key Words Used:

cond stdio contents with-stream time trim strip length

Loading the files

The first task is to create a simple URL loader. Which is basically reading a list of URLs from a file. In factor, you begin by opening the file reader stream and then getting the contents.

For example:

: load-all-urls ( filename -- seq )
<file-reader> [
stdio get contents
] with-stream
"n" split [ trim ] map ;

So, an array of URL strings are returned if you provide a filename. Here is example usage:

: run ( -- )
#! Load the list of URLs to test from the text file,
#! iterate through them and run the test url word
"apps/http-test/test_urls.txt" load-all-urls
[ [ load-test-url ] each ] time
test-report ;

The time word is defined in the factor api and is used to get the processing time for a particular block of code. The test-report word definition is defined in our library to print the number valid and invalid URLs. load-test-url contains the core code of our library which is used to connect to a single URL and extract the HTTP content.

: load-test-url ( url -- )
dup trim length 0 > [
[ "loading url... [ " write write " ]" print ] keep
[ test-http-get http-test-response nl ] time
] [ drop ] if ;

Clearly, the most interesting section of our code is contained in the http-test-response word. The input parameters include the HTTP status code, the HTTP headers returned and the content returned.

: http-test-response ( code headers string -- )
#! Process the returned HTTP code, headers and the content including
#! incrementing the failure and passed counts
-rot swap
dup "* Status Code: " write number>string write nl dup
{ {
[ 500 = ] [
[ [ swap "Key: " write write " | Value: " write print ] hash-each ] when
"* Response Content: " write print "------ End Content" write nl
inc-failed
] } {
[ 404 = ] [
"Warning: 404 File not Found error" print
dup [ [ swap "Key: " write write " | Value: " write print ] hash-each ] when
"* Response Content: " write print "------ End Content" write nl
inc-failed
] } {
[ t ] [
dup [ [ swap "Key: " write write " | Value: " write print ] hash-each ] when
drop
"Warning: Not printing content (successful download)" write
inc-passed
] } } cond ;

Redefinition of HTTP word definitions; I needed to redefine a couple of the default http client words in order to support
HTTP 1.1 and to include my own processing. Here are the HTTP oriented words:

: test-http-request ( host resource method -- )
write " " write write " HTTP/1.1" write crlf
"Host: " write write crlf
default-request-headers ;

: test-get-request ( host resource -- )
"GET" test-http-request crlf ;

: test-read-response ( -- code header )
#! After sending a GET or POST we read a response line and header.
flush readln parse-response read-header ;

That describes the most interesting word definitions.

The Full Source: (Note: the source is saved to one file, you will need to break it up into load.factor, driver.factor, httptest.factor)

(http-test framework in factor) http://docs.google.com/Doc?id=dq6cjjg_48273xs7

To Test the Application:

To test the application, download Factor and unzip the Factor version for your particular environment. This example was tested with Factor 0.88 under win32.

http://www.factor.org

Updates on the game, data structures, the camera

This will be a short post, basically wanted to keep you up-to-date on the game progress and some other interesting tidbits. I am basically trying to convert the C game which mentioned in the earlier post to Factor; it is interesting taking C structs and converting them to tuples. Removing things like 'global' variables and replacing them with stack manipulation. Here is the structure of my camera object:

C: mechbot-camera ( -- gadget )
[ { 0.0 0.0 0.0 } swap set-mechbot-camera-position ] keep
[ { 0.0 0.0 0.0 } swap set-mechbot-camera-angle ] keep
[ { 0.0 0.0 0.0 } swap set-mechbot-camera-rotation ] keep
[ -1 swap set-mechbot-camera-id ] keep
[ CAMERA_STATIC swap set-mechbot-camera-type ] keep
[ 0.0 swap set-mechbot-camera-centerx ] keep
[ 0.0 swap set-mechbot-camera-centery ] keep
[ 0.0 swap set-mechbot-camera-centerz ] keep
[ 3.2 swap set-mechbot-camera-zoom ] keep
[ -0.01 swap set-mechbot-camera-_tmpx ] keep
[ 4.1 swap set-mechbot-camera-_tmpy ] keep ;

It looks like a mess, but I was trying recreate the structure I used with the C game.
The C version:

typedef struct tagCamera
{
float position[3];
float angle[3];
float rotation[3];
float centerx;
float centery;
float centerz;
float Yaw;
float Pitch;
float Roll;
float zoom_factor;
float old_zoom;
int id;
int type;
} DriverCamera;

And so to update the opengl display with the camera position and look, we will use gluLookAt: The goal with this word is basically to extract the values from the camera object and place the values on the stack for gluLookAt.

: mechbot-glulookat ( v -- )
[ first ] keep [ second ] keep third
_TMP_LOOK_X POS_ZERO _TMP_LOOK_Y
POS_ZERO POS_ONE POS_ZERO
gluLookAt ;

: third-person ( mechbot-camera -- )
#! Fill gluLookAt from camera object
#! gluLookAt(eyeX, eyeY, eyeZ, centerX, centerY, centerZ, upX, upY, upZ)
[ mechbot-camera-_tmpx ] keep
[ mechbot-camera-_tmpy ] keep
[ CAMERA_HEIGHT swap mechbot-camera-zoom * ] keep
>r swap 3array r>
[ swap pos-camera ] keep
mechbot-camera-position
mechbot-glulookat ;

For actually moving the camera and handling keyboard input, the gestures are set for the mechbot gadget we are using.

mechbot-gadget H{
{ T{ key-down f f "UP" } [ mechbot-gadget-mechbot handle-camera-keys ] }
{ T{ key-down f f "LEFT" } [ mechbot-gadget-mechbot handle-camera-keys ] }
{ T{ key-down f f "DOWN" } [ mechbot-gadget-mechbot handle-camera-keys ] }
{ T{ key-down f f "RIGHT" } [ mechbot-gadget-mechbot handle-camera-keys ] }
} set-gestures