January 4, 2012
OMG Jury Duty

Back and at a loss. I have been in Jury Duty since … well my last post! Now I have a new child on the way, a pending real estate deal, and a job to get back to … it may be awhile =/

October 5, 2011
over(heard) & under(stated): I have some news to share.

I just wanted to highlight the great write up our head of new products put out with the release of WSJ Social:

mbaratz:

Today, I’m excited to unveil a project I’ve been working on over the last few months: WSJ Social, a standalone news publication on Facebook.

Having recently moved from startup hub San Francisco — where I mused about the future of newspapers in the app economy — to media hub NYC, where I…

September 20, 2011
WSJ Social, my latest and greatest…

Ok so here it is, the reason I have been so absent in social media. We have been working day and night!

WSJ Social

So I won’t go into to much detail but I did want to give a high level overview of the stack and performance.

In the browser: HTML5, CSS3 and Dojo

"Web Servers", 2 tiers

  1. Node 4.11 - calling RESTful services and rendering views (ejs) Express, Underscore, Klass, Step, and several others

  2. Java - RESTful services written with Restlet and Spring, hosted on Virgo OSGi

Database - Mongodb =)

Continuous integration w/ Maven and Jenkins

All manner of testing: functional, unit, static analysis JUnit, Vows, JSLint, Findbugs, etc …

Monitoring -

Monit monitoring every server process, Cluster monitoring node instances, Jenkins doing some url testing of external services. Basically if someone sneezes we get a very detailed report. We can act before the user can pick up the phone =)

All requests served to browser < 100 ms and capable of handling thousands of request per second with a small amount of infrastructure

I also want to thank the small but dedicated team that worked to build this awesome new platform for the Journal. They are all brilliant, this project went from design to production in less than 3 months.

It is still beta but pretty solid. We still have many things to work out, like SSL, in the next couple weeks, so stay tuned!

July 18, 2011
Never Fear

Just an update. I am still here and this remains my goto place for long form posts. I just have been busy as usual. I’m still very active around the web and you can check me out in the following places:

http://www.linkedin.com/in/scottrahner
http://github.com/scott2449
http://j.mp/scottrahner (Google +)
http://www.twitter.com/scott2449
http://about.me/scottrahner

Working on some juicy stuff, so if I get a chance to breath maybe I can write! Also feel free to check out my dozen or so previous posts, some interesting stuff there (I think)

April 25, 2011
Dojo JsonRestStore

I recently have been doing a lot of work in Dojo and sadly much of that work is modification/augmentation of the existing code to meet our needs, both extending and monkey patching.. First, I have to say when it comes to JavaScript frameworks the Dojo Data api and now Dojo Object Stores are not the best abstract data layer available, but they are also not he worst. They have made a good effort and honestly it is hard to avoid the “code my back end to match the framework JSON” model. If you want my opinion Ext/Sencha is still much stronger in this department, but Dojo is their only serious competitor and it’s also totally free!

Considering the lack of documentation and flexibility of some of the core parts of the data api I thought I would share some applicable work here (especially when you notice the dirth of user generated help on this topic as well.) In this case, some of the recent changes I made are very specific but some are widely re-usable. There is a lot I learned while messing around with this stuff, so as usual if you have any questions or just want to discuss things, send me a DM @scott2449 or comment below.

So what am I sharing? Well two bits of code that overcome weakness or enhance the JsonRestStore specifically. This component is located in the dojox part of the library and provides the tools to take a Restful web api that supports JSON as a return/content-type and will pump that into various dijits (dojo widgets) “automagically” for you. The idea is great and sort of avoids the “code my back end to match the framework JSON” problem I mention above by selecting two well known standard to help lock in the interface, JSON and REST. Problems are 1) Rest is not well defined and many people break what rules and principals exist 2) JSON is not strict semantically. This means that there is still no contract locking in URLs, return code, JSON structure etc… Considering that, the JsonRestStore makes two assumptions: 1) Restful URIs can only be of the form /Resources/id, obviously many out there do, but this is a wild simplification 2) Your JSON must have an items collection, as in, {items:[{},{}]} (This is actually a restriction that most dijits enforce, not the store)

Phew, so finally on to my fixes 1) I solved the items issue but not using any dijits and instead creating my own, which is for another story 2) I needed URIs of the form /Resource/id/NestedResouce/id/etc…. 3) I needed custom headers, which no Dojo Data classes support!

After digging and thinking for a while here is what I came up with:

1) I extend JsonRestStore to allow a custom request object, something that is built into the Rest service but that the JsonRestStore fails to leverage:

You’ll notice that all I am doing is overriding the creation of the Rest service to pass 2 additional arguments, namely the request information as a constructor param so that you can modify it however you like: dojox.rpc.Rest(options.target, true, null, options.getRequest), see usage.

2) A monkey patch replacing the resolveJson method of dojox.json.ref. dojox.json.ref is the workhorse of the JsonRestStore, is does a complex toJson, that walks the json and creates an index to track dirty records etc … It is also used to generate the restful URIs, so the patch below, which is about 5 line into the resolveJson method, generates more complex ids:

If you read carefully you will see that with JSON like this:

{a:{
    id:1,
    b:{
        id:2,
        c:{id:3}
    }
}}

will be indexed as follows:

/a/1             {id:1,b:{id:2,c:{id:3}}}
/a/1/b/2         {id:2,c:{id:3}}
/a/1/b/2/c/3     {id:3}

If you were to say modify c, it would now result in a PUT to /a/1/b/2/c/3. Dojo, left to it’s own devices would do a PUT to /target/id which turns out to be /a/3 which is totally WRONG! Instead we traverse the JSON structure and produce a matching URI, of course it still is “code my back end to match the framework JSON”, however, it is a very common and reusable standard.

Any questions? Let me know!

April 21, 2011
Soon

I have been so busy … but I want to make a promise to write about the following:

  1. OSGi, part 2
  2. A node server I wrote to parse the Google Buzz fire hose
  3. Patches I made to Dojo to help deal w/ Restful apis/interfaces

Other ideas:

Jenkins? Hosting on maven central repos?

I am hoping the public statement will make me feel obligated =)

March 30, 2011
OSGi, Part I

My long overdue post on OSGi… I decided trying to cover all the details on my schedule would be ludicrous, so instead I am going to give brief intro and then show you how to let maven help you create OSGi compatible jars.

First, what is OSGi. That answer is actually quite simple, it is a container. Much like EJB, Servlet, or JEE, OSGI is just another java based container that adds value and makes doing certain things in java easier (and some harder). Just as servlets let your java code function easily on the web by handling all the nity gritty of sockets, http, and syncrhonization for you, the OSGi container tries to overcome a few specific weaknesses of the java classloader. Mainly allowing the following:

  1. Multiple version of the same classes and packages to exist in the same JVM
  2. Manage dependencies between different versions
  3. Allow code to be easily swapped in and out w/o ever stopping the JVM

Now there are a varity of features in OSGi that help achieve this, the service registry, life cycle management, etc. To put it simply however, all code is split up into special jars which OSGi calls bundles. These bundles are just regular jars with a bit more data in there MANIFEST.MF file. The container gives each bundles it’s own classloader, much like a servlet container gives each ear it’s own classloader. Then it tells each classloader about only the packages that bundle imports via the extra tags in MANIFEST.MF.

An example: Let’s say you have package A and package B and you want all code in A to use logback version 0.8.x and all code in B to use version 0.9.x. As you know in a traditional JVM the last version in the classpath will be assigned to both A and B. However if you were to make A, B, and both versions of logback OSGi bundles you could easily say A import-package:ch.qos.logback;version=”[0.8.0,0.9.0)” and B import-package:ch.qos.logback;version=”[0.9.0)”. This means A can use any version from 0.8.0 up to but not including 0.9.0 while B can use anything from 0.9.0 up to but not including 1.0.0. With me so far? If you need some more reading you can check the wiki http://en.wikipedia.org/wiki/OSGi or some of the great docs from any of the OSGi container sites. Also before I go on, just like with servlets where you have IBM, Sun, JBoss, and many others making servlet and JEE containers, there are also many providers of OSGi containers. Some popular implementations include: Spring DM a.k.a. Virgo, Felix, and Equinox. Equinox is the OSGi container that runs the popular IDE Eclipse…. yes many of the popular java based IDEs and servers you use are running on OSGi.. very meta!

Why wait! I’ll post this now and get out part 2 soon.

February 28, 2011
nodeJS: an experience

I just spent about 4 hours diving into node seriously for the first time. It is very cool, very slick, and very fast … but it has poor documentation, in part because the devs iterate so damn fast. You’ll see my resulting code below, it’s a framework for something I need at work. It’s tiny and sounds sad that it took so long .. but I had to search all the docs, on github, and ask a few questions on irc to get it all straight.

Enough babbling Here’s the code, w/ comments so there can be a real example in google results for some common node use cases. I’ll list my specifc troubles building it below.

Simple right? Well in order for me to understand all this I had to learn about, express, middleware, EJS templates, partials, clientResponse, etc….

Here’s a break down of the time sucks:

  1. So now I had my data, which I knew needed to be rendered with a template, but all I could find in the documentation was about referencing models w/ the locals variable. I figured that couldn’t be all there was, ONE WAY to add data to a template! After 20 mins searching, the painful answer was yes … so you have to get remote data yourself and shoe horn it into locals… which brings me to my next item.
  2. Now I needed the equivalent of URLConnection in node to pull my data out of the cloud:
    • I spent 15 minutes trying to figure out if express had this functionality, nope, it only helps with serving content
    • I spent 15 minutes looking in node JS documentation and found nothing
    • I asked on the IRC channel and got an immediate answer, http.get (which can be confusing considering server.get is the opposite)
    • went back to the documentation (another 15 min) to see if I could find it now, still no … I mean it’s there, in pieces, with no context about the kinds of problems it solves.
  3. Now I needed to figure out the event callbacks for get. I am very familiar with javascript, callbacks, and events. Still, it took me another 15-20 mins to get right. I was probably just overthinking the chucked encoding part.
  4. For at least 30 minutes JSON.parse was broken for me, or so I thought. nodes error messages are very poor so I thought the library was not being imported. I was installing, uninstalling, reading, trying random crap. Turned out I made the dumb mistake of having var body; instead of var body = “”; which was putting undefined into the string. In the browser this would have been immediately apparent .. maybe I just need to learn more about node exception handling and logging.
  5. Layouts make sense and are simple but I spent more time searching around the web on the assumption that they were more powerful or complex, nope… just a place to stick your html and head tags =(
  6. Just as I was thrown off by variations of get, I was also thrown by server reponse vs client response.. I have been network programming for a long time, but because of javascript namespace ambiguity it just didn’t dawn on me.

Like I said very cool, but for a collection of subjects I am very familiar with: javascript, networking, etc … man, this learning curve was unexpected. Hopefully you can all learn from my experiences and have an easier time. Anyway I learned many more lessons during and since, so feel free to dm me @scott2449 or ‘disqus’ below.

February 12, 2011
Lexers, and Parsers, and Grammars! Oh my!

Ok, so for the first time since my career started I have actually developed something using what I learned in school. Well, ok, I mean I learned Java in school, but this is certainly the first time using some so … academic.

It started with the fact that I need to develop something very similar to http://code.google.com/apis/gdata/docs/2.0/reference.html#PartialResponse in order to limit large XML/JSON responses down to something manageable inside a set of restful services. Unfortunately Google doesn’t have anything available to parse the syntax and nothing else (like XPath/XSLT) can function to select multiple nodes (at different depths) all at once inside a data structure.

So what to do? Write my own language .. no problem! I had been exposed to ANTLR briefly before and I vaguely remembered BNF Grammars from compiler class so I figured what the heck. Here are the results. First I needed a grammar, so I download ANTLRWorks and after some time spent learning the ANTLR sytax and some debugging I wound up with:

Kinda hard to explain in a blog post how this works, but if you read all the stuff above after some time it should make sense how this grammar would take something like this:

name,id,othernodes(name),nodes(other,nodes(id))

and parse it into an AST like this:

AST

Now after clicking a few buttons inside ANLTRWorks it will generate the lexer and parser classes in Java (or whatever language you like). I won’t post those here, as with all generated code it’s not very readable anyway. Plus it’s encouragement to try yourself! Your welcome to dump my grammar in there.

Now the last step is to create what’s know as a “walker” in compiler speak. A set of functionality that can crawl the parsed material and take action or translate to another representation … just like a C compiler re-writes your code to assembler. So my walker code though, recursively, “straight forward” is probably too large to post here, but … the idea is it takes my new filtering language:

name,id,othernodes(name),nodes(other,nodes(id))

and any nested set of simple java objects:

and produces a JSON representation:

If you do want to look at the code, just send me a tweet @scott2449. You can also take a look at some tree walker info here: http://www.antlr.org/article/1170602723163/treewalkers.html

January 24, 2011
I swear …

The original point of this “blog” was to talk about development work and technologies. I promise to talk about some of the things I have been working with like OSGi, Restlet, Freemarker, and Dojo/Dijit.

Liked posts on Tumblr: More liked posts »