Kris Zyp's Questionable Assumptions

September 22nd, 2008
JSON a powerful simple expressive format for data structures with a high level of interoperability; implementations exist in virtually every popular language. However, there are certainly situations where developers often want additional constructs for effectively representing data that are not natively supported in JSON. Perhaps the most common usage of JSON is for the consumption of data in the browser. Most JavaScript libraries, parse JSON by using eval, and consequently are actually capable of full JavaScript expression evaluation, of which JSON is a subset. JavaScript expressions support a much wider range of constructs than pure JSON. Usually a simple JSON/JavaScript expression parser looks like:

function parseJson(jsonText){
return eval(‘(‘ + jsonText + ‘)’);
}
One of the most oft-desired data type that JSON doesn’t provide is a date type. Numerous creative, bizarre, weird, and silly techniques have been proposed for expressing dates in JavaScript. These methods often require extra parsing or walking strategies. Douglas Crockford’s reference library for JavaScript JSON serialization, serializes dates to strings by ISO format. I have written about deserializing these ISO dates. But, on deserialization it is not possible to determine if a value is actually a string or is really intended to be an actual date object. However, if the recipient of JSON is known to support full JavaScript evaluation (like the browser with a library using an eval), the solution for delivering date type value is simple, we can just use the normal JavaScript constructor:

{“mydate”:new Date(1222057313264)}
There are also several numeric entities that JavaScript provides that are not available, including Infinite, -Infinite, and NaN. I am not sure why you would need it, but undefined can also be transferred with JavaScript. Functions can be included as well:

{
“reallyBig”: Infinite,
“parsedString”: NaN,
“whatValue?”: undefined,
“doSomething”:function(arg){return doSomethingElse{arg};}
}
The data representations thus demonstrated are not JSON, they are JavaScript using the same object/array literal that JSON is based on. However, certainly one of the biggest benefits of JSON is it’s interoperability. If your data is going to be consumed by more than just JavaScript-parsing JavaScript libraries, you must make your data available in pure JSON format as well. This is well-handled through content negotiation. If you are using JavaScript expressions to transfer data, you should make sure your requests from the browser actually are specifying they can handle JavaScript:

Accept: text/javascript
Your server should be prepared to handle requests from clients that indicate that they only understand pure JSON:

Accept: application/json
Persevere, the JavaScript/JSON application and storage server, also provides support for parsing and storing extra constructs including non-finite numbers (NaN, Infinite), functions. Persevere can output the data as JSON or JavaScript (expression).

JSON is certainly powerful, expressive data format. This post is by no means an attempt to expand or lobby for the modification JSON. However, when it is known that consumers are actually JavaScript capable clients, it can often be advantageous to use the full power of JavaScript to represent data, while still providing JSON as representation for non-JavaScript capable clients.

February 12th, 2008
I suppose this post would be more prophetic a decade or two ago. It was in the 90’s that the HTTP protocol really became the Great protocol. It is foundation of the World Wide Web, and is language on which browsers were able to really open the doorway to the Internet for us. So am I little behind the times to suggest that HTTP now has an emerging future relevance? Is HTTP a relic of the past or does it have something to contribute to the future?

One of the distinctives of current Internet technological advance is in the growing realm of open sharing and utilizing data from disparate sources. Facilitating this progress is one of the principle goals of my work and this site. I want the Open Web to be more than just a bunch of pages that are developed without proprietary constraints, but for the Open Web to be the environment for open flow of information with intelligible interconnection of data that can give participants unprecedented leverage and permutations of capabilities. Mashups are a buzzword to describe this process. However, in order for information to flow rapidly, there must be commonly understood communication. JSON has enormous potential because it is so simple, expressive, and pervasive that it forms a excellent syntax for expressing data. However, JSON is not a transport. Two agents that wish to dialogue may understand JSON data, but they still need a mechanism to communicate and transfer that information. HTTP is almost ubiquitously the right choice for the transport. The incredible adoption of HTTP is main reason for this. No transport is more widely understood

What is wrong with HTTP?

Before going into the benefits of HTTP, let us look at the problems with the HTTP, or more specifically, THE problem with HTTP. The most fundamental problem with HTTP is that it requires that every single response must be preceding by one corresponding request. The specification describes HTTP as request/response protocol. This constraint has an enormous impact on the capabilities of HTTP. The first major impact is in hindering performance optimizations. In order to load a web page, every resource must be requested before the server can send the resources. This creates a signficant latency problems. There can be large gaps in transfers while servers are waiting to receive requests. However, a server could easily determine the most likely resources that a user agent will need and send them before the request if this constraint was not in place.

This constraint is also the fundamental cause of consternation with Comet development. Comet consists of efforts to allow servers to send messages to clients asynchronously instead of in immediate response to a request. Doing this requires creating an outstanding HTTP request that the server can respond to when it wants to. Comet push capabilities could easily be achieved if servers could simply send messages to the client without requiring a preceding request.

Throwing away the entire protocol because of a single issue is absurd, rather let’s fix or enhance the protocol. In recent articles I have discussed how non-request/response-bound HTTP messages can be sent within HTTP messages in order to deal with this problem using existing infrastructure.

So why is HTTP the right choice for future information transport?

It is so pervasive – HTTP is everywhere. It is how all browsers communicate with web servers. HTTP is understood by an overwhelming amount of software. Attempts to reinvent the functionality and capabilities of HTTP is essentially asking for this broadly understood language to be ignored in lieu of new one. Without very significant advantageous to new semantics and vocabulary, such attempts are generally either doomed to obscurity or worse, a cause in division in multiple semantics for the same thing, causing increased code complexity and costs.
It has the constructs for tomorrow – With ever increasing interchange of data in the future, more sophisticated and robust techniques for communicating data are needed. Many of these techniques already exist in HTTP, but have simply not yet been needed with yesterday’s technology. The future of high performance, intelligent data interchange will hinge on capabilities that already exist in HTTP including:
Content negotiation – HTTP includes vocabulary for negotiating between different formats.
Partial data transfers – HTTP has an extensible mechanism for sending a range of information.
Robust error handling – HTTP includes a comprehensive set of errors.
Parallel scalability – Perhaps one of the most impressive features of HTTP is how carefully designed such that demand can be easily scaled across numerous machines as HTTP proxy servers.
REST/CRUD semantics – HTTP provides semantics for basic create, read, update, and delete operations.
Performance improvements – HTTP pipelining has only begun to be utilized (only Opera has it turned on by default). Substantial performance improvements can be realized through pipelining.
Emerging technologies give us new leverage with HTTP – With traditional web application development, much of the workings of HTTP were hidden away by the browser and the server. However, the Ajax revolution gives developers far greater control of HTTP messaging. Most Ajax developers have simply used XMLHttpRequest as a means for communicating simple payloads of data back and forth to server, but XHR has given developers new access to the HTTP capabilities through their header metadata, and leverage to utilize the full semantics of HTTP for more meaningful communication.
Furthermore, with new XHR capabilities coming soon (FF3 will have cross site XHR support), XHR communication will involve much more than simply communicating with your own server. Communicating with your own server does not necessarily require widely understood vocabulary, but when communicating with other servers, the cost and efficiency of integration will be directly related to how much shared vocabulary can be utilized to provide jointly understood communication. New ways to utilize, extend, and leverage HTTP are being developed as well like the Atom Publishing Protocol.
Does it really matter what is on the wire? Can’t we simply distribute API based communication handlers? Consider this, is it easier to setup a TV to tune into the available TV stations, or is it easier to setup a printer to work with your new operating system? TV stations have standardized on a single format for broadcasting content. Connecting a TV to a station is as simple as turning it on choosing your station. On other hand, printers have no standardized communication with servers. Devices like printers use API based communication handlers (AKA drivers). With a huge number of different protocols for each printer, and different operating systems to interact with, there are an enormous amount of permutations of different drivers that must be developed, very prone to incompatibilities. While the situation has improved, many have experienced the frustrating effort that can go into trying to find the right driver for your OS/printer combination.

But in the realm of browser communication does it matter since we are all using JavaScript? Absolutely. There may be a single language on the browser, but there are different JavaScript libraries, with may prefer different APIs. In addition the browser environment can be very bandwidth constrained. Requiring another library for each data endpoint that you want to connect to, does not scale well. And as we move towards more service oriented architectures, browsers will not be the only consumers of information. Many other clients must be considered as well.

We need to be examining the HTTP specifications and learning how to leverage the power that is available, to maximize our potential in coming world of extensive data interchange and mashups. Next time you need to create Ajax communication to trigger CRUD operations, consider using the RESTful HTTP methods (PUT, POST, and DELETE). Do you need to deliver multiple formats of the resource? Consider using HTTP content negotiation. Do you want Comet capabilities in your application? Consider using an HTTP standards based approach. The more we can utilize what is there, the more widely we can be understood, and the more efficiently we can utilize the infrastructure of the web. The full utilization of HTTP can provide a solid foundation for the future of data interchange.

December 24th, 2007
JSONP is a proposal for performing cross site JSON data interchange by using script tag insertion with a callback to deliver the data payload. JSONP can be used in place of XmlHttpRequest in cross site situations. However, when using XmlHttpRequest, there are situations where the JSON data in content body is not the only relevant information, but the response headers, as well other aspects of the HTTP request and response may also contain important information. I propose that JSONP can extended in a small and simple manner to accommodate the transfer of header information in suitably interoperable manner.

I propose when custom headers need to be included in the request, that headers simply be included as parameters. When servers that are responding to JSONP requests want to include response header information in the response, they may optionally respond with a second argument that contains header information in the form of an JSON object/map. The object should contain properties with the names corresponding to header names, and values corresponding to header values. The request format can remain the same:

http://myserver/requestJSON?callback=mycallback&Accept=application%2Fjson
And the response would look like:

mycallback({“name”:”foo”},{“Content-Type”:”application/json”,”Expires”:”Mon, 24 Dec 2007 17:09:04 GMT”})
This addition to JSONP should generally not break existing JSONP clients, as normally only one parameters is used. This allows JSONP to emulate more semantically correct manners of transferring information that conceptually belongs in headers.

Usages could include transferring the “Last-Modified” header, so clients could determine the freshness of the data, sending If-Modified-Since header in requests and many more.

In addition to transferring request and response headers, I propose that additional HTTP information could also be transferred in pseudo headers. The following pseudo headers may be used:

request
httpMethod – Indicates the HTTP method used (GET, POST, PUT, DELETE, etc).
httpContent – Indicates the body of the content of the HTTP request (such as the POST body)
httpNoCache – With non-idempotent methods, you should generally include an additional parameter with a random unique value to ensure that the request is not cached.
response
httpStatusCode – Indicates the HTTP status code.
httpStatusText – Indicates the HTTP status text.
A more sophisticated example request could be:

http://myserver/requestJSON?httpMethod=POST&httpNoCache=23n9zs92l&httpContent={“name”:”bar”}&callback=call1
And the response could be:

call1([{“name”:”foo”},{“name”:”bar”}],{”httpStatusCode”:200,”httpStatusText”:”OK”,”Date”:”Mon, 24 Dec 2007 17:09:04 GMT”})
Of course, this enables a greater level of interaction, and therefore the prerequisite security warnings must be heeded. You can get yourself in a lot of trouble with XSS if you are not using proper explicit authentication schemes.

This weekend I released CrossSafe. Briefly, CrossSafe provides secure cross domain JSON requests and partially implements the JSONRequest specification (the get and cancel methods). Ajaxian covered the release and you can also read more about at the project page, see a demo, or download it. Rather than repeat the project description here, I thought it would be more interesting to describe the approach and elicit feedback on the future of secure cross site request.

CrossSafe uses nested iframes with a different domain than the parent window to setup a secure channel of communication with cross site servers. This works by passing a JavaScript object to a child frame and then using an alternate host name (like webservice.json.com) to prevent cookie access and changing the document.domain to prevent window, DOM and other JS environment access. By bringing this together, the dynamic script tag/cross site scripting approach can be used to retrieve cross site JSON data, and the scripts that are loaded from the other site are sandboxed. The parent window can retrieve data from these scripts, but the scripts can not access the parent window. This approach is also described here, but as far as I know, this is the first implementation. This implementation also follows the JSONRequest API specification, which allows you to use a standard API, and the library defers to a native implementation when it becomes available.

There is another approach for accessing cross site data securely that uses iframe proxies called fragment identifier messaging (FIM). Dojo has a good implementation of this approach. However, I believe this approach suffers from a couple of problems. First, it requires a level of server cooperation that has not been widely implemented yet. Servers must have Dojo’s iframe proxy script available on their site. On the otherhand, CrossSafe requires that server implement callback parameters which is already available with web services from Yahoo, Flickr, Codinginparadise.org’s transclusions, and JSPON. FIM also relies on polling to transfer data. Second, I have not done any tests to verify this, but I would be inclined to believe this is a slower approach as well. However, that said, I am interested in possibly implementing the JSONRequest.post method using this approach. The JSONP/XSS technique is only capable of making GET requests. The FIM approach on the otherhand does support POST. If I integrated Dojo’s FIM implementation in CrossSafe, all three JSONRequest methods could be available, and it could just be recommended to use the get method whenever possible because of the performance and interopability advantages.

Another issue with these approaches is that there are no real standards about how to do these requests since they require server cooperation. With the JSONP/XSS callback approach has seen various callback parameter names used including jsonp, callback, and jsoncallback. CrossSafe supports changing the parameter name, but it would be great if we could standardize this. Despite the fact that the original JSONP article proposed jsonp, I propose that we use the parameter name of callback. This is very succint and clear, and let’s face it, Yahoo is the most significant provider for JSON out there, and this is the parameter name they use.

The FIM approach could also use standardization. Dojo has the best implementation that I know of, but I believe OpenAjax.org is currently working on standardizing this as we speak. At least I hope…

Let me know if you have any thoughts.

testing

It yet another attempt to do a trackback, I am responding to my sister’s post “10 reasons I Love my Life”. Seems only appropriate to respond with a list of thankfulness:

1. The wonder of forever increasingly experiencing the glory and awe of God
2. The priviledge of forever interacting with the true king and hero, His Son
3. The cutest little girl in the world is most daughter
4. The most beautiful girl in the world is my wife
5. I am starting a really cool new self-directed R&D job, where I can work from home
6. I am moving to Utah, the most beautiful place on earth :). (OK, there are some more impressive places, but it ranks close to the top)
7. I have a wonderful family that loves me

• Here is my wish list for Christmas 2007: 
• Anything from the GFA catalog
• Utah Atlas: Nevermind, I found it
• The Wasatch Rock Climbing Guidebook
• Any other guidebooks for the area: Hiking Utah, Western Uinta Guidebook, American Fork Climbing Guide
• Renewal of my magazines, Climbing and Pray! (it is a rather ironic combination, isn’t it?)
• Enrollment at Momentum Climbing
• Stoli Blueberi

I am pretty sure I speak for the minority with this viewpoint. It seems as American as apple pie to believe that all good citizens should vote. However, I believe that there is fundamental flaw in logic that takes us from the founding principal of democracy that everyone be allowed to vote, to the idea that everyone should vote. That everyone can vote is essential to democracy. It is what ensures representation of the people in a fair and appropriate manner. However, is high voter turnout really important for democracy, or is informed voters more important?

The goal of democracy is that good decisions will be made by the government that properly represent the desires of the people. There are two parts to this: good decisions and representation. Good decisions require thorough understanding. In practice, democracy on a large scale is simply impractical. We must resort to a representational democracy. There are simply too many decisions that must be made by the government to ask everyone to thoroughly understand the issues. However, this concept is not just a necessary evil, that should be avoided as much as possible. In representational democracy, most decisions are made by people that study and research the issues. This is actually very beneficial. Rather than decisions being made by people that have only quickly glanced at the issues, decisions are made by representatives that understand the issues. This same concept should also apply to issues that are brought to the general public. My main point is simply that we should not be encouraging more people to vote, but rather that we should encourage people to only exercise their right to vote if they are really willing to research the issues.

The second goal of democracy is representation. If there are fewer voters will proper representation be lost. This can primarly be frame as a statistics question, with decent mathematical answers. In large populations, a small voter turnout has very little effect on representational accuracy. If a population of one million registered voters vote, and voter turnout dropped from %40 (typical for America), to %20, the impact on representational accuracy would be a fraction of a percentage. In short, if only highly informed people voted on issues, better decisions would be made, without signficant loss in representation.

James Bouvard said, “Democracy must be something more than 2 wolves and a sheep  voting on what to have for dinner.” How does democracy fail us in this situation? High voter turnout does not help this situation. Of course we can simply blame the wolves for the lack of consideration in voting, but in reality there are many situations, where the ethics are not so clearcut. The important aspect of successful representation that this example illustrates is that people’s opinions are not necessarily equal in the forcefulness. Some people are very close to indifferent on issues, the issue does not seem to have much impact to them. Others maybe deeply impacted by the issues. By encouraging everyone to vote, you are actually encouraging those who are impacted to lesser degree to a equal plane with high degrees of impact. By avoiding such encouragement, you allowing the natural process of less impact to result in lower turnout. Back to our example, if we had many wolves, there are probably lots of options for what to have for dinner. It is not a big deal for them, to choose something besides the sheep. Consequently, the natural inclination is for low voter turnout by wolves on this issue. On the otherhand, the sheep will naturally turn out in droves to vote on this issue. By simply allowing the natural process of allowing people to vote, but pushing them into, it is quite reasonable that voter turnout by wolves could be %30, and the sheep can win the vote (as they should).

Another aspect of voting is simply the cost of becoming informed enough to vote. If politics was the end goal of life, and good decisions by the government was the key to our happiness, we should certainly all spend every available moment researching issues and preparing to vote. However, this is not the situation. In fact, the time spent researching issues could be diverted to other activities, there is a vast amount of good that could be done. If half of voters decided not to vote and spend time researching issues, but rather taking that time and volunteering at the nearest rescue mission, what would be the impact? It would be hugely positive, with negligible impact on accurate representation.

Most Christians seem to think it is their spiritual duty to vote, but is that really what the Bible teaches? Actually this can go both ways. The Bible clearly teaches to protect the innocent. When issues come up where we can see the need to stand up for the innocent, we should clearly make our voices heard and cast our votes. Without getting into the issue, the Christian’s motivation for voting pro-life should be to protect the innocent. However, are most of the issues before us really about protecting the innocent? Or are they about making it easier for us to be Christian’s, or trying to get others to be more moral? Jesus said that if someone asks for your robe to give it him and more, and if someone asks you to go a mile with them, go two. I don’t believe this is just for individuals. The political implication: the church should be taking every opportunity to defer to the desires of the people around them.