A Simpler Ajax Path

by Matthew Eernisse
05/19/2005

I began working with web applications back in the bad old days, when making an application behave like a desktop app meant wrestling with byzantine table-based layouts nested five and six levels deep, and horrid, hackish frame sets within frame sets within frame sets. Those were the days.

Things have steadily improved for web developers with the advent of standards-compliant browsers, CSS, DHTML, and the DOM. Pervasive broadband access has made web apps feel a lot snappier. Now something called the XMLHttpRequest object makes it even easier to develop full-blown, superinteractive applications to deploy in the browser.

While not exactly new, the XMLHttpRequest object is receiving more attention lately as the linchpin in a new approach to web app development, most recently dubbed Ajax (asynchronous JavaScript and XML), which powers the cool features found on sites like Flickr, Amazon's A9.com, and the new poster children for whizzy web-based interactivity, Google Maps and Google Suggest. The snazzy Ajax moniker seems to be getting some momentum--it's popping up in all sorts of places, including the Ajaxian weblog and the recent Ajax Summit put together by O'Reilly Media and Adaptive Path.

Cool acronym or not, when I decided a while back to add a long overdue Search Playlist feature to my webcast radio station, EpiphanyRadio, it seemed like a good opportunity to show off some of the features the XMLHttpRequest object offers. The Search feature accesses a PostgreSQL database of the tracks in current rotation and allows listeners to search by artist, song title, and other criteria.

Related Reading XML Hacks 100 Industrial-Strength Tips and Tools By Michael Fitzgerald

As it turns out, it's pretty easy to take advantage of the XMLHttpRequest object to make a web app act more like a desktop app--while still using traditional tools like web forms for collecting user input. I also found some great ways to handle server-side errors to make debugging less of a headache.

Introducing the Object

The XMLHttpRequest object allows client-side JavaScript to make HTTP requests (both GET and POST) to the server without reloading pages in the browser or resorting to iframe tricks or other ad hockery. Microsoft implemented the XMLHttpRequest object in Internet Explorer on Windows as an ActiveX object beginning with version 5. The Mozilla project added it in Mozilla 1.0 as a native object with a compatible API. Apple added it to Safari in version 1.2.

Note that despite its name, you can use the XMLHttpRequest object with more than just XML. You can use it to request or send any kind of data--keep in mind, though, that JavaScript processes the response from the server. The following example returns the data to the browser in a simple DSV (delimiter-separated values) format.

Preparing Form Data to POST

Other resources online covering the XMLHttpRequest object demonstrate how to use it with simple GET requests. The object is also capable of doing POSTs, which makes it a much more useful tool for creating web applications.

To use the POST method, pass data to the XMLHttpRequest object in query-string format (for example, ArtistName=Kolida&SongName=Wishes), which the object then sends to the server just like a normal form POST. You can use JavaScript to pull data a piece at a time out of web-form elements, and then format the data into a query string. If I wanted to do this for my playlist search function, I could use something like this:

var searchForm  = document.forms['searchForm'];
var artistName  = searchForm.ArtistName.value;
var songName    = searchForm.SongName.value;
var queryString = '?ArtistName=' + escape(artistName) + '&SongName=' +
        escape(songName);

Note: Be sure to escape (URL encode) the values.

Of course, keeping in mind that one of Larry Wall's three great virtues of a programmer is laziness (and in that respect I'm really virtuous), I wrote a general function to automate this process, to put all the data from a form into a query string. This function allows me to POST form data with the XMLHttpRequest object without having to do a lot of extra work every time. That makes it significantly easier to integrate the use of the object with my existing application code.

Here's a quick look at the first part of the function:

function formData2QueryString(docForm) {

        var strSubmit       = '';
        var formElem;
        var strLastElemName = '';
        
        for (i = 0; i < docForm.elements.length; i++) {
                formElem = docForm.elements[i];
                switch (formElem.type) {
                        // Text, select, hidden, password, textarea elements
                        case 'text':
                        case 'select-one':
                        case 'hidden':
                        case 'password':
                        case 'textarea':
                                strSubmit += formElem.name + 
                                '=' + escape(formElem.value) + '&'
                        break;

The variable docForm is a reference to the form from which to pull the data. This lets me reuse the function in other places.

The function iterates through the form's elements collection, using the type of each element to figure out how to retrieve its value. For each distinctly named element, it appends the name and value onto the query string variable strSubmit. It then hands off that string to the XMLHttpRequest object for the POST.

For most types of form elements, looking up its value property does the trick. However, radio buttons and check boxes require a bit more work. For check box sets, I create a comma-delimited string for the value, but you can handle them in whatever way suits you best. This function is a huge time-saver when working with the XMLHttpRequest object and collecting user data from a form.

The entire function is available for download if you'd like to try it.

Creating the Object

To create the object in JavaScript for IE, use new ActiveXObject("Microsoft.XMLHTTP"). In Mozilla/Firefox and Safari, use new XMLHttpRequest(). The first half of a simple function to create and use the object looks like this:

function xmlhttpPost(strURL, strSubmit, strResultFunc) {

        var xmlHttpReq = false;
        
        // Mozilla/Safari
        if (window.XMLHttpRequest) {
                xmlHttpReq = new XMLHttpRequest();
                xmlHttpReq.overrideMimeType('text/xml');
        }
        // IE
        else if (window.ActiveXObject) {
                xmlHttpReq = new ActiveXObject("Microsoft.XMLHTTP");
        }

To make this a reusable, generic function, I pass in three parameters: the URL for the processing page on the server, the query-string formatted data to submit, and the name of the JavaScript function that will process the response from the server (to invoke later through eval).

Note the addition of the overrideMimeType method call in the Mozilla/Safari code. Without this, some people have reported that some versions of Mozilla lock up when the server returns anything other than XML. (I cannot confirm this issue, as I have not experienced this problem myself.)

Also, if you want to support older browsers, you can then test the xmlHttpReq variable and fall back to other methods of submitting data if the object is not present.

POSTing the Data

Here's the rest of the function, which submits the request to the server:

     xmlHttpReq.open('POST', strURL, true);
        xmlHttpReq.setRequestHeader('Content-Type', 
		     'application/x-www-form-urlencoded');
        xmlHttpReq.onreadystatechange = function() {
                if (xmlHttpReq.readyState == 4) {
                        eval(strResultFunc + '(xmlHttpReq.responseText;);');
                }
        }
        xmlHttpReq.send(strSubmit);
}

The open method here takes three parameters: the first sets the request method, the second is the processing page, and the third sets the async flag, governing whether the function continues executing immediately after sending the request or waits for a reply before continuing.

Note: According to the HTTP 1.1 spec, the request method is case-sensitive. It doesn't seem to matter using Internet Explorer, but in Mozilla if you enter the request method in lowercase, the request will fail.

The onreadystatechange property sets a callback function (for example, xmlHttpReq.onreadystatechange = handleResponse;) to execute when the readyState property changes. When readyState changes to a value of 4, the request has completed. The code above uses an anonymous function (instead of passing off the result to a separate function) that watches until the response comes back and passes the result as a string to the processing function.

Finally, the send method actually sends the request. It takes one parameter: the data to submit to the server. In the function here, this is strSubmit, the query string created from the form data with the formData2QueryString function earlier.

The Server Response

I mentioned earlier that despite its name, the XMLHttpRequest object works with other types of data besides XML. This makes me happy, because the search function for my internet radio station returns very simple tabular data of track listings. Returning results as DSV instead of XML significantly reduces the size and complexity of the return data and simplifies parsing. (As Eric S. Raymond notes in the Data File Metaformats chapter of The Art of Unix Programming, "XML is well suited for complex data formats ... though overkill for simpler ones.") You can use whatever separator you want, but for this I used a pipe (|) character.

The back-end code for the Search Playlist feature is Ruby code running under mod_ruby, as is most of the site. Ruby may not be as familiar to developers as PHP or Perl, but its flexibility, extensibility, and clean syntax make it an ideal web development platform.

nRowCount    = sth.size
strContent += nRowCount.to_s + 10.chr + 10.chr
sth.each do |row|
        strContent += row['artist'] + '|' + row['song'] + '|' + 
                row['album'] + '|' + row['comment'] + 10.chr
end

In this example, sth is an array containing the result of the database query performed with the Ruby DBI module. The sth.each do |row| line may look a bit odd to the non-Rubyist, but it's an example of an iterator/block combination, one of the many interesting and powerful features Ruby offers. In this case, as you may have guessed, it's pretty much equivalent to a foreach in other languages.

The 10.chr is a linefeed character. Essentially this chunk of code writes out the row count followed by two linefeeds, then writes out each returned row on a single line with the fields separated by pipe characters. A sample search result looks like this:

4

Kush|New Life With Electricity|The Temptation Sessions||
Kush|Plaster Paris (Part Two)|The Temptation Sessions||
Kush|Reverse (Part One)|The Temptation Sessions||
Kush|The Beauty of Machines at Work|The Temptation Sessions||

The two pipe characters together at the end indicate a blank field for the comment column.

Processing the Response

When the response comes back from the server, the XMLHttpRequest object can access it through two properties: responseXML (as an XML document) and responseText (as a string). Because I eschewed the unnecessary complexity of XML here, the code passes the responseText off to a JavaScript function for the processing and display of the returned data with this line from the original function:

eval(strResultFunc + '(xmlHttpReq.responseText;);');

This takes the function name passed into the xmlhttpPost function and executes it using eval, passing the XMLHttpRequest object's responseText as a parameter.

After the code has split the string into an array, you have several ways to use it to populate a table for display. Not being a huge fan of the DOM table modification methods (like XML, they're just too verbose for my tastes), I generally take the straightforward approach of using innerHTML. Here's a sketch of the JavaScript I use to process the result for my playlist search:

function displayResult(strIn) {

        var strContent = '<table>';
        var strPrompt = '';
        var nRowCount = 0;
        var strResponseArray;
        var strContentArray;
        var objTrack;
        
        // Split row count / main results
        strResponseArray = strIn.split('\n\n');
        
        // Get row count, set prompt text
        nRowCount = strResponseArray[0];
        strPrompt = nRowCount + ' row(s) returned.';
        
        // Actual records are in second array item --
        // Split them into the array of DB rows
        strContentArray = strResponseArray[1].split('\n');
        
        // Create table rows
        for (var i = 0; i < strContentArray.length-1; i++) {
                // Create track object for each row
                objTrack = new trackListing(strContentArray[i]);
                // ----------
                // Add code here to create rows -- 
                // with objTrack.arist, objTrack.title, etc.
                // ----------
        }
        
        strContent += '</table>';
        // ----------
        // Use innerHTML to display the prompt with rowcount and results
        // ----------
}

The Ruby code on the server separated the row count from the actual data rows by two linefeed characters; this function thus pulls out the count by splitting the entire results string on two linefeeds and using the first item in the resulting array.

The actual data rows are in the second item in that array. The data rows have single linefeeds separating them, so another split on a single linefeed of that item creates the main array of the data to write to the page. To create the table content, the code iterates over that array, creating a trackListing object for each row and using that object to make an HTML table row. The trackListing function creates trackListing objects:

function trackListing(strEntry) {
        var strEntryArray = strEntry.split('|');
        this.artist       = strEntryArray[0];
        this.title        = strEntryArray[1];
        this.album        = strEntryArray[2];
        this.label        = strEntryArray[3];
}

It splits the pipe-delimited string for each row and sets named properties for each object matching the column names in the database. You could omit this bit and instead use numbered array items for each column back in the main function, but I think it's easier to refer to them with names.

Handling Errors

The XMLHttpRequest object has a huge upside: it allows JavaScript to communicate directly with the server without loading a page in the browser. Unfortunately, however, that also becomes its downside when the inevitable Bad Things happen on the back end. If you usually work with languages that can return errors directly in the browser window, it can feel like flying blind when you're trying to debug a page that uses the XMLHttpRequest object--especially in environments where you don't have easy access to server error logs.

The object does have a status property, which contains the numeric code returned by the server (for example, 404, 500, and 200), and there is an accompanying statusText property, which is a brief string message. In the case of a server-side (code 500) error, this message merely states the obvious "Internal Server Error," which might be OK to display to the user but is pretty worthless for debugging.

The normal 500 error page returned from the server quite often contains extremely helpful debug information such as the error type, the line number on which the error occurred, and even a full backtrace of the error. Unfortunately, with the XMLHttpRequest object, all that stuff ends up buried in a JavaScript string variable.

It's actually fairly simple to retrieve the full-page 500 error messages and to accommodate debugging in a more elegant way. To accomplish this I had to add code to the original function for creating and using the XMLHttpRequest object:

if (xmlHttpReq.readyState == 4) {
           strResponse = xmlHttpReq.responseText;
           switch (xmlHttpReq.status) {
                   // Page-not-found error
                   case 404:
                           alert('Error: Not Found. The requested URL ' + 
                                   strURL + ' could not be found.');
                           break;
                   // Display results in a full window for server-side errors
                   case 500:
                           handleErrFullPage(strResponse);
                           break;
                   default:
                           // Call JS alert for custom error or debug messages
                           if (strResponse.indexOf('Error:') > -1 || 
                                   strResponse.indexOf('Debug:') > -1) {
                                   alert(strResponse);
                           }
                           // Call the desired result function
                           else {
                                   eval(strResultFunc + '(strResponse);');
                           }
                           break;
           }
   }

The case statement handles the response from the server differently depending on the xmlHttpReq.status value. It hands off the response to an error-handling function in the case of a full-blown error, but it still allows for a simple JavaScript alert to display a simple error message to the user (perhaps, "Error: e-mail address does not match the one for this account."), or to print a little debug message for the developer. (You could also print out those types of errors in nicely formatted text to a div somewhere on the page.)

Here's the function that creates the full-screen 500 error page:

function handleErrFullPage(strIn) {

        var errorWin;

        // Create new window and display error
        try {
                errorWin = window.open('', 'errorWin');
                errorWin.document.body.innerHTML = strIn;
        }
        // If pop-up gets blocked, inform user
        catch(e) {
                alert('An error occurred, but the error message cannot be' +
                        ' displayed because of your browser\'s pop-up blocker.\n' +
                        'Please allow pop-ups from this Web site.');
        }
}

The try/catch is important because of the ubiquity of pop-up blockers. If the user has blocked pop-ups, at least she has the option of allowing them so that she can see and report the error properly.

If the user allows pop-ups, a server-side 500 error will produce a new window containing the all-too-familiar page displaying all that helpful info you need to track down and squash the bug. (You might decide you'd rather display the error message in the current window, but my playlist search is a small pop-up window that will not display the entire message properly. Most standard-issue 500 error pages assume a full-size window.)

Notes

The XMLHttpRequest object transmits cookies with its requests, just like normal requests from the browser, so you don't have to do any special gymnastics to make server-side sessions work.

I haven't had much luck using the same object for multiple requests, particularly when invoking it from multiple windows. Use a new XMLHttpRequest object for each request. That will also save you trouble when doing multiple, asynchronous requests.

Requests made with the XMLHttpRequest object do not affect the browser history. That can cause serious user confusion, because clicking on the browser's Back button may not change things back to their previous state. In cases where you need history steps to correspond to user actions, you're better off using an iframe to make HTTP requests, because it creates history entries for each request.

Conclusion and Future Plans

The XMLHttpRequest object provides some serious kung fu to help web developers make web applications more responsive and dynamic--and make them perform more like desktop apps. With a little setup beforehand, such as the formData2QueryString and handleErrFullPage functions to handle the quirks of working with the object, you can get in on the Ajax action without having to alter your development process significantly.

Once you've started down the Ajax path, of course, you begin to see other places in your app where it would be a great fit. I can already see using it with the Previously Played feature on EpiphanyRadio. It would be a snap to have the XMLHttpRequest object poll the Song History page on the SHOUTcast server periodically. I could then have JavaScript pull the song list out of the HTML page response and write it out with DHTML to a div. The list of previously played tracks would continuously update without ever having to reload the page in the browser.

I could use it on the listener log-in page, to display nicely formatted log-in error messages on the original log-in screen instead of redirecting to a completely different page. I could use it to provide a continuous update of the number of current listeners by polling the main SHOUTcast site and pulling the numbers out of the response using a smart regular expression.

Other References

• Dynamic HTML and XML: The XMLHttpRequest Object
• Using the XML HTTP Request object
• XMLHttpRequest
• Using XMLHTTPRequest in Mozilla
• IE7 XML Extras (This is a third-party JavaScript add-on for IE 6.)

Matthew Eernisse works for the Open Source Applications Foundation doing Web application development. He has been working and playing with Linux and other open-source software since 1999.

---

Return to ONLamp.com.