ONLamp.com: Exploring E4X with Ruby

Published on ONLamp.com (http://www.onlamp.com/)
See this if you're having trouble printing code examples

Exploring E4X with Ruby

by Jack Herrington
08/12/2004

E4X is a new standard for XML access in ECMAScript (JavaScript). Recently, I had the good fortune to meet some folks on the steering committee and to sample a demo. The idea is simple -- XML access using SAX or DOM is too difficult, so make it easier. Take this simple XML as an example:

<transactions>
   <account id="a">
       <transaction amount="500" />
       <transaction amount="1200" />
   </account>
   <account id="b">
       <transaction amount="600" />
       <transaction amount="800" />
       <transaction amount="2000" />
   </account>
</transactions>

With DOM, you would read the tree into memory and then use methods such as getChildren to fetch all of the nodes of the root, and then traverse those nodes using iterators and more methods. Often, you would also need to write code simply to avoid the whitespace nodes inserted into the tree to preserve formatting.

There's nothing inherently wrong with the DOM API. The problem is that the API is heavyweight and often obscures the algorithm being implemented. Take this example, which reads and traverses the example XML file in Ruby using the REXML API:

require 'REXML/Document'
out = {}
doc = REXML::Document.new( File.open( 'test_data.xml' ) )

doc.root.each_element( 'account') { |account|
   out[ account.attributes[ "id" ].to_s ] = 0
   account.each_element( 'transaction' ) { |trans|
       out [ account.attributes[ "id" ].to_s ]
			+= trans.attributes[ "amount" ].to_s.to_i
   }
}

p out

REXML is an excellent XML API that is very easy to use. However, it still obscures the basic algorithm, which in this case sums up all of the amounts by ID and print the result. This program generates this output:

{"a"=>1700, "b"=>3400}

It's not much to look at, but it is the right answer. The sum of account a is 1,700 dollars and account b is 3,400.

If we step back a little bit and look at XML in the large, isn't there a better way to read and write XML?

A Little XPath

One possibility is to use XPath, which make it easier find nodes in the tree. An example use of XPath that is similar to the original code is:

require 'REXML/Document'
doc   = REXML::Document.new( File.open( 'test_data.xml' ) )
total = 0

REXML::XPath.each( doc, '//@amount' ) { |amount| total += amount.to_s.to_i }
print "#{total}\n"

In this case, we are just totaling up all of the amounts into a single number. (We'll talk a little more about XPath later.) Suffice it to say that even with XPath, working with XML doesn't feel like working with data objects.

E4X

The E4X working group obviously thought so, and they figured that ECMAScript was the right language to start with. The resulting standard extends the ECMAScript language with syntax-level extensions for XML, which turn a DOM into a "dot notation"-capable data structure. For example, this code, in E4X ECMAScript:

var id = doc.account[0].id;

sets the id to a, assuming that the doc variable contains the example XML above.

You can use standard iterators such as for each on the XML variables, and you can set a variable to XML as easily as you can read it. For example:

doc.account[0].id  = "foo";

would set the id. This code:

doc.account[0] = <account id="a"><transaction  amount="200" /></account>;

would rewrite that entire first account node with new XML. Notice how you can even write XML as clear text in the code.

The great value of E4X is that it opens up XML to a wider audience by making it simpler to use. For an XML fan like me, that is music to my ears. How can you use it today? E4X is still in-process, and there is only one example implementation on top of the Rhino JavaScript engine.

E4X for Ruby

Perhaps I could use Ruby, my favorite scripting language, to implement something close to E4X. Why Ruby? For several reasons:

The excellent REXML API for reading and writing XML makes a fine starting point.
Ruby supports extensive operator overloading.
Ruby supports overriding the behavior of the language when a method is not found on a class, which makes it easy for a class to mold itself to the structure of the XML data.
Ruby's latent typing will make it a lot easier to wrap and contain the elements of REXML I will use.
Ruby is cool.

Of course, it's this last reason I like the best.

Ruby is very similar to Perl and Python. Its major advantage is its readability. It's often been called "executable pseudo-code." To learn more about Ruby, check out the official Ruby site as well as Dave Thomas' and Andrew Hunt's excellent book Programming Ruby. The book is available for free online.

Let's jump in and see what we can do to make reading XML easier with Ruby.

Reading XML the Easy Way

To test how much we have simplified reading, I will use the original amount-totaling example on the test data file and rewrite it to use a new API:

out = {}
doc = NodeWrapper.new( REXML::Document.new( File.open( 'test_data.xml' ) ) )

doc.transactions.account.each { |account|
   amount = 0
   account.transaction.each { |item| amount += item._amount.to_i }
   out[ account._id ] = amount
}

p out

Instead of calling each_element, I can now just use the node names in the XML as if they were attributes of the class. How do I do that? By wrapping REXML nodes with my own class:

Figure 1. Wrapping REXML nodes

This wrapper will then allow access to the node using the dot notation syntax. Here is the Ruby code for the wrapper:

class NodeWrapper
   def method_missing( name, *args ) 
       name = name.to_s
       if ( name =~ /^_/ )
           name.gsub!( /^_/, "" )
           return @node.attributes[ name ].to_s
       else
           out = NodeListWrapper.new()
           @node.each_element( name ) { |elem|
               out.push( NodeWrapper.new( elem ) )
           }
           return out
       end
   end

   def initialize( node )
       @node = node
   end

   def to_s() @node.to_s; end

   def to_i() @node.to_i; end

end

That was easy, but what does it mean? The trick is in the missing_method class. Ruby is a dynamic language, so it doesn't require a class to define all of its methods at compile time. In fact, there is no compile time at all. When Ruby attempts to call a method on an object, it first looks up the method directly. If it's defined, then it calls it. If not, then it looks in the base class, and so on. If there are no methods defined with that name, it calls method_missing with the method name and the arguments.

This is where our code comes into play. We look for the names of child nodes and attributes and make them appear as real methods by overriding missing_method.

Why does that work for attributes? One would think that in order to use this missing_method process, that the client code would have to read:

doc.transactions().account().each

However, in Ruby everything is a method call. Calling the method transactions without the parentheses is the same as calling it with them. That's why the simple dot notation works:

doc.transactions.account.each

Not everything is so simple. Notice that we can run the each method on the account attribute. Why? Because we don't return a node. We actually return a node list that wraps an array. This is an array of nodes:

Figure 2. An array of nodes

The code for the node list looks like this:

class NodeListWrapper < Array
   def method_missing( name, *args )
       name = name.to_s
       self[0].send( name, args )
   end
end

The trick here is to make the default method lookup apply to the first element of the array. This is how:

doc.transactions[0].account[0]._id

is equivalent to:

doc.transactions.account._id

I compromised a little by using the underscore to hint the system as to what is an attribute versus a node, but I think it's a small price to pay.

The next step is to enhance the model to handle writing as well as reading.

XML Writing Made Easy

I really liked E4X's model of putting XML inline into the code. I couldn't push Ruby that far, because the language already defines the less-than and greater-than symbols. I settled on:

doc += xml <<XMLEND

<account id="bar">
   <transaction amount="100" />
   <transaction amount="200" />
</account>
XMLEND

This will add a chunk of XML to the document using the plus operator. The new xml keyword is really just a function:

def xml( xmldata ) NodeWrapper.new( REXML::Document.new( xmldata ).root ); end

It creates a new REXML document, and then wraps its root node in a NodeWrapper to make it easy to access. To make the plus happen, I had to add some methods to NodeWrapper:

class NodeWrapper
   def method_missing( name, *args )
       name = name.to_s
       if ( name =~ /^_/ )
           name.gsub!( /^_/, "" )
           if ( name =~ /=$/ )
               name.gsub!( /=$/, "" )
               _write_attribute( name, args[0] )
           else
               _read_attribute( name )
           end
       else
           xpath( name )
       end
   end

   def initialize( node )
       @node = node
   end

   def to_s() @node.to_s; end

   def to_i() @node.to_s.to_i; end

   def _add ( nodes )
       @node << nodes._get_node
       self
   end

   alias :<< :_add

   alias :+ :_add

   def _get_node() @node; end

def xpath( name )
       children = NodeListWrapper.new()
       REXML::XPath.each( @node, name ) { |elem|
           children.push( NodeWrapper.new( elem ) )
       }
       children
   end

private

   def _read_attribute( name )
       @node.attributes[ name ].to_s
   end

   def _write_attribute( name, value )
       @node.attributes[ name ] = value
   end

end

I broke out method_missing to make it a little clearer about what it does. I also aliased << and + to the _add method. This method in turn uses REXML's << method on a node to add a set of nodes from one tree into another.

Now, to test the upgraded NodeWrapper class, I will add a new account into the tree after reading the file:

out = {}
doc = readxml( 'test_data.xml' )
doc += xml <<XMLEND
<account id="bar">
   <transaction amount="100" />
   <transaction amount="200" />
</account>
XMLEND

doc.account.each { |account|
   amount = 0
   account.transaction.each { |item| amount += item._amount.to_i }
   out[ account._id ] = amount
}

p out

The xml method creates the new tree, and the plus operator handles adding it into the document. I also added a new readxml function, which takes a path name and returns a wrapper to the root node of the XML object.

One last step is to integrate XPath to make things even easier.

Adding XPath Support

Our new NodeWrapper supports an XPath method that returns a node list of wrappers:

   def xpath( name )
       children = NodeListWrapper.new()
       REXML::XPath.each( @node, name ) { |elem|
           children.push( NodeWrapper.new( elem ) )
       }
       children
   end

XPath allows you to specify a set of nodes in an XML document in a way similar to specifying files in an operating system. As paths are to a file system, an XPath is a path within an XML document. Every node and attribute in any tree has a unique XPath.

XPath also supports wildcards and will return a set of nodes that match. For example, this code:

total = 0
doc.xpath( "account[@id='a']//@amount" ).each { |amount| total += amount.to_i }
print "#{total}\n"

returns the total for just the account with the id value of a. This code:

total = 0
doc.xpath( "//@amount" ).each { |amount| total += amount.to_i }
print "#{total}\n"

returns the amount sum for the entire document, regardless of account.

This just barely scratches the surface of the power of XPath. It's important to have easy access to XPath features in any XML API.

Caveats

This article was an experiment in creating an E4X-style API by using the power of the Ruby language. It doesn't cover the entire standard, but it does provide some perspective both on the value of E4X and on the flexibility of scripting languages. Perl and Python both provide the equivalent of the missing method system shown here, so it's possible to do something similar to this in either of those languages.

With statically typed languages, such C++ or Java, you will run into the problem that the nodes and attributes are not defined at compile time. One alternative solution is to use a code generator to build classes from an XML schema definition that will provide dot-notation syntax for read and write access. Unfortunately, the code will be specific to one particular XML schema. For run-time flexibility, you will need to use the DOM or SAX method of reading and writing.

Finally, there is very little published information about E4X so far. This article relies on what I could glean from the hour-long presentation I attended. If I have made some mistakes in the E4X syntax, I apologize. I'm pretty sure I nailed the highlights, even if the specifics may vary somewhat.

Conclusion

I've written plenty of articles recently using Java as the language. In comparison, writing the code for this article was a blast. One of the great things about Ruby is that it makes writing code really fun because it works the way we think.

Writing code and working with computers should be fun. I hope this article gives some reasons to try and simplify XML access to make it fun for everyone. If that helps Ruby out a little bit in the process, so much the better.

Jack Herrington is an engineer, author and presenter who lives and works in the Bay Area. His mission is to expose his fellow engineers to new technologies. That covers a broad spectrum, from demonstrating programs that write other programs in the book Code Generation in Action. Providing techniques for building customer centered web sites in PHP Hacks. All the way writing a how-to on audio blogging called Podcasting Hacks.

Return to ONLamp.com