When it comes to Linux and open source issues, people slink, cringing, past my office for fear of accidentally triggering off an evangelic outburst. The last person to mention Microsoft in my presence is still receiving counselling. When a number of my colleagues announced their intention to conduct further investigations into the recently-released shared source C# compiler, code-named Rotor [1], I was appalled that they would be restricted to working on Windows or FreeBSD. "It already runs on one Unix base," I thought, "how hard could it be to make it compile under a different flavor?"
[1] Incidentally, as a paraglider pilot, the definition of rotor that I am most familiar with is: the wind turbulence behind obstacles that causes your wing to collapse, and you to crash and die.
I tracked down the release on Microsoft's Web site, surprisingly getting to the download page without my browser expiring. Not unexpectedly, I am expected to actively acknowledge a license agreement. After careful scrutiny, it appears that I am to be allowed to read the code without forfeiting all future rights to approach within 100 meters of a computer. (Although since writing this article, I have already had one warning from Microsoft about the manner in which I am complying with the terms of this license.)
I continue with the download. Tarred and gzipped ... this is looking
promising. Perhaps someone is getting a feel for this open source
business after all? Once expanded, the 10MB file grows to a 100MB monster
(and finally reaches 900MB once the build is complete). At initial
glance, it doesn't look too bad -- the use of autoconf is promising,
and it is with optimism that I run a quick configure, followed
by the instructed buildall.
This is when things stop looking so good. The first line of the build process seems to be trying to link C files into an object file. Justifiably, the system complains. At this point it is clear that some manual intervention is required. I check the documentation -- and start to get some idea of the way the system is structured.
The first part of the build process compiles a section known as the Platform Adaptation Layer (PAL). This essentially makes any other platform appear to be Windows -- at least from the point of view of the standard C library functions. The PAL comprises some 60 files -- and represents the area where the most effort needs to be expended when porting the system to a new platform.
I apply a bit of manual intervention at this point, grateful that compiling before linking has its benefits. Each file in the PAL demonstrates some quirks. Problems during this process fall into three categories:
There is an impressive piece of header file engineering in place,
in the form of rotor_pal.h and palinternal.h, which
rename, undefine, and redefine functions to support all permutations
of requirements. The working of these is complicated by Linux having
its own overlapping set of functions that sneak through some of the
gaps, leaving one never quite sure of which function one is using
or where it has been defined. I settle for the option of commenting
out all offending lines. These will be brought back in on an individual
basis as required.
Once all of the pieces have compiled and linked, I try out the single example in the directory examples. After a bit of tweaking,
it starts (and I finally find out who is responsible for calling main
in a C program -- for the answer send a self-address, stamped envelope
to ...), and stops, and then even prints out the debugging message
I added. A bit of self-congratulation is applied at this point, and
I explore further to see what is next.
At this point, I encounter something I haven't seen very often with open source systems: compliance tests that actually run the various pieces of code and check to see if they perform as expected. In this context, these are wonderful -- trying to build the rest of the system with the port of the PAL functions in an uncertain state would have been well-nigh impossible.
For interest, the number of test failures during the porting of the PAL is shown in Figure 1.
Figure 1. PAL Compliance Test Results.
The brief setback at trial 4 is a result of finally figuring out how the header files work, a process that initially broke more things than it fixed.
The compliance tests are comprehensive, and informative as to the nature of any problems. I have only spotted a single test with which I refuse to comply -- the one for random numbers that insists that no element may recur in a series of 10 random numbers.
Correcting some of the other problems makes me aware of incompatibilities
between FreeBSD and Linux. The format string %05d for the
printf function is fairly universally accepted as printing
an integer, with leading zeroes. The string %05s is apparently
somewhat more ambiguous: do you still add the leading zeroes (required
for PAL compliance) or provide leading spaces (as assumed by Linux)?
Other similar printing issues also keep manifesting in the test failures,
which eventually prompts me to grab the source of these functions
from that nearby FreeBSD system and to use these to replace calls
to the native Linux print routines. Once again, a triumph for open
source. The mix of GNU and Shared Source licenses should keep the
lawyers employed for a while.
Other areas that require work are the networking functions and process management. Most of the problems with the networking functions involve translating the Linux error value into the value expected
by the compliance test for that scenario. Process management requires
some changes in signal names to their Linux equivalent, and some rather
processor-specific register reporting. More challenging is the lack
of a suspend option in the Linux pthreads package. This fortunately
can be replaced by some juggling with signal handlers -- and has the
beneficial side effect of uncovering a rather sneaky race condition
in the process termination code.
Fixing the various problems is a process that extends over a number of weeks. Since each individual problem is relatively small and can be corrected in a short time, this process provides a delightful "hacking fix" between preparing lectures, marking tutorials, and terrorizing research students.
A number of outstanding issues still remain in the PAL: input equivalents of the print problems described earlier, still more networking constant re-labelling issues, and about three serious problems that will require substantial re-engineering (which I decide to defer until there appears to be justification for the effort).
By the time I complete this exercise, I have identified the pieces
required to convince autoconf to build working makefiles,
and the PAL is building to completion in an aesthetically pleasing
manner. I now go back to the Rotor root directory and unleash the
build process on the rest of the system.
To my horror, I discover that autoconf is merely a thin veneer
over something more primeval. It is starting to look as though the
PAL is required just so that nmake and the other utilities
reminiscent of Visual C can be compiled. These then take over the
rest of the compilation process and drive the construction of the C#
compiler and interpreter.
Between some exciting debugging and repeatedly running off to find another machine from which to kill the processes that have gone berserk, draining my machine of every spare byte [2], I eventually get these tools compiled and begin to let them plough through the 100 billion lines [3] of code making up the bulk of the system.
[2] There must be a technical term for this operation.[3] I exaggerate. Slightly. At last count I saw about 580 000 lines of C code, another 320 000 lines of header files, and 660 000 lines of C#.
I cringe as the compiler (good old g++) begins throwing up syntax
errors. The prospect of individually hand-fixing that amount of code
is daunting. Fortunately, closer observation shows that most of the
problems fall into a small number of categories:
g++ has always
been smarter than me in dealing with these, anyway. These messages do,
however, seem to indicate a rather intimidating dependence on word
size and address space of a particular architecture.
# and ##
are used. [4] These are used to turn preprocessor tokens
into strings and concatenate them.
It seems recent versions of the preprocessor have imposed new restrictions
on what can be used between the tokens, and this code violates them.
Fortunately, the problems are confined to a small set of header files;
an exhaustive search of all possible ways to arrange all of the
non-alphanumeric symbols on the keyboard reveals a formulation that
is acceptable to the preprocessor without altering the original intent
of the header files.
[4] I resist the urge to comment about the use of # following
a single uppercase C.
or as shorthand for object reference.
I hadn't realized that this is a reserved word, but g++ certainly
did. A global search and replace is tempting, but I quickly realize
that a two-letter string would occur in too many other places -- and
while it was tempting to have every file copyrighted by the Microsoft
Cobjrefpobjrefation, I eventually settle on the path of tedious
supervised substitution.
At this point, I start the build process and a few hours
pass before I need to exercise any manual intervention. Executable files
are starting to materialize, and I spot several that I need to reach
my ultimate target: the hello world program mentioned as
the second step in the quick start process.
The build seems to be jamming consistently, so I decide to short-circuit
the process and jump directly to the goal. Invoking csc hello.cs
immediately informs me that I am missing some DLLs. Missing DLLs on
a Linux box! At this point in the past, previous software products (most notably
an early commercial office package, purportedly for Linux) have been
drag-and-dropped to the (physical) trash can. Strangely tolerant in
this case, I apply a little manual intervention in the build process
and the missing pieces begin to appear.
With these additional pieces present, the compiler now churns happily
(it's quite snappy, actually) and disgorges something entitled hello.exe.
[5] The application of another utility, clix,
is required to
produce the anticipated global greeting message. Running it as instructed
produces ... nothing. No message printed, alas. On the other
hand, no ominous error message requiring backtracking through
two million lines of code. In fact, not even the command line prompt
-- the interpreter has wandered off into never-never land.
[5] Tirades relating to .exe files on a Linux box have been deleted in interests of sticking to the point.
At this point, I am on the verge of discarding the whole thing. The error is either in the compiler, meaning I might have to wade through the complex data structures involved in that sub-system, or in the interpreter, which means dealing with the virtual machine -- the part responsible for the bulk of the compilation time.
A few days later a synapse fires and I realize that the executable
produced should be portable across all of the supported platforms. A
number of my colleagues are subjected to a barrage of hello.exe,
and they quickly confirm that the executable not only works on their
systems, but is also byte-for-byte identical to their version of the
executable program. Kudos to the compiler writers -- cross-platform
compatibility on the first attempt.
Enter gdb, the Gnu debugger. At this point, a number of things
become clear:
Eventually, I am triumphantly able to announce:
Hello World!
(As copied and pasted from the output of a genuine C# program running on a single processor RedHat Linux 7.3 system.)
A number of issues still remain to be resolved, but the port is currently usable for its primary purpose: experimenting with the system. Issues of networking conformance in the PAL, and of security in the compiler/interpreter, will need to be resolved before serious work can be done using these aspects.
Having got to this point, I have realized that it is possible to port a system of this nature without needing an understanding of exactly what it does. I have invoked several of the applications, including the compiler and interpreter, with a very limited knowledge of C#, the nature of the intermediate language, compilation strategies, or execution environment. With the debugger, I have waded through morasses of code while still being able to identify the point at which things start misbehaving. Such successes I attribute to the nature of the programming; for the most part, sensible use of variable and function names make the code quite readable. For code explicitly acknowledged as pre-release quality, I am rather impressed. In many cases, it is clear where provision has been made for adapting and building extensions to the system, in the form of debugging structures, comments around the appropriate pieces, and test suites for relevant modules.
Another significant advantage when performing this port has been the availability of the compliance tests. When available in other systems, I have found these vaguely interesting but of limited usefulness. [6] Successfully porting the PAL would have been impossible without the comprehensive set of tests provided. In addition, the compliance tests have one unanticipated benefit. It is extremely rewarding to be able to make alterations to code and have one's changes immediately tested, and to receive confirmation in quantitative form that one is actually getting measurably closer to one's desired goal. Porting this code is fun.
[6] In one previous case while porting a virtual reality system to another language on a different architecture, the compliance tests indicated that the compiler was unable to distinguish between positive and negative numbers. I was somewhat amused to discover the system that I was porting (except for one minor module) continued working happily.
The Linux port of Rotor is now publicly available. Thanks to the rapid response and assistance from Brian Jepson at the O'Reilly Network, both the code and announcement of its availability have been spread further than I would have been able to do personally. I have received positive feedback from a number of sources -- including offers of assistance from several members of the Rotor team at Microsoft. A number of people have offered to host a CVS archive to coordinate further work on the system. Patches are already starting to filter in, supporting Linux distributions and working environments that I wouldn't have been able to test myself. The benefits of an open source development process are already clear.
Shaun Bangay is employed as an associate professor in the Computer Science Department at Rhodes University, where he teaches courses in Computer Graphics, Networks and Operating Systems.
Return to .NET DevCenter
Copyright © 2009 O'Reilly Media, Inc.