The T-Files


Thu, 15 May 2008

YAPC::Asia 2008 Day One

For the third year in a row, the Shibuya Perl Mongers brought YAPC::Asia to Tokyo, this time on campus at the Tokyo Institute of Technology, which I hereby proclaim to be the best venue yet. Today's Day One was the second day actually, as there were also some additional talks yesterday (including one by co-worker-on-sabbatical Faiz), but I was tied up at work and could not go. Equally unfortunately, there is no T-shirt this year, and I could not get WiFi to work on the XO-1.

Larry Wall: A Standard That Is Meant To Be Broken.
A keynote that was a bit more on the technical side (they are usually more political or philosophical), in which $Larry talked about internals of the Perl6 parser he has been working over the last year. He stressed that there are not really any built-ins, and that the parser can be extended or otherwise modified at runtime to support all kinds of languages. And there is still a lot of dwimmery going on, such as the longest token matcher, which provides a sensible default behaviour when trying to parse a complex grammar.
Kang-min Liu: Continuous Testing.
A presentation of Test::Continuous, which polls your source directory for changed files, so that relevant tests can be run when you update them.
Lunch break
Another accidental lunch with Larry Wall (and his wife, and Faiz), at Matsuya.
Jose Castro: Perl Black Magic.
A very entertaining introduction to obfuscation and Perl Golf.
Ingy döt Net: JavaScript Love for Perl Hackers.
Ingy has ported the very popular jQuery to Perl, so that you can do web scraping with this powerful DOM query language. He also introduced his latest slideshow system Vroom, and proposed to use CPAN to distribute JavaScript libraries as well.
Leon Brocard: Working in the cloud.
Leon explained how his company is operating without any hardware himself, having outsourced everything to Amazon, Google and the like.
Jesse Vincent: Step 3 - Prophet.
In what was the most exciting talk for me, Jesse talked about Best Practical's experimental peer-to-peer replicated database. This fits in very nicely both with my recent musings about the future of databases, and my recent migration from Subversion to the distribute Mercurial source version control system, which works very similarly. The biggest challenge of these peer-to-peer systems is how to resolve conflicting updates.
Chia-liang Kao: Running A Perlish Small Business.
CL in his spare time runs AIINK, a small printing company, and explained how he quickly cobbled together the infrastructure for it with Perl.
Makoto Kuwata: The Fastest Template Engine is Perl World.
Yet another templating engine, but these things get incrementally better, and this one is pure Perl, very fast, feature-rich, and has an extremely clever (if you know Japanese) name: Tenjin - Template Engine.
Lightning Talks
More templating engines, YAPC financials, RSS readers, Perl6 signatures in sane Perl5, Japanese CPAN authors, from POE to Erlang (prologue), an OS written in Perl, web services, and crazy HTML hacks
Thu, 10 Apr 2008

Are relational databases on the way out ?

For decades, the default choice when it comes to storing application data have been relational databases. Recently however, we see a lot of alternative approaches gaining widespread exposure (not sure about acceptance yet), especially as part of Web 2.0 platforms. Think Amazon's SimpleDB, Google's BigTables, or Apache CouchDB.

Cluster architecture: RDBMS have traditionally always been client-server oriented, meaning that you can have multiple clients access the same database concurrently over a network. This alone is an enormous improvement over file-based storage, and it is also useful for three-tier web application, as it allows to scale out the number of application servers. In order not to have the single database server as a bottleneck and single point of failure, you eventually will want to spread its functionality over a cluster of machines. This is a more advanced option that most RDBMS have added in one form or another, but it seems these new web databases were designed specifically to run on distributed nodes.

Schema-free: RDBMS rely on data schema definitions (tables with typed columns) and have great difficulties to handle unstructured documents. In particular, a relational system offers no way to query data other than by column value, and makes it very difficult to query data across tables. Again, most RDBMS now have non-relational extensions like XML query capabilities or full text search. In contrast, the newcomers appear to be very document-centric, where every document can have its own set of attributes. One could argue that a data schema is part of the data integrity validation that a database system should perform. On the other hand, most people seem happy with doing that in the application instead, and in any case, it seems like it should be an optional feature. One could also argue that a fixed schema makes for more efficient storage and access paths. In this case, the schema is seen more as a necessary evil, and one would be happy to give up on it if any performance problems can be avoided some other way.

Impedance mismatch: A big complication when using an RDBMS for storing application data is that everything has to be broken down and mapped to tables and columns using only the rather primitive (scalar) data types of the RDBMS. This gets complex very quickly, both conceptually and also in regards to how the resulting data will be stored, retrieved and queried. Multi-table joins are not easy to understand, and also not especially fast to execute.

Transactions: Probably the main selling point for an RDBMS is that they pass the famous ACID test: Atomicity (all or nothing: no incomplete updates), Consistency (the state of the database does not get corrupted at any time, even in the presence of crashes), Isolation (no one can see the results of a transaction before it is committed), Durability (no committed update can be lost). These properties are essential for many applications, but they come at a cost. In particular, they make it difficult to efficiently replicate or distribute the system. The newer non-relational databases tend to relax these constraints considerably, which makes them unusable when you really need a transactional database. But if you don't ...

Performance: One would assume that RDBMS with all their compacted and normalised storage schemes and their indices are the fastest way to go. And I guess that they do offer the fastest possible way to sort fifty million records, but how often do you really need to do that? Especially if sorting these fifty million records in the fastest possible fashion is still too slow for an interactive application, you start looking at alternative approaches such as an intelligent hierarchy of pre-computed aggregated data. In the RDBMS world this is called data warehousing. Once you get used to the idea that ad-hoc queries are impossible anyway, and that anticipated queries can be satisfied using clever indexing (that may not even need to be completely up-to-date), the performance benefits of operations that you can avoid become less important.

So, in summary, I think that these new databases are obviously not able to replace an RDBMS in its traditional field of operation (record processing where consistent read and writes, transaction isolation, and atomic updates are critical), but they may very well take a sizable chunk of the huge market where RDBMS are currently being used solely because there have been no other choices. There may be no need for an RDBMS in the usual web application stack after all.

Mon, 07 Apr 2008

Hemorrhoid Pictures

Confusing gmail ad of the month. Usually, those targeted ads are actually really close to the contents of the mail thread that they are displayed for. But I really cannot see how big bad hemorrhoids (I am not giving you a link here, and I am certainly not going to click on it myself, being in the middle of dinner and all) are related to the following conversation...

-------------
sakura pictures
 
   :-)
   attached: IMGA0110.JPG IMGA0112.JPG
   
-------------
Thanks Thilo,

And here is the video of the S. Carolina beauty contestant:
http://youtube.com/watch?v=WALIARHHLII

Wed, 12 Mar 2008

More JDBC Microbenchmarks

Now that I managed to log in to my OTN account, here are the results of Saturday's test suite for Oracle XE on Windows XP:

run0[ms]run1[ms]run2[ms]run3[ms]updates/sec

Oracle

A3490338233393402296

B1578147814441452686

C1272125312411474756

D6606606576591518

E4745434123256

  • With Oracle, using prepared statements makes a lot of difference, going from interpolated variables to bind variables more than doubles the throughput, and reusing the same statement adds another ten percent. This is good news in more than one way, because that first part (low-hanging fruits for a programmer) brings such a big gain that you can argue against the need for the extra few percent fro the second improvement, which is trickier to implement in a general fashion (although you could turn on statement-caching in the driver, I need to try to measure that some time).
  • Using the batch-update interface when applicable gives a spectacular boost, in this case it is about 15 times faster. Further testing is needed to how this plays out with different batch sizes, specifically if there are upper and lower limits for when it makes sense to use the feature.
  • As for how much time it takes for getting a connection from the pool, it depends if you turn on the validation feature of the pool, which checks if the connection is still alive before giving it out. With validation turned off, there is basically no overhead, with validation it adds a few milliseconds every time you get a connection, in my case (I only tested this with Oracle, the times are not included in the charts) one to two ms.

After these measurements for a thousand updates, I also took timings for a different scenario:

  1. SELECT non-existing row
  2. INSERT the row
  3. SELECT again
  4. UPDATE the row
  5. SELECT again
  6. DELETE the row
  7. SELECT the now missing row again

This pattern was run in two variations (as shown above and without the selects) in two different implementations (using bind variables or not using them). Each of these four routines was run interleaved (ABCDABCD...) for a total of 101 times, with the first iteration results discarded, and the times it took for each iteration becomes the benchmark result. The connection was in auto-commit mode the whole time.

[ms/run]PostgresqlMySQLOracle

Insert, update, delete (no binds)4.8328.8

Insert, update, delete (binds)5.3325.3

Plus selects (no binds)9.93714.6

Plus selects (binds)10.3369

Again, we see prepared statements making a big difference on Oracle, not so much (even a slight slow-down?) on the open source databases, and that MySQL suffers because of the slow commits (of course, it should still be fast enough, that part is unlikely to become the bottleneck).

Potential follow-ups to this would be to properly profile the connection pool's validation feature, to include Hibernate into the mix and measure its overhead, to record the strain on the server, and to use multiple threads to see how bind variables affect scalability. But I promise that if I do that, I will not bore you with the results here on my blog (one thing that I do want to put here, though, are the results of running these two benchmarks on the same machines in Perl instead of Java).

Sat, 08 Mar 2008

JDBC Microbenchmark

What is the overhead of getting a fresh connection from the connection pool instead of passing the connection around? How much faster are repeated SQL statements when using a fixed query string with bind variables as opposed to directly interpolating the data into the query string? How much faster when re-using the same prepared statement? How much faster when using a batched update?

I ran a little benchmark.

  • A) 1000x [getConnection createStatement executeUpdate commit]
  • B) 1000x [getConnection prepareStatement executeUpdate commit]
  • C) getConnection prepareStatement 1000x [executeUpdate commit]
  • D) getConnection prepareStatement 1000x [executeUpdate] commit
  • E) getConnection prepareStatement 1000x [addBatch] executeBatch commit

I wanted to test Oracle XE on Ubuntu, but did not get either installed (the eMachine did not like the Ubuntu CD, and Oracle's web-site was unresponsive), so I went with Postgresql 8.3 and MySQL5(InnoDB) instead. The databases were running on Windows XP, both fresh installs using the default settings, accessed from the Java test program on a Mac mini via local ethernet network.

run0 [ms]run1 [ms]run2[ms]run3[ms]updates/sec

Postgresql

A2013194517551809545

B2088179118751731556

C1667172916581714588

D1213119811791169846

E7697807677661297

MySQL

A9479937992999479107

B9382935792649274108

C9314937193949222107

D6326176746411553

E6506506136341581

  • Commits against MySQL are amazingly slow. I assume that this is a problem with my setup, or with Windows. This also probably only affects the transactional InnoDB backend.
  • With MySQL, there is no speed difference between methods A, B, and C, and hence no visible performance advantage to prepared statements. Maybe the JDBC driver does not implement the feature. With Postgresql it seems to improve throughput, but not by much. The Oracle figures should be interesting here.
  • Committing only once instead of separately after every update makes a big difference, especially with MySQL (see above). Of course, performance considerations should not be a factor in deciding what a transaction is.
  • Bulk updates give another big boost to Postgresql, not so much to MySQL.
Wed, 05 Mar 2008

From X to O: Price

The XO-1 was supposed to be the $100 Laptop, but unfortunately went over budget and in its current version costs $188. The OLPC project hopes that an increased output combined with price drops in its off-the-shelf components will bring production costs down enough to reach $100 by the end of this year. Considering that OLPC mainly targets the world's poorest countries, and that for example Intel sells more conventional computers (together with Windows licenses, training, and support) starting for less than $300 in these markets, the price tag could easily become the decisive factor for OLPC's success, regardless of the educational and social concepts that they also have to offer.

Sat, 01 Mar 2008

AVCHD

We have a video camera (Panasonic HDC-SD5) shooting in Full HD resolution, which is more of a down payment towards a future home entertainment system than anything we can really use right now (nothing in the house can play back at a 1920x1080 resolution). So for now I am just left with these huge files that take up lots of disk space to store and ages of CPU cycles to process.

The camera records in AVCHD, which is a highly compressed format that still takes about 1GB per 10 minutes. Things get worse when importing them into iMovie, because iMovie insists to decompress the files, resulting in 1GB per minute, and the decompression is painfully CPU-intensive, running at about real-time. I want to store the movies exclusively in their native AVCHD until I feel like actually editing them, but there seems to be no viewer application for that format (which is weird, as iMovie's import wizard can preview them). At least I can copy them off the camera onto the hard disk to free up the memory card.

Sun, 24 Feb 2008

From X To O: Questions and Answers

Well, I have no topic for Q, so here is my pick of five Frequently Asked Questions from the OLPC site:

Who is behind these XO laptop computers?
The XO laptop computer is being developed by One Laptop per Child, a non-profit organization founded by MIT professor Nicholas Negroponte and a team of educators, developers and technologists dedicated to educating children in developing countries with the goal of eradicating poverty. One Laptop Per Child is based on principles expressed by MIT Media Lab Professor Seymour Papert in the 1960s, and later elaborated upon by Alan Kay, and complemented by the principles articulated by Nicholas Negroponte in his book, Being Digital. Partner corporations including Advanced Micro Devices (AMD), Brightstar, eBay, Google, Marvell, News Corporation, Nortel, Quanta, Chi Mei Group, Red Hat, and SES Astra are involved in this initiative.
Why do children in developing nations need laptop computers?
Nicholas Negroponte (OLPC's Chairman and founder) thinks of the XO laptop not as just a piece of equipment, but as an educational opportunity. Laptop computers can be a window and a tool—a window into the world and a tool with which to think. Computers are a wonderful way for all children to learn through independent interaction and exploration.
How is it possible to get the cost of the laptop so low?
First, by dramatically lowering the cost of the display. The first-generation computer will have a novel, dual-mode display that represents improvements to the LCD displays commonly found in inexpensive DVD players. Second, we have also worked to get the fat out of our software systems. In other words, our laptop computers operate more efficiently. The XO’s operating system is based on a free and open source version of GNU/Linux. Third, One Laptop per Child is a non-profit organization that is not obligated to any investors. Finally, One Laptop per Child uses large scale orders to minimize marketing and distribution costs and to bulk order components to drive prices down.
Does my laptop come with a hand crank? Solar panels?
The Get One XO does not ship with any human powered device or solar panel, which are designed to support the XO laptops shipped to developing countries. Future peripheral availability is discussed on the Product News page. In addition to plugging the laptop into an electrical outlet (110-240 volts AC), the XO laptop can take a DC input ranging from 11 volts to 40 volts, a range that’s far more flexible than most portable devices. The XO laptop is remarkably energy efficient, generally using only 5-10 percent of the average wattage of a standard laptop.
How long does the battery last?
Battery life is approximately 3 to 6 hours, depending on which Activities are used. The sleep feature is not enabled in software provided on XOs shipped starting December 2007. A software upgrade early in 2008 will support suspend/resume sleep features, for much improved battery life.
Wed, 13 Feb 2008

Developers Summit 2008

Nice event, and free to attend, too (thanks to all the Platinum and Gold and Silver and other sponsors).

Joel Spolsky: How to write popular software
He would have surely signed my copy of his book, too, but I misplaced it.
Mr. Kurihara: Aren't we forgetting the data
About the importance of proper data modelling, data storage, access path design, validation and other data-centric topics, and how they are often just an afterthought to application development, which leads to all kinds of nasty problems. I always feel at home with these kind of talks, reminds me of my university days and makes me want to move into the data management space, away from application development with its ill-defined and ever-changing requirements.
Lunch break
The Japanese Squeak user group had an OLPC (not the XO-1, but an almost identical developer pre-release machine with a Spanish keyboard) and we could try out the mesh network functionality, chatting and sharing a text editing session. Great fun.
The latest on rich clients
A panel discussion with representatives from Adobe (Flash/Flex/AIR), Microsoft (.NET/Silverlight) and Curl (Curl).
Jeffrey Fredrick (Agitar Software): Continuous Integration
An introduction to Continuous Integration and how it can benefit the development process by increasing product quality and shortening development cycles.
Next generation web application frameworks
By next generation, he meant frameworks that try to completely hide the stateless nature of HTTP requests with sessions and continuations. JBoss Seam and Wicket were presented as examples. There was also a lively discussion about how robust and complete these new approaches already are.
Mr. Ikehara (Infragistics) Line-of-Business development WPF
An introduction to Vista's new Windows Presentation Foundation and how it can be used to create attractive applications for a company's vital back-office operations, including a rather impressive live coding session showcasing Infragistics WPF components.
Programming Languages of the Past, Present and Future
A discussion between Yukihiro Matsumoto (creator of Ruby) and Daigo Hamura (test leader for C#) about language design, development process, life cycles, backwards compatibility, and other issues the maintainers of programming languages face.
Sun, 10 Feb 2008

From X To O: Reading

The main scenario I had in mind for the XO-1 was as a portable reader, both online and offline, on the road and also in bed (it is so inconvenient to have to get up and walk over to the living room for my daily fill of news websites). Technology issues aside, a laptop in bed is not going to happen, because Cissy does not tolerate that level of laziness and won't hear any of it (even an iPod is pushing it).

Online reading via the built-in browser works famously. Thanks to its Mozilla code base, web sites render as they should, images, JavaScript and even Flash (except for video clips) works, fonts look great (free Japanese fonts, by the way, are available on the Japanese government web-site). There is no email client, but web-mail should work (no problem with GMail). One downside is the inability to open links in a new tab or window, so that you are limited to linear browsing and cannot queue up pages for later access. You can bookmark pages, but not before actually opening them.

Offline reading with the currently available software is tricky. At first I thought I could use the PDF reader and downloaded me a couple of Cory Doctorow novels. As it turns out, PDF is not really a good format for screen readers, because the layout is fixed and cannot be adjusted to display size or orientation. Zooming in to make the characters legible leaves to few of the visible at the same time to read without scrolling around like crazy. The next choice was HTML, which the reader can re-flow and re-page to fit the screen, and while that works well online, I could not figure out how to save a file for offline access from the browser. Manually placing the file on the disk using the command line (wget) does not do much good either, because unless the file is managed by the Journal activity, it does not appear in the user interface at all. It seems the easiest (only?) way to do this is to prepare content for the XO-1 on a USB memory stick or SD card, similar to how iPods work (an SD card is better, because it disappears inside the laptop, whereas USB devices stick out from the side).

The XO-1 can be put into tablet mode by folding the screen over the keyboard. This has the potential for a nice book reading experience. The only keys you can still use in this case are the power button, a cursor pad, four game-pad buttons, and the display orientation switch. That should be plenty to zoom, scroll and page around a book, and maybe even click links on a web site and move a mouse pointer directly. Unfortunately, and this is a real shame, the software does not make use of the buttons in any useful way, so that you need to go back to the inaccessible keyboard for many basic functions. Also, the screen rotation feature is not co-ordinated with the cursor pad, so that pressing Up will sometimes scroll right (or left, or down).

Another problem is the long time it takes for the XO-1 to boot up. It should be possible to put the machine into sleep mode and very quickly resume from there, but apparently this is not implemented, which is surprising considering how much thought went into energy preservation for the hardware. As it is, having to power it down completely makes it inconvenient as a book reader.

Fri, 08 Feb 2008

Johnny can't add, part 8

External USB hard drives were on sale, so I picked up 500 GB from I-O Data and thanks to Time Machine now have a decent backup system in place. An interesting fact about hard disks is that there are two ways to count data volumes. Usually a kilobyte has 1024 bytes, with admittedly is somewhat at odds with the metric system. Hard disk makers call 1000 bytes a kilobyte. That is not much of a difference (2.35%), so you can ignore that as a marketing oddity, but the discrepancy grows dramatically as we get towards terabyte disks. A 1 MB hard disk really only holds 95.3% of a megabyte, a 1 GB disk only 93.1% of a GB, and my new 500 GB disk has only 465 GB.

PS: Caption inspired by Mike Camino Pinkerton's remarks on mathematics, marketing and McDonalds.

DTrace Envy

Probably the most acclaimed technology to come out of Sun Microsystems recently is DTrace, the dynamic tracing framework for Solaris (which Apple also ported for and included in Leopard). DTrace allows to inject so-called probes into the running operating system and applications in order to gather data to tune or troubleshoot them. Probes are written in a simple programming language called D, creating probes requires no changes to the code being probed, and DTrace support has no performance impact unless the probes are actually enabled.

I really want to have something similar for Java, in order to troubleshoot tricky support issues at customer sites. At least a simple version (with only JVM-level and no application-specific instrumentation points) should be possible using the existing JVM tracing and debugging interfaces. I picture something like inserting a little script that fires when a specific method is called with a specific set of arguments and then logs a message or starts a timer or increases a counter. Half a day of searching did unfortunately not turn up an easy-to-use and painless-to-deploy tool, although I am sure there must be something.

As it is, we have to rely on logging, the catch being, of course, that appropriate trace messages need to be in the code in the first place, which more often than not they are not, requiring a patch release with all the costs and delays associated to it. And even if the messages are there, they of course only appear when trace logging is enabled, and even when only enabling tracing for the bare minimum of log categories required, there will usually be a lot of output unrelated to the problem at hand that need to be manually filtered.

Update: I am aware that DTrace is available on the latest JVM for Solaris and Mac OS X, but that will not help me with Java5 on RedHat, which is what we deploy on. I was looking for a Javaland solution that works with older/other JVM/OS.

Mon, 04 Feb 2008

From X to O: Screen

The most expensive and most innovative component of the XO-1 is its display. It is a 7.5 inch TFT with two operating modes: With the backlight turned on it can display approximately 800x600 coloured (6bit: 262k colours) pixels. With the backlight turned off, it becomes monochrome (6bit: 64 shades), but the resolution goes up to 1200x900 and the screen is usable in bright sunlight.

This peculiar behaviour is due to the interesting way the display is constructed: The LCD is a 1200x900 grid of square pixels with 64 levels of opacity. Behind it is a reflector, that sends incoming ambient light back through the LED. So when the backlight is off, all you see is the high-resolution monochrome screen. When the backlight is turned on, it first goes through a fixed colour filter that tints every pixel either red or green or blue. In a completely dark room, all you see are 1200x900 pixels of fixed colour (but 64 shades of opacity), or, if you group three of them together to count them as one coloured pixel, approximately 800x600. In most situations, and depending on your viewing angle, you see a combination of both.

Another interesting feature of the display is that its controller has enough memory to keep the screen alive while turning off the CPU and other parts of the motherboard. This can save power in book reader mode.

Speaking of book reader mode, the display can be pivoted, turned around and folded so that it closes like a lid over the keyboard, turning the XO-1 into a tablet (without the touchscreen). A game pad and cursor keys located next to the display can then be used in lieu of keyboard and trackpad for simple navigation tasks. There is also a very handy button that rotates the contents of the screen in 90 degrees intervals (normal / portrait / upside-down landscape / reverse portrait) to allow you to adjust to how you hold the device.

Sat, 02 Feb 2008

From X To O: Toy laptop

Child's toy computer is how Martin declared the XO-1 in the customs form when sending it here. It does include a smattering of educational software:

  • TurtleArt is a graphical Logo implementation: You get to paint by programming a little pen-wielding turtle to wander over the screen. The programming is done by dragging instruction blocks (such as move forward, turn, arc, raise the pen, change the pen colour) from the tool box onto the screen. This is great fun, I can see myself not finishing this post on Friday because I cannot stop playing TurtleArt right now.
  • Etoys is a Squeak Smalltalk media-rich authoring environment with a simple, powerful scripted object model for many kinds of objects. I have played with Squeak before, which is a platform is equally interesting and esoteric. The XO-1 version is completely compatible with everything out there in Squeakland.
  • Pippy is an interactive Python interpreter that aims to teach programming in the language most of the higher-level programs on the laptop are written.
  • Measure is for the physics lab and provides an interface to display and record signals produced by external sensors as waveforms, like an oscilloscope. You would need to build those sensors and connect them to the audio input port, so that I could not really test this activity.
  • TamTam is a suite of music composition programs. TamTam Mini is a simple activity where you can select an instrument and use the keyboard to play notes. TamTam Edit is a five-track editor that can be used to arrange and record whole songs. Young Man's GarageBand. TamTam Jam seems to be somewhere in between Mini and Edit. TamTam SynthLab is an advanced synthesiser application that I think is used to create new instruments for TamTam Edit.

In addition to using these activities in creative ways, the child is also encouraged to modify the software itself. A good part of it is being written in Python and other dynamic languages, and thus lends itself to live hacking. There is even supposed to be a View Source button to reveal the underlying codebase of the running activity and allow changes to it, but that seems to be not implemented yet. As programming skills grow, one could even tackle the base system itself, all of which is Open Source (Linux and OpenFirmware), although that kind of development can probably not be done on the XO itself.

As for games, there is Memorize, the game of matching pairs, with an editor to create your own set of tiles, others are available for download, and Electronic Arts has donated SimCity.

Thu, 31 Jan 2008

From X To O: Unix Underpinnings

[olpc@xo-11-19-06 ~]$ uname -a

Linux xo-11-19-06.localdomain 2.6.22-20071121.7.olpc.af3dd731d18bc39 #1
PREEMPT Wed Nov 21 00:39:06 EST 2007 i586 i586 i386 GNU/Linux
[olpc@xo-11-19-06 ~]$ df -k

Filesystem           1K-blocks      Used Available Use% Mounted on
mtd0                   1048576    359044    689532  35% /
tmpfs                    35676         0     35676   0% /dev/shm
[olpc@xo-11-19-06 ~]$ cat /proc/cpuinfo

processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 5
model		: 10
model name	: Geode(TM) Integrated Processor by AMD PCS
stepping	: 2
cpu MHz		: 430.952
cache size	: 128 KB
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu de pse tsc msr cx8 sep pge cmov clflush mmx mmxext
3dnowext 3dnow
bogomips	: 862.97
clflush size	: 32
 
[olpc@xo-11-19-06 ~]$ ps auxw

USER      PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

root        1  0.7  1.7   4952  4144 ?        Ss   17:36   0:04 oatc    
          /init
root        2  0.0  0.0      0     0 ?        S<   17:36   0:00
[kthreadd]
root        3  0.0  0.0      0     0 ?        SN   17:36   0:00
[ksoftirqd/0]
root        4  0.0  0.0      0     0 ?        S<   17:36   0:00
[watchdog/0]
root        5  0.1  0.0      0     0 ?        S<   17:36   0:00
[events/0]
root        6  0.0  0.0      0     0 ?        S<   17:36   0:00
[khelper]
root       47  0.0  0.0      0     0 ?        S<   17:36   0:00
[kblockd/0]
root       48  0.0  0.0      0     0 ?        S<   17:36   0:00
[ksuspend_usbd]
root       51  0.0  0.0      0     0 ?        S<   17:36   0:00
[khubd]
root       53  0.0  0.0      0     0 ?        S<   17:36   0:00
[kseriod]
root      115  0.0  0.0      0     0 ?        S    17:36   0:00
[pdflush]
root      116  0.0  0.0      0     0 ?        S    17:36   0:00
[pdflush]
root      117  0.0  0.0      0     0 ?        S<   17:36   0:00
[kswapd0]
root      118  0.0  0.0      0     0 ?        S<   17:36   0:00
[aio/0]
root      517  0.0  0.0      0     0 ?        S<   17:36   0:00
[kpsmoused]
root      560  0.0  0.0      0     0 ?        S<   17:36   0:00
[kmmcd]
root      617  0.0  0.0      0     0 ?        Z    17:36   0:00 [init]
<defunct>
root      667  5.2  0.0      0     0 ?        SN   17:36   0:35
[jffs2_gcd_mtd0]
root      670  0.0  0.2   2140   664 ?        Ss   17:36   0:00 init [5]
root      757  0.0  0.2   2260   592 ?        S<s  17:36   0:00
/sbin/udevd -d
root      886  0.0  0.0      0     0 ?        S<   17:36   0:00
[libertas_main]
root      887  0.0  0.0      0     0 ?        S<   17:36   0:00
[libertas_worker]
root     1263  0.0  0.2   1804   596 ?        Ss   17:37   0:00 syslogd
-m 0
root     1266  0.0  0.1   1744   404 ?        Ss   17:37   0:00 klogd -x
dbus     1289  0.1  0.4   2956  1008 ?        Rs   17:37   0:01
dbus-daemon --system
root     1311  0.0  2.8  13660  6704 ?        Ss   17:37   0:00
/usr/bin/python /usr/sbin/rainbow-daemon --daemon
root     1326  0.0  0.3   5392   940 ?        Ss   17:37   0:00
/usr/sbin/sshd
root     1338  0.0  0.9   7568  2196 ?        Ssl  17:37   0:00
console-kit-daemon
root     1378  0.0  0.4   3296  1104 ?        Ss   17:37   0:00 crond
root     1423  0.0  0.2   1736   616 ?        SNs  17:37   0:00 anacron
-s
root     1435  0.1  0.9  28740  2152 ?        Ssl  17:37   0:00
NetworkManager --pid-file=/var/run/NetworkManager/NetworkManager.pid
68       1462  0.2  1.1   4552  2640 ?        Ss   17:37   0:01 hald
root     1463  0.0  0.4   3084   972 ?        S    17:37   0:00
hald-runner
root     1486  0.0  0.4   3176  1080 ?        S    17:37   0:00
hald-addon-input: Listening on /dev/input/event0 /dev/input/event1 /
root     1491  0.0  1.4   7092  3372 ?        S    17:37   0:00
/usr/bin/python /usr/bin/olpc-hardware-manager
avahi    1519  0.0  0.5   2656  1420 ?        Ss   17:37   0:00
avahi-daemon: running [xo-11-19-06.local]
avahi    1522  0.0  0.1   2656   432 ?        Ss   17:37   0:00
avahi-daemon: chroot helper
root     1540  0.0  0.1   1728   464 ?        Ss   17:37   0:00
/sbin/mingetty --noclear tty1
root     1543  0.0  0.1   1724   456 tty2     Ss+  17:37   0:00
/sbin/mingetty tty2
root     1544  0.0  0.2   1736   504 ttyS0    Ss+  17:37   0:00
/sbin/agetty ttyS0 115200 vt100
root     1545  0.0  0.5   2880  1200 ?        Ss   17:37   0:00
/usr/sbin/olpc-dm
olpc     1557  0.0  0.4   2552  1064 tty1     Ss+  17:37   0:00 /bin/sh
/usr/bin/startx /usr/bin/olpc-session -- -fp built-ins -wr
olpc     1574  0.0  0.3   3076   928 tty1     S+   17:37   0:00 xinit
/usr/bin/olpc-session -- /usr/bin/X -fp built-ins -wr -auth /h
root     1575  5.2  8.5  26072 20400 tty3     Ss+  17:37   0:34
/usr/bin/X :0 -fp built-ins -wr -auth /home/olpc/.serverauth.1557
olpc     1585  7.5 12.9  58040 30920 ?        Ss   17:37   0:49 python
/usr/bin/sugar-shell
olpc     1593  0.1  0.5   3480  1408 ?        Ss   17:37   0:01
/bin/dbus-daemon --fork --print-pid 4 --print-address 6 --session
olpc     1594  0.0  0.2   2840   616 ?        S    17:37   0:00
dbus-launch --exit-with-session sugar-shell
olpc     1605  0.1  2.1  11100  5004 ?        S    17:37   0:00
matchbox-window-manager -use_titlebar no -theme sugar -kbdconfig /us
olpc     1607  0.6  4.5  17152 10740 ?        S    17:37   0:04 python
/usr/bin/sugar-presence-service
olpc     1609  0.0  1.1   8068  2808 ?        S    17:37   0:00
/usr/libexec/telepathy-salut
olpc     1611  0.3  3.7  15320  8884 ?        S    17:37   0:01 python
/usr/bin/sugar-shell-service
olpc     1613  2.1  5.8  32012 13872 ?        Sl   17:37   0:13 python
/usr/bin/datastore-service
olpc     1623  0.0  0.1   3780   472 ?        S    17:37   0:00 /bin/cat
olpc     1625  4.7 10.1  42892 24136 ?        S    17:37   0:29 python
/usr/bin/sugar-activity journalactivity.JournalActivity -b or
olpc     1660  1.3 10.5  55076 25140 ?        Sl   17:38   0:07 python
/usr/bin/sugar-activity terminal.TerminalActivity -s -b org.l
olpc     1670  0.0  0.2   2420   628 ?        S    17:39   0:00
gnome-pty-helper
olpc     1671  0.0  0.6   4688  1536 pts/0    Ss   17:39   0:00
/bin/bash
499      1699  0.0  0.2   1820   704 ?        S    17:39   0:00
avahi-autoipd: [msh0] bound 169.254.2.85
root     1700  0.0  0.1   1772   352 ?        Ss   17:39   0:00
avahi-autoipd: [msh0] callout dispatcher
olpc     1786  0.0  0.2   3788   608 pts/0    S+   17:46   0:00 script
olpc     1787  0.0  0.1   3792   400 pts/0    S+   17:46   0:00 script
olpc     1788  0.1  0.6   4688  1540 pts/1    Ss   17:46   0:00 bash -i
olpc     1802  7.0  0.3   4512   944 pts/1    R+   17:48   0:00 ps auxw
Wed, 30 Jan 2008

From X To O: Video

The first thing to try with any web-enabled device is to check if it can play back Youtube. Well, the XO-1 cannot, which I guess is because of the GNU Flash player (Adobe's official one cannot be included as it does not meet the open source criterion) lacking the codec that is being used. Even if it was there, I wonder if the CPU would be up to the task.

Not being able to watch Youtube is of course a decidedly First World Problem, and not really relevant to OLPC (or maybe it is, it seems the Internet is rapidly becoming an important alley for authentic individual voices that the traditional media outlets would just ignore). Anyway, there is a built-in camera and a Record activity (applications are called activities) that can be used to capture video clips of up to sixty seconds. Unfortunately, I could not make that work, either. I could record a clip and play it back again in Record, but after closing the activity could find no way to view it again. The video does appear in the Journal, but when trying to open it, the browser launches, shows a download dialog and then a blank page. Again, the software is not quite there yet.

It does not make matters easier that the XO-1 tries to get rid off the concept of a traditional file system: Rather than the user explicitly saving and loading files, the Journal is supposed to automatically keep track of everything you do (including older revisions of documents you change, but I am not sure if that is already implemented). Until that works reliably, it would be nice to know where the files actually are.

Update: The same problem hit me after writing this article -- first blog post from the machine itself! After finally finding the file I typed in the Write activity (based on AbiWord) as /home/olpc/.sugar/default/datastore/store/78dc0364-be9c-4886-8329-ee62063f28fb I decided to drop out of Sugar and into vi to finish it (praised be the person who agreed to keep the underlying standard Linux tools around and readily accessible).

Tue, 29 Jan 2008

From X To O: Wireless networking

The OLPC project aims to enable children to study and create in a collaborative environment that makes it natural to share documents and work on them together. To facilitate this, the XO-1 has built-in networking capabilities, most importantly an IEEE 802.1 chipset for wireless communications. Notable features here are the low energy consumption (paid for with a relatively low speed of 2 Mbits) and the ability to form a self-organised mesh network in the absence of an access point. The chipset is autonomous enough to allow the XO-1 to act as a repeater in the mesh even when the laptop CPU is powered down.

Since OLPC laptops are intended for deployment in areas where there will likely be no traditional telecom infrastructure, there are is no ethernet adaptor or modem (I wonder if there was similar reasoning behind the MacBook Air). Other means of getting data in and out are the USB ports and the SD card reader. And then there is the Acoustic Tape Measure program, that can be used to measure the distance between two laptops by measuring the time it takes for them to shout at each-other.

Whereas the hardware does not fail to impress, some areas of the XO-1 software in its current form feel unfinished and flakey. Being software, of course, it should be relatively easy to resolve these issues over time. One instance that caused me quite a bit of frustration at first was getting the WiFi to work. Although I would very much prefer a wired connection (faster, less radiation) I do have an access point that I occasionally switch on for the Wii, but the XO-1 would not connect to it. Turns out that the version of the OS that it came with has critical bugs here, but there was a new version (poetically named build 653) available for download. The process required some computer literacy, another computer with a working Internet connection and an SD card (or USB stick), but now it works.

Mon, 28 Jan 2008

From X To O: XO-1

The XO-1 is the first version of the One Laptop Per Child. It comes in compact (242x228x32 mm) and rugged green package, with an innovative screen (7.5 inch), a membrane keyboard (sized for children), a touchpad with adjacent stylus areas (reserved for future use), three USB ports, an internal SD card reader, a built-in microphone, speakers and video camera.

The computer is powered by a 433 MHz AMD CPU that runs at only 800 mW, the display uses between 100 and 1000 mW, so that the overall power consumption is less than 3 W. It does not come with a hand-crank, but a selection of hand- or foot-operated power units is optionally available.

Networking is an important part of the educational concept behind the project and the XO-1 sports a Wi-Fi chipset that allows connections using 802.11 with an access point or by forming an ad-hoc mesh network with other XO-1. The latter allows sharing a single Internet connection for a the whole classroom.

All software on the XO-1 is free and open source. The operating system is based on Fedora Linux and a Python-powered GUI called Sugar. A central component is the Journal, which keeps a chronological logs of all activities, such as changed files or accessed applications, and allows to resume those activities (open the file or application again). The bundled software is mostly educational programs to encourage creative exploration into the realms of music and programming, but there is also a general purpose (Mozilla-based) browser, a word processor, and an RSS reader. I could not find a scheduling application, or even a calendar, though, clearly it is not aimed at business people.

Thu, 24 Jan 2008

One Laptop Per Man-Child

My XO laptop is finally almost here. After having travelled to New York and California (Give One, Get One would not ship to outside the USA and Canada, so I had to ask Martin in New York to be the middle-man), the postman came by to drop it off yesterday (but no one was at home), and will come again on the weekend. Stay tuned for a review then, for now read what Martin had to report and watch what David Pogue has to say.

Here is the "x", the "ooooh" happens when you see it ...
preinstalled: GUI apps, vi, gzip, python, ssh, scp
    not installed: emacs, perl, gcc, latex
          memory: 1GB disk, 237MB RAM
                camera built-in
                    light
               long battery life
         processor: AMD Geode, 420MHz
  http://wiki.laptop.org/go/WPA_Manual_Setting
fun geek toy, sturdy linux laptop, altruistic deed
Wed, 09 Jan 2008

Ubuntu: A pleased costumer's report

Once every year or so I install a recent Linux distribution and am every time impressed about by progress it has made as a desktop OS.

I have a dual-boot machine at work with Windows XP and Debian, the first of which I have not touched for months, but it still got corrupted (could not even start up anymore), so that I re-installed Windows from scratch (CD-ROM, scratch-free actually) yesterday. Unfortunately, Windows does not really work out-of-the-box on the machine, it apparently requires some OEM patches from Dell (without extra drivers or something the video card would not go beyond 640x480 which is totally useless). So I gave up again on Windows for the moment and tried to restart Debian to not fall completely behind schedule, but of course, the Windows installer had clobbered the boot menu. I suppose I should be grateful that the partitions are still there.

So I popped in an Ubuntu Live CD that I found in my desk drawer, with the honest intent to just restore the boot menu to be able to get back to my Debian environment. What immediately struck me is how pretty Ubuntu is, and how it justed booted and worked without me configuring anything. Said graphics card posed no problems, for one thing, and all my partitions were nicely mounted and just a click away in the very nice-looking file manager. I remember that this used to be the other way around: Whereas Windows just worked, getting Linux and especially X11 up an running required real work. I fiddled with the command line to recover my GRUB settings from the Debian file system and put it back onto the boot record, but this somehow failed, and I could not really be bothered to research how these tools work, so I just decided to go ahead and also install Ubuntu.

The next pleasant surprise was the Ubuntu installer. It is completely graphical, it can shrink existing NTFS partitions to make space for its own use, it finds existing OS installations and migrates wallpapers, bookmarks, address books, music libraries and such from Windows and Debian accounts. The resulting boot menu included Ubuntu, Windows, and all the options for the old Debian.

I am now in the process of migrating the rest of my stuff over from Debian as I go along, but compared with previous OS updates this has been very painless so far: Eclipse installation and workspaces can just be run directly from the mounted Debian filesystem, Java VM can be copied over, Oracle XE can be installed via the Debian/Ubuntu package manager now, which also provides other extras such as Japanese IME (the fonts, by the way, look much better now, too).

I am also in the process of discovering all the nice applications that make up the Ubuntu desktop. Not having worked with or even seen most of them, it is very pleasant to note how natural it feels to use them. Basically they work exactly how one would expect and want them, too. UI usuability and consistency is leaps above my admittedly very bare-bones Debian install (which I very much doubt I will see again), and honestly, if asked where the Mac still comes out in front, I need to think for a while.

Very nicely put together.