Blog Closed

This blog has moved to Github. This page will not be updated and is not open for comments. Please go to the new site for updated content.

Saturday, May 30, 2009

Linux Security

I was reading an interesting blog post about the effects of malware on your computer, and found two gems of comments. The second is a direct response to the first:

The suggestion:
Anyone running Windows volutarily is asking to be hacked. Do yourself a favor- install Linux on a spare machine and try it out. It's friendly, and free! Check out or
The response:
Changing your operating system isn't the answer. You don't get security through obscurity. For most people, what you propose would add a great deal of complexity (different ecosystem, learning curve of different system, different hardware/software compatibility, etc.) You would be better served learning how to properly secure and operate a Windows based system. Start with the list below:

1. Use a non-admin (limited user) account for daily use
2. Use a firewall (preferably a hardware firewall at the perimeter and a software firewall on each computer)
3. Keep the system fully patched (includes ALL software)
4. Use Antivirus/Antispyware software that is configured to update itself DAILY
5. Practice safe computing (ex. use caution with downloaded files and e-mail attachments, don't click on links in e-mail, browse wisely, etc.)
6. Routinely (at least monthly) backup your data to external media (CD-R, DVD-R, external hard drive, etc.)
7. Install ONLY required software (reduces system attack surface and minimizes patching). AVOID file sharing software! (too risky to system)
8. Optional (but highly recommended):
a.Use a blocking HOSTS file (
b.Enable Windows Automatic Updates (Auto download and install)
c. Use an e-mail client instead of webmail, configure it to “Read all e-mail in plain text”

There are a few points here in the response that are so rediculous and so ignorant that they require public mockery. First off, Linux is not "security by obscurity", that's exactly the security model used by proprietary software like Microsoft. The hope is that if criminals can't see the source code, that they won't be able to find the holes. Apparently, some people think this model works well enough. Here are the rest of the points:

  1. Talk about the troubles of changing environments, try using Windows as a non-adminstrator user sometime! It's like night and day in terms of usability, especially if you're not the kind of power user who knows how to get into administrator mode to do stuff like install new software. Let me put this into perspective: My grandfather has used a computer for a few years but is by no means "computer literate", much less "computer saavy". His old Win95 box was so overrun by viruses and malware that it was absolutely unusable. So I reformatted the computer to use Fedora instead, and now he has no problems whatsoever. I set up his email account in thunderbird, and I set up his list of bookmarks in firefox, and he was up and running immediately, with no learning curve and no hardships whatsoever. Not only was the transition painless and immediate, but his computing experience from that point forward was far superior to what it would have been otherwise: No invasive anti-virus to deal with. I'll talk more about the evils of antivirus later.
  2. A hardware firewall "at the perimeter" and a software firewall "on each machine"? Seems like a little much for Joe-Schmoe PC user. Maybe we should all be digging foxholes too, and building reinforced concrete bunkers for our networks. And he leaves out the point about how these firewalls would need to be properly configured because the "default allow" methodology behind most firewalls is inherently faulty anyway. Plus, when the firewall decides that it's not going to allow the social networking application du jure, your average non-techie computer user will be pretty unhappy with their computing experience. Sorry, but throwing more firewalls at the problem is not the solution for most people.
  3. Do me a favor: Figure out where "ALL software" on your system comes from, and figure out how to independently update all of it. I'm a pretty saavy computer user, and I doubt I could track down the sources for all the software that's on my computer currently, much less find ways to reliably update them all. There are a few programs on my machine that automatically update themselves, but then you run into problems where your automatic updater runs afowl of your firewall (as happens with my Java updater here at work). Also, tell me where you turn when an update to a device driver breaks one of your programs? Or when updating an application "broke the internet"? In most Linux distros, it's as easy as opening the update manger and letting it track and update all your software for you (and you can even set it to run automatically without interference from the system firewall). On Windows, you're up shit creek with this, and there's no two ways about it.
  4. Antivirus software is the bane of modern computers. I refuse to use it myself, because of the horrible performance penalties you have to pay with it. To give some perspective, my work computer, which is an Intel Core 2 Duo with plenty of RAM boots up in just under two minutes without antivirus installed. With Antivirus, it takes nearly 10 minutes just to turn on. Plus, when it does turn on it's less responsive in many situations. All the while it's popping up nagware "reminders" about the state of my subscription. This completely ignores the fact that antivirus software is almost perpetually out of date because threats usually aren't identified and neutralized until after they are released into the internet. This creates the illusion that you're safe, when so often you are not. Sorry, but antivirus is so much worse then the problem it claims to solve.
  5. The definition of "safe" computing differs from platform to platform, but in general this does hold equally for all systems. This so far is really the only good advice this poster has provided, and it is in no way an argument for using Windows over Linux.
  6. Backing up really has nothing to do with security. This is especially true when you think about how often Windows' "autorun" feature has been used as an attack vector by viruses. I might never put another USB flash drive into a Windows computer for as long as I live, I've learned that lesson the hard way.
  7. What counts as "required"? I do image editing every now and again, is my photo editor a "requirement"? Is FireFox a requirement when we already have IE installed? What about a code editor that's any more advanced then notepad, is that required? What about the tools that I use to stay in touch: Instant messaging, IRC, and Skype? what about office/productivity software? And what about people who's primary reason for having a computer is to share media with friends and family? With a single-purpose server machine, that's one thing. With a family media-oriented PC, that's another entirely. We all know that the safest way to use your computer is to not use it at all, but in this case abstinence is not the solution to the problem.
  8. Monkeying with your HOSTS file is far too advanced and too risky for your average non-power-user, which is 99% of everybody. Automatic updates are great until they download WGA without your consent and then falsely identify your computer as being pirated, and then restrict your access. I've seen this happen on more then one occasion to people who owned legitimate copies and were technically inable to resolve the situation themselves. I can't imagine that using an email client is any safer then using a good webmail app, especially when most webmail clients have built-in abilities to open Microsoft Office documents, which themselves can be sources of viruses. Gmail for example lets you preview documents like these on the web without having to download them to your computer and run malicious macros. That sounds like the safer way to me.
So that's my little response to this little ignorant comment. Your average PC user is not so tied to their system that they couldn't play succesfully with Linux, and I think most people, for most needs, would be up and running on Linux very quickly indeed, especially since there are so few security settings that need "tweaking" on Windows. I don't want to say that either Windows or Linux are superior, I use both and am very happy with that. However, I also won't hear that a good Linux distribution poses any sort of meaningful usability hurdle to the average user, or that it is in any appreciable way less secure then Windows is. I suggest that people try Linux, because trying it costs nothing but some old commodity hardware, and you might find that you like it more.

Friday, May 29, 2009

IO Work Proceeds, with Questions

Earlier this week I kicked off work on a revamp of the Parrot IO system in the io_rewiring branch. The purposes of the branch are multifold:
  1. To improve speed, by decreasing reliance on PCCINVOKE calls to perform IO operations
  2. To improve flexibility, by realizing that IO-related PMCs do not all hold a common interface
  3. To make basic IO PMC types properly subclassible
The io_rewiring branch is being used primarily for tinkering right now. We still have a lot of questions open as to how we are going to fix everything. However, we have some general concepts that we are trying:
  1. IO API functions like Parrot_io_* are now called by the methods in IO PMCs, instead of the other way around. Most IO operations now, except those called from methods in PIR directly, do not use PCCINVOKE.
  2. IO PMCs subscribe to a number of roles that determine what operations they can and cannot be expected to do. A PMC that does "file" can 'seek' but cannot 'connect'. A PMC opened for writing only does "write", but does not does "read".
  3. The PMC inheritance hierarchy is getting a little bit more sane. Pipes are not going to be FileHandles with a special flag set anymore. Sockets are not subclassed FileHandles either. At the moment, all IO Objects are derived from a Handle type, but this may change. The imporant part is that they all does "IO".
  4. Buffering logic is (probably) being abstracted into a separate PMC type somewhere.
So what my goal is for this branch, in a nutshell, is to break a little bit of encapsulation for major gains in performance, flexibility, and subclassability. Instead of blindly calling a method to perform operations, we check the roles of the PMC and depending on what roles it implements we access attributes and methods as necessary. The reason I want to try this approach is simple: We have a series of PMCs, each with an internal implementation and an external-facing API. The problem is that none of the PMC types are going to have the same API: Not all PMCs implement 'seek', or 'connect' or 'fcntl' for example. So it doesn't make sense for all PMCs to blindly be implementing these methods, or having to have a million API methods to satisfy every need of every potential type. So the question comes down to this: Do we maintain a large standard API with more methods then any one PMC type needs, do we maintain a small standard API and shoehorn all IO PMCs into it, or do we not maintain any specific API, and assume all PMCs of a particular type share common internals that we can poke into directly?

And we're not really poking into the "internals", really. We're using named attributes which are easy to subclass from PIR and are being treated as public fields. We're also trying to use VTABLEs where appropriate. In fact, I have a major complaint about the massive overabundance of arithmetic-related VTABLEs as compared to the dearth of IO-related VTABLEs, but that's a different rant for a different day. If we make the rule that we only acces VTABLEs and named attributes, and if we properly document which of each the different IO PMC Roles require, I think this method should be fine.

Whether the work in this branch ever satisfies all my goals or not, and if so whether we get community approval to merge it into trunk, is up in the air. It certainly is fertile ground for exploration, however, and I'm taking the opportunity to explore in great detail.

Tuesday, May 26, 2009

Lots of IO work to do

It's no secret that I've been planning pretty aggressively to implement an asynchronous IO subsystem for Parrot. However, this is really just one part of a larger group of projects that the IO system needs.

Today in the weekly Parrotsketch meeting, a lot of IO-related issues were raised. Several people mentioned performance issues, which seems to be a hot button issue in Parrot-world and for good reason. The way the IO system API is currently implemented is as a thin wrapper layer around PCCINVOKE method calls for the FileHandle and related PMCs. This implementation works decently, and was necessary for a time to support performing IO operations on a wide array of hypothetical IO PMCs, but there is a huge overhead involved in calling all those PMC methods. In fact, it was this very issue that prompted the current round of PCC refactors. Instead of having IO PMCs related simply by sharing a common set of methods, it's probably much better for these PMCs to be related together by inheritance and to use a common set of C functions to access their common VTABLEs and attributes directly.

So Infinoid and I have been putting together a plan tonight and I'm pretty excited about it. Here are some of the main points:
  1. We're refactoring the PMC hiearchy. IO types will all derive from a common Handle type that will have a few common attributes, vtables, and methods.
  2. We're going to refactor FileHandle to subclass from Handle
  3. We're going to fix Socket so that it subclasses Handle, and not FileHandle
  4. We're going to use roles and the VTABLE_does to determine the capabilities of each type of IO PMC. Roles like "read", "write", "file", "socket" and "pipe" will determine capabilities. Then we can also have subroles, like "connection" and "connectionless" for different types of sockets, etc.
  5. Try to fix attributes, where possible, to be inheritable so we can subclass all these things from PIR reliably
  6. Change the IO API to use C functions instead of PCCINVOKE methods to perform operations. We sacrifice a little bit of encapsulation for big performance gains, but I think it's a worthwhile change.
After this, Infinoid is really looking to get an IO multiplexing "Select" PMC put together, and I'm looking at getting AIO in place. I was planning to do something like Select as part of the AIO work, but I'm not sure that my idea is the best for general purpose use. I'm still considering using an "IOListener" that will check IO events as part of the scheduler, but that could be the asynchronous equivalent of a manually-polled Select PMC. Maybe we need both, maybe we only need one. We'll see how it goes.

My goal is to get this cleanup work done within 1-2 weeks, and with a great coder like Infinoid in my team I think it's completely possible. After that, I have the ambitious goal of getting AIO implemented, or at least a working subset of it, by the 1.3 release (that I am the release manager for). Certainly by 1.5 Parrot is going to have a world-class IO system available, which is perfect timing because that's when a few of my other tasklist items become unblocked.

Sunday, May 24, 2009

Blocked Tasks and Starting AIO

I'm trying to take stock of my task list, and I have surprisingly few things available to do. The list is pretty large (and wasn't even complete, I added a new task this morning), but I am able to work on only a small handful of items right now.
  1. I am adding a new job to my task list: removing the deprecated stacks functions. I added in the deprecation notice in r39144, which means I can't work on this stuff until the 1.5 release in 3 months.
  2. The calling conventions work is all blocking, probably until 1.4 in two months. This is a pretty big issue and getting bigger, because I'm finding all sorts of new items that are related to this that I wasn't aware of previously. Most of the items on my list are blocked because of this.
The IO system is in better condition then I thought when I last looked at it, so it doesn't really require any cleanups, so check that item off my list. Also, yesterday I fixed the problem Coke reported about Objects, so that item is off too. So, what's left on my list that I can do between now and 1.4?
  1. Create the incremental GC
  2. The Asynchronous IO system
  3. Documentation (especially the book)
I do some idle writing on the book every now and again, but that hardly scratches my "need to code" itch. Plus, it's scheduled to go to the printer within the month, which leaves me with only two tasklist items to work on post-1.3.

So basically I have the incremental GC and the AIO systems to work on for now. Since I'm not ready yet to do the GC, I've started the AIO work this morning and have already started implementing some of the necessary PMC types. This branch isn't going to make any changes anywhere else in the code base, it will only add prototypes of new PMC types that can be added to trunk without breaking anything. Once these things are implemented, I will open new branches to start integrating them into Parrot. I'm leaving notes throughout to keep track of what the integration will require. There are also obviously still a few open questions to answer, and I'm hoping that these will get resolved as I start implementing things.

Saturday, May 23, 2009

Wikibooks Templates

I've created a handful of new books in my time at Wikibooks, and I find that there are some processes that are common to each: We need to create a table of contents, we need to create various templates and categories, etc. Being a programmer and therefore very lazy, I try to find ways to automate these kinds of processes as well as I am able. To this end, I've created a series of what I call meta-templates, or templates that are used to create other templates. The goal of these meta templates is two-fold: First, to make it easy to create new templates quickly, and to hide a lot of the "dirty" formatting stuff that new users shouldn't have to worry about. For new and inexperienced users this is a huge boon. They can substitute one meta-template and get a full-featured header template to use in their new book. Certainly beats having to make a template from scratch, especially since our documentation in this area is still far from complete.

Let's look at some of the things that a good header template will want to have:
  1. The name of the book, which will probably double as a hyperlink to the TOC
  2. "Next" and "Back" links, for easy navigation
  3. Categories. The template should put all pages where it is transcluded into a category for all pages in the book.
  4. Some documentation, to explain to users how to use the template.
A lot of these items should only appear conditionally: Next and Back links should only appear if needed. There is no "Back" on the first page of the book, and no "Next" on the last page. Documentation about the template should appear on the template page but should not be transcluded with the template into every page in the book. The entire template will also not want to be displayed in a printed version, for various reasons. Let's look at how to implement that last requirement in wikitext:

<includeonly><no</includeonly>include>[[Category:Exclude in print]]</includeonly></no<includeonly>include></includeonly>

This is quite a mess! But that's the purpose of the meta template, I write this mess once and nobody else ever has to do it again. When we substitute the meta template to create a new navigation template, we get this:

<noinclude>[[Category:Exclude in print]]</noinclude>

So that makes more sense to see the result. We have to do all the mess above because if we didn't break up the noinclude tags in the meta template, they would disappear when we substituted into the navigation template.

Let's look at the related, but more complicated, problem of having next and back links that appear only when values are provided. Here's the code that I have to use in the meta-template:

<includeonly>{{</includeonly>#if: {{{last|}}} | ← [[{{{book}}}/{{{last}}}|{{{last}}}]] | }}

Again, without the includeonly tags put in those weird places, the {{#if:}} tag would disappear when we substituted. Now when we substitute, the includeonly tags disappear, leaving a perfectly-formed tag in their place:

{{#if: {{{last|}}} | ← [[{{{book}}}/{{{last}}}|{{{last}}}]] | }}

This string of stuff isn't exactly clean and isn't easy to read either, but at least the new users don't have to write it themselves. Anything we can do to decrease the barrier to entry for aspiring new book authors is always a good thing.

Friday, May 22, 2009

My Personal Roadmap for Parrot 2.0

I've drawn up a handy little roadmap of the tasks I want to get completed before Parrot 2.0 is released. This is by no means a comprehensive list, nor is it some kind of sign of task ownership: These are items that I want to see implemented, and if nobody does them first I will try to do them myself. I've helpfully color-coded this graph. Green items I finished yesterday, Red items I am currently blocking on, and Blue items are things that I would like but that I don't reasonably expect will make it in before 2.0.

The arrows represent a general flow for these projects. They don't necessarily show strict dependencies, just a vague notion I have that combines dependencies, priorities, and my personal work flow. There are some items that I am just going to do in a certain order.

There are four main channels for this work:
  1. GC: Yesterday I merged in the branch with my most recent round of GC cleanups. Next on the task list is to get a proper incremental GC core implemented and integrated. There's some cleanup work to do first to support it, but I think I have a good plan to get it started.
  2. PCC: Once Allison gets done with her major refactor that's in the works (she estimated they will land before 1.4) I want to get my grubbly little fingers deep into the PCC system. First step is turning Contexts into PMCs, and then lots of refactoring, optimizing, and bugfixing after that. A big part of all this is getting subclassing working seamlessly in PIR, an area where we have a few outstanding problems. We also want to make sure we get some things implemented that Rakudo is blocking on.
  3. IO: There are some cleanups needed by the subsystem, but not many. The big task here is getting an asynchronous IO system implemented.
  4. Documentation: I've found a number of places in the PDDs that need updating, and some areas in the book that need cleaning before it heads to press in June. After that, we want to expand the book from a simple "Users Guide" to a full-on "Developers Guide".
So that's my basic list of things that, life willing, I'll try to have implemented by Parrot 2.0. This isn't really a comprehensive list either, I imagine there are a few other things that I'll end up working on between now and then too.

Thursday, May 21, 2009

New GC Branch Started

Last night after the release had shipped I created a new branch for continuing my cleanup work in Parrot's garbage collector. The goal of this particular branch is to tackle some of the items in the GC tasklist on the Trac wiki. You'll notice that many items related to GC have been completed already, and I've done a lot of GC work recently that wasn't even planned on this tasklist page. Most of the things left on this list, besides the understated "Create an incremental tri-color mark GC module" task involve simply moving files around. Moving those files around (and hopefully cleaning and streamlining everything just a little bit more) is the goal of the current branch. Once that's done I'll be able to focus more on the Incremental GC core that I've been planning and idly working on for a while now.

The initial implementation of the incremental GC core is going to have some similarities to my GSOC work, but won't be so ambitious. I won't be using separate headers, for instance, and I will be relying on the PObj flags for marking instead of trying to use a separate card for the purpose. Having an incremental GC means that we will need a way to save state between increments, and I'm not entirely certain how I want to do that yet. The current MS core uses a trick to overlay a linked-list structure onto the PMC to maintain a free list. However, we can't really reuse that trick to put items on a "partially marked" list because we can't be overwriting data in the structure willy-nilly. We could add a new pointer field to every PMC for this purpose, but that seems like a bit of a waste to me.

Also, I'm still looking into using the Boehm collector for Parrot, it would be a cool little experiment to see if we could get that working and if so what the performance would look like. Just one more item to toss onto the top of my to-do list!

Wednesday, May 20, 2009

Calling Conventions Work

Yesterdays ParrotSketch meeting was quite interesting. We got through the reports quickly, and I asked a bunch of questions that got quick answers, and then we moved on to a discussion about Calling Conventions. My planning for Context PMCs is just a small part in an ongoing stream of refactors, changes, and improvements to this critical subsystem.

Allison, as I've mentioned before is getting this party started with some serious refactors to the way function calls happen. All calling paths are being migrated to use CallSignature PMCs, which contain the call signature, the list of passed arguments, the list of returns, and assorted other information. Patrick and Jonathan were also asking for some features that have been on the TODO list for a while: named-only parameters, named-if-possible parameters, and an ability to override the invoke VTABLE from PIR without TEH FAILZ.

The current semantics of named parameters is to fill in the list of parameters with the given positionals first, and then fill in any extra named parameters that have been provided. However, this system doesn't support a case where a parameter should not be filled by a positional parameter in any circumstances; that is if the parameter is "named only". It also doesn't count the case where we want a parameter to attempt to be be filled by a named if available, and a positional otherwise (a reverse of the current semantics). Apparently this issue was discussed at length on the mailing list, and I need to review all that. I think, based on what we talked about yesterday, that the parameter declaration flags are going to look like this:
  • :named, parameters that can be filled in by a positional argument or a named argument otherwise
  • :namedonly, parameters that can only be filled in by a named argument
  • :lookahead, Parameters that can be filled by named arguments preferably, or positional arguments otherwise (reverse of :named)
Another issue that was brought up, which is near and dear to my heart, is improving PIR overrides of the "invoke" VTABLE. The things currently work, the invoke VTABLE does not have access to the self keyword. Without being able to easily get a hold of the PMC reference that was invoked, it is useless and nonsensical to try and override invoke from PIR.

The reasons why we have such difficulty with this particular issue because of two reasons: The way that we pass arguments around in PIR, and the semantic differences between different notions of the word "self". Consider two cases:



In the first case, the $P0 object is being invoked, which will call the invoke VTABLE on it. In the second case, a method on $P0 is invoked instead. These are obviously two different cases, so it doesnt make sense that within the subroutine body we refer to this quantity by the same name both times. In the first example, $P0 is the subroutine being invoked, and in the second it is the invocant on which a subroutine (method) is invoked. If we invoke a method directly as a subroutine without an invocant, do we expect that self will refer to anything at all, much less the Sub object itself? And when we do invoke the method on the invocant, we obviously expect that self refers to the invocant. Seeing the difference here, we have to assume that Sub should not change it's behavior based on the way it is called, or should even be aware of the way it's invoked. That's bad for encapsulation.

The solution here is to provide two separate access points for the two separate concepts: a self keyword which represents the invocant, and something else (like, say, this_sub) that will refer to the currently executing Sub PMC. Normal methods will use the former without breaking anything. The invoke VTABLE can use the later to suddenly become useful.

There are two methods on the horizon to make this possible: Allisons work where we will have a CallSignature PMC made available from which the current invocant and the current Sub PMCs can be extracted, and my work where we will have access to a Context PMC which contains these same pieces of information. In the long term future, as I mentioned in my previous post, these two PMC types are likely to be merged together.

Allison says her stuff is scheduled to land by the 1.4 release. I'll definitely be doing my Context work shortly after that (I don't want to be developing two separate in-depth branches on the same subsystem simultaneously, to prevent TEH FAILZ). So things will get moving soon on all this work, and there will be much rejoicing.

Tuesday, May 19, 2009

GC and Other Miscellaneous Cleanups.

Spent most of my time this weekend working on more GC cleanups. The work I talked about previously was in improving the API, today I am working to clean up everything behind the curtain. It wasn't without it's bumps, but I think all this cleanup work is having a real effect on the quality of the GC, and the ability for us to eventually implement a better core for it. There are still some warts here and there, and I'm perpetually unhappy that several unused and unworking code files can't be deleted yet, but it is getting better.

Today is the Parrot 1.2 release, and whenever a release comes up on the horizon I basically put all my work on hold. I don't want to be committing changes to trunk except bug fixes, and I know branches tend to get merged in before or after a release, so I try not to have one of those outstanding to avoid merger messes. I do my testing, report bugs, and do what I can to clean up low-hanging bug tickets.

One particular ticket that had been stuck in my craw was a problem about subclassing an ExceptionHandler with a PIR-defined type. I had been examining the issue in the debugger for a while without finding anything that looked too out-of-the-ordinary. Finally yesterday I had my epiphany: The throw routine was attempting to access a non-inherited attribute in a way that wasn't throwing an exception. I added in a small fix to avoid this and magically the test program started running as expected. There are definitely some limitations to this method of course, but it's a step in the right direction. It's worth noticing here that if the attribute in question was a Context PMC instead of being a struct Parrot_Constext *, the attribute would be inheritable and everything would have worked as expected without much change. So that's more impetus for us to get that work under way ASAP.

I'm the release manager for June, and will be pushing 1.3 out the door on the 16th. A few more GC fixes for me to tackle after the release, and a few other grandios plans at the top of my task list are also scheduled for June. Plus, I'm going to be at YAPC. June is going to be quite the busy month for me!

Monday, May 18, 2009

Linux On the Desktop

Slashdotted this morning was a very interesting list of reasons for why Linux is "Not Ready" for general use on the desktop. It's quite an interesting list, although I would remove some things from it and add others.

Far from being an attack on Linux, I see this list as being a labor of love. Consider the parallel of chromatic criticizing Perl 5, and there almost isn't anybody I can think of who loves Perl more then chromatic does. He has a lot of good things to say about Perl, but he's also very vocal that things need to change to keep it strong and modern.

It's like punishing your children for misbehavior: You love your children, but sometimes you need to teach them important lessons the hard way.

I started using Linux a long time ago. I made what I now see as a mistake, taking a crash course using the "Core" linux distro. Core is a minimalist distro where users are expected to be power users who build the rest of the distro from source. By starting at the bottom, I was hoping I would learn quick, and I did. I quickly learned to scrap Core and move to a more full-featured distro. From there I moved on to Red Hat (back before it was Fedora, but not long before) and used that for a little while. At the time all this experimentation was happening my family was using Windows XP for their computing experience, and I have to say that the differences were staggering. Their computer was more usable on average (it did get virus'd up pretty regularly and needed cleaning and reformatting on a regular schedule), and it was easier to make configuration changes. Sure I had all the power in the world to monkey around in configuration files, but when you compare the time it would take me to look up a configuration change on the internet, find the appropriate config file, make my changes, reboot to make sure I didn't destroy my computer, etc, to the time it would take to make the same change in Windows it was almost comical that I was using Linux at all. Hell, Windows power users stopped having to edit autoconfig.bat by hand back in Windows 95.

My third foray into Linux was with Ubuntu, and I am much happer with it now as a full-featured replacement for Windows. With RedHat several years ago I still felt like I needed to keep a Windows PC around. Imagine trying to make changes to your internet configuration when you needed to look up all the complex documentation for it on the internet, to get a feeling for why I needed the extra machine. But with Ubuntu, the internet just works every time in any configuration I need.

That said, Ubuntu is still not perfect and I still run into the rare occasion when I see that Windows has implemented some particular thing better. The Asynchronous IO stuff that I wrote about a few days ago is an example of a place where I think Windows has a better approach at the fundamental level then Linux does. Of course that's not something that directly affects the experience of the average desktop user although for particularly sluggish programs it might help boost performance. I've also run into the kinds of problems that the list author mentions about audio and video configuration. I had a few really lousy Skype calls from my laptop before I was able to find the bad settings that were making everything sound bad (and I'll give you a hint, they weren't in the audio mixer that has the icon on the task bar).

I use Linux on my only computer, and I use it for almost all my needs. I do double-boot into Windows Vista sometimes, mostly for testing Parrot on that platform prior to a release. However, I do use my computer in a way that your average PC consumer does not: I do very little with audio or video and I'm willing to do a lot of stuff from the commandline. Is Linux ready for general acceptance by the PC consumer public? Maybe not yet but it is getting there in a hurry. I think the work that's being done by Canonical on Ubuntu is quite encouraging and maybe in a few more releases they will have reached that milestone where, for the average Joe Schmoe computer user, Linux is more usable immediately then Windows is. I hope that time comes soon.

Sunday, May 17, 2009


It's official, I'm going to YAPC::NA this year. I had been planning on it for a while, but finally this morning I managed to buy my ticket and reserve my room.

I reserved a room for Sat, Sun, and Mon nights. I'll be coming home Tuesday afternoon. I'm going to try like hell to get there by 9AM so I can make the whole PVM workshop. It will be cool to finally meet some of these people!

I'm going to be driving from Philly to Pittsburg, so in the unlikely event that somebody else wants to carpool let me know.

Friday, May 15, 2009

IO System: I Was Wrong

Today I opened a ticket about how the layering portion of the IO system specification needed to be implemented. It turns out I was wrong: Despite the fact that the PDD claimed a layering system would be used and the fact that there were some hints about it in the code, it is no longer the current plan that the IO system use a layering approach. So tonight after talking to Allison about it, I went in and ripped the confusing passages out of the PDD. I also went through and removed some unused code that wasn't being used.

So there really isn't anything standing in the way of a proper AIO implementation, except maybe a handful of enhancments to the threading and scheduling systems (I haven't dug into either too deeply yet, so I can't say for sure). That's good news, fewer things in the way means it's more likely we're going to get proper AIO by 2.0.

GC Next Steps

The GC is properly encapsulated now, at least for the most part, and now it's time to move on to the next steps in the process. Encapsulation of an interface is one thing, having the interface be usable and even elegant is something else entirely. Parrot's current GC API isn't really either of those things: It is specifically tailored to the current stop-the-world GC and is far from being an elegant solution. Before we can know what we want the interface to look like ideally, we need to know where we intend to go with the GC in the long term. I've already talked about a few things, but I think it's worthwhile now to recapitulate on a few of them:
  1. Incremental Behavior. The GC as it is right now does the whole shibang at once. It marks everything, from the most global root down to the most insignificant property, and then it sweeps everything. This can cause distinct pauses in Parrot's execution that the user might become aware of. Instead of doing everything at once, we can try to break it up. Do small bits here and there until the whole job is done. In this case, we're still blocking for the same amount of total time, but it's spread out and therefore less noticable. Being able to stop and restart the sweep is going to require a tricolor marking scheme (in the most simple case).
  2. Asynchronous sweep. Once we have the set of all PMCs marked and clearly demarcated as being either alive or dead, there is no reason why we should have to wait for the sweep to run synchronously. The sweep could be contained in a separate thread (OS thread, not Parrot thread) and run asynchronously. After all, once something has been marked as "dead" we don't need to worry that it would suddenly come back to life while we are sweeping. We do need to worry that newly-created objects are created as being alive so they don't get swept as soon as they are created, but that's a small issue.
  3. Generational Behavior. By separating out PMCs into multiple generations based on longevity, we can intelligently choose to avoid sweeping PMCs that are not likely to be dead. This helps the GC sweep to take less total time because we are sweeping fewer things. The downside is that there are false positives with this method, a PMC that is not alive may not be swept in a timely manner. We would likely need to use some form of hackery to make sure PMCs that needed timely destruction did not end up in an older generation.
  4. Copying/Compacting Behavior. We want to be able to keep our memory tidy, and be able to return unused resources to the OS in an organized and timely way. The method for doing this is to use copying/compacting collector. Objects that are marked as being alive can be moved to a single location, freeing up space elsewhere in the pool.
  5. Ordered Destruction. With ordered destruction we make sure that a PMC is only freed after all the PMCs that it relies on are freed. We would need a way to specify these kinds of reliance relationships, and then we need to be able to adhere to them.

It's worth pointing out here that not all of these features are going to be implemented in every core. Some cores are going to be incremental. Some are going to be generational. Some are going to be copying/compacting. Some are going to combine two or more of these attributes. It's also worth noting that once we make a guarantee of ordered destruction, all cores are going to have to subscribe to that. It's a feature where once we have it things are going to start relying on it.

These ideas on the table, we need to think about some ways that the GC API is going to need to change to support them in the future. Here are some ideas that I have:

  1. We have a function Parrot_gc_mark_and_sweep that performs a stop-the-world run. We should probably break this up into Parrot_gc_do_mark() and Parrot_gc_do_sweep. This will enable us to support collectors that do things like asynchronous sweeping where the mark and the sweep are handled in very different ways at different times. It will also help to streamline and consolidate places where we need to check the mark and sweep block levels.
  2. The function Parrot_gc_mark_and_sweep (or the separate Parrot_gc_do_mark and Parrot_gc_do_sweep) are not really well-suited for incremental usage. They get called from places that expect results, such as in the PMC allocator when there are no more headers available so we run the gc to try and free some up. If the GC is incremental, we will only run an increment when we are in need of new headers, and we will end up allocating lots of new arenas by the time we finally get around to sweeping all our accumulated dead. If we have an incremental version of this function (which non-incremental cores can simply choose not to implement), we can call that from places like inside the concurrency scheduler that happen more frequently.
  3. I'm thinking about an interface where we can "store" PMCs that aren't really being used currently. So we would call Parrot_gc_checkin_header(), which will store the header internally, possibly allowing it to be moved/copied/compacted, and will return an integer receipt. Later, we could pass that integer receipt to a function like Parrot_gc_checkout_header() to return a pointer to the PMC where it is now. Another idea would be to use a single function like Parrot_gc_relocate_header() that would take a pointer to a PMC and return a new pointer to the same structure that may have been moved in memory. This could be used to explicitly compact a PMC when we know that there is only one reference to it in existance (so a PMC could call it on its internal members on VTABLE_mark for instance, or we could extend Parrot_gc_mark_PObj_alive to take an extra boolean flag that says whether the PMC is known safe to move or not. Lots of possibilities here, but it would provide us a way to smoothly transition Parrot to a copying/compacting GC scheme.

There are lots more ideas that I have (and will talk about at length) concerning the GC, and I'd love to hear any other opinions or ideas that other people have about it as well. Maybe, if I can find the time I will try to get started working on it some more soon.

Thursday, May 14, 2009

On "Science"

I found an interesting blog post the other day (via a link from that talked about the decline of intellectualism in the conservative movement. It's quite an interesting read, I think, regardless of your own political bent. I did notice that they've gone out of their way to avoid terms like "Fiscal Conservative" and "Moral Conservatives", because it does sound like they are referring to one group as being "intellectual" and the other not. It's an interesting read in either case.

What I found most interesting was not the blog post itself but instead some of the comments that this post generated. Of course, this is true of any partially-controversial blog with decent exposure: All sorts of people come out to share their hastily-formed opinions anonymously. Immediately interesting is that some of the most ignorant of these comments are coming from the very "emotional conservatives" that the blog post itself is lamenting the rise of. Here's one particularly interesting comment, edited for brevity and clarity:

I love when Democrats pretend they are intelligent on the global warming issue. They love to pretend Republicans ignore science on this subject and that science is the be-all end-all for policy.

But then they ignore the scientific fact that life begins before birth. They use judicial activism to come up with an UNSCIENTIFIC definition for life.

Partial Birth Abortion is murder. This is a scientific fact. Barrack Obama is the worst abortionist to ever step foot in the white house. While we debate how to "stimulate" the economy with his Keynesian, Barrack is sending our money overseas to fund abortion. The man supported infanticide. At what point, do all you brilliant democrats who preach science wake up and realize how crazy you all are...?

What's most interesting here, and something that I find to be interesting throughout these interwebs at large is the fundamental misunderstanding of what "science" is. Science generally has two components: The terminology which attempts to be very particular and precise (even on occasion being in stark contrast to the common understandings of certain words) and the methods which aim for precision and repeatability. Saying something like "partial birth abortion is murder, that's a scientific fact" is really not a true statement, because it's not scientific. I won't suggest an opinion about whether or not the procedure is actually murder or not, this isn't a political blog. "murder" is a legal term, not a scientific one. It doesn't have a precise definition in such a way that any person can look at any case and say "yes this is a case of murder" or "no this is not a case of murder" for all possible cases. There are gray areas: Think of self defense, or accidental homicide, or war. Even though these are cases where one person ends the life of another person, I doubt we would brand all of them "murder", at least not all the time. Most people would also not call the act of ending life in general as "murder", killing a plant or an animal doesn't usually label somebody as a murderer.

When it comes to global warming there are several facts that can be verified scientifically: global Carbon Dioxide gas levels have risen steadily since the industrial revolution, global temperatures have also risen during that time in a way that correlates positively to the level of carbon in the atmosphere, other known causes of global temperature variation (solar cycles, orbit variation, volcanic activity, etc) are measured and accounted for and do not by themselves account for 100% of the measured temperature differences.

I use the word "correlates" in the sentence above for a particular reason: Correlation does not always imply causation, but in the absence of other causes a strong correlation can point us towards the likely cause of an observed effect. We can think about a trial where a suspect is convicted based on the correlation of the facts of the crime and the means, motive, and opportunity of the suspect. Imagine that some people can be given the death penalty (which some would say is a form of "murder") based only on the correlation of certain pieces of evidence and the lack of a better explanation. We can take away a person's life or lively hood based on correlated evidence, but we can't look at a wide array of evidence of man-made climate change and say "you know, this makes good sense, we should explore this issue a little further". At the very least, it's worth exploration with an open mind, and if we can improve ourselves by becoming more mindful and considerate of the ecosystem at large (whether it's an absolute necessity or not) all the better.

I never understood the resistance to the idea of global warming that some people get so fervent about, but then again I know that proponents of the idea aren't always the best salespeople. Is it really so unthinkable that humanity has the power and ability to modify the climate? Is it unthinkable that we could have been inadvertently exercising that power without having given it the proper forethought and consideration? It's ignorant to think that our chemistry, our waste products, our deforestation and our geoengineering projects don't have some net effect on the atmosphere, the climate, and the planet at large. It's ignorant to think that humans don't have the power to change the planet, and it's ignorant to think that we as a species would never be so ignorant as to allow it to happen in the first place. The question is whether we are we ultimately doing these things or not, and I think the answer is "There's a lot of evidence to support it, and a lot of bright thoughtful people believe it to be so".

So that's my rant for today. I try not to let my blog get political but political discussions, especially those carried by anonymous internet users, tend to display the kinds of ignorance, unreasonableness, and pseudo-science quackery that really gets under my skin. I'm in Houston today on business, and should be back in good-ole' PA tomorrow.

Wednesday, May 13, 2009

Software Engineering Licenses

Ovid wrote a very interesting blog piece about the idea of software engineering being subjected to consumer protection laws in Europe. He is seemingly in the minority by supporting this idea. I was writing a comment to agree with him, but as the comment got longer and longer I decided it would be better served as a complete blog post.

In the USA there is a licensing for engineers called "Professional Engineer", or "PE". The idea isn't specific to the USA, but that is the only version that I am familiar with. Getting a PE license requires an apprenticeship under an existing PE and a strenuous and comprehensive exam (actually two exams, one to qualify before the apprenticeship and one afterward). Once you have it you are able to do certain jobs that you legally could not do otherwise. For instance, almost all Civil engineers must have a PE because public infrastructure work almost always requires it. Most Mechanical engineers get it as well, at least that I am aware of. For electrical engineers the number is smaller, typically only people who work on public power or transportion systems get it. I know that when I was in school for EE and looking to pursue PE licensing for myself, it was very difficult for me to find potential mentors to apprentice under, and it was even harder for me to justify the effort considering I wasn't going to be in a field that required it. However, that's not to say that it's impossible to get a PE in EE, and there are plenty of other per-project certification schemes such as Intrinsic Safety that EEs need to follow (I know we spend a lot of effort at work pursuing IS certifications for our products, and it's far more strenuous of process then any regression testing of software that I've ever experienced). In software engineering, which is where I am now, it's almost impossible to get PE even if you needed it (there are basically no PEs in software engineering to apprentice under), and I'm not really aware of any certifications for having software approved as being "safe" in the legal sense.

PE specifies a few things: First, you must demonstrate a level of competency in your work. Second, you are only approved to do licensed work in your area of expertise. A Civil PE cannot do mechanical or electrical work that requires a PE, for instance. Third, there is legal responsibility. Plans and projects need to be approved by a PE before they can be implemented, and the PE is responsible for the outcome. The PE license also specifies who can be licensed: You must have at least a bachelors degree from an accredited institution, you must pass a qualification exam, you must apprentice under a licensed PE, and then you must pass a comprehensive final exam. And the accreditation, as I will tell you, can be quite strenuous. The bozo schools get weeded out by accreditation agencies before they can graduate bozo engineers. If you picked up a degree from John's Basement University, I'm sorry but you don't get to design bridges (at least not ones that people drive on).

There's a difference between consumer software and infrastructure software. If your software can have a negative impact on a person's health or well-being, and it fails, there should be some resource and some place to lay legal responsibility. We don't want Joe Schmoe with a AS in Information Technology writing the software to control your antilock breaks. We don't want a former Mechanical or Electrical engineer who decided to "switch to software" coding the safety fallbacks for a nuclear power plant. We don't want people who aren't properly licensed writing the autopilot software on a plane or the failsafe software on a missle, or the routing software of the electrical grid. We want people who are properly trained, competent, and accountable to be writing all these things. I've seen enough shitty code written by "software engineers" already, people who don't even specialize in it can't be expected to do any better (although some certainly can do it very well indeed).

If a bridge collapses or if a power line lands on a playground, you can be damn sure an engineer somewhere will be held responsible for that. And knowing that their asses are on the line will prompt these people to be more careful, more conservative, and more circumspective before signing off on a shoddy design. Why should it be any different for safety-critical software? Why shouldn't software engineers in safety-critical industries be held accountable for their own bad designs and bad, uninformed decisions? Why wouldn't we as an industry want to cover our own asses like that?

On the flip side, if there is a legal requirement that critical software be approved by a licensed software engineer, suddenly those engineers have more power over deployment: software doesn't get shipped unless it is properly approved, and no amount of business politics will force a deployment before that. Good developers with proper licenses will be able to earn more money for their work too, because they become a legal necessity
. And there will be more job security for these people as companies won't be able to replace good licensed developers with new graduates, and won't be able to outsource all their coding needs to the bottom bidders around the world. It's both amazing and depressing the kinds of people who are allowed to write critical software in this world, and more often the costs of software development right now are treated as overwhelmingly more important then the long term costs in terms of money or public safety. We can see this same principal at work in all sorts of other industries too (Think about the banking industry where the pursuit of short-term gains brought the whole world into recession). Do we want to allow that kind of process creating the software that we all rely on?

I think that requiring software engineers to be licensed in some cases and to hold them accountable for their work will be a good thing overall. Require that important software only gets written by good software writers. Require that decisions about deployment of important software get made by the engineers, not the marketing department. Require that important software not be an exercise in "failure probability", and give people guarantees that things that are supposed to just work do, in fact, just work. It's time that software engineering graduate from a field of amateurs and non-accountable dabblers and become a field of precision, science, and results.

Tuesday, May 12, 2009

Contexts into PMCs: The plan

Turning Parrot_Context structures into PMCs is not going to be a trivial process. There are a number of reasons for this:
  1. Contexts are very complicated structures
  2. Contexts are very central to the operation of Parrot, so any conversion will affect the majority of the codebase
  3. The VTABLE interface is designed for data items that serve a singular purpose, while Contexts serve several purposes in Parrot. There isn't going to be a good fit between what needs to be done with Contexts to the VTABLE interface.
Since VTABLEs are relatively limited, any Context PMC is likely to require a number of direct accesses to the various internal data fields using the {GET|SET}_ATTR_* macros. If we tried to use a STRING-based property lookup to access internal members, it would be a major performance bottleneck for Parrot. A STRING-based property lookup could also be available to allow introspection from the PIR level, but that's separate issue entirely (although a nice perk to think about!)

Here are some ideas for what the VTABLE interface might be able to be used for:
  • init: This will throw an exception (at least at first) because we don't want Contexts being created from PIR initially. However, eventually we are probably going to want to be able, not only to access Contexts directly from PIR, but actually to subclass and extend them. I'll post more information about this idea tomorrow.
  • {get|set}_*_keyed_int: These will provide accesses to the various register sets. So "VTABLE_set_string_keyed_int" will set the Nth S register to the given value, "VTABLE_get_pmc_keyed_int" will get the Nth P register value.
  • {push|pop}_*: These will provide ways to pass positional arguments and return values, and then retrieve them again.
  • {get|set}_*_keyed_string: The hash interface, used to set/retrieve named parameters and returns. Note that this could also be used to supplant LexPad/LexInfo PMCs as the way to handle lexical variables. Imagine for a second that named parameters to a subroutine could be a direct subset of named lexical variables, and accessed in the exact same way. I won't get too much into speculation today, but I will in my post tomorrow.
  • invoke: Not entirely sure about this one yet. Since we are intending to unify Contexts and RetContinuations, maybe an invoke on the Context will perform the return. I don't know if we want to do anything else with this too.
  • {get|set}_pmc: There are a number of PMC values pointed to by the Context: The current Sub, the current Invocant object, the associated LexPad, the current namespace, the returns signature, and the lexically-scoped list of handlers. Which one would we access through get_pmc, if any? I have no idea. In reality, we will probably not use get_pmc, and instead use the ATTR accessor macros to deal with all of these values.
  • add_parent: There are two different Contexts that could be considered the "parent": The previous Context in the call chain, and the outer Context in the case of a closure. Either could be the "parent" in this case.
So that's a brief overview about what the VTABLE interface to the Context PMC will look like. Obviously we will also implement basics, like destroy, init_pmc, visit, mark, freeze, thaw, etc. These will all do the kinds of operations we expect of any PMC.

Here are some of the initial steps that I think we can take to replace struct Parrot_Context with Context PMCs. I think that if we can follow these steps we should be able to keep Parrot stable throughout the conversion process without allowing anything to be broken for too long:
  1. Encapsulate all the existing fields into a new PMC with no vtables and no methods
  2. Modify the PMC to remove fields used for ref-counting, and replace pointers to other contexts with pointers to Context PMCs.
  3. Modify the Context allocation/initialization routines in src/gc/register.c to allocate a new Context PMC instead
  4. Modify the CONTEXT(interp) macro to be ((Parrot_Context_attributes)PMC_data((interp)->ctx)) instead of ((interp)->ctx.state). At this point, the switch over to the new Context PMC should be complete, and Parrot should be using the PMC directly (although with very little respect for encapsulation) for all it's context-related needs.
  5. Implement various VTABLE methods to handle some common operations, and replace direct ATTR accesses with VTABLE calls instead.
  6. Optimize like a crazed elephant
  7. Profit!

What's really interesting in all this, which is something I've alluded to earlier, is that we open the possibility for Contexts to be manipulated in PIR code and introspected from there as well, things that are not currently possible. What kinds of cool control structures and situations will people develop once we have this?

Monday, May 11, 2009

Reading Data from the Serial Port

At work I'm building a program that is, in part, a serial port console like HyperTerminal. It's not a pure drop-in replacement, we have some specific requirements that HyperTerminal didn't quite meet. However, a lot of the port-handling logic in my program could probably be repurposed for creating a more faithful drop-in replacement if the need arose.

The biggest problem for me at the beginning was deciding how to read data from the port. Obviously, we can't do the reading synchronously, in the case of a TTY-like program that just doesn't make any sense. I had two real options for asychronous operation:
  1. Create a worker thread that reads from the port, sleeps for a certain timeout, and then repeats.
  2. Use the DataReceived handler on the SerialPort object to automatically call a callback when incoming data is received.
People who read my discussion about AIO in Parrot will probably recognize that item #1 does not appeal to me at all. First there is the issue of synchronizing the thread and getting it to start and stop when I need it to do each. I also need to be able to pass settings to the reader, which would require cross-thread synchronization logic that could get really messy really quickly. Plus, if we have a thread timeout of 1second for example, there could be up to a one second delay between the time data is received and when it becomes visible to the user. We could easily set a timeout to zero seconds, but then we have a background thread endlessly looping over the port, usually reading nothing. Not exactly a performance killer, but still needlessly wasteful. Nothing we are doing is so time-critical that a 1second sleep timer for instance between reads would kill us, but it's still a clunky-feeling mechanism that will make the interface look amateur.

On top of all those reasons, as if I need another one, I feel like using an explicitly-managed thread here is a particularly inelegant solution. In short, I decided to use the DataReceived event handler to do my port reading (when I put my port into asynchronous read mode, that is).

That question out of the way, I needed to decide how to do input buffering. Do I buffer by line (SerialPort.ReadLine), "buffer" by character (SerialPort.ReadChar), or do I not buffer (SerialPort.ReadExisting)? On the one hand I need to support both buffered and non-buffered input. On the other hand I don't want to be stacking up DataReceived events with Timeout events endlessly, because that could create a stability problem.

My design goes as follows: I use the DataReceived event handler to signal me when data is ready. From within the handler I use SerialPort.ReadExisting to read the incoming data into a buffer, and from there slice and dice the input into the forms needed by modules higher up in my program. I admit that this may not be the most attractive or elegant code I have ever written, but it has demonstrated itself to be very performant and robust solution for my needs:
private string readBuffer;
private SerialPort port;
private bool unBuffNeedNewline = false;

private void InitPort()
this.port = new SerialPort();
this.port.DataReceived += new SerialDataReceivedHandler(DataReceiver_Handler);

private void DataReceiver_Handler(object sender, SerialDataReceivedEventArgs e)
if(this.bufferMode == ConnectionHelpers.ReadBuffering.LineBuffered)

private void DataReceiverLineBuffered()
try {
string x;
lock(this.readBuffer) {
x = this.readBuffer + this.port.ReadExisting();
char[] delim = new char[2] {'\n', '\r'};
while(x.Length > 1 && (x.Contains("\n") || x.Contains("\r"))) {
String[] ary = x.Split(delim, 2);
this.ReadHandler(ary[0], true);
x = ary[1] + this.port.ReadExisting();
lock(this.readBuffer) {
this.readBuffer = x;
} catch(Exception e) {
this.StatusReporter("Serial Read Error: " + e.ToString());

private void DataReceiverCharBuffered()
try {
string x;
lock(this.readBuffer) {
x = this.readBuffer + this.port.ReadExisting();
while(x.Length >= 1) {
string c = x.Substring(0, 1);
if(c == "\r" || c == "\n") {
this.unBuffNeedNewline = true;
x = x.Substring(1) + this.port.ReadExisting();
else if(this.unBuffNeedNewline) {
this.ReadHandler("", true);
this.unBuffNeedNewline = false;
} else {
this.ReadHandler(c, false);
x = x.Substring(1) + this.port.ReadExisting();
} catch(Exception e) {
this.StatusReporter("Serial Read Error: " + e.ToString());

The function "ReadHandler" takes two arguments: The string value that's read (without carriage return or linefeed characters) and a boolean value indicating whether the given string is followed by a newline or not. Further up the call chain the display function will take that information to display timestamps and do other per-line formatting fanciness. It's worth noticing that since we are calling from within an event handler, ReadHandler probably needs to call BeginInvoke to pass it's incoming data to the GUI (that's what my code does, anyway).

The way I lock the readBuffer repeatedly was something that I figured out later. When data is incoming quickly (like 115200 baud or higher) we can get multiple DataReceived events being triggered and stacking up if we lock through the entire function. This can cause noticeable program slowdowns when a number of event handlers acquire the lock subsequently. Instead, I lock smaller parts of the code and let additional handler instances run concurrently but with no input. Throughput performance is better in these cases, and is not noticeably different otherwise.

So that's my implementation of a buffered asynchronous serial port reader in C#. It's not perfect, but it gets the job done, has pretty good throughput, and is relatively fault tolerant.

Sunday, May 10, 2009

Turning Contexts into PMCs

I merged the GC branch in yesterday, and though the work wasn't without it's issues the GC API is now in a much better condition then it has been. That out of the way, and AIO still on the back burner, it's time to start looking towards the next project in my list.

This week, the project is converting Parrot's context structures, which at the moment are just normal C structures, into garbage-collectable "Context" PMCs. TT #596 contains some information about the work, although it's currently very sparse. This is not a new idea by any stretch: I've been thinking about it since last summer when I was doing my GC work, and my ambitions were so high I was convinced that all memory allocations in Parrot could be done through the GC—I still think that, but I'm more humbled by the sheer amount of effort required to make that happen. Not only does this project make it easier to manage contexts without all sorts of manual reference counting, there is also going to be a lot of potential for streamlining and optimizing parts of the codebase.

Parrot_Context structures form a linked list of sorts that represent the current calling context and all parent contexts. The context structure contains the array of I, N, S, and P registers too. It also contains a few data fields to help manage the CPS calling system. In short, Contexts are pretty central to Parrot and are very important.

The calling conventions system as it currently stands is pretty inefficient and messy. This isn't a new sentiment, I've blogged about it before (although I can't find a link right now). There are about half a dozen different ways to execute a PIR subroutine, many of those have multiple special-purpose interfaces. Allison is doing some great work in getting things to be more unified, and once she gets her work merged into trunk we will be able to start optimizing the new unified pipeline. A major part of those optimizations is going to be converting Contexts into PMCs.

For an idea of why, consider what happens in the unified calling path when we make a call (note that these aren't exactly in order):
  1. Arguments come in as an array, either as a va_list variadic argument array or as an opcode_t* serialization in bytecode. We also get an invocant object (if any) and a return address. The reason for the difference here is because we can be invoking a subroutine from PIR (via the invoke opcode) or via C (using Parrot_pcc_invoke_from_sig_object)
  2. Arguments and the invocant are added into a CallSignature PMC, which needs to be allocated and initialized.
  3. A new context is created, initialized, and a reference is made to it.
  4. The arguments from the CallSignature PMC are added into the registers of the Context
  5. The invocant, if any, is extracted from the CallSignature PMC and put into the interpreter structure as the current "self".
  6. A new RetContinuation PMC is created to handle a return call and stored in the Context.
  7. Using a series of Call_State structures, and a long loop, extract arguments from the context registers and associate them with parameters in the called sub.
RetContinuations are like Continuations, but single-purpose and lighter-weight, and they're only really used in these cases to provide the behavior of the "return" opcode. If Contexts are a PMC type, we can combine this with the RetContinuation PMC, since the later is never really used without the former. This kills step # 6 in the list above. We can then combine the Context PMC with the CallSignature PMC (since again, the one is almost never used without the other), we can get rid of steps 3, 4, and 5. Finally, combining the Call_State structure with the Context PMC and creating a custom iterator for it will leave us with only this:
  1. Arguments come in as an array, either as a va_list variadic argument array or as an opcode_t* serialization in bytecode. We also get an invocant object (if any) and a return address.
  2. The arguments, the invocant, and the return address are added to the Context PMC
  3. A custom iterator is created for the Context PMC and the subroutine parameters are extracted from it directly.
Quite a bit more streamlined, no? In addition to having fewer steps in the algorithm we also have a number of other significant savings: Fewer PMC allocations, Context PMCs can be allocated from Parrot's memory pools instead of needing to be malloc'd from the OS, and a large number of copying operations can be avoided since the data is put into one place and kept there. Plus, since the Context PMC would suddenly become garbage collectable, we can get rid of all the reference-counting code, and the custom GC hooks to clean up dead context memory. This is only just the beginning, there are major optimizations to be had throughout Parrot once this change happens. It all starts with turning contexts into PMCs though.

Some of my recent work on the GC API refactor has really exposed the current Context system API, and has made it easier to pull individual functions out into the new Context PMC VTABLEs. This is going to be a big project, but it's not unmanageably big. I'm probably going to get started on this work sometime after the next release (which is going to be on May 19th I think). I'll definitely be planning and prototyping before that, however.

Saturday, May 9, 2009

Parrot GC Refactor work

I spent a lot of time yesterday and this morning working on the GC API refactor for Parrot, and I've made some pretty good progress so far. One cool result from this work is that I am finding all sorts of places where the GC internal data structures were being manipulated throughout the codebase, some instances of which are very likely to have been causing the problems I experienced over the summer.

That aside, I'm pleased to say that this morning I managed to get the GC almost entirely properly encapsulated. All external-facing GC functions are in the API, and all other functions are kept private. In addition, data structures like struct Arenas, struct Small_Object_Pool and struct Small_Object_Arena are now only accessible from within the GC subsystem. This prevents the rest of Parrot from monkeying around in these structures, but also makes it easier for alternate GC cores to redefine them as necessary. The API isn't particularly clean or elegant right now, but it is encapsulated and that's a start. It's certainly more then we've had since I've been a Parroteer.

The downside to this work is that the number of functions in the GC API has increased pretty substantially, both from creating new functions and by moving old functions into it. Many of the new functions are simple accessors for internal datastructures. I've managed not to have all the new functions listed as "PARROT_EXPORT", which is a win for dynamic link time, so thats good at least. Some of the macros that controlled blocking and unblocking the GC are now redefined as API functions, which is something I've wanted to do for a long time now.

In my last GC post I guessed at which functions would be necessary in the API. Now that I've finished encapsulating it, I can tell you what functions are actually in the API now:
  1. Function to mark an object as being alive.
  2. Functions to initialize and deinitialize the GC subsystem.
  3. Functions to perform mark/sweep and compact actions
  4. Functions to manipulate pools between interpreters, such as merging pools, moving shared objects between pools, etc.
  5. Header allocation functions for PMCs, STRINGs, and other Buffer-like objects (ListChunks, Stack_Chunks, etc).
  6. Header reclamation functions to take unused PMCs, STRINGs, and other Buffer-like objects and add them back to the free list.
  7. Functions to add extension objects (pmc_ext and Sync) to existing PMCs, and functions to remove them again.
  8. Memory block allocation functions for allocating buffers of arbitrary sizes, such as storage for STRINGs. Also, functions to deallocate these memory blocks, and functions to resize them.
  9. Functions to determine if a given pointer is in a particular pool
  10. Functions to block and unblock various GC phases, and functions to determine if they are blocked currently
  11. Accessor functions to get information from the Arenas structure.
  12. Function to mark PMCs that need timely destruction
So this is a basic list of the things we have in the API right now. I'm sure there are more features we would like to add here eventually, but these are the operations that are currently used by the rest of Parrot from the GC.

I'm still not entirely happy with the encapsulation status as it stands now. There are a few places such as the freeze/thaw subsystem where accesses are made to the memory system internals that I really wish weren't being made. But, that's a relatively small issue because most of those routines happen at interpreter destruction time after the GC has already been blocked/finalized.

I want to get my branch merged back into trunk soon. I've hit a good milestone now and I don't want this branch to live for too long lest new problems arise. Probably some small cleanup and documentation work to do first, but it's not far off now.

Friday, May 8, 2009

Darbelo's New Blog

Darbelo, the GSOC student that I am backup-mentoring, has finally started his blog. The blog will be used throughout the summer to post updates about his project. I can say that his progress so far is pretty fantastic, and some initial prototypes of various things are already checked into his googlecode repo.

It's going to be an interesting project to watch, and the stage is set for a pretty exciting and productive summer.

Inheritable Menus in C#

I've been coding a program at work that is supposed to interface with and control a wireless sensor network. It's a heterogeneous network where each node may be one of several different types with different capabilities, represented internally by child classes of an abstract "WirelessSensorNode" parent class. To keep track of things, I display all the nodes and their children (things, including other nodes that connect through them) in a TreeView.

The problem I ran into was trying to create a context menu that would be displayed when I right-clicked on a TreeNode in the tree. Because each node class was different, I needed to display slightly different menus for each. I started the naive way, trying to create a new ContextMenuStrip item for each different class, and I ended up with a huge messy piece of logic like this:

object o = myTreeView.SelectedNode.Tag;
if(o is NodeType1) menu1.Show();
else if(o is NodeType2) menu2.Show();
else if(o is NodeType3) menu3.Show();

And each menu, because there were some options common to all nodes, needed to contain some duplicate logic. This was, in short, a mess and extending this system to add new node types (and therefore new menus) was a real pain. What I really wanted to do was something like this, and leave the logic of menu building to the classes themselves:

WirelessSensorNode n = (WirelessSensorNode)myTreeView.SelectedNode.Tag;
ContextMenuStrip menu = n.GetContextMenu();
if(menu != null) menu.Show();

So I decided instead to fix this system up yesterday to do the right thing in a better and more encapsulated way. The node classes themselves know what menu items they are capable of showing, my GUI should not be in charge of figuring that information out. So, I added this code to create a new attribute type, ContextMenuHandler for methods in a class, and a routine to read all methods with this attribute and add them to a menu:
[AttributeUsage(AttributeTargets.Method, AllowMultiple=true)]
public class ContextMenuHandler : Attribute
public string name;
public ContextMenuHandler(string name)
{ = name;

public static void AddContextMenuHandlers(System.Type c, object parent, ContextMenuStrip menu)
BindingFlags flags = BindingFlags.NonPublic | BindingFlags.DeclaredOnly |
foreach (MethodInfo method in c.GetMethods(flags)) {
foreach(object attribute in method.GetCustomAttributes(
typeof(ContextMenuHandler), false)) {
string name = ((ContextMenuHandler)attribute).name;
menu.Items.Add(new ToolStripMenuItem(name, null, new EventHandler(
delegate(object o, EventArgs e) {
method.Invoke(parent, null);

And I added this code into the abstract parent class:

public virtual ContextMenuStrip GetContextMenu()
ContextMenuStrip menu = new ContextMenuStrip();
Helpers.AddContextMenuHandlers(typeof(WirelessSensorNode), this, menu);
Helpers.AddContextMenuHandlers(this.GetType(), this, menu);
return menu;

Not a whole lot of code to write for the flexibility that the system gives me. For people who aren't familiar with attributes, they are metadata items that can be added to methods (or other things) and can be examined at runtime using reflection. With this system, to add a new item to the menu of a particular class, I only need to create a new private method in that class with the ContextMenuHandler attribute:

[ContextMenuHandler("Menu Item 1")]
private void Menu_Item_1_ContextMenuHandler()
MessageBox.Show("You clicked 'Menu Item 1'");

No need to "install" the new menu item anywhere, the simple existance of this method in the class will enable the item to be shown in that menu. This is great for encapsulation because I don't want to be having to modify all sorts of existing code, especially not unrelated existing code, every time I want to add an option to my right-click menu. Plus, it gives me the ability to inherit menu items from the parent class into the menus of the child types, without having to write code in each child class to duplicate them!

There are some limitations to this method, obviously my class hierarchy is only 2-levels deep so this method as-is doesn't extend to arbitrarily deep class hierarchies. I'm sure it could be extended to do that with a little bit more System.Reflection magic, but I don't need to do it so I'm not going to spend the time. Despite the limitations my goals were met: context menus are inherited between classes, and I can edit/expand menus without having to modify all sorts of existing, working code.

Wednesday, May 6, 2009

Switching Gears: GC

I talked briefly with Allison today via email, and decided that the best idea for me is to switch gears and not focus on the Asynchronous IO system just yet. Though I've been trying to keep it out of my mind for a little while the GC really is the most pressing need we have right now, so I'm going to start working on that instead of the AIO system, for now. Given several projects that I am both willing and able to work on, I will tend to work on the one that's the most needed by the project at large. I say "tend" because this is far from a hard-and-fast rule, but it does explain why I am willing to change focus in this case so readily.

My loyal followers (the null set) will remember back to last summer when I was working on a new GC system for Parrot as part of the Google Summer of Code project. People should also remember that my project was ultimately unsuccessful: I was not able to produce a GC that ran reliably. Part of it was my fault, the GC turns out to have been a bigger problem then I was capable of building successfully at that time. However, we've also learned the lesson that Parrots GC API is particularly messy and needs to be cleaned significantly before we can make another honest attempt at an improved GC core.

My task, should I choose to accept it, is to clean up the GC API so we can eventually have pluggable GC cores. This is going to be a bigger task then most people realize, for a variety of reasons. The current GC is intimately intertwined through the whole codebase. There isn't clear encapsulation between the GC and other parts of the system, and as a consequence it's almost impossible to create a new GC core because we don't even know what all the old core is doing (and where it's doing it at).

What we need to define is:
  1. What interface functions (and macros) the GC should provide to do it's work.
  2. What functions from the rest of Parrot the GC is permitted to call. This should be a very small subset of all functions Parrot provides.
  3. What data structures the GC provides for access by the rest of the system.
  4. What data structures the GC is allowed access to.
Not a small task at all but potentially a very rewarding one if it's done right.

Yesterday I created the "gc_api" branch to start the work, and my first task is to rename of the functions from "foo" to "Parrot_gc_foo" following the naming convention used by the rest of the Parrot subsystems. This isn't going to be a simple prefixing operation, some functions are going to be completely renamed to be more honest about what they do.

Next task is to work out what functions the GC needs to provide and export to the rest of the system. These functions will be located in src/gc/api.c only. Any function not in that file will not be intended for use by the rest of Parrot. Here is a partial list off the top of my head of functions that I think the GC needs to provide:
  1. Initialize and Deinitialize the system
  2. Allocate new STRING and PMC structures from the proper pools
  3. Allocate new pmc_ext and sync structures (or, given a PMC, append these structures to it)
  4. Allocate new raw data items from the fixed-size pools (or from the system, or whatever)
  5. Mark an object as being "alive"
  6. Mark an object as being "dead"
  7. Create a new pool
  8. Perform a mark phase
  9. Perform a sweep phase
  10. Perform a combined mark/sweep collection run.
There are probably a few more basic operations that I am forgetting right now, and there are definitely some "wishlist" items that I'm not mentioning here too for brevity. This does seem like a reasonably complete yet concise interface for the GC to implement. I'll post updates as my work progresses.

Tuesday, May 5, 2009

Path to AIO on Parrot

In the last few blog posts I've put out about Asynchronous IO (AIO) on Parrot, I've looked at some issues including how things work on Windows and Linux, and how things could start looking on Parrot. The series started with this post and ends with this one. Notice that I haven't provided all the answers here, so I'm still hoping to get some feedback from readers. I'm also pretty well convinced that we will learn some lessons quickly as work progresses. The biggest hassle is going to be creating a system which is semantically transparent on Windows and Linux, so that PIR users don't need to care about the differences.

PDD22 contains some information about AIO although it's all speculative. I've been pouring over this document for a few days and I think I'm really understanding a lot of the nuances about it and some of the methodology that the original designers had in mind. I think it's a decent approach, and I think we will be able to follow it reasonably closely without too many issues popping up. I showed some speculative PIR examples in a previous blog post, but I'm starting to think now that I will follow the PDD more and pursue that idea less.

The PDD has this to say about the relationship between synchronous and asynchronous opcodes:
Parrot only implements synchronous I/O operations. Initially,the asynchronous operations will be implemented separately from the synchronous ones. There may be an implementation that uses one variant to implement the other someday, but it's not an immediate priority.
and it says this about how the interface to PIR will look:
Synchronous opcodes are differentiated from asynchronous opcodes by the presence of a callback argument in the asynchronous calls. Asynchronous calls that don't supply callbacks (perhaps if the user wants to manually check later if the operation succeeded) are enough of a fringe case that they don't need opcodes.
The main point to take away from this is that asynchronous IO requests are not implemented as a PMC (at least not as one that the user needs to be aware of) and they are not handled using Parrot's existing concurrency opcodes like schedule. Here's a PIR example for something that, I think, will perform an asynchronous file write according to the specification in the PDD:

$P0 = open "/path/to/file.txt", "w"
$P1 = find_global("io_callback_sub")
$P2 = print $P0, "hello world", $P1

In the example above, $P0 is the FileHandle PMC that we are writing to, $P1 is the callback subroutine, and $P2 is the IO request status object that keeps track of the status of the request. The PDD says that there should be a division between the synchronous and asynchronous opcodes. However, on Windows platforms at least the file handle will need to be opened in asynchronous mode before any asynchronous operations can be performed on it. I don't see any real way to avoid immediate unification in this case, unless:
  1. The print opcode closes the filehandle and reopens it in asynchronous mode (bad)
  2. We add an additional specifier to the open opcode that specifies that the file handle should be opened in asynchronous mode ("wn" would be "write, non-blocking", for instance).
  3. We always open file handles in asynchronous mode and just implement all the blocking opcodes in terms of underlying asynchronous operations.
I like idea #3 the best personally, but I imagine there is going to be some significant support for #2 as well. I'm just worried about cases where a non-asynchronous filehandle object is passed to an asynchronous opcode, or any other related combination of things that can and will happen.

Here are some notes that I'm following in general (I will discuss some caveats below):
  1. All streams are opened in asynchronous mode by default, except for streams that don't support it.
  2. Asynchronous opcodes will accept as a callback either a Sub, a Continuation, or PMCNULL. In the case of a Sub, the opcode will perform asynchronously and call the sub in a separate thread when an event occurs. If it's a Continuation, the operation will block until completed and then resume execution at the Continuation. If PMCNULL, it will launch the asynchronous request and ignore any results.
  3. We need to create a new PMC for the AIO status object. I'm thinking we call it "IORequest". The IORequest object will have interfaces to check for the current status (in progress, complete, error) and the result (number of bytes read/written on success). I am not sure how we will handle errors, there are a few options for this that I won't talk about here.
The problem with this approach that I can see is that nowhere in here do we interact with the concurrency scheduler at all, unless the C-level callback function of the request schedules the PIR-level callback Task instead of executing it directly, or for systems that don't support AIO directly and we need to fudge it. Relying on the direct callbacks is definitely better performance-wise then polling a result flag internally. However, I've been told now by a handful of people that the better way to go in Linux anyway is probably a poll loop using epoll anyway.

An alternative idea, that I personally like less but which might cause fewer headaches, would be to create the IORequest object as a simple flag accessor. The scheduler would need to poll the pending requests regularly (using epoll in linux and an IO Completion Port in Windows) and when it finds one that needs handling, it would update the flag in the IO request object and schedule the callback for us (if one is provided). The difference to this approach, of course, is who is checking the status and who is scheduling the callback (kernel vs Parrot). I feel like this way is going to have a lot of performance drawbacks, but then again the biggest performance drawback in any IO system isn't going to be the callback scheduler anyway. The differences might be negligible.

So we don't even know how we're going to do simple read/write operations yet, but we might be able to nail down the specifics of a few other tasks. A listener object or a Select/Poll object for instance might be registered with the scheduler to repeatedly check their status and call callbacks when an incoming event occurs. This would be useful on a server-side network app where we could listen on an array of sockets for incoming data passively and let the scheduler take care of checking flags and scheduling callbacks.

Here's a straight-forward example of code to create a passive listener (which some Unix/Perl folk would probably prefer we called "Select") PMC which calls a callback when an incoming event occurs:

$P0 = new 'Socket'
# ... Connect the socket here
$P1 = new 'IOListener'
push $P1, $P0
schedule $P1

So at the end of that snippet, we have a listener object in the the scheduler. The scheduler will poll the IOListener, which in turn will poll the Socket and all other PMCs that it contains, and for every event that it finds it will add the corresponding callback to the scheduler for execution.

So the PDD definitely offers a nice base to start working on, but there are a ton of questions to be answered about some of the implementation specifics. We'll know as we start breaking ground and writing code what does work and what doesn't. I'm sure I'll be reporting on these issues as they arise.