Catching up with NetBSD

For a great many years, and certainly during the "glory days" of this website (if it's even remotely sensible to talk about such days existing), I was a pretty devout enthusiast of the NetBSD operating system. This site was hosted on NetBSD machines up until relatively recently (2010, maybe?), and for a lot of its life the most popular page was a "NetBSD Survival Guide" (now hopelessly out of date). However, it's been a very long time now since I used NetBSD on a regular basis or followed news about its development closely.

In April of 2011, while in Japan, I bought a refurbished Thinkpad X60s in DenDen Town (Osaka's equivalent of Tokyo's imfamous Akihabara), and one of the first things I looked into doing was installing NetBSD on it. I was shocked at how much the project appeared to have changed since I had last checked in on it. Now about a year later I am finally getting around to blogging about some of this.

Maybe the first really shocking thing I discovered was that the Lua scripting language had been added to the NetBSD base system. My immediate response to this was utter dismay. One of the things I always loved about NetBSD was its clean minimalism. The base system genuinely was a base system, containing only what was needed to make the system run and to give you the ability to add extra stuff when and as you wanted it. It didn't come with a web server or browser (like OpenBSD), and it didn't come with a scripting language (like many Linux distributions do). I was confused and upset when a web server was added to base for 5.0. To learn that now there was going to be a scripting language too made me feel like NetBSD was completely giving up on minimalism. At the very least, it was no longer going to be anywhere as special as it used to be.

I was even more taken aback to learn that there was talk of integrating Lua into the kernel!!! I had never - and still have not - heard of anything like this before. I don't quite know what to make of it. On the one hand, it sounds like an interesting idea, and it's really good to see genuinely innovative OS resarch taking place in NetBSD. It's something of a sign of good health. I have no previous experience with Lua, but I do imagine that it's easier to write and maintain Lua code for a lot of tasks than the corresponding C code, so this could lead to significant increases in kernel development efficiency. On the other hand, I have to be a little wary. I don't really understand enough about OS development to know with confidence what the implications of such a move are for the size, speed, stability and security of the kernel. Security may increase, I suppose, since parts of the kernel written in Lua will not be vulnerable to stack smashing attacks, and similarly stability migh improve because tricky bugs related to pointers and array bounds and dynamic memory allocation and all that will disappear. But at the same time, the complex NetBSD kernel will now be interacting with the complex Lua environment, such that there's probably an awful lot of new scope for things to break, and the net effect on stability and security might be negative. It certainly seems likely that the basic kernel will not be made smaller or faster by this. And I certainly am not one of those people who worships speed in software - speed should be sacrificed, within reason, where it can lead to more simplicity, stability, security or other good things. But the fact that NetBSD runs on such a wide variety of machines, many of them obscure and/or old and with limited resources, has always been a defining feature of the OS (although seems less of a priority these days). If NetBSD ends up getting tightly integrated with Lua, to the point that extensive kernel and userland use of Lua is basically mandatory, then a lot of supported old machines with 16MB of memory are going to suffer for it.

Another significant technical change that caught my eye is the introduction of "rump", or Runnable Userspace Meta Programs. As far as I can tell, this is a way for you to run NetBSD kernels - or even parts of NetBSD kernels, like the TCP/IP system - as userland processes. This is apparently yet another genuinely novel idea which is only implemented in NetBSD, which it's good to see. This doesn't really give me an uneasy feeling like Lua-in-the-kernel does, because it seems like something that it will always be possible to not use if you don't want it.

One last surprising technical change: NetBSD is getting a new firewall, called NPF. This represents a new entry into a fairly crowded "marketplace". NetBSD already supports Darren Reed's IPFilter and the OpenBSD project's successor to it, PF. FreeBSD supports both of these and also it's own firewall, IPFW. With the arrival of NPF, the three major BSDs will now have a total of 4 distinct firewall options available. This feels like a bit of a waste of resources, honestly, although then again some would say that about the existence of 3 BSDs in the first place (not sure I'd agree with that, though). It looks like what NPF is bringing to the table in terms of uniqueness is good multiprocessing support, something that NetBSD has shown quite a bit of strength with lately.

With regard to the overall running of the project, and the vibrancy of the community, NetBSD seems to have gone a bit downhill here. When I was last really active in the community, there used to be a great user-generated wiki at netbsd.se. This is now gone, it seems, and there is now an official wiki at wiki.netbsd.org. It seems that the netbsd.se wiki was taken down in order to let the official wiki take the stage - I've not been able to determine whether or not the person who ran the netbsd.se wiki voluntarily did this out of good will or if they were strong-armed into it by people from the NetBSD Foundation trying to protect their trademark (something that accusations of have surfaced before). What anyone can determine is that the official wiki is pretty much dead in the water. It was first set up over 2 years ago, with the front page saying "Welcome to the NetBSD developer wiki! Only NetBSD developers can edit. Anyone can read; when there's enough useful content we'll make a public announcement". As far as I can tell, this hasn't happened yet, and there's still very little useful content in there. Certainly nothing to compare with what the old netbsd.se wiki used to be. The current official wiki page claims that "I estimate that two solid weekends of my time should be enough to push it out the door and make a public announcement". This claim has been standing there for over a year. Now, I realise NetBSD is a volunteer project - the work is hard, the developers are busy, and the users are not entitled to anything. But when a new wiki is allowed to to have a sign on its front page saying it should be ready in two weeks for a year, that's a little embarassing. It makes the project look severely undermanned and poorly managed.

The wiki isn't the only place this kind of thing has happened. A mailing list user pointed out last year that a late-2009 presentation mentioned that mid-2010 was around the time that NetBSD 6.0 would hopefully be released, which obviously didn't happen. In the replies, Alistair Crooks claimed that he'd "really like to see a 6.0 release in 2011. Really", which also hasn't happened. Now, again, I know this is a volunteer project, and really it isn't a big deal if 6.0 comes out arbitrarily late, as long as the 5.0 series works and is having security bugs fixed. But there is something slightly concerning about the fact that people high up in the project are making public statements about when they think important milestones will be reached and those dates are then missed by literally years. I first got involved in NetBSD basically immediately after the release of 2.0 in late 2004, and in the years following that release the pace of development was nowhere near as slow as it seems to be these days. It feels like NetBSD these days has a lot less manpower behind it than is needed for the project's goals to happen at the pace that people want them to happen, which is a real shame.

Finally, I've noted with interest that Minix 3, the latest incarnation of Andrew Tanenbaum's well known operating system which was the original inspiration for Linux, is looking very lively these days and is borrowing very heavily from NetBSD. This includes using pkgsrc for package management, borrowing a lot of NetBSD's userland utilities, using NetBSD's libc, and they seem to only want to continue this trend! It's very interesting to see Minix 3 choose such a niche OS as NetBSD from which to import so much stuff. I assume NetBSD was singled out because, despite my complaints above, it's still pretty much the lightest, simplest and cleanest free Unix out there which is fairly modern and full-featured. I have fond memories of installing Minix 2 and programs for it onto a 486-era computer using huge stacks of 3.25 inch floppies as a highschooler, and I'll be interested to see how Minix 3 progresses.