Leaving Sun^HOracle

I joined Sun out of the OpenSolaris community. In 2008 I was offered an internship and jumped at the chance. The internship became a full-time position, and here I am.

Sun has been one of the best experiences of my employment life. I feel honoured to have worked among the ranks of some of the brightest engineers in the world. I've learned so much from each person I've had the pleasure of meeting here. The ranks are too innumerable to call out individually, so if you know who I am it's you. Thank you so much.

But all good things must come to an end, and today is my last day as an Oracle employee. I'm excitedly moving on to Joyent starting next week for a new adventure with some new friends (and a couple old friends.)

KCA 2009

A little late, I know but a couple weeks ago I got back from Kernel Conf Australia in lovely Brisbane, Australia at the Queensland Brain Institute. Talks were great and ranged from the userland to the deep kernel, from introductions to hard technical details.

The thing that stood out the most to me was the collaboration between the communities. I was pleased to see OpenBSD, FreeBSD, (Open)Solaris and Linux people mingling and talking about the relative solutions to various problems without turning in to the one-upmanship often seen in online debates on the subject.

It was also a much smaller conference from what I've attended before and that's a good thing, the attendees got a chance to meet and interact with each other much more than at for example CommunityONE.

One of my favourite talks was Pawel's talk on GEOM, largely because I didn't know it exists and the talk was on how it works. Also David Gwynn's slides, for the sole reason of using Lego to represent data structures. They were good talks anyways, but the Lego pushed it over the top.

I did a talk on the basics of writing drivers in (Open)Solaris which I thought was well received. I got several questions afterwards asking for suggestions on unsupported hardware to give a start.

Obligatory pictures:

Local fauna

Local Flora:

Town square:

Pancake Church (serves liquor until midnight):

Lunch!

Brendan Gregg, Garrett D'Amore & Alan Hargreaves debating something:

Submitting packages to pending/

The OpenSolaris pending/ and contrib/ repos are open.

As a result, I was asked to document the procedure for turning a bunch of useless source in to a useful package. After wading through a bunch of occasionally outdated information, here's what I came up with.

Step 1:
Install the JDS cbe from http://dlc.sun.com/osol/jds/downloads/cbe/
update: and run cbe-install ( thanks Luca )

$ cd /opt
$ wget http://dlc.sun.com/osol/jds/downloads/cbe/jds-cbe-1.6.2-i386.tar.bz2 -O /tmp/jdscbe.tar.bz2
$ bzcat /tmp/jdscbe.tar.bz2 | tar -xf -
$ cd jdscbe-1.6.2; ./cbe-install 

Step 2:
Make your spec file.

The Fedora project publishes a collection of them, props to them for that. I didn't find much utility in them simply because OpenSolaris and Fedora are quite different, but I'd just as soon chalk that up to a personal failing. I found the spec-files-extra repository to be a much more useful resource for templates.

You will need to strip out plenty of %include directives, since they're not really relevant to not-sfe files. The sections should be pretty self-explanatory, and the people on the mailing lists and irc channels are helpful if you don't understand something.

Just for posterity, the spec file I submitted is here. It just copies some files in to apache's wwwroot.

Step 3:
Set up your environment, and build with pkgtool ( Let's use Drupal as an example)

$ . /opt/jdsbld/bin/env.sh
$ pkgtool build --download drupal.spec

TEST

then uninstall the SysVR4 package

$ pfexec pkgrm drupal

Step 4:

make a local package repo:

$ pfexec svccfg -s pkg/server "setprop pkg/port=10000"
$ pfexec svcadm refresh pkg/server
$ pfexec svcadm enable pkg/server
$ pfexec svcadm restart pkg/server

and add your local repo as a pkg(5) authority

$ pfexec pkg set-authority -O http://localhost:10000 localhost

Step 5:

add your package to your local repo.

$ eval `pkgsend open drupal@6.8`
$ pkgsend import /export/home/johns/packages/PKGS/drupal/
$ pkgsend close

and install your package with pkg(1) and test again.

$ pfexec pkg install drupal

Step 6:

Everything work? License in the .spec file is kosher? Excellent.

You're ready to submit your .spec file to sw-porters-discuss@opensolaris.org (you'll need to subscribe first as it is set to auto-reject nonsubscribers). Send a friendly email including your .spec file and they'll take it from there.

OpenSolaris on a Macbook

I have a Macbook.

*jeers from the audience*

Just for the record it's the white model with 2Ghz CPU's and Wireless-N ( which makes it a Merom, I think )

I decided to install OpenSolaris on it. It being Apple, it has to be more obtuse and difficult than any other PC on the market.

Ultimately, a Mac is just like any other PC so there shouldn't be a whole lot of problem except for one thing: EFI.

EFI is Intel's BIOS replacement, because even though OpenFirmware/OpenBoot have been around forever, work well, and are open, it wasn't invented there. Nothing really boots off EFI yet except Linux, OSX, and the various HP operating systems. GRUB doens't work with EFI, for example, which leads to a number of problems getting OpenSolaris installed.

So, what the prospective OpenSolaris user will need to do is this:

Step 1: install a first stage EFI bootloader. This almost certainly means downloading and installing rEFIt. rEFIt is really easy to install, you just run the installer in the disk image.

Step 2: Toggle rEFIt's on switch. Open a terminal ( /Applications/Utilities/Terminal.app ) and type this:

/efi/refit/enable-always.sh

Step 3: You need to use Disk Utility to give yourself a free partition. Set it up as an MS-DOS (FAT) volume.

Step 4: Now, because the partition is marked as a DOS partition, OpenSolaris isn't going to want to install on it, so you'll need to mark it as a Solaris partition. This is done roughly the same way you'd do it anywhere else, that is to say, you'll be using fdisk. Open a terminal window and open fdisk on your drive, typically disk0. The -e flag makes it an interactive session.

$ fdisk -e /dev/disk0

It being Apple, the syntax is different from any of the other fdisk's you're likely to have seen. Find your partition in the list ( it'll be the one not marked HFS+ ) and to change it use:

setpid <partition number>

It'll prompt you for a type number in hex without the leading 0x. You'll be using type number 'BF' for Solaris2 ( because 0x82: Solaris was stolen for Linux swap... ). Then just 'write' to write the table, 'quit' to exit out of fdisk.

Step 5: Insert your CD, and reboot. You'll not get the familiar grey apple logo, instead you'll be presented with a rEFIt menu. Since rEFIt doesn't really know about the partition table well enough to present the operating system with anything sane, use the Partition Tool to set things up. Just use the defaults that it asks you about. Once that's written to disk, back in the rEFIt menu there'll be an option to boot OSX, or the CD. Choose the CD.

Step 6: Now you're running the OpenSolaris livecd, play around, do whatever you're going to do with it. Networking won't work because myk isn't installed. When you decide to install, there's some considerations. First off, you can't write to the partition table, rEFIt doesn't allow it (and it doesn't make much sense anyways). Since the installer really, really wants to write to the partition table ( bug 6413235 ) you'll need to trick it.
First off, you need to create a new file in your home directory named 'fdisk'. It should contain this:

#!/bin/sh
echo "$*" | grep -- "-F" > /dev/null
if [ $? = 1 ] ; then
/sbin/fdisk $*
fi

( Thanks to Jim Walker for the script )

make it executable

$ chmod 755 ./fdisk

and rather than running the installer icon on the desktop, you'll need to trick it in to using this new fdisk script with $PATH.

$ PATH=$PWD:$PATH pfexec 

it'll do it's thing, fill out the information, click Next> a half dozen times. When it prompts you where to install, don't muck about with the partitions, there should be one Solaris partition that it'll try to install on by default, just click Next> through that screen. Now it's installing

Step 7: Reboot. rEFIt will present you with the option to boot OSX or Linux. rEFIt is dumb and thinks anything that uses GRUB is Linux. Let's ignore this slight for now, but bide our time and destroy everyone and everything they love, leaving them a sad hollow shell wandering the wasteland wishing for death, later. Choose the not-Linux Linux option, and OpenSolaris will boot. ( check here later for instructions on how to change the icon if it bothers you )

Step 8: Networking doesn't work. Pick up Murayama's myk driver. Put it on a USB stick. It comes with a Readme.txt, but you can ignore most of it. All you really need to do is this part:

$ pfexec make install
$ pfexec ./adddrv.sh
$ pfexec devfsadm -i myk
$ pfexec ifconfig myk0 plumb

at this point NWAM should realize that a new NIC exists, and do it's thing accordingly.

Now, wired networking works. Wireless does not because the new ath binary blob hasn't been integrated so the wireless-N chipsets are left in the cold. Sound is hdaudio, and the single button mouse is annoying, so I reccomend using an external one.

Other than that though, you're all good, up and running. Enjoy.

(and to all the other mac-heads that works on OpenSolaris or in marketing for it... now you have no more excuse ;) )

Presenting: chsh

I logged in to the Solaris (9) machine at school the other day and was presented with this:

bash-2.05$

Bash. Gross.
Then it occurred to me that there's no real sane way for a user to change his or her shell on Solaris by default.
Being the diligent late-night hacker with an itch that I am, I solved the problem.

So, without further ado I present to you: chsh.

Usage:

chsh SHELL

or, to list valid shells

chsh -l

if you have the solaris.admin.usermgr role ( it is role aware, so root is unnecessary ) you can change any arbitrary user's shell

chsh -u USERNAME SHELL

It requires root to install, as it is a suid binary, so if you don't already have root-like privileges on the box it's not going to be a lot of help, but your users will thank you. Much like usermod(1M) it only modifies the local /etc/passwd file, so NIS+ and LDAP entries are unaffected.

I'm not sure about the process for getting something integrated in Solaris from scratch, and I'm pretty sure I missed the 2008.11 deadline so it won't be shipping but hopefully you'll see it in a Nevada build and IPS sooner or later.

Tips & Tricks for a new sponsoree

I've recently been moved behind the firewall and am intern-ing at Sun.

One of the tasks of this job that I've been charged with is to sponsor community code contributions going in to OpenSolaris and I noticed a bunch of things that'd just make it less work & by extension time to go from an email with a .diff in it to integration. Let's begin:

So you want to go that extra mile for your sponsor:

You've signed your SCA , found a bug, sent your email to the list, and are ready to roll.
Awesome! OpenSolaris is huge and we appreciate the extra hands, plus you help make it more community driven.

So the next step is to go back and forth with your sponsor trying to find the best possible solution to the bug. Once you've done that, you can get down to work, fix the bug, and then build & test it.

Then you just send the code to your sponsor, and you're done right?
Well, you could do that. It's technically correct and is the standard way of doing things but if you really want to wow your sponsor and go above and beyond, there are a couple things that the gatekeepers require that you can do to make his or her job ( taking your code, committing it to the tree ) a breeze:

• unified diffs : from the top of the tree, running a diff -u gives you a single patch file that can be applied to the tree, rather than a diff of every affected file. Better still is an hg export file. The standard comment for HG exports is :

  Contributed by {your name} <{your email}>
  {bug id} {bug synopsis}

  note the single space between bug id and bug synopsis
  
• recent gate checkout : a bunch of stuff may have changed in the meantime, so it's best to apply your code to a copy of the ON gate as recent as possible, so that the diff doesn't break
  
• cstyle clean it : onbld has a tool in /opt/onbld/bin called 'cstyle' which checks to make sure that the pendantic little nits like indentation standards are adhered to. You only need to make the stuff you changed cstyle clean, don't worry about all the old stuff
• change the copyright : you'll notice every file owned by Sun in the ON consolidation that has a line with Sun's copyright in it like:
  

  /*
  * Copyright 2008 Sun Microsystems, Inc. All rights reserved.
  * Use is subject to license terms.
  */
  

  These all need to be updated to reflect the current year ( in this case it's 2008 ) whenever the file changes.
• Update the CDDL block: Some older files have a CDDL block that's no longer acceptable, so if you run across it you may need to update it. You'll be able to tell, because it gives a version number ( 1.0 ) of the license. So, update this:

  The contents of this file are subject to the terms of the
  Common Development and Distribution License, Version 1.0 only
  (the "License"). You may not use this file except in compliance
  with the License.

  To this:

  The contents of this file are subject to the terms of the
  Common Development and Distribution License (the "License").
  You may not use this file except in compliance with the License.

  
  
• SCCS keywords : the ON gate used to use a different source-code management system (teamware). The gate is now managed by Mercurial. As a result, there's still some vestigial teamware bits lying around. Before a change can be integrated someone ( typically the sponsor, but it could be you, which is why we're here ) needs to remove the old SCCS keywords. They look like this:

  #pragma ident "%Z%%M% %I% %E% SMI"

  This whole line can just be removed from the file entirely. It's not needed anymore.

With these simple steps, you can cut (depending on the size of the patch) hours off your sponsor's task and surprise them with how thorough you are

Emancipation Community

I proposed the other day that the emancipation project be promoted to the Emancipation Community.

The way I see it, Jason & Steven's work on the ce driver, Roland's work on the xpg/posix stuff
and my libc_i18n work are all loosely related in that they are all reimplementations of closed source stuff, but aren't really as closely micro-coupled as much as one might think a project is ( we don't share an hg repo set, for example )

Here's what Plocher and I came up with for a charter:

Emancipation Community Charter ( rev 1 )

CG Problem statement

The OpenSolaris operating system is not completely open
because several components that are required to build and
boot the OS are only available in the "closed bin" archives.

Scope:
Initially, the focus will be on selected high-value efforts,
such as self-hosting an open ON, drivers, posix utils, but
the long range intent is to eliminate the need for (and use

of) closed source software on the opensolaris OE.

Goals/milestones:

Quarterly progress reports will be produced and sent to the
OS-announce alias to keep the larger community informed of
our progress.

In order of priority:

Goal 0: Replace the components needed to build and boot

the ON consolidation with whatever shims, hacks
and scaffolding is needed to produce a proof of

concept that self-hosts and boots, followed by a
reimplementation of the userland utilities as per same.

Goal 1: Determine the best way to replace the above hacks

with a permanent solution, including decision
making architectural and design guidelines that
can be used in similar situations elsewhere in
the emancipation effort (i.e., should we reuse
from some particular other open OS, roll our own,
do without; what makes a good -vs- poor choice,
how do we choose without causing unnecessary

strife, ...? Collaboration with the ARC community
is implied during this stage.

Goal 2: Develop and push the changes prototyped in phase

0 and formalized in phase 1 into the ON (and other)
consolidation(s) and remove the associated closed
bin pieces.

Goal 3: Seek replacement for high-use closed source software
such as media players, rich web players, etc.

Goal 4: After completion of goals 0 - 3, disband the community

To facilitate this community, the initial list of core contributors (
should they accept ) shall be:

Faciltator:
John Sonnenschein ( error404 )
Core Contributors:

Jason King ( jbk )
Steve Stallion ( stallion )
Roland Mainz ( gisburn )

Joerg Schilling ( joerg )
John Sonnenschein ( error404 ) ( note: i'm not sure if this is
implied by "facilitator" )
Garret D'Amore ( gdamore )
Contributors:
John Plocher ( plocher )