Emancipation

Leaving Sun^HOracle

2010-09-24T08:26:00.000-07:00

I joined Sun out of the OpenSolaris community. In 2008 I was offered an internship and jumped at the chance. The internship became a full-time position, and here I am.

Sun has been one of the best experiences of my employment life. I feel honoured to have worked among the ranks of some of the brightest engineers in the world. I've learned so much from each person I've had the pleasure of meeting here. The ranks are too innumerable to call out individually, so if you know who I am it's you. Thank you so much.

But all good things must come to an end, and today is my last day as an Oracle employee. I'm excitedly moving on to Joyent starting next week for a new adventure with some new friends (and a couple old friends.)

KCA 2009

2009-08-08T00:16:00.000-07:00

A little late, I know but a couple weeks ago I got back from Kernel Conf Australia in lovely Brisbane, Australia at the Queensland Brain Institute. Talks were great and ranged from the userland to the deep kernel, from introductions to hard technical details.

The thing that stood out the most to me was the collaboration between the communities. I was pleased to see OpenBSD, FreeBSD, (Open)Solaris and Linux people mingling and talking about the relative solutions to various problems without turning in to the one-upmanship often seen in online debates on the subject.

It was also a much smaller conference from what I've attended before and that's a good thing, the attendees got a chance to meet and interact with each other much more than at for example CommunityONE.

One of my favourite talks was Pawel's talk on GEOM, largely because I didn't know it exists and the talk was on how it works. Also David Gwynn's slides, for the sole reason of using Lego to represent data structures. They were good talks anyways, but the Lego pushed it over the top.

I did a talk on the basics of writing drivers in (Open)Solaris which I thought was well received. I got several questions afterwards asking for suggestions on unsupported hardware to give a start.

Obligatory pictures:

Local fauna

Local Flora:

Town square:

Pancake Church (serves liquor until midnight):

Lunch!

Brendan Gregg, Garrett D'Amore & Alan Hargreaves debating something:

Submitting packages to pending/

2009-01-12T14:26:00.001-08:00

The OpenSolaris pending/ and contrib/ repos are open.

As a result, I was asked to document the procedure for turning a bunch of useless source in to a useful package. After wading through a bunch of occasionally outdated information, here's what I came up with.

Step 1:
Install the JDS cbe from http://dlc.sun.com/osol/jds/downloads/cbe/
update: and run cbe-install ( thanks Luca )


$ cd /opt
$ wget http://dlc.sun.com/osol/jds/downloads/cbe/jds-cbe-1.6.2-i386.tar.bz2 -O /tmp/jdscbe.tar.bz2
$ bzcat /tmp/jdscbe.tar.bz2 | tar -xf -
$ cd jdscbe-1.6.2; ./cbe-install

Step 2:
Make your spec file.

The Fedora project publishes a collection of them, props to them for that. I didn't find much utility in them simply because OpenSolaris and Fedora are quite different, but I'd just as soon chalk that up to a personal failing. I found the spec-files-extra repository to be a much more useful resource for templates.

You will need to strip out plenty of %include directives, since they're not really relevant to not-sfe files. The sections should be pretty self-explanatory, and the people on the mailing lists and irc channels are helpful if you don't understand something.

Just for posterity, the spec file I submitted is here. It just copies some files in to apache's wwwroot.

Step 3:
Set up your environment, and build with pkgtool ( Let's use Drupal as an example)


$ . /opt/jdsbld/bin/env.sh
$ pkgtool build --download drupal.spec

TEST

then uninstall the SysVR4 package


$ pfexec pkgrm drupal

Step 4:

make a local package repo:


$ pfexec svccfg -s pkg/server "setprop pkg/port=10000"
$ pfexec svcadm refresh pkg/server
$ pfexec svcadm enable pkg/server
$ pfexec svcadm restart pkg/server

and add your local repo as a pkg(5) authority


$ pfexec pkg set-authority -O http://localhost:10000 localhost

Step 5:

add your package to your local repo.


$ eval `pkgsend open drupal@6.8`
$ pkgsend import /export/home/johns/packages/PKGS/drupal/
$ pkgsend close

and install your package with pkg(1) and test again.


$ pfexec pkg install drupal

Step 6:

Everything work? License in the .spec file is kosher? Excellent.

You're ready to submit your .spec file to sw-porters-discuss@opensolaris.org (you'll need to subscribe first as it is set to auto-reject nonsubscribers). Send a friendly email including your .spec file and they'll take it from there.

OpenSolaris on a Macbook

2008-11-11T09:56:00.000-08:00

I have a Macbook.

*jeers from the audience*

Just for the record it's the white model with 2Ghz CPU's and Wireless-N ( which makes it a Merom, I think )

I decided to install OpenSolaris on it. It being Apple, it has to be more obtuse and difficult than any other PC on the market.

Ultimately, a Mac is just like any other PC so there shouldn't be a whole lot of problem except for one thing: EFI.

EFI is Intel's BIOS replacement, because even though OpenFirmware/OpenBoot have been around forever, work well, and are open, it wasn't invented there. Nothing really boots off EFI yet except Linux, OSX, and the various HP operating systems. GRUB doens't work with EFI, for example, which leads to a number of problems getting OpenSolaris installed.

So, what the prospective OpenSolaris user will need to do is this:

Step 1: install a first stage EFI bootloader. This almost certainly means downloading and installing rEFIt. rEFIt is really easy to install, you just run the installer in the disk image.

Step 2: Toggle rEFIt's on switch. Open a terminal ( /Applications/Utilities/Terminal.app ) and type this:


/efi/refit/enable-always.sh

Step 3: You need to use Disk Utility to give yourself a free partition. Set it up as an MS-DOS (FAT) volume.

Step 4: Now, because the partition is marked as a DOS partition, OpenSolaris isn't going to want to install on it, so you'll need to mark it as a Solaris partition. This is done roughly the same way you'd do it anywhere else, that is to say, you'll be using fdisk. Open a terminal window and open fdisk on your drive, typically disk0. The -e flag makes it an interactive session.


$ fdisk -e /dev/disk0

It being Apple, the syntax is different from any of the other fdisk's you're likely to have seen. Find your partition in the list ( it'll be the one not marked HFS+ ) and to change it use:


setpid <partition number>

It'll prompt you for a type number in hex without the leading 0x. You'll be using type number 'BF' for Solaris2 ( because 0x82: Solaris was stolen for Linux swap... ). Then just 'write' to write the table, 'quit' to exit out of fdisk.

Step 5: Insert your CD, and reboot. You'll not get the familiar grey apple logo, instead you'll be presented with a rEFIt menu. Since rEFIt doesn't really know about the partition table well enough to present the operating system with anything sane, use the Partition Tool to set things up. Just use the defaults that it asks you about. Once that's written to disk, back in the rEFIt menu there'll be an option to boot OSX, or the CD. Choose the CD.

Step 6: Now you're running the OpenSolaris livecd, play around, do whatever you're going to do with it. Networking won't work because myk isn't installed. When you decide to install, there's some considerations. First off, you can't write to the partition table, rEFIt doesn't allow it (and it doesn't make much sense anyways). Since the installer really, really wants to write to the partition table ( bug 6413235 ) you'll need to trick it.
First off, you need to create a new file in your home directory named 'fdisk'. It should contain this:


#!/bin/sh
echo "$*" | grep -- "-F" > /dev/null
if [ $? = 1 ] ; then
/sbin/fdisk $*
fi

( Thanks to Jim Walker for the script )

make it executable


$ chmod 755 ./fdisk

and rather than running the installer icon on the desktop, you'll need to trick it in to using this new fdisk script with $PATH.


$ PATH=$PWD:$PATH pfexec

it'll do it's thing, fill out the information, click Next> a half dozen times. When it prompts you where to install, don't muck about with the partitions, there should be one Solaris partition that it'll try to install on by default, just click Next> through that screen. Now it's installing

Step 7: Reboot. rEFIt will present you with the option to boot OSX or Linux. rEFIt is dumb and thinks anything that uses GRUB is Linux. Let's ignore this slight for now, but bide our time and destroy everyone and everything they love, leaving them a sad hollow shell wandering the wasteland wishing for death, later. Choose the not-Linux Linux option, and OpenSolaris will boot. ( check here later for instructions on how to change the icon if it bothers you )

Step 8: Networking doesn't work. Pick up Murayama's myk driver. Put it on a USB stick. It comes with a Readme.txt, but you can ignore most of it. All you really need to do is this part:


$ pfexec make install
$ pfexec ./adddrv.sh
$ pfexec devfsadm -i myk
$ pfexec ifconfig myk0 plumb

at this point NWAM should realize that a new NIC exists, and do it's thing accordingly.

Now, wired networking works. Wireless does not because the new ath binary blob hasn't been integrated so the wireless-N chipsets are left in the cold. Sound is hdaudio, and the single button mouse is annoying, so I reccomend using an external one.

Other than that though, you're all good, up and running. Enjoy.

(and to all the other mac-heads that works on OpenSolaris or in marketing for it... now you have no more excuse ;) )

Presenting: chsh

2008-09-28T01:18:00.000-07:00

I logged in to the Solaris (9) machine at school the other day and was presented with this:

bash-2.05$

Bash. Gross.
Then it occurred to me that there's no real sane way for a user to change his or her shell on Solaris by default.
Being the diligent late-night hacker with an itch that I am, I solved the problem.

So, without further ado I present to you: chsh.

Usage:

chsh SHELL

or, to list valid shells

chsh -l

if you have the solaris.admin.usermgr role ( it is role aware, so root is unnecessary ) you can change any arbitrary user's shell

chsh -u USERNAME SHELL

It requires root to install, as it is a suid binary, so if you don't already have root-like privileges on the box it's not going to be a lot of help, but your users will thank you. Much like usermod(1M) it only modifies the local /etc/passwd file, so NIS+ and LDAP entries are unaffected.

I'm not sure about the process for getting something integrated in Solaris from scratch, and I'm pretty sure I missed the 2008.11 deadline so it won't be shipping but hopefully you'll see it in a Nevada build and IPS sooner or later.

Tips & Tricks for a new sponsoree

2008-08-21T13:01:00.000-07:00

I've recently been moved behind the firewall and am intern-ing at Sun.

One of the tasks of this job that I've been charged with is to sponsor community code contributions going in to OpenSolaris and I noticed a bunch of things that'd just make it less work & by extension time to go from an email with a .diff in it to integration. Let's begin:

So you want to go that extra mile for your sponsor:

You've signed your SCA , found a bug, sent your email to the list, and are ready to roll.
Awesome! OpenSolaris is huge and we appreciate the extra hands, plus you help make it more community driven.

So the next step is to go back and forth with your sponsor trying to find the best possible solution to the bug. Once you've done that, you can get down to work, fix the bug, and then build & test it.

Then you just send the code to your sponsor, and you're done right?
Well, you could do that. It's technically correct and is the standard way of doing things but if you really want to wow your sponsor and go above and beyond, there are a couple things that the gatekeepers require that you can do to make his or her job ( taking your code, committing it to the tree ) a breeze:

unified diffs : from the top of the tree, running a diff -u gives you a single patch file that can be applied to the tree, rather than a diff of every affected file. Better still is an hg export file. The standard comment for HG exports is :
Contributed by {your name} <{your email}>
{bug id} {bug synopsis}
note the single space between bug id and bug synopsis
recent gate checkout : a bunch of stuff may have changed in the meantime, so it's best to apply your code to a copy of the ON gate as recent as possible, so that the diff doesn't break
cstyle clean it : onbld has a tool in /opt/onbld/bin called 'cstyle' which checks to make sure that the pendantic little nits like indentation standards are adhered to. You only need to make the stuff you changed cstyle clean, don't worry about all the old stuff
change the copyright : you'll notice every file owned by Sun in the ON consolidation that has a line with Sun's copyright in it like:
/*
* Copyright 2008 Sun Microsystems, Inc. All rights reserved.
* Use is subject to license terms.
*/
These all need to be updated to reflect the current year ( in this case it's 2008 ) whenever the file changes.
Update the CDDL block: Some older files have a CDDL block that's no longer acceptable, so if you run across it you may need to update it. You'll be able to tell, because it gives a version number ( 1.0 ) of the license. So, update this:
The contents of this file are subject to the terms of the
Common Development and Distribution License, Version 1.0 only
(the "License"). You may not use this file except in compliance
with the License.
To this:
The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
SCCS keywords : the ON gate used to use a different source-code management system (teamware). The gate is now managed by Mercurial. As a result, there's still some vestigial teamware bits lying around. Before a change can be integrated someone ( typically the sponsor, but it could be you, which is why we're here ) needs to remove the old SCCS keywords. They look like this:
#pragma ident "%Z%%M% %I% %E% SMI"
This whole line can just be removed from the file entirely. It's not needed anymore.

With these simple steps, you can cut (depending on the size of the patch) hours off your sponsor's task and surprise them with how thorough you are

Emancipation Community

2008-05-13T15:08:00.000-07:00

I proposed the other day that the emancipation project be promoted to the Emancipation Community.

The way I see it, Jason & Steven's work on the ce driver, Roland's work on the xpg/posix stuff
and my libc_i18n work are all loosely related in that they are all reimplementations of closed source stuff, but aren't really as closely micro-coupled as much as one might think a project is ( we don't share an hg repo set, for example )

Here's what Plocher and I came up with for a charter:

Emancipation Community Charter ( rev 1 )
CG Problem statement

The OpenSolaris operating system is not completely open
because several components that are required to build and
boot the OS are only available in the "closed bin" archives.

Scope:
Initially, the focus will be on selected high-value efforts,
such as self-hosting an open ON, drivers, posix utils, but
the long range intent is to eliminate the need for (and use
of) closed source software on the opensolaris OE.

Goals/milestones:

Quarterly progress reports will be produced and sent to the
OS-announce alias to keep the larger community informed of
our progress.

In order of priority:

Goal 0: Replace the components needed to build and boot
the ON consolidation with whatever shims, hacks
and scaffolding is needed to produce a proof of
concept that self-hosts and boots, followed by a
reimplementation of the userland utilities as per same.

Goal 1: Determine the best way to replace the above hacks
with a permanent solution, including decision
making architectural and design guidelines that
can be used in similar situations elsewhere in
the emancipation effort (i.e., should we reuse
from some particular other open OS, roll our own,
do without; what makes a good -vs- poor choice,
how do we choose without causing unnecessary
strife, ...? Collaboration with the ARC community
is implied during this stage.

Goal 2: Develop and push the changes prototyped in phase
0 and formalized in phase 1 into the ON (and other)
consolidation(s) and remove the associated closed
bin pieces.

Goal 3: Seek replacement for high-use closed source software
such as media players, rich web players, etc.

Goal 4: After completion of goals 0 - 3, disband the community

To facilitate this community, the initial list of core contributors (
should they accept ) shall be:

Faciltator:
John Sonnenschein ( error404 )
Core Contributors:
Jason King ( jbk )
Steve Stallion ( stallion )
Roland Mainz ( gisburn )
Joerg Schilling ( joerg )
John Sonnenschein ( error404 ) ( note: i'm not sure if this is
implied by "facilitator" )
Garret D'Amore ( gdamore )
Contributors:
John Plocher ( plocher )

Bio & Affiliation

2008-03-12T09:41:00.000-07:00

hey, so this is a little on the late side, but not too late I hope

Disclosure: I currently work for an MS Windows based contracting firm that has no commercial interest in OpenSolaris. I therefore am under no obligations that would prevent me from fully representing the community if elected to the OGB.

Bio: I'm an undergraduate computer science student at the University of Northern BC and a part-time software developer . I joined OpenSolaris approximately 3 or 4 months after the inception of the project, being a former 7 year Linux user. I was drawn to OSol because of the tradition of unrelenting engineering excellence that I knew Sun & her engineers for.
I envision the OGB's primary charter for the time being as being the liaison with the benefactor of OpenSolaris ( that is, Sun ). The OGB ought to be the go-between of Sun's management and the rough-and-tumble of the community. Sun has immense influence on the community as it is a community built around their code, and it's important for the OGB to act on the community's behalf when faced with decisions regarding steering that community ( for example, the recent trademark debacle. I believe the OGB could have had a much more influential role negotiating with Sun during that incident).

How to turn a mirror in to a RAID

2008-01-02T20:02:00.000-08:00

People occasionally ask on the mailing lists and in #opensolaris how to add a disk to a zfs mirror to make a raidz. Today, I received in the mail a new SATA controller and a new disk, so I was left in the same circumstance.

There's a drought of information on the topic on the internet, probably due in large part to the typical deployment of ZFS ( i.e. large shops that have a ton of spare disks laying around, or have otherwise planned out a migration path beforehand ), rather than the small home user.

So, here's what I did:

On a high level, we have to remember what sort of replication we've got for any given RAID level. More accurately, we need to know how many disks can be broken before the whole thing falls apart.

When we've got a single drive, that drive can't die, or we lose everything (obvious). With a mirror, we can't have 2 drives die. A 3-disk RAIDZ ( raid5 ) requires at least 2 operational disks out of 3. So, when moving from a 2 disk mirror to a 3 disk raidz, we obviously don't have enough disks to have both of them operational in full, even if we break the mirror in to a single disk.

But, if we count the number of disks allowed to be dead ( 2 ) at any given time, and the number we have ( 3 ), we can spread them out such that two degraded pools exist. One single-disk ( broken mirror ) and one degraded zpool ( 2 disks + NULL ).

So the procedure we'd use to attain this state is break the mirror, create a zpool with the new disk and the old mirror drive, copy the data over, destroy the old mirror, attach the old second mirror disk to the new raidz.

For the purpose of demonstration, I'll use the disks I've got attached to the system, c2d0, c3d0, and c4d1 .

first, the starting condition:


$ zpool status

  pool: xenophanes
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        xenophanes  ONLINE       0     0     0
          mirror
            c3d0      ONLINE       0     0     0
            c2d0      ONLINE       0     0     0

errors: No known data errors

Now, to break the mirror:

# zpool detach c2d0

so, what I've got now is a single-disk zpool comprised of c3d0, and two free disks, c2d0 and c4d1.

To create a raidz, you need 3 devices. We only have two. We can solve this problem, however, with sparse files and loopback.

Loopback allows you to use a file the same way you'd use any other block device in /dev. Linux has it ( mount -o loop ), Solaris has it ( lofiadm ). It's a pretty common thing.
A sparse file is a type of file where the filesystem only stores it's beginning and end pointer information, and a size. The actual contents of the file aren't stored until you begin to write to them. This allows us to do things like create a 140GB file on a 140GB disk with plenty of room to spare. And that's precisely what we'll do.

You can create a sparse file easily with dd(1) like so:

$ dd if=/dev/zero of=/xenophanes/disk.img bs=1024k seek=149k count=1

bs is block size, 1kb. seek is the number of blocks to skip ( and is equal to the size of the drive in kb, because of the previous bs= line ), and count tells dd(1) to copy one block.

and we can create a device like so:

# lofiadm -a /xenophanes/disk.img
/dev/lofi/1

So, to recap, here's what we have. We have a zpool, two spare disks ( c2d0 and c4d1 ) and a sparse file the size of those disks hooked up with lofi. And if you'll notice, that's precisely what we need.

From here out, we need to create the raidz, degrade it ( otherwise we'll fill up a sparse file that's the same size as the other disk, it'll run out of space, stuff will break... it won't be pretty )

# zpool create heraclitus raidz c2d0 c4d1 /dev/lofi/1

ta da! a raidz. Now let's break it.

# zpool offline heraclitus /dev/lofi/1 && lofiadm -d /dev/lofi/1 && rm /xenophanes/disk.img

and here's what we get:


# zpool status
  pool: heraclitus
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
 scrub: none requested
config:

        NAME             STATE     READ WRITE CKSUM
        heraclitus       DEGRADED     0     0     0
          raidz1         DEGRADED     0     0     0
            /dev/lofi/1  OFFLINE      0     0     0
            c4d1         ONLINE       0     0     0
            c2d0         ONLINE       0     0     0

errors: No known data errors

  pool: xenophanes
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        xenophanes  ONLINE       0     0     0
          c3d0      ONLINE       0     0     0

errors: No known data errors

as you can see, heraclitus is degraded, but operational.

so, we can just create our filesystems and copy data over

# zfs create heraclitus/home && zfs create heraclitus/opt
# cd /xenophanes/home && cp -@Rp * /heraclitus/home/ && cd /xenophanes/opt && cp -@Rp * /heraclitus/opt

and go have a cup of coffee or 12. When that's complete, we destroy the old pool

# zpool destroy xenophanes

and replace the lofi disk with the old zpool's disk

# zpool replace heraclitus /dev/lofi/1 c3d0

and there you have it. a 3-disk raidz out of a 2-disk mirror. No data juggling, tape drives, or extra disks necessary

KDE & DTrace

2007-12-11T13:09:00.000-08:00

At the moment I'm looking at adding dtrace probes to Apache/RogueWave's libstdcxx ( the C++ standard library) and I'm coming up against a couple hurdles, not least of which is the requirement for C++ name mangling.

C++ implements templates, namespaces, function overloading, inheritance and a myriad of other things that make plaintext names for functions unfeasible. So, in order to properly probe a function, we can't use the standard provider:module:function:probe naming scheme ( since the function will be something meaningless like __1cEswap4Ci_6FrTA1_v_ for a function named 'swap'

My current thought is that, since the function name is meaningless, perhaps we ought to exploit the probe names.

Talking with Damian on irc, we came up with a naming scheme for our probes that looks something like this: namespace_[class, if any]_function__probename

So a function like will have a probe named 'entry' that looks something like global-swap-TT-TT-entry.

The other option is to pretend the problem doesn't exist, use the fbt provider as normal, and pipe dtrace -l through c++filt but I don't really like that.

Indiana

2007-11-01T14:08:00.000-07:00

The proverbial cat is out of the bag now, and the Indiana project has released something.

I downloaded it last night and installed it in Fusion and Parallels. The installer is still pretty braindead, but I'm sure it'll improve to have options eventually. It is alpha after all.

I ran in to a weird bug where I haven't managed to coerce it in to letting me log in, so aside from that I haven't had a chance to fiddle enough to pass judgement on it.

It unfortunately had to come with what I perceive to be a slap in the face to the OpenSolaris community, when it was pronounced from on high that it shall be called OpenSolaris to the exclusion of all others.

This really gets my goat, so to speak (I'm rocking the animal metaphors today I guess) I don't particularly agree with the name change to begin with. I think OpenSolaris can exist as a noun in and of itself much as Linux does today, or as Joerg put it, like a screwdriver that you build things with, and not a particular screwdriver.

Ian's ranting about being confused how to download "OpenSolaris" notwithstanding (it being nothing but pure rhetoric to serve a purpose anyways), I think the target audience is intelligent and well-accustomed to the idea of distributions by now that calling OpenSolaris a codebase and nothing more isn't a real issue.

The slap in the face part comes primarily from the way that the name was chosen. Had there been a vote which suggested that Indiana be called OpenSolaris, then fine, fair enough. In this case however, the name came from executive fiat from Murdock quite aside from how the community feels about the issue.

Fortunately the OGB seems to have collectively grown a pair (individual members have been outspoken already. They're good.) and seem to be seriously discussing condemning this action by Murdock and the rest of the marketing crew and imploring Sun to hold a vote on the matter, which is something I support 100%.

So, in conclusion
Congrats to the indiana team for pushing something out the door. The marketing team's actions aren't your fault.

Large cats, ZFS and you

2007-10-26T22:38:00.001-07:00

Like many, I got my copy of OSX 10.5 "Leopard" today to install on my MacBook. The upgrade went relatively smoothly and I was presented with the new UI in all it's flashy glory.

After doing the standard explorations: Coverflow in my images/ folder, new Mail.app. OSX supports ACL's now, that makes me pretty happy. Then, I decided to start playing with the OpenSolaris-derived features.

Dtrace. works like a charm. Colloquy spends an inordinate amount of time in syscall::read. A lot of Mac apps don't use syscall:::, that's interesting to note but not terribly surprising considering the way XNU is designed.

ZFS. Now, the version of ZFS that ships with OSX is read-only. My server, an Athlon64, and my workstation, a Blade1000 run SX:CE b74.

So, I pull out my USB stick, put it in the SPARC, and turn it in to a zpool named 'test'. export.
Put it in my MacBook.
$ sudo zfs import -a
The ZFS version is too new. OSX can't import it.

So I decide "what the hell..." and I log in to my ADC account, download the read/write beta of ZFS from Apple. Install it. Reboot. Put the USB stick in.
$ sudo zfs create newpool disk1
'newpool' shows up on my desktop. So far, so good.
$ sudo zfs create newpool/test
still good so far. I copy a small image over to the new pool, it does what you'd expect. So I unmount the volume. Won't unmount because it contains other volumes. that's silly, but okay... it's a beta and OSX doesn't really grok ZFS yet. Fire up Terminal.app again
$ sudo zpool export -f newpool
no dice, dataset busy. unmount it's mountpoint then. Try again. No feedback, it must've worked.
yank the USB key. The Mac gives me the multilingual kernel panic window ( it's not a blue screen, that means it's better than Windows, right? ). Whatever.
I plug the key in to my Blade1k and run a
# zpool import -a
I/O error reading dataset 'test'. Kernel Panic. That's interesting... shouldn't the old dataset been destroyed when the new dataset took the device over?

So, I bring it back over to the Mac and run disk util, maybe I can just destroy the EFI partition table and try again from scratch. Disk util somehow mounts 'test' to my desktop and sits there spinning forever. I can't unmount test at all because it's in use. Force Quit Disk Utility. Nothing. kill -9 it. Nothing.
$ sudo umount -f /Volumes/test
could not unmount. yank the drive. kernel panic.

At about this point I decide that my best course of action with respect to this USB key is to find a machine that doesn't support ZFS and format it there, and give up trying to coax OSX to behave like Solaris.

Casualties: one USB key.
Conclusion: When Apple says "beta", they mean it.

Aftermath

2007-10-18T11:40:00.001-07:00

I appear to have caught some sort of nasty California virus on the airplane back that's left me quite incapacitated.

I've managed to wake up this morning, but that's about the extent of my ability.

Developer Summit: Day 2

2007-10-14T19:45:00.000-07:00

Day 2 went well, with the exception of the naming&branding discussion which ended up a little bit heated. Garrett did a GLDv3 conversion talk & there was a talk on moving core Sun processes such as ARC out in the open, streamlining them, etc.

It felt a bit short today. I'm not sure why, because we didn't get out that much earlier, but it did. Not to say today was entirely unproductive though, because it wasn't.

I think overall, this weekend was a good thing. Whether or not decisions were made that wouldn't have otherwise been is irrelevant, I think it was a fantastic idea to get everyone in the same place at the same time so that the community could all meet each other face to face.

I know I for one learned a lot, and I imagine others found some useful information to take out of the conference. It's also done a lot to dispell some of my fears with respect to Project Indiana.

Dev Summit. Day 1

2007-10-13T22:27:00.000-07:00

Today was the first day of the developer summit & I think it went over fantastic. Jesse, Sara, Glynn and the rest of them have done a fantastic job organizing the thing.
Stephen Hahn's presentation on the packaging system was quite informative and did a lot to calm some of my fears of the chaos of network install. After that I left the main presentation room in favour of seeing Garrett's presentation on fixing a driver to include suspend/resume funcionality, which was great, and then proceeded to Sun Labs' room to plan out the next course of action for the PowerPC port.
I'll organize some notes later & publish them

Early morning Chaos time

2007-10-12T07:55:00.001-07:00

I'm sitting on the incredibly overpriced wifi at YXS. I should be on an aircraft headed to YVR, but the airport shuttle heard "7am" instead of "6am".

Sara was an awesome help, so my flight was changed around and instead of YXS->YVR->LAX->SJC I'll be flying straight from Vancouver to SFO and meet someone at the Sun Menlo Park offices.

Dealing with problems at this hour is never fun. Thank you so much guys...

Closer and closer

2007-10-06T01:06:00.000-07:00

nightly(1) now successfully compiles through my code on x86.

It dies with the linker now, which is a vast improvement over the previous situation.

I want this done, debugged & integrated or at least the ARC process started by Christmas, if only to be able to give it as an xmas present to OpenSolaris.

Status, conferences and the likes

2007-10-04T12:44:00.000-07:00

So, In case anyone was curious, nightly manages to build most of emancipation-gate's libc, but I've been monkeying around with it because evidently O/N demands warning free code, and that's causing my code to not build. Lots of it comes from FreeBSD where they use gcc, for one, and are less strict about the build throwing warnings (especially in the regex stuff)

That out of the way, I'm excited to be going to the OSol Dev Summit next week, particularly in light of the recent thread on the mailing list which seems to imply a bunch of us want to break off from the packaging/install/community stuff in favour of more hard technical discussions.

It's been a while

2007-09-26T11:39:00.000-07:00

I know it's been a while since I've updated (or really done anything for that matter), but the school year started so it's been a bit hectic.

Good news though is that I'm no longer building code in a sandbox with ksh for loops, packing it up as libc_i18n, and putting it in place of the old code; Now I've got my code shoved inside an O/N fork, and ON attempts to build it. Emancipation has a new repo at hg.opensolaris.org/hg/emancipation/emancipation-gate if anyone wants to check it out. One caveat though is that this is a full ON tree, so it'll be quite large.

It's not building at the moment, and I'm not sure why, but since I just coaxed make(1) in to even trying last night at about 2am, I'll figgure it out and fix it. It's probably just something stupid anyways.

And, after a week off

2007-09-04T18:53:00.000-07:00

I took the last week off because I'm no longer time constrained & it's nice to take a break every once in a while.

last night, I removed a chunk of the GCC-isms that were preventing my code from compiling with studio and removed them ( __inline and __restrict were the main culprits ), so now it ought to work with both compilers, given gcc is passed the correct flags to comply with the standards in question. The code's in the Mercurial repo.

I was going to try to get it to link in this week, but it occured to me that it'd probably be a much better plan to shove it in an ON branch, so I think I'm going to play with makefiles for a while & remove the stupid compile step i've got going on right now.

cheers

Repo?

2007-08-26T10:52:00.000-07:00

I've had a number of people ask me about where exactly to find the code so far.

As of this morning some time, I migrated my code from cr.opensolaris.org over to the Project Emancipation Mercurial repo.

You can create a local clone with Mercurial. If you haven't ever done this before, do this:

$ hg clone ssh://anon@opensolaris.org/hg/emancipation/libc_i18n

and you'll have a local copy of the code

End of GSoC, but not of emancipation

2007-08-20T10:38:00.000-07:00

Well, so it's the 20th. Which means, as of noon today, PST none of the work I'm doing counts towards the Summer of Code.

I've got my code building, although it still won't link completely with libc. The linker doesn't see fit to recognize that I do, in fact, have getwc() and friends in an object file.

Regardless, I'm not going to be satisfied until this code integrates with OpenSolaris, so the end of Summer of Code really doesn't mean that much.

new Toys & Progress

2007-08-09T10:54:00.000-07:00

I got this machine on account of I need to break libc in fabulous ways & I need to make sure that it works on both Solaris platforms.

Tons of thanks to everyone that helped me find some SPARC kit to break!

As for progress, I tried to build my code in to O/N last night & it failed on symbol duplication errors, which is about the best error I could've gotten from the build.

I'm going to resolve that & see if I can't get a build of O/N that at least compiles with my code in it by Monday, and move on to testing from there.

Almost...

2007-08-06T09:26:00.000-07:00

I spent the last week visiting family in Vancouver, so I didn't do anything last week.

Regardless, with the exception of some private functions that I don't see in use anywhere, I think I'm pretty much done the initial coding.

Now I just need to force it in to libc and put together a workstation to break for testing.

Mid-program Update

2007-07-26T02:11:00.001-07:00

So, it seems my updates are getting less frequent as time goes on, but I'm sure that if you graphed it logarithmically, it'd still look somewhat linear at first glance ;)

Anyhow, The middle of the program has come and gone this week, I've gotten the midterm payment from Google and here's where I stand at the moment:

Most of the functions in my list are complete, with the exception of date/time stuff, which I'll work on next week. I finished off getwc/setwc and friends this week.

The subsequent week ( or next week, we'll see how things go ) I'll be shoving my libc_i18n.a in place of the one shipped with closed_bins and seeing what happens. Hopefully it won't blow up in my face, but that's what testing hardware is for.

Halfway there...

2007-07-04T13:50:00.000-07:00

It's the 4th of July today, which means happy birthday America to everyone in the country to my immediate south.

It also means that mid-term evaluations for the Summer of Code begin in two days.

Here's my own personal evaluation of myself:

I've got well over 3/4 of the major function groups done ( "function groups" meaning collate, regex, get/set characters/lines, binary is(alpha | number | English | punctuation ), etc ) and compiling cleanly, with some minor testing (albeit not on edge cases) completing successfully. I have a printout of libc_i18n.a's elfdump(1) symbol table output that I've been crossing off with a Sharpie, and it's getting more and more black by the day.

I feel as though I can complete the coding in the next few weeks, spend the remainder of the program testing & doing the minor tweaking required, and then work on integrating the code to the main Solaris tree ( which, as I understand it, is not necessary for the Summer of Code, so I didn't budget time for it )

hmm....

2007-06-22T11:54:00.001-07:00

I think I got strcoll & friends working this week...

It just loops through the string comparing each character and returning 1 or -1 if they're different as per the manpage, but I dunno, it seems like it should be more complex than that

weekly report

2007-06-15T23:31:00.000-07:00

All the low-lying fruit is done, as it turns out I could steal most of the EUC stuff from FreeBSD too. Still no idea what the "dense" functions are for ( obviously something densely packed ) but I'll keep looking...

What I have left are things like collate, which I plan to work on this weekend and a week or two afterwards, then regexes (how would you properly pluralize that anyways? ) for a little bit, then getwc & putwc and friends, and then I should be more or less done, save for shoehorning it in to the current Solaris framework and making sure it all works properly.

Perhaps a bit optimistic with the "I've only got to do..." type language, but I'm fairly confident that the project should be completed mostly on time.

Beginning of program update

2007-06-02T21:06:00.000-07:00

So, the Summer of Code program has now officially begun ( didn't stop me from writing stuff before then, but there have you ).

I've written a couple more functions & I've been testing stuff as I go along, no real exciting breakthroughs or any such thing. Things seem to be working more or less as expected ( though changing locales just explodes in a fire of glory )

Cheers

growing...

2007-05-19T03:42:00.000-07:00

I've successfully migrated FreeBSD's internationalization code outside of their tree, and I went to do a symbol count today ( from elfdump(1)'s symbol table ).

I've got 269 defined symbols, libc_i18n.a has 287. Given, FreeBSD has a bunch of functions for fiddling with their "rune" data structure, so as a progress bar the number comparison is more or less useless, I just found it mildly interesting

Unfortunately it all depends on gcc at the moment, and I'd like to get it cleaned up to compile with SUNWspro, but a Studio cleanup is at the bottom of the queue for now.

Weekly Status

2007-05-05T15:16:00.000-07:00

So, I figure it's Saturday, and I should probably make an update as to my progress thus far.

I've got all of FreeBSD's libc/locale subsystem compiling outside of the fBSD tree with the exception of localeconv.c, nl_langinfo.c & setlocale.c which I will finish for tonight or tomorrow evening most likely.

That was all mostly just a bunch of boring monkey work, but it'll make work later a lot easier. I've also been pondering about the best way to link together these objects as something I can encapsulate within a larger library that'll export the symbols I actually need (my linker-foo is weak).

I also need to inform myself about a way to convert the locale on-disk format that Solaris uses in to something that these fBSD functions expect. Since I'm almost certainly not allowed to break things, I'm thinking it'll have to be a runtime conversion with some wrapper functions. Given, I haven't looked too far in to things, so I may be completely off-base with what I need to do here.

first compile

2007-04-26T17:41:00.000-07:00

Like I mentioned in an earlier post, I've found FreeBSD's source to contain piles of eminently useful code.

I'm slowly starting to migrate fBSD's locale stuff to a standalone repo, and I've got a couple files to compile without error.

Whether they work or not is a different story, and I would be surprised if they did to be honest, but it is a start.

At this point I'm thinking that the best course of action is getting the proverbial square peg of fBSD's locale subsystem working on it's own, and writing wrappers to fit it in to Solaris' similarly proverbial round hole with proper type mangling first, and when that's working, move on to the rest of the libc_i18n functions that FreeBSD doesn't implement.

BSD for the win

2007-04-22T16:12:00.000-07:00

Dear OpenGrok group:

Thank you.

I set up an OpenGrok instance on my home machine that indexes NetBSD & FreeBSD, and started searching.

As it turns out, there's a lot in there that I can take, at least as a basic framework anyways, and modify to fit Solaris' locale formats

minus one distraction

2007-04-20T20:26:00.000-07:00

I had my last final exam today, and studying for exams was mostly what's been slowing me down.

Now that that's over with, I should be able to work far more regularly. I still have work to contend with (since at the end of the summer I'll still need a job, so I can't just quit), but that shouldn't take up the bulk of my time, and I can be pretty good about asking for less hours if it does start to interfere.

I took a look at NetBSD & FreeBSD's sources for implementation details, and they both look fairly promising. I should theoretically be able to pull a bunch of code out of there.

stay tuned.

Progress

2007-04-16T23:19:00.000-07:00

I started actual coding work today, I seem to work better when I sequester myself in to little cubicals in the library ( did I just actually say I like cubicals? ).

wcwidth() is now complete (minus testing) and wtype() is nearly finished (whenever I discover /createthe correct enums), which isn't much but it's a start, and wtype() leads to a whole bunch of other functions ( isalpha(), for example )

Accepted!

2007-04-15T16:11:00.000-07:00

So, I've created this blog because I was accepted to participate in Google's Summer of Code program, which is awesome!

The project I'll be working on is to re-implement OpenSolaris' internationalization functions that are in closed_bins at the moment, meaning they're not open-source.

The reason I chose this project is because one, I believe in (Open)Solaris, I think it's an unbeatable platform, and two, because I believe in open-source.
I think OpenSolaris will really begin to take off when it no longer relies on Sun Micro. to build. As great a company as they are, I can see how people can get scared off by the fact that there are still closed binary components and you can't really build an OSol distribution without the bi-weekly closed binaries.

I'm not hip to this whole blogging thing, but I'll try to remember to write an entry fairly regularly through the summer, to sort of document the whole experience.