Tuesday, November 11, 2008

OpenSolaris on a Macbook

I have a Macbook.

*jeers from the audience*

Just for the record it's the white model with 2Ghz CPU's and Wireless-N ( which makes it a Merom, I think )

I decided to install OpenSolaris on it. It being Apple, it has to be more obtuse and difficult than any other PC on the market.

Ultimately, a Mac is just like any other PC so there shouldn't be a whole lot of problem except for one thing: EFI.

EFI is Intel's BIOS replacement, because even though OpenFirmware/OpenBoot have been around forever, work well, and are open, it wasn't invented there. Nothing really boots off EFI yet except Linux, OSX, and the various HP operating systems. GRUB doens't work with EFI, for example, which leads to a number of problems getting OpenSolaris installed.

So, what the prospective OpenSolaris user will need to do is this:

Step 1: install a first stage EFI bootloader. This almost certainly means downloading and installing rEFIt. rEFIt is really easy to install, you just run the installer in the disk image.

Step 2: Toggle rEFIt's on switch. Open a terminal ( /Applications/Utilities/Terminal.app ) and type this:

/efi/refit/enable-always.sh

Step 3: You need to use Disk Utility to give yourself a free partition. Set it up as an MS-DOS (FAT) volume.

Step 4: Now, because the partition is marked as a DOS partition, OpenSolaris isn't going to want to install on it, so you'll need to mark it as a Solaris partition. This is done roughly the same way you'd do it anywhere else, that is to say, you'll be using fdisk. Open a terminal window and open fdisk on your drive, typically disk0. The -e flag makes it an interactive session.

$ fdisk -e /dev/disk0

It being Apple, the syntax is different from any of the other fdisk's you're likely to have seen. Find your partition in the list ( it'll be the one not marked HFS+ ) and to change it use:

setpid <partition number>

It'll prompt you for a type number in hex without the leading 0x. You'll be using type number 'BF' for Solaris2 ( because 0x82: Solaris was stolen for Linux swap... ). Then just 'write' to write the table, 'quit' to exit out of fdisk.

Step 5: Insert your CD, and reboot. You'll not get the familiar grey apple logo, instead you'll be presented with a rEFIt menu. Since rEFIt doesn't really know about the partition table well enough to present the operating system with anything sane, use the Partition Tool to set things up. Just use the defaults that it asks you about. Once that's written to disk, back in the rEFIt menu there'll be an option to boot OSX, or the CD. Choose the CD.

Step 6: Now you're running the OpenSolaris livecd, play around, do whatever you're going to do with it. Networking won't work because myk isn't installed. When you decide to install, there's some considerations. First off, you can't write to the partition table, rEFIt doesn't allow it (and it doesn't make much sense anyways). Since the installer really, really wants to write to the partition table ( bug 6413235 ) you'll need to trick it.
First off, you need to create a new file in your home directory named 'fdisk'. It should contain this:

#!/bin/sh
echo "$*" | grep -- "-F" > /dev/null
if [ $? = 1 ] ; then
/sbin/fdisk $*
fi

( Thanks to Jim Walker for the script )

make it executable

$ chmod 755 ./fdisk


and rather than running the installer icon on the desktop, you'll need to trick it in to using this new fdisk script with $PATH.

$ PATH=$PWD:$PATH pfexec

it'll do it's thing, fill out the information, click Next> a half dozen times. When it prompts you where to install, don't muck about with the partitions, there should be one Solaris partition that it'll try to install on by default, just click Next> through that screen. Now it's installing

Step 7: Reboot. rEFIt will present you with the option to boot OSX or Linux. rEFIt is dumb and thinks anything that uses GRUB is Linux. Let's ignore this slight for now, but bide our time and destroy everyone and everything they love, leaving them a sad hollow shell wandering the wasteland wishing for death, later. Choose the not-Linux Linux option, and OpenSolaris will boot. ( check here later for instructions on how to change the icon if it bothers you )

Step 8: Networking doesn't work. Pick up Murayama's myk driver. Put it on a USB stick. It comes with a Readme.txt, but you can ignore most of it. All you really need to do is this part:

$ pfexec make install
$ pfexec ./adddrv.sh
$ pfexec devfsadm -i myk
$ pfexec ifconfig myk0 plumb

at this point NWAM should realize that a new NIC exists, and do it's thing accordingly.

Now, wired networking works. Wireless does not because the new ath binary blob hasn't been integrated so the wireless-N chipsets are left in the cold. Sound is hdaudio, and the single button mouse is annoying, so I reccomend using an external one.

Other than that though, you're all good, up and running. Enjoy.

(and to all the other mac-heads that works on OpenSolaris or in marketing for it... now you have no more excuse ;) )

Sunday, September 28, 2008

Presenting: chsh

I logged in to the Solaris (9) machine at school the other day and was presented with this:

bash-2.05$

Bash. Gross.
Then it occurred to me that there's no real sane way for a user to change his or her shell on Solaris by default.
Being the diligent late-night hacker with an itch that I am, I solved the problem.

So, without further ado I present to you: chsh.

Usage:

chsh SHELL


or, to list valid shells

chsh -l


if you have the solaris.admin.usermgr role ( it is role aware, so root is unnecessary ) you can change any arbitrary user's shell

chsh -u USERNAME SHELL

It requires root to install, as it is a suid binary, so if you don't already have root-like privileges on the box it's not going to be a lot of help, but your users will thank you. Much like usermod(1M) it only modifies the local /etc/passwd file, so NIS+ and LDAP entries are unaffected.

I'm not sure about the process for getting something integrated in Solaris from scratch, and I'm pretty sure I missed the 2008.11 deadline so it won't be shipping but hopefully you'll see it in a Nevada build and IPS sooner or later.

Thursday, August 21, 2008

Tips & Tricks for a new sponsoree

I've recently been moved behind the firewall and am intern-ing at Sun.

One of the tasks of this job that I've been charged with is to sponsor community code contributions going in to OpenSolaris and I noticed a bunch of things that'd just make it less work & by extension time to go from an email with a .diff in it to integration. Let's begin:

So you want to go that extra mile for your sponsor:

You've signed your SCA , found a bug, sent your email to the list, and are ready to roll.
Awesome! OpenSolaris is huge and we appreciate the extra hands, plus you help make it more community driven.

So the next step is to go back and forth with your sponsor trying to find the best possible solution to the bug. Once you've done that, you can get down to work, fix the bug, and then build & test it.

Then you just send the code to your sponsor, and you're done right?
Well, you could do that. It's technically correct and is the standard way of doing things but if you really want to wow your sponsor and go above and beyond, there are a couple things that the gatekeepers require that you can do to make his or her job ( taking your code, committing it to the tree ) a breeze:
  • unified diffs : from the top of the tree, running a diff -u gives you a single patch file that can be applied to the tree, rather than a diff of every affected file. Better still is an hg export file. The standard comment for HG exports is :
    Contributed by {your name} <{your email}>
    {bug id} {bug synopsis}
    note the single space between bug id and bug synopsis
  • recent gate checkout : a bunch of stuff may have changed in the meantime, so it's best to apply your code to a copy of the ON gate as recent as possible, so that the diff doesn't break
  • cstyle clean it : onbld has a tool in /opt/onbld/bin called 'cstyle' which checks to make sure that the pendantic little nits like indentation standards are adhered to. You only need to make the stuff you changed cstyle clean, don't worry about all the old stuff
  • change the copyright : you'll notice every file owned by Sun in the ON consolidation that has a line with Sun's copyright in it like:
    /*
    * Copyright 2008 Sun Microsystems, Inc. All rights reserved.
    * Use is subject to license terms.
    */
    These all need to be updated to reflect the current year ( in this case it's 2008 ) whenever the file changes.
  • Update the CDDL block: Some older files have a CDDL block that's no longer acceptable, so if you run across it you may need to update it. You'll be able to tell, because it gives a version number ( 1.0 ) of the license. So, update this:
    The contents of this file are subject to the terms of the
    Common Development and Distribution License, Version 1.0 only
    (the "License"). You may not use this file except in compliance
    with the License.
    To this:
    The contents of this file are subject to the terms of the
    Common Development and Distribution License (the "License").
    You may not use this file except in compliance with the License.

  • SCCS keywords : the ON gate used to use a different source-code management system (teamware). The gate is now managed by Mercurial. As a result, there's still some vestigial teamware bits lying around. Before a change can be integrated someone ( typically the sponsor, but it could be you, which is why we're here ) needs to remove the old SCCS keywords. They look like this:
    #pragma ident "%Z%%M% %I% %E% SMI"
    This whole line can just be removed from the file entirely. It's not needed anymore.
With these simple steps, you can cut (depending on the size of the patch) hours off your sponsor's task and surprise them with how thorough you are

Tuesday, May 13, 2008

Emancipation Community

I proposed the other day that the emancipation project be promoted to the Emancipation Community.

The way I see it, Jason & Steven's work on the ce driver, Roland's work on the xpg/posix stuff
and my libc_i18n work are all loosely related in that they are all reimplementations of closed source stuff, but aren't really as closely micro-coupled as much as one might think a project is ( we don't share an hg repo set, for example )

Here's what Plocher and I came up with for a charter:

Emancipation Community Charter ( rev 1 )
CG Problem statement

The OpenSolaris operating system is not completely open
because several components that are required to build and
boot the OS are only available in the "closed bin" archives.

Scope:
Initially, the focus will be on selected high-value efforts,
such as self-hosting an open ON, drivers, posix utils, but
the long range intent is to eliminate the need for (and use
of) closed source software on the opensolaris OE.

Goals/milestones:

Quarterly progress reports will be produced and sent to the
OS-announce alias to keep the larger community informed of
our progress.

In order of priority:

Goal 0: Replace the components needed to build and boot
the ON consolidation with whatever shims, hacks
and scaffolding is needed to produce a proof of
concept that self-hosts and boots, followed by a
reimplementation of the userland utilities as per same.

Goal 1: Determine the best way to replace the above hacks
with a permanent solution, including decision
making architectural and design guidelines that
can be used in similar situations elsewhere in
the emancipation effort (i.e., should we reuse
from some particular other open OS, roll our own,
do without; what makes a good -vs- poor choice,
how do we choose without causing unnecessary
strife, ...? Collaboration with the ARC community
is implied during this stage.

Goal 2: Develop and push the changes prototyped in phase
0 and formalized in phase 1 into the ON (and other)
consolidation(s) and remove the associated closed
bin pieces.

Goal 3: Seek replacement for high-use closed source software
such as media players, rich web players, etc.

Goal 4: After completion of goals 0 - 3, disband the community

To facilitate this community, the initial list of core contributors (
should they accept ) shall be:

Faciltator:
John Sonnenschein ( error404 )
Core Contributors:
Jason King ( jbk )
Steve Stallion ( stallion )
Roland Mainz ( gisburn )
Joerg Schilling ( joerg )
John Sonnenschein ( error404 ) ( note: i'm not sure if this is
implied by "facilitator" )
Garret D'Amore ( gdamore )
Contributors:
John Plocher ( plocher )

Wednesday, March 12, 2008

Bio & Affiliation

hey, so this is a little on the late side, but not too late I hope

Disclosure: I currently work for an MS Windows based contracting firm that has no commercial interest in OpenSolaris. I therefore am under no obligations that would prevent me from fully representing the community if elected to the OGB.

Bio: I'm an undergraduate computer science student at the University of Northern BC and a part-time software developer . I joined OpenSolaris approximately 3 or 4 months after the inception of the project, being a former 7 year Linux user. I was drawn to OSol because of the tradition of unrelenting engineering excellence that I knew Sun & her engineers for.
I envision the OGB's primary charter for the time being as being the liaison with the benefactor of OpenSolaris ( that is, Sun ). The OGB ought to be the go-between of Sun's management and the rough-and-tumble of the community. Sun has immense influence on the community as it is a community built around their code, and it's important for the OGB to act on the community's behalf when faced with decisions regarding steering that community ( for example, the recent trademark debacle. I believe the OGB could have had a much more influential role negotiating with Sun during that incident).

Wednesday, January 2, 2008

How to turn a mirror in to a RAID

People occasionally ask on the mailing lists and in #opensolaris how to add a disk to a zfs mirror to make a raidz. Today, I received in the mail a new SATA controller and a new disk, so I was left in the same circumstance.

There's a drought of information on the topic on the internet, probably due in large part to the typical deployment of ZFS ( i.e. large shops that have a ton of spare disks laying around, or have otherwise planned out a migration path beforehand ), rather than the small home user.

So, here's what I did:

On a high level, we have to remember what sort of replication we've got for any given RAID level. More accurately, we need to know how many disks can be broken before the whole thing falls apart.

When we've got a single drive, that drive can't die, or we lose everything (obvious). With a mirror, we can't have 2 drives die. A 3-disk RAIDZ ( raid5 ) requires at least 2 operational disks out of 3. So, when moving from a 2 disk mirror to a 3 disk raidz, we obviously don't have enough disks to have both of them operational in full, even if we break the mirror in to a single disk.

But, if we count the number of disks allowed to be dead ( 2 ) at any given time, and the number we have ( 3 ), we can spread them out such that two degraded pools exist. One single-disk ( broken mirror ) and one degraded zpool ( 2 disks + NULL ).

So the procedure we'd use to attain this state is break the mirror, create a zpool with the new disk and the old mirror drive, copy the data over, destroy the old mirror, attach the old second mirror disk to the new raidz.

For the purpose of demonstration, I'll use the disks I've got attached to the system, c2d0, c3d0, and c4d1 .

first, the starting condition:

$ zpool status

pool: xenophanes
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
xenophanes ONLINE 0 0 0
mirror
c3d0 ONLINE 0 0 0
c2d0 ONLINE 0 0 0

errors: No known data errors


Now, to break the mirror:

# zpool detach c2d0


so, what I've got now is a single-disk zpool comprised of c3d0, and two free disks, c2d0 and c4d1.

To create a raidz, you need 3 devices. We only have two. We can solve this problem, however, with sparse files and loopback.

Loopback allows you to use a file the same way you'd use any other block device in /dev. Linux has it ( mount -o loop ), Solaris has it ( lofiadm ). It's a pretty common thing.
A sparse file is a type of file where the filesystem only stores it's beginning and end pointer information, and a size. The actual contents of the file aren't stored until you begin to write to them. This allows us to do things like create a 140GB file on a 140GB disk with plenty of room to spare. And that's precisely what we'll do.

You can create a sparse file easily with dd(1) like so:

$ dd if=/dev/zero of=/xenophanes/disk.img bs=1024k seek=149k count=1


bs is block size, 1kb. seek is the number of blocks to skip ( and is equal to the size of the drive in kb, because of the previous bs= line ), and count tells dd(1) to copy one block.

and we can create a device like so:

# lofiadm -a /xenophanes/disk.img
/dev/lofi/1


So, to recap, here's what we have. We have a zpool, two spare disks ( c2d0 and c4d1 ) and a sparse file the size of those disks hooked up with lofi. And if you'll notice, that's precisely what we need.

From here out, we need to create the raidz, degrade it ( otherwise we'll fill up a sparse file that's the same size as the other disk, it'll run out of space, stuff will break... it won't be pretty )

# zpool create heraclitus raidz c2d0 c4d1 /dev/lofi/1

ta da! a raidz. Now let's break it.

# zpool offline heraclitus /dev/lofi/1 && lofiadm -d /dev/lofi/1 && rm /xenophanes/disk.img

and here's what we get:

# zpool status
pool: heraclitus
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
heraclitus DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
/dev/lofi/1 OFFLINE 0 0 0
c4d1 ONLINE 0 0 0
c2d0 ONLINE 0 0 0

errors: No known data errors

pool: xenophanes
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
xenophanes ONLINE 0 0 0
c3d0 ONLINE 0 0 0

errors: No known data errors


as you can see, heraclitus is degraded, but operational.

so, we can just create our filesystems and copy data over

# zfs create heraclitus/home && zfs create heraclitus/opt
# cd /xenophanes/home && cp -@Rp * /heraclitus/home/ && cd /xenophanes/opt && cp -@Rp * /heraclitus/opt


and go have a cup of coffee or 12. When that's complete, we destroy the old pool

# zpool destroy xenophanes


and replace the lofi disk with the old zpool's disk

# zpool replace heraclitus /dev/lofi/1 c3d0


and there you have it. a 3-disk raidz out of a 2-disk mirror. No data juggling, tape drives, or extra disks necessary