Sunday 15 May 2022

The journey to ZFS raidz1 with different sized disks (On NetBSD) (Wheelbarrow optional)

The joy of having a redundant remote backup machine is that if it dies you do not immediately lose any data, just some redundancy.

On the other hand, sufficient remoteness can make the process of rebuilding annoying enough to encourage a burning desire to avoid repeating it.

So... I find myself wanting to setup a NetBSD machine with a ZFS raidz1 - but without sufficient equally sized large disks or enough drive bays to pack in many smaller disks. In the world of ZFS, that's a pretty big liability.

So, what are our assets?

  • Five drive bays
  • One 8TB disk
  • Two 6TB disks
  • A seemingly unlimited number of 2TB and smaller disks

That's it? Impossible. If I had a month to plan, maybe I could come up with something. But this... I mean, if we only had a wheelbarrow, that would be something.

Now the question of how to use different sized disks with ZFS comes up reasonably often, and the answer is invariably "You don't", quite often followed by:

  • Various ways in which you would not be able to do it
  • One way in which you could do it, but everything pretends to be the size of the smallest disk, so you have much less space
  • Some options to build a stack of ZFS on something on ZFS on something on ... with a note that you may want to supply the disk data blocks with good reading material and detailed maps in case they get lost on the long trek to and from the disk
  • (and my personal favourite) Split up all the devices into the largest divisible unit common to all disks, determine just the right way to stack and assemble all the units (I think that would come to 12 separate units in my case), wrap the result up in a holocaust cloak, set it on fire and push it towards a castle gate

These all seemed to either not give the desired result, or to be the wrong type (*) of complexity (or quite often both), so I thought I'd give it a go

*: Criticising other solutions as too complex while I contemplate my own particular combination of cogs, pulleys, and string, would be... unsporting

Anyway, I did consider using RAIDframe, but to bolt together each 6 & 2 disk to pretend to be an 8, but then... I recalled ccd - a delightful little artisanal driver from the last couple of decades of the 1900s. ccd can be used to stripe together multiple devices, concatenate them, or even a combination (stripe until one until it runs out then just use the other).

Since I'm just trying to pretend to ZFS that "No, no, really you have three identically sized disks. Just Trust Me Here", I elected to concatenate them.

And as a final gust of pretending this Might All Be A Serious Attempt At Making Something Useful, I decided to use labelled wedges, for both the ccd and the ZFS elements, so when something inevitably fails and devices just go away, or everything gets randomly crammed into another box the system might have a vague chance of determining how to fit together whatever pieces remained.

So, before the script (there always has to be a script, encoding the necessary magic with cryptic, misleading and occasionally completely incorrect comments for the edification of one's future self), did it work?

It seems to work fine, taking:
wd0: 1863 GB, 3876021 cyl, 16 head, 63 sec, 512 bytes/sect x 3907029168 sectors
wd1: 7452 GB, 15504021 cyl, 16 head, 63 sec, 512 bytes/sect x 15628053168 sectors
wd2: 1863 GB, 3876021 cyl, 16 head, 63 sec, 512 bytes/sect x 3907029168 sectors
wd3: 5589 GB, 11628021 cyl, 16 head, 63 sec, 512 bytes/sect x 11721045168 sectors
wd4: 5589 GB, 11628021 cyl, 16 head, 63 sec, 512 bytes/sect x 11721045168 sectors

... into

# zpool list -v  
NAME               SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
onyx              21.8T  5.94M  21.7T         -     0%     0%  1.00x  ONLINE  -
  raidz1          21.8T  3.83M  21.7T         -     0%     0%
    wedges/onyx0      -      -      -         -      -      -
    wedges/onyx1      -      -      -         -      -      -
    wedges/onyx2      -      -      -         -      -      -

Right now its booted off an old USB key which appears to think its a QL microdrive, so syncthing is busy maxing out IOPs to that while its downloading the data to rebuild the backup, but that's a problem for another day.

Oh, yes - the script. I suspect primarily for my benefit so as when something goes horribly wrong in three years time and I have no memory of how I set things up:

#!/bin/sh -e
# Setup raidz1 pool from one large disk, and two smaller paired disks using ccd
# - everything created in labelled wedges  
# - vdev size is capped to the first large disk

DISK0=wd1
DISK1a=wd3
DISK1b=wd2
DISK2a=wd4
DISK2b=wd0
DISKS="$DISK0 $DISK1a $DISK1b $DISK2a $DISK2b"

calculate_needed()
{
disk1_size="$(gptsize $1)"
disk2_size="$(gptsize $2)"
# Add 4096 for ccd header
expr $disk1_size - $disk2_size + 4096
}

create_ccd()
{
wedge=$1
ccd=$2
diska=$3
diskb=$4
ccdconfig $ccd 0 0 NAME=$diska NAME=$diskb
# gpt destroy falls over if reusing with different ccd size
(yes y | fdisk -i $ccd) > /dev/null
gpt destroy $ccd || true
gpt create -f $ccd
gpt add -a 1m -t fbsd-zfs -l $wedge $ccd
}

gptsize()
{
gpt show -i 1 $1 | awk '/Size:/{print $2}'
}

zpool destroy onyx || true
ccdconfig -u ccd1 || true
ccdconfig -u ccd2 || true
for disk in $DISKS; do
  gpt destroy $disk || true
  gpt create $disk
done
gpt add -a 1m -t fbsd-zfs -l onyx0 $DISK0

gpt add -a 1m -t ccd -l onyx1a $DISK1a
extrasize="$(calculate_needed $DISK0 $DISK1a)"
gpt add -a 1m -t ccd -l onyx1b -s $extrasize $DISK1b
create_ccd onyx1 ccd1 onyx1a onyx1b

gpt add -a 1m -t ccd -l onyx2a $DISK2a
extrasize="$(calculate_needed $DISK0 $DISK2a)"
gpt add -a 1m -t ccd -l onyx2b -s $extrasize $DISK2b
create_ccd onyx2 ccd2 onyx2a onyx2b

sleep 1 # Allow devpubd to catch up
zpool create onyx raidz1 /dev/wedges/onyx0 /dev/wedges/onyx1 /dev/wedges/onyx2

# Calculate ccd size, remembering 4096 header
# Cannot use "ccdconfig -g ccd1 ccd2" as it loses wedge names
disk0_size=$(gptsize $DISK0)
ccd_size=$(expr \( $disk0_size + 4096 \) \* 512)

cat > /etc/ccd.conf <<END
ccd1            0      none  NAME=onyx1a NAME=onyx1b
ccd2            0      none  NAME=onyx2a NAME=onyx2b
END

Wednesday 24 September 2014

CentOS 6 DOMU under NetBSD/xen in 15 minutes

A while ago I wrote a quick post on setting up CentOS 5.x under NetBSD/xen
I've long since updated all the CentOS instances to 6.x, but I found I needed to install a fresh version and as the procedure was slightly different, here are some notes :)

Converting an existing NetBSD install to Xen DOM0

  • Install xenkernel42 and xentools42 from pkgsrc
  • Extract  kern-XEN3_DOM0.tgz into /netbsd.dom0 (leave /netbsd alone)
  • Add the following to /boot.conf - adjust 512M to the amount to reserve for the DOM0
menu=Xen 512M:load /netbsd.dom0 console=pc;multiboot /usr/pkg/xen41-kernel/xen.gz dom0_mem=512M noreboot
  • Add the following to /etc/rc.conf
xencommons=YES
xenwatchdog=YES
  • Create /etc/ifconfig.bridge0 with the following - replace bge0 with a suitable Ethernet interface
create
!brconfig $int add bge0 up
  • Reboot and select the Xen option
  • Edit /boot.cfg and change the default= entry to the number of the Xen option

To mis-quote Ken Arnold: "Now that we have a NetBSD DOM0, it would be nice to do something with it, unless you have the social life of a kumquat in which case just running the DOM0 may be enough in itself :)"

Onto setting up the DOMU!

Setting up a CentOS 6 DOMU

The easiest option it to run the standard CentOS install iso in a xen hvm (Hardware Virtual Machine) guest, and then after install switch it to a pv (ParaVirtualised) guest. The big advantage of this is it should work for pretty much any guest OS - if the guest does not support xen, just leave it as an hvm guest!

The follow assume all files are in /opt/xen. Adjust to taste
  • Download the latest CentOS 6 install iso
  • Create a file onto which to install. The below is 20G (20480M)
dd if=/dev/zero of=centos6.img bs=1m count=20480
  • Create a config file 'centos6.cfg' for installing
name = "centos"
memory = "512"
disk = [ 'file:/opt/xen/centos6.img,xvda,w', 'file:/opt/xen/boot.iso,hdc:cdrom,r'
]
vif = [ mac=02:00:00:00:00:01, 'bridge=bridge0', ]
vcpus = 2
builder='hvm'
boot='d'
vnc=1
vncunused=1
  • Start the domain to install, and then connect to the console using vncviewer. If you are running this remotely on a server you can 'ssh -Y in to proxy the X connection back to your local host
xm create centos && vncviewer :0.0
  • Complete the CentOS install, then adjust centos.cfg as vmlinuz & initrd are not needed
name = "centos"
memory = "512"
disk = [ 'file:/opt/xen/centos6.img,xvda,w', ]
vif = [ mac=02:00:00:00:00:01, 'bridge=bridge0', ]
vcpus = 2

Points to note

  • NetBSD/xen does not yet handle modules. If you need pf or ipf you will need to compile your own DOM0 kernel
  • Not all X drivers run as well under DOM0, the above was done on a server where X was not required
  • If the original NetBSD installation was before 5.0 and has been upgraded you will need to ensure the boot program must be updated
  • If anyone hits any issues with any of the above, drop me a message & I'll be happy to try to help :)

Friday 21 September 2012

Netbooting NetBSD on a PC Engines Alix2

Alix 2d2 board
I've been looking for a small box to act as a firewall and OpenVPN server, and someone pointed me at the PC Engines Alix2 - a delightful little AMD Geode board, with up to 256M, three Ethernet ports, USB, CompactFlash, miniPCI and a serial console.

The Geode is a 486 class processor, so will just run NetBSD/i386 directly.

I picked the 2D2, which just has two Ethernet ports, primarily because the 2D3 was out of stock and I wanted something now :)

Installing NetBSD

The easiest option would be to put the Compact Flash card into a normal PC, install as normal, and set the boot blocks to use the serial console. Of course this assumes that: (a) you haven't already put the CF card into the Alix and assembled the case, and (b) you're not in the process of redecorating  your study, the contents of which have been decanted into a huge pile in the kitchen, with your only CF equipped machine at the bottom.

So, a good opportunity to test netbooting and install, which I've been doing on Suns and other machines since early 1990's, but this is possibly the first time I've done it on an x86 box. I'm going to use my T500 running NetBSD/amd64 as the server.

1) Console setup

 

The Alix 2 has a serial console defaulting to 38400 bps. As serial ports are a distant memory for most modern machines, my T500 needs a USB to serial adaptor, plus the obvious null model cable. On NetBSD the USB serial port shows up as ttyU0 (or dtyU0 for the 'direct' device), so to connect:
# cu -l dtyU0 -s 38400
Powering on the ALIX rewards me with:
PC Engines ALIX.2 v0.99h
640 KB Base Memory
261120 KB Extended Memory

01F0 Master 848A TS8GCF133                              
Phys C/H/S 15538/16/63 Log C/H/S 974/255/63 LBA

BIOS setup:

(9) 9600 baud (2) 19200 baud *3* 38400 baud (5) 57600 baud (1) 115200 baud
*C* CHS mode (L) LBA mode (W) HDD wait (V) HDD slave (U) UDMA enable
(M) MFGPT workaround
(P) late PCI init
*R* Serial console enable
(E) PXE boot enable
(X) Xmodem upload
(Q) Quit
All that should be required now is to select PXE boot and away we go.

It is at this point that I will reveal that while the NetBSD PXE boot program is more than happy to speak to the console at the speed it is given, when it goes to load the kernel it seems to switch to 9600 baud (at least as on NetBSD 6.0_RC1). Rather than relate the actual sequence of events (which involved the NetBSD boot startup, go to load the kernel, then some time of watching a non-responsive screen, followed by a "hmmm", and a retry... closely followed by another retry with tcpdump running, a quizzical expression and finally a minor baud rate epiphany, I'll continue as if I already knew about the baud rate issue (purely to make me seem more forward thinking).

So, that would be a [9], [e], [q] and a [y] to save, followed by [RETURN][~][.] to exit cu, and reconnecting at the new speed with
# cu -l dtyU0 -s 9600
This time you should be rewarded with a message ending something like:
CLIENT MAC ADDR: 00 0D B9 DD EE FE
which we need for the netboot server.

2) The netboot server


For this we need three services, conveniently all of which ship with NetBSD:
  1. A DHCP server to provide an IP address and details of where to load the boot program (dhcpd)
  2. A tftp server to serve the boot program (tftpd)
  3. An NFS server to provide the NetBSD kernel which the boot program loads (rpdbind, mountd, nfsd)
We could extract a full NetBSD/i386 distribution onto the NFS server and export it to the Alix, but a diskless network routing box which requires an NFS server to run seems a little... perverse, so we'll just use a NetBSD INSTALL kernel.

That means we need two files from a NetBSD/i386 distribution:
  1. The PXE boot program - installation/misc/pxeboot_ia32.bin
  2. An INSTALL kernel with ramdisk - binary/kernel/netbsd-INSTALL.gz
Other items to note, and potentially change in your configuration
  • The gateway to the Internet is at 192.168.2.1
  • The server is at 192.168.2.2
  • The Alix is at 192.168.2.3 and named "alix"
  • The Alix "hardware ethernet" address is copied from above
  • The directory to NFS export will be /export/alix"
So, starting with the dhcpd setup
  • Create an /etc/dhcpd.conf as below
  • Add "dhcpd=YES" to /etc/rc.conf
  • Start dhcpd with /etc/rc.d/dhcpd start
# dhcpd.conf
ddns-update-style none;
subnet 192.168.2.0 netmask 255.255.255.0 {  option routers 192.168.2.1;
  option domain-name-servers 192.168.2.1;
  host alix {
    hardware ethernet 00:0d:b9:dd:ee:fe;
    filename "pxeboot_ia32.bin";
    fixed-address 192.168.2.3;
    next-server 192.168.2.2;
    option host-name "alix";
    option root-path "/export/alix";
  }
}
(If its never been run before you will also need to touch /var/db/dhcpd.leases)

Next the PXE boot program
  • Copy pxeboot_ia32.bin into /tftpboot/pxeboot_ia32.bin
  • Edit /etc/inetd.conf, uncomment the line starting "#tftp"
  • Restart inetd with /etc/rc.d/inetd restart

Finally the kernel
  • Copy netbsd-INSTALL.gz in /export/alix as netbsd (no need to uncompress it)
  • Edit /etc/exports and add a line "/export/alix 192.168.2.3"
  • Add the following lines to /etc/rc.conf "mountd=YES", "nfsd=YES", "nfs_server=YES"
  • Start the daemons: /etc/rc.d/rpcbind start; /etc/rc.d/mountd start; /etc/rc.d/nfsd start

3) Netbooting the Alix


Once all the above is in place we can connect the console, switch on the Alix and netboot it. We need to interrupt the PXE boot program to tell it to use the serial console, but apart from that it should just boot directly into the NetBSD sysinst installer. Log below:
PC Engines ALIX.2 v0.99h
640 KB Base Memory
261120 KB Extended Memory

01F0 Master 848A TS8GCF133
Phys C/H/S 15538/16/63 Log C/H/S 974/255/63 LBA

Intel UNDI, PXE-2.0 (build 082)
Copyright (C) 1997,1998,1999 Intel Corporation
VIA Rhine III Management Adapter v2.43 (2005/12/15)

CLIENT MAC ADDR: 00 0D B9 DD EE FE
CLIENT IP: 192.168.2.3 MASK: 255.255.255.0 DHCP IP: 192.168.2.2
GATEWAY IP: 192.168.2.1


>> NetBSD/x86 PXE boot, Revision 5.1 (from NetBSD 6.0_POST_RC1)
>> Memory: 555/261120 k
Press return to boot now, any other key for boot menu
booting netbsd - starting in 5 seconds. [SPACE]
type "?" or "help" for help.
> consdev com0

>> NetBSD/x86 PXE boot, Revision 5.1 (from NetBSD 6.0_POST_RC1)
>> Memory: 555/261120 k
> boot netbsd
PXE BIOS Version 2.1
Using PCI device at bus 0 device 9 function 0
Ethernet address 00:0d:b9:dd:ee:fe
10741836+5665276+443200 [573760+563454]=0x11296ac
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.

NetBSD 6.0_POST_RC1 (INSTALL)
total memory = 255 MB
avail memory = 234 MB
RTC BIOS diagnostic error 0x80<clock_battery>
mainbus0 (root)
acpi_probe: failed to initialize tables
cpu0 at mainbus0: Geode(TM) Integrated Processor by AMD PCS, id 0x5a2
pci0 at mainbus0 bus 0: configuration mode 1
pchb0 at pci0 dev 1 function 0: vendor 0x1022 product 0x2080 (rev. 0x33)
glxsb0 at pci0 dev 1 function 2: RNG AES
vr0 at pci0 dev 9 function 0: vendor 0x1106 product 0x3053 (rev. 0x96)
vr0: interrupting at irq 10
vr0: Ethernet address: 00:0d:b9:dd:ee:fe
ukphy0 at vr0 phy 1: OUI 0x0002c6, model 0x0034, rev. 3
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vr1 at pci0 dev 11 function 0: vendor 0x1106 product 0x3053 (rev. 0x96)
vr1: interrupting at irq 15
vr1: Ethernet address: 00:0d:b9:dd:ee:ff
ukphy1 at vr1 phy 1: OUI 0x0002c6, model 0x0034, rev. 3
ukphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
gcscpcib0 at pci0 dev 15 function 0: vendor 0x1022 product 0x2090 (rev. 0x03)
gcscpcib0: Watchdog Timer via MFGPT0, GPIO
gpio0 at gcscpcib0: 32 pins
viaide0 at pci0 dev 15 function 2: AMD CS5536 IDE Controller (rev. 0x01)
viaide0: primary channel interrupting at irq 14
atabus0 at viaide0 channel 0
viaide0: secondary channel ignored (disabled)
ohci0 at pci0 dev 15 function 4: vendor 0x1022 product 0x2094 (rev. 0x02)
ohci0: interrupting at irq 12
ohci0: OHCI version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
gcscehci0 at pci0 dev 15 function 5: vendor 0x1022 product 0x2095 (rev. 0x02)
gcscehci0: interrupting at irq 12
gcscehci0: companion controller, 4 ports each: ohci0
usb1 at gcscehci0: USB revision 2.0
isa0 at gcscpcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com0: console
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
attimer0 at isa0 port 0x40-0x43
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker
sysbeep0 at pcppi0
isapnp0 at isa0 port 0x279
npx0 at isa0 port 0xf0-0xff
attimer0: attached to pcppi0
uhub0 at usb0: vendor 0x1022 OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1 at usb1: vendor 0x1022 EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
wd0 at atabus0 drive 0
wd0: <TS8GCF133>
wd0: 7647 MB, 15538 cyl, 16 head, 63 sec, 512 bytes/sect x 15662304 sectors
boot device: vr0
root on md0a dumps on md0b
root file system type: ffs
WARNING: clock lost 4632 days
WARNING: using filesystem time
WARNING: CHECK AND RESET THE DATE!
warning: no /dev/console
Created tmpfs /dev (1474560 byte, 2848 inodes)
Alix running sysinst (via T500)
Followed by the sysinst screen. The only thing to remember is that when prompted to install the boot blocks, select the serial console version, and your preferred speed (I went back to 38400 at that point).

Friday 17 August 2012

Instant wifi - just add "Edimax EW-7811UN"

Archimedes could use one of these with a pivot
Ah, the 3Com OfficeConnect 3CRUSB10075. I remember it well...

About five years ago this would have been sold as a USB wifi adaptor, though its actual purpose was more to test the structural integrity of laptop USB ports. Extending no less than three inches, it was also cleverly more than an inch wide - providing the added bonus of preventing laptops with vertical USB ports from sitting flat.

Slightly larger than the average car's indicator controls, and also slightly easier to hit, it was close to guaranteed that at some point *something* would strike the OfficeConnect a lateral blow, resulting in  two goodly sized pieces of grey plastic, one still securely connected to the USB port.

Of course the unlucky would find themselves with one very large piece of plastic, still securely connected to your USB port, and a usb sized hole in the side of the laptop.

For mild amusement, I plugged it into my Thinkpad (after turning the laptop on its opposing side, then using both hands and alignment rails to guide it into place) and NetBSD's dmesg blurted out:
zyd0 at uhub3 port 1
zyd0: 3COM 3CRUSB10075, rev 2.00/43.30, addr 2
zyd0: HMAC ZD1211, FW 46.05, RF RFMD, PA 0, address 00:0f:cb:c0:8c:d6
zyd0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
zyd0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
If I recall it even came with a cap - which itself was around three times the size of the other USB wifi adaptor in the picture - an Edimax EW-7811UN.
For the princely sum of GBP8.95 delivered this diminutive device provides wifi connectivity without significantly adjusting your machine's centre of gravity:
urtwn0: Realtek 802.11n WLAN Adapter, rev 2.00/2.00, addr 2
urtwn0: MAC/BB RTL8188CUS, RF 6052 1T1R, address 80:1f:02:61:13:b0
urtwn0: 1 rx pipe, 2 tx pipes
urtwn0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
urtwn0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
I had hoped to provide some benchmarks between the two USB devices and my Thinkpad's built in Intel PRO/Wireless 5100 AGN:
iwn0 at pci3 dev 0 function 0: vendor 0x8086 product 0x4237 (rev. 0x00)
iwn0: interrupting at ioapic0 pin 17
iwn0: MIMO 1T2R, MoW, address 00:16:ea:c5:bb:10
iwn0: 11a rates: 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
iwn0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
iwn0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
... but unfortunately the OfficeConnect appears to no longer be able to associate with my WPA2 network. Possibly some of its valves have worn out, or maybe a small family of mice have set up residence inside - I would shake it to see if it rattles but I fear I might put my back out...

For reference the PRO/Wireless 5100 reported ttcp rates of 17.2 Mbit/sec send and 16.9 Mbit/sec receive, while the Edimax managed 12.7 Mbit/sec and 12.2 Mbit/sec.

Not earth shattering, but not too bad either.

So what prompted this trip down "USB wifi adaptors the size of vinyl records" memory lane? Someone on a mailing list was running NetBSD on a macbook and it wasn't recognising the wifi. I suggested trying a nano USB wifi dongle, but I wanted to recommend one that worked - so I bought one to see it work.

In the process I found out that while NetBSD-current included the urtwn driver and the firmware, NetBSD-6 BETA2 only had the driver and required a manual download of the firmware, so I submitted a pullup request to get the firmware into the NetBSD-6 release - its always nice to have stuff work Out Of the Box :)

About the only thing bad I could have to say about the Edimax is its use of one of those violently blue LEDs - you know the one's suitable for landing aircraft in heavy fog or signalling direct to the Curiosity rover on Mars. It doesn't have any gaps or transparency in the case - the LED just shines directly through the black plastic (and I suspect the metal USB plug if it was far enough out of the machine).

(some time elapses, as it does)

Now *that* is interesting... in a "random profanity and threatening of inanimate objects" sort of way...

Presumably the OfficeConnect did not appreciate my less than glowing comments about it - that, or possibly because I forgot to use the correct guide rails and block-and-tackle to remove it from the machine - when I unplugged it my laptop hung for a moment and then rebooted.

More deserving of the profanity however, on reboot the laptop hung when probing the CPU. Repeated. Powered off, leaving for a minute and back on and it *still* hung on probing the CPU. Eventually I removed the battery and power cable, held the power button down for 30 seconds, then reassembled and successfully booted.

So it appears the OfficeConnect is capable of deranging the USB hardware on a Thinkpad sufficiently well to require a complete disconnection and draining of all power. Handy... if you like the sensation (albeit briefly) of all blood draining from your face as you consider the prospect of your laptop having just been completely fried.

Things you would prefer not to fall out of your battery
As an added bonus, when I removed the battery I found myself in possession of a small spring and broken black plastic tag, which appear to have fallen out of the battery.

It could be a subtle suggestion that its time to replace the battery, or a somewhat less subtle hint that I really should drop the Thinkpad less..

Anyway, that concludes today's "Fun With WiFi And NetBSD" - and today's top hints are:
  • Edimax make a cheap and usable tiny wifi adaptor
  • If you have an old wifi adaptor large enough use as furniture, just carefully dispose of it
  • While repeatedly dropping a Thinkpad may make it gradually lighter over time, its not a good idea

Saturday 4 August 2012

Final Dreamcastery nonsense at the end of Retrochallenge 2012

Retrochallenge 2012 has come to an end, so this is a catchup post of the mainly Dreamcast related foolery over the past week.

What every Dreamcast needs
Quite some time ago Kiyoshi IKEHARA designed a DCEXT board enabling connection of an IDE disk and an ISA (typically network) card to NetBSD/dreamcast, (I believe Izumi Tsutsui wrote the code). The patches never made it back into the main NetBSD tree and have just been maintained by Izumi.
 
Now someone is trying to add support for it to DreamShell, an alternative gaming OS for the Dreamcast, so I found a copy of the patches for netbsd-5, tweaked them for current, built a test kernel posted it to their forum.

Annoyingly the email notification of subsequent comments didn't seem to work for me, so I missed the very quick response asking for a kernel with additional debug. Just seen it now, so just built & uploaded it now :)

Elsewhere in my world of retro-challenge I still don't have public IPs for wopr (VAX) and orac (Dreamcast) after the office move, so wopr is very lonely on an internal network and orac is at home (where at least I can poke at it).

Unfortunately my "at home" Dreamcast CD-R burning opportunities are somewhat limited:
  • My thinkpad DVD drive is more than a little unwell. It writes CDs that not even it can read, and had a tendency to just randomly eject at any point during the day. Hence its been replaced by an sata drive bay
  • Michelle has a macbook air, which is beautiful, but optical-drive-less
  • My gaming PC has a blueray drive, and none of the CD-Rs it generates work with the DC. I have a whole stack of coasters that prove adjusting speed, burn options, or switching NetBSD/Windows has no effect
  • I have a Mac-mini which has a drive, but only OSX installed. (thinks). I *could* port dc-burn-netbsd to OSX, or at least the "burn existing ISO" option...
Anyway...

I had a play with telling NetBSD to add unused video ram to the normal kernel memory. Works perfectly on gxemul, but real hardware was a little pickier, so that it going to take some more work (if its possible).

Nick Hudson posted me a Dreamcast serial cable he was now longer using, which enabled me to find out that getty and remote are *really* fussy about using a read-only /dev, so I'm probably going to have to switch my live CD to use the standard tmpfs /dev trick. That is annoying to the tune of around 300K of wasted memory, but not the end of the world.

My current "goal of randomness" is to get the live CD to show a prompt in an xterm. Thats readonly-root-on-iso9660, with misc tmpfs & union mounts, running multi-user in 16M without swap, with the X server and one xterm running.

I've tested it on root-on-nfs without swap, so I think it should be doable... just. I wonder if I should add an option to dc-burn-netbsd to create a root filesystem suitable for testing over NFS - to avoid a fresh coaster per test (and the feeling that I can hear the Dreamcast CD seek mechanism expiring before my ears).

As if the VAX & Dreamcast retro-madness isn't enough I've just collected a PowerBook Duo 270c from someone who ran out of space. 12M RAM, 240M disk, 68030 & a 9" 640x480 screen. Its so *tiny* and cute that I actively want to do something useful with it, unfortunately its only I/O consists of two serial ports and a dock connector.

Did I mention I also picked up a Duo Dock II with it, which has a vast selection of ports, floppy drive, space for an addition disk, an FPU chip, additional VRAM, and two nubus slots. It addresses the "tiny, cute but no ports" aspect of the Duo 270c in a big way. However...

Do not make Dock angry. Dock crush!
  1. It is the size and weight of a small bus. I keep having to check underneath for wheels
  2. The network port does not appear to be fitted - so the *one* thing I need most, it doesn't do
  3. It requires the laptop to be closed and inserted - so I now need to find a an external ADB mouse, keyboard, and a mac compatible monitor (or adaptor)

Don't worry, its Apple approved
There is a beautiful scene from the film Brazil, where Robert De Nero - playing a renegade special forces equipped air conditioning engineer - pulls out a tiny component and says "thats your problem". When asked if he can fix it he says "no, but I can bypass it with *this*" and proceeds to take from his bag what can only be described as a cross between a cybernetic cthulhu and the spinal column of a robot made from valves.

I look at the tiny Duo 270c and the across at the docking station from hell, and think... "Thank you Robert, that will do just fine".

Friday 27 July 2012

Nice NetBSD/dreamcast shell script, but can you do it under Windows?

I posted about the dc-burn-netbsd script on one of the Dreamcast forums, and someone asked if they could run the script under Windows.

I suspect pointing him at Microsoft's current *nix-on-Windows solution (Interix? Windows Services for Unix?  Subsystem for UNIX-based Applications? Grudgingly Provided but Subtly Maladapted Compatibility Subsystem for Customers Who Otherwise Would Use Some OS Begining with L?)... might have (quite rightly) have been met with a vigorous but somewhat impolite reply, so I decided to generate an image for him to burn directly.

I also took the opportunity to cleanup dc-burn-netbsd a little more:
  • split out 'extract sets' from 'setup extracted data for live usage' (now -l) - after all, why should I assume my live setup is the One True Way?
  • tweaked live setup further (do we *really* want to try to rebuilding the X11 fontcache? I think not), and /etc/motd was not reporting the kernel version (tsk, fer-shame)
  • making -l default to the right kernel and minimal sets. I have a deep seated antipathy towards typing many options on a command line. (If I'm building a live CD then it should be able to work out that it needs some sets and a non ramdisk kernel - but only as a default of course, if the user wants to explicitly select something apparently stupid... "here is rope")
  • Adjusting default behaviour to generate but not burn the image (added -b to burn). Ah, now I have a naming crisis as it no longer "does what it says on the tin" by default. Foo. I'll come back to that later
He also asked another question - to paraphrase it - "why?"

Currently the motivation for all this has been "because its there" - while NetBSD/dreamcast can run a whole bunch of standard unix software (see http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/dreamcast/5.1_2012Q1/All/ for example), not all of it necessarily makes sense on such a platform.

But again... "here is rope" :)



I also put together a README (I'll quote it below - mainly on the grounds I've already spent time writing it, got distracted watching the Olympics opening ceremony, and its a cheap way to pad out the rest of this entry)


netbsd-dreamcast-6_BETA2.iso.README

Sample NetBSD/dreamcast live image (Generated by "dc-burn-netbsd -l -n"):

Requirements
- Dreamcast or gxemul emulator (*)
- If using a real Dreamcast, blank CD-R & Dreamcast keyboard
- Optional: Dreamcast BBA (broadband adaptor) for network access

Usage:
- Download netbsd-dreamcast-6_BETA2.iso.7z and uncompress to get .iso image
- To burn to CD for booting on Dreamcast:
  - On Windows use http://code.google.com/p/bootdreams/downloads
  - On NetBSD no need to download this, use http://pkgsrc.se/sysutils/dc-tools
    and run 'dc-burn-netbsd -l' directly
- To boot in gxemul (*) run
  "gxemul -XEdreamcast -d co23965696:netbsd-dreamcast-6_BETA2.iso"
- Once booted enter "gdrom0" and return three times when prompted
- Login as "root"

This is still a work in progess, in particular future goals for future versions
include:
- Including installed binary packages such as some from
  http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/dreamcast/5.1_2012Q1/All/
- Switching to a shell with command line editing
- Support for Serial-SD and potentially Serial-Ethernet adaptors
- Better X support (currently insufficient memory to effectively run)

(*) Running under gxemul requires gxemul-0.6.0nb3 or later from pkgsrc, or the patch-src_devices_dev__dreamcast__gdrom.cc patch applied to gxemul-0.6.0

Interestingly X doesn't  start under gxemul - I'll need to test on real hardware before I dig further into that, but as my DC is in my office, and I'm currently in a Days Inn in Winchester (not Inglewood - that was a rather different experience), that will have to wait...

Thursday 26 July 2012

Virtual Dreamcast - Virtual (a)Live?

There has been something of a hiatus in my Retrochallenge 2012 postings.

Last Tuesday evening I visited London Hackspace and... made it home in the decidedly early hours of Wednesday morning, rather neatly occupying the time during which I would have tried to do something Retrochallenge related and then have written about it.

Then as ever, "stuff happened", but finally I have made time to get back to it!

"...Last episode on Retrochallenge2012, I was doing something with a Dreamcast, or a VAX - I really can't remember now, possibly even combining the two in a gratuitous and potentially amusing fashion, but I did remember that trying to create a Dreamcast live CD was quite an annoying process, requiring as it did burning a CD-R each time to test.

Enter gxemul, a very nice set of emulators which included one that could boot NetBSD/dreamcast, including a rather elderly NetBSD-3.0 live CD image. Unfortunately... I was unable to get a recent NetBSD CD image to mount - the kernel would load, I could view the start of the CD image partition which looked fine, but it would just refuse to mount the ISO9660 filesystem.

Sprinkling some debugging prints in both the kernel and gxemul (it is *so* much nicer to be able to debug from both sides), revealed the NetBSD kernel trying to access sectors somewhere in the multigigabyte range (a neat trick on a medium with a maxiumum of 900MB capacity).

The issue, it transpires, was gxemul only fakes up enough of a CD ToC (Table of Contents) to allow NetBSD to boot... specifically NetBSD 3.0. NetBSD 4.0 and later have stricter requirements. Actually the code is very similar and I have a horrible feeling that the fake ToC caused the NetBSD 3.0 gdrom code to fall off the start of an array and read some nulls. Nasty.

So I went searching for the CD ToC specifications, and eventually determined that I needed the Yellow and Red Book standards, written by Sony & Philips and apparently still to this day considered commercially privileged and not freely available. However... their contents match the freely available (and catchy named) Standard ECMA-130: Data Interchange on Read-only 120 mm Optical Data Disks (CD-ROM) which has insane level of detail on the ToC (among other things).

So I read the relevant section, rubbed my eyes a few times and read it again, then took out a pad & paper and thought about drawing diagrams... and then read it again. I considered the strange need for some engineers to produce overly complicated and involved solutions, and about technical writers who actively despised their potential readers.

I just have to share Figure 15 at this point. Looks nice and simple? Just split up 96 bits into sections? Only... bits 0..97 makes *98* bits, and the Control bits have two extra bits (for free). Don't even get me started on how each track has 4 bytes in the ToC data returned, but something like 10 bytes in the spec.


Then I experimented with building gxemul with some values that should have made sense... but apparently didn't. In the end I burned a NetBSD/dreamcast disk with a modified kernel which dumped the ToC values, then updated gxemul to fake a more convincing ToC...

Final result is an updated gxemul which will boot NetBSD/dreamcast 3, 4, 5 & 6 kernels, a NetBSD/dreamcast kernel option to dump the gdrom ToC, and a tweak to the gdrom parsing code to reject too-fake ToC values.

Once the above was reached it only took a few quick iterations to get dc-burn-netbsd building fully functional Dreamcast live CDs, including the ability to login to both the graphical and the serial consoles - literally multiple users on a virtual Dreamcast without any network interfaces or persistent storage, virtual or otherwise. And they say retrocomputing emulation has no practical use? Pffff...