Gzip with parallel compression support: pigz

The speed at which files are compressed with gzip is currently almost always determined by the speed of the CPU. However, standard unix gzip is single-threaded and only uses a single CPU (core).

However, the maintainer of the zlib library has released 'pigz' or 'pig-zee' whichs adds just that: support for parallel compression. This dramatically improves the speed at which a file can be gzipped.

In this example, a 3 GB compressable file is gzipped:

Gzip:

root@Core7i:~# time gzip pigz.bin

real 1m58.994s

user 1m56.480s

sys 0m1.820s

Pigz:

root@Core7i:~# time pigz pigz.bin

real 0m31.524s

user 2m54.890s

sys 0m2.900s

This simple and a bit unscientific example shows a 400% speed improvement. Since the Core i7 has four real cores, this shows that pigz scales nicely.

How to determine which process causes IO ?

There is a nifty little program called 'iotop'. Iotop is part of Debian or Ubuntu and can be installed with a simple apt-get.

Once you have determined with 'top' that the system is waiting on IO-access, It is nice to know

which process is responsibe for this IO. Therefore, you want a list of processes, just like top provides, but instead of CPU or RAM usage, it shows the IO a process is generating. This is exactly what 'iotop' provides.

This is ideal when troubleshooting performance problems caused by heavy IO.

http://guichaz.free.fr/iotop/

Making cowpatty recognize a four-way handshake

I was unable to get cowpatty working with a packet capture that actually

contains a four-way handshake of a WPA session.

I got it working like this:

First, download cowpatty 4.6 right here, within the source directory of cowpatty.

Extract cowpatty and apply this patch using these instructions

Then build couwpatty just with 'make' and 'make install'.

I created a test setup with a known password. However, this patched

version did not find the passphrase using a dictionary file.

I then used genpmk to create a precomputed hash database like this:

genpmk -f ./aircrack-ng-1.0/test/password.lst -d hashes -s default

Please note that I added the correct passphrase to this password list to

make sure that cowpatty works.

Finally, using the -d option on the hash file, cowpatty

managed to crack the PSK.

Monitor power usage with your UPS

If a system is connected to a UPS (Uninterruptible Power Supply), it is possible to determine how much power it consumes. For this purpose, I wrote a small script:

Host:~# ./ups.sh

---------------------------------

UPS model: Back-UPS RS 1200 LCD

APC model: Back-UPS RS 1200 LC

----------------------------------

Capacity: 720 Watt

Load: 18 Percent

Usage: 129 Watt

Time left: 33 Minutes

Status: ONLINE

----------------------------------

Host:~# ./ups.sh

----------------------------------

UPS model: Back-UPS RS 1200 LCD

APC model: Back-UPS RS 1200 LC

----------------------------------

Capacity: 720 Watt

Load: 19 Percent

Usage: 136 Watt

Time left: 22 Minutes

Status: ONBATT

----------------------------------

This script assumes that:

1. You are running a unix
2. You run apcupsd

The script can be downloaded here:

24 TB based on Norco RPC-4020 and Linux Software RAID

Just a quick link:

Some person build basically the same setup, including identical controller, providing 28 TB of storage: Take a look here

The main difference is that this person uses 1.5 TB disk, thus achieving more storage.

Linux Mac Mini - temperature monitoring with lm-sensors

This post is about getting temperature monitoring to work with a Mac Mini running Linux.

Using Debian Lenny, out of the box, lm-sensors is not working. No sensors can be found. This is how temperature monitoring and fan speed monitoring can be made to work:

modprobe applesmc

If you run "sensors-detect" after this, and do a:

modprobe coretemp

Then "sensors" will give you ouput like this:

Mini:/sys/devices/platform# sensors

applesmc-isa-0300

Adapter: ISA adapter

Master : 2151 RPM (min = 1500 RPM)

temp1: +83.2°C

temp2: +68.0°C

Some software to control the fan speed:

http://stargate.solsys.org/mod.php?mod=faq&op=extlist&topicid=27&expand=yes#118

Linux on Mac Mini - boot after power failure

When using a Mac Mini as a server or router, it is very nice if the machine automatically boots if a power failure has occurred.

User chirhoxi on the ubuntu forum found out how this can be achieved:

http://ubuntuforums.org/showthread.php?t=1209576

Basically you need one of these commands:

Original Mac Mini:

setpci -s 00:03.0 0xa4.b=0

Newer Mac Mini:

setpci -s 00:03.0 0x7b.b=19

These commands might not work with the latest Mac Minis but the thread discusses how to determine for yourself what the appropriate command is.

How to run Debian Linux on an Intel based Mac Mini

The Mac Mini is just a gorgeous device. It is beautiful, small, silent, powerfull yet energy efficient. When idle, it uses around 20 watts. I'm using one of the first Intel-based Minis with an Intel Core Duo chip, running at 1.6 Ghz.

I want to use this mini as an expensive router and download host. I could have used something embedded, such as one of those router boxes that costs about 70 euros, but no, I want to do some more with my router, such as downloading, etc. It is the only device in my house that is allowed to run 24/7 so it has to be a bit more powerful if I want more than just routing. I know that this mini was like 600 euros or something back in the days, and that is quite some money to spend on something that is now only a router. However, when I was still running Mac OS X on it, I didn't do much more with it than I will now, it will actually do more.

I am assuming that you want to run Linux exclusively on the Mac and that Mac OS X will be wiped off.

To get this puppy running Debian Linux (Lenny), you need to first boot the Mac with the (Snow) Leopard OS X boot CD and startup the diskutilily.

You need to create at least two partitions: one for the root file system and one for swap. The most important step is to select 'options' under the partition layout screen, and select Master Boot Record partitioning instead of the other 2 options. Do NOT use GUID or Apple Partition Map.

Now, boot your regular Debian Linux boot CD, I use the regular network installation CD. When you get to the partitioning screen, do NOT auto-partition the hard disk. Just reconfigure the existing partitions you just made using Diskutility. So the large partition will be configured as "/" and made bootable. The small partition must be configured as swap.

After the installation finishes, just install GRUB in the MBR and reboot. If all went alright, you will see a non-blinking folder on a gray background for a couple of seconds, after which Linux will boot. If you get a blinking gray folder with a question mark, something went wrong.

It seems that if configured properly, after the EFI boot mechanism fails to find a system folder on some Mac partition, the legacy BIOS emulation seems to kick in, and star to search for something to boot.

The Mini has only one network card, so another one is necessary to run it as a router. I bought some no brand USB2 to 100 MBIT NIC (Bus 005 Device 003: ID 9710:7830 MosChip Semiconductor MCS7830 Ethernet) which seems to run smoothly.

UPDATE: According to Campr, it is possible to replace the Wireless mini PCI-e card with a dual gigabit card.

I guess you will need to mod the Mini but it will allow true gigabit speeds on all interfaces.

The security risk of vendor-supplied default SSL certificates

Often, software comes supplied with some default SSL certificate, for testing purposes, such as those 'snake oil' certificates (they are called snake oil certificates for a reason). In practice, I often encounter usage of such certificates. People may seem to think that as long SSL is used, authentication and thus credentials are safe, but nothing could be further from the truth.

If you encounter a service that uses a default vendor-supplied SSL certificate, decryption of communication is trivial. Just obtain a copy of this vendor software and grab the private key. This private key can be loaded into Wireshark to decrypt any captured SSL traffic that has been encrypted with this certificate. Please read this link about decrypting SSL with Wireshark.

So it is important to always replace default SSL certificates with a freshly generated, no matter if it is self-signed or not.

Blu Ray is dead

HD-DVD may be dead, but Blu Ray is just as dead. The whole concept of optical media is dead. Honestly, who is still burning CD's or DVD's nowadays? (If you are, why for Christ sake? I can't think of a single good reason) And at 10 euros ($100) for a single Blu Ray disk, you must be totally bonkers to buy one of them recorders.

I mean, let's face it, CD's were really cool in an age where 650 MB was way more than the 40 or 80 MB hard drive in your computer. It made a difference. That was already less so with a DVD, with a capacity of 'only' 4 GB. However, in the early years of the DVD, you could backup your entire hard drive on two, maybe three disks, since a hard drive averaged around 4 to 10 GB at that time.

Then finally came Blu Ray and HD-DVD. A whopping 25 GB on a disk. No shit!. You mean, like I need no less than 40 Blu Ray disks (400 euros) and an eon of burning disks to backup one of my 1 TB hard drives that cost me like 70 euros?

What must they have been thinking when they developed Blu Ray? As a backup medium it is useless, but it was ofcourse intended as a carrier for movies, I know. But why should I go outside, through the cold and the rain to go to some shitty video store that only has last years block busters? Why bother with 40 Mbit downstream and the Internet at your disposal? Downloading a movie will take me as much time as going to the video store and selecting something so dreadful even the DVD player will refuse to play it. What else is there to choose from. The Internet provides us with the most rare and obscure but most beautifull movies you ever saw. And the more main stream movies can be obtained in full HD 1080p.

Current Internet connections are of such quality, that physical media such as Blu Ray disk are becomming irrelevant. A normal DVD is downloaded within 20 minutes at 4 MB/s. And when Internet connections will reach 100 Mbit or 12 MB/s, even a HD 1080p movie will be downloaded within the hour.

Storage is not a problem. If you can store 40 HD movies on a single 1 TB disk, then you will pay 1,5 euros for each movie. Beats any Blu Ray disk in price and time. If you even care about them that much, buy or build a NAS and store them on some redundant storage.

In time people will do with DVDs and CDs what most people already do with CDs: rip them to some format your computer understands (to transfer it to your MP3 player) and get rid of that CD that will become scratched and useless even if you don't touch it. No, you don't want to transcode your HD movies to some iPod or something, but you may want to stream it to your media player in the living room? A jukebox full of films, just as you have a jukebox full of music.

Give the CD, DVD and Blu Ray a little push, and let them fall into the grave. It is an outdated technology for a problem that existed 10 years ago. The Internet has made it irrelevant. Be done with it.