Louwrentius.com

This blog has been moved to louwrentius.com

2011-01-01T06:32:00.000-08:00

I decided to move away from blogger and host my own blog at another location. The new location for my blog is:

http://louwrentius.com

Visit my new blog for a motivation.

All comments have also been migrated.

LFS - Linux Firewall Script released

2010-12-27T17:15:00.000-08:00

I started a small new Google project for a new script I wrote called LFS. It stands for Linux Firewall Script.

I run a small Linux box as an internet router that doubles as a firewall. The firewall is configured using iptables. In my opinion, iptables is not the easiest tool to use and may have a steep learning curve for people new to it.

The goal of LFS is to provide an easier interface to iptables. It also adds some features that by default are not or difficult to setup using only iptables. The most important additional feature is the use of objects and groups. Object groups can be used to make a single rule affect multiple hosts, networks or services.

LFS uses a single configuration file which contains the firewall rules. Rules look like this:

nat "$INTERNAL_NETWORK" "$EXTERNAL_IP" "$NAT_INTERFACE"
port_forward "$EXTERNAL_IP" "$INTERNAL_HTTP_SERVER" "80/tcp" "8080/tcp"

Please visit the project page for some examples.

http://code.google.com/p/lfs/

Why filtering DHCP traffic is not always possible with iptables

2010-12-27T06:57:00.000-08:00

When configuring my new firewall using iptables, I noticed something very peculiar. Even if all input, forward and output traffic was dropped, DHCP traffic to and from my DHCP server was not blocked even if there were no rules permitting this traffic.

I even flushed all rules, put a drop all rule on all chains and only allowed SSH to the box. It did not matter. The DHCP server received the DHCP requests and happily answered back.

How on earth is this possible? In my opinion, a firewall should block all traffic no matter what.

But at least I found out the cause of this peculiar behaviour. The ISC DHCP daemon does not use the TCP/UDP/IP stack of the kernel. It uses RAW sockets. Raw sockets bypass the whole netfilter mechanism and thus the firewall.

So remember: applications using RAW sockets cannot be fire walled by default. Applications need root privileges to use RAW sockets, so RAW sockets thankfully cannot be used by arbitrary unprivileged users on a system, but never the less. Be aware of this issue.

Please understand that if a serious security vulnerability is found in the ISC DHCP daemon, you cannot protect your daemon with a local firewall on your system. Patching or disabling would then be the only solution.

Belkin Gigabit USB 2.0 adapter works perfectly with Linux

2010-12-08T12:53:00.000-08:00

My ISP upgraded my internet connection speed to a whopping 120 Mbit. I am using a mac mini as my internet router. As you may be aware, the mini has only one network interface, so I added a second interface using a USB to ethernet adapter. This adapter was limited to 100 Mbit, so to make full use of the 120 Mbit connection, I had to upgrade this adapter.

I took the gamble and bought the Belkin Gigabit USB 2.0 adapter. I could not figure out if it would work with Linux, but on the box it officially supports Mac OS X, which is always a good sign.

This adapter is recognized by Debian Linux without a hitch:

Mini:~# ethtool -i eth0

driver: asix

version: 14-Jun-2006

firmware-version: ASIX AX88178 USB 2.0 Ethernet

bus-info: usb-0000:00:1d.7-5

lsusb output:

Bus 005 Device 004: ID 050d:5055 Belkin Components F5D5055 Gigabit Network Adapter [AX88xxx]

I did not test the actuall performance of this adapter, but it at least goes beyond the 100 Mbit (it does 120 Mbit at least). I expect it to be limited at say max 300 Mbit, being constrained by the maximum speed of USB 2.0.

The minimum requirements for a secure system

2010-11-26T11:42:00.000-08:00

The most secure server system is a system that is not connected to a network and turned off. However, little work seems to be getting done this way. So we want to turn systems on and connect them to a network, or even (God forbid) the internet.

The thing is this. A system connected to a network without any running services is almost as as secure as a system that is turned off. They also share a common property: they are useless. A system starts to get useful if you start running services on them. And make these services accessible from the network for clients.

Services

Security on a technical level is all about securing those services. Every service that you enable is an opportunity for an attacker to compromise your system. If a service is not installed or running on your system, it cannot be used to compromise your server.

If a service is enabled and accessible through the network, it is logically of vital importance that you know:

what does this service do?
what can it be used for?
what steps needs to be taken to properly secure it?

If you know what a service does, you can understand the potential security risks. If you understand the product you are using, you can secure it properly. Security is all about understanding. If you don't understand what you are running, then it can't be secure.

Firewalls

So if you only run required services, why do you need to run a firewall? You don't. Yes that's right. Think about it. A firewall protects services that should not be accessible and allows access to services that should be accessible. If you just disable those services that should not be accessible from the outside, why use a local firewall? You don't want the Internet to access the SNMP-service on your system, you say? But then why not bind it only to the management interface instead of the production interface? You have a separate management network, right?

Of course, firewalls are a good thing. They are an ADDITIONAL line of defense. They mostly protect you against yourself. If you make a mistake and, by accident, enable some vulnerable service on a system, a properly configured firewall will prevent access to it and save your behind. That is the purpose of a firewall.

People often wrongly see the firewall as the first line of defense. If you do, you are wrong. The first line of defense is to secure your services.

The whole point is that there are holes in your firewalls. Those holes allow access to services. Those services may be necessary, like a web server, but nevertheless holes. You are exposing services to the Internet.

Web applications (or web-based back doors?)

We are now mostly running web-based applications on the services that we make accessible for the network or the internet. Those applications run on application servers. Yes, these application servers, like Apache Tomcat or IIS ASP.NET need to be secured, but nowadays, they are almost secure by default.

All security depends on the level of security of the application you are running on your application server. Is your application written well, with security principles in mind? Does it protect against SQL-injection or cross-site scripting? Are sessions predictable? Can a user access data of another user?

Firewalls don't protect against vulnerabilities in your web applications. You need to do it right at the core level: the application itself. Just like how you harden a system. You must run secure code.

And be aware that if you run third-party code, watch out for security news. There have been many worms exploiting vulnerable commodity software such as phpBB, Wordpress or similar products.

This is the really hard part. Deploying secure software and keeping it secure during the development life cycle.

Patches
The last fundamental principle of keeping systems secure is keeping up with security patches. Many security vulnerabilities are often only exploitable under specific conditions and may not be that important. But the most important thing is to be aware of vulnerabilities and available patches. Then you can decide for yourself how to act.

There is always a risk that a security patch breaks functionality. But that's not a real problem, because you have this test environment so you can check first, right?

Keep up with security patches and non-security patches. If you first have to install 100+ patches to be able to install the latest high-risk security patch, something might break. So then it's choosing between staying vulnerable or going off-line until you have fixed everything.

Conclusion

So what are the most basic ingredients for secure systems?

only run required services
harden those required services
deploy a firewall as an additional defense layer
deploy secure application code
keep up-to-date with security patches
Audit and review your systems and application code on a regular basis.

With this small number of steps, you will be able to protect against a lot of security threats. I don't say this is everything that is necessary. But it is a good foundation to build on. You still have to identify risks that may apply to your particular situation. These risks may require you to take (additional) measures not discussed here.

Linux: using disk labels to counter storage device name changes

2010-11-21T23:08:00.000-08:00

My router decided to change the device name for some USB storage devices. So /dev/sdc was swapped for /dev/sdd and vice versa. The result was some file system corruption on /dev/sdc, because it was used on a remote system through iSCSI, using a different file system from /dev/sdd.

With regular internal disks, attached with PATA, SATA or SAS, the chances are very small that such an event will occur, but it is possible, especially if you start adding/subtracting disks. With USB devices the risk is substantially bigger.

To prevent your system from mixing up drives because there device names change, use file system labels. All information that follows have been stolen from this location. Since this blog is also my personal notepad, the relevant bits are reproduced here.

There are three steps involved, the third being optional:

add a label to the file system
add the label to /etc/fstab
update grub boot manager (optional)

1. Add a label to the file system

Setting a label when the file system is created

mkfs.ext3 -L ROOT /dev/sda1
mkfs.xfs -L BIGRAID /dev/sde

Set label for existing file system

EXT3:
e2label /dev/sda1 PRIMARY_ROOT
e2label /dev/sda1

XFS:
xfs_admin -L DATA1 /dev/sdf
xfs_admin /dev/sdf

Set label for swap partition

mkswap -L SWAP0 /dev/sdb5

2. add the label to fstab

Example of contents of fstab:

LABEL=ROOT / ext3 defaults 1 1
LABEL=BOOT /boot ext3 defaults 1 2
LABEL=SWAP swap swap defaults 0 0
LABEL=HOME /home ext3 nosuid,auto 1 2

3. Update the grub boot manager

title server
root (hd0,0)
kernel (hd0,0)/vmlinuz ro root=LABEL=SERVER_ROOT0 rhgb quiet
initrd (hd0,0)/initrd.img

Secure programming: how to implement user account management

2010-11-18T13:40:00.000-08:00

Most web applications work like this:

The application uses a single database account to perform all actions. Users are just some records in a table. Account privileges and roles are part of this table, or separate tables.

This implies that all security must be designed and build by the application developer. I think this is entirely wrong. There is a big risk:

In such applications, SQL-injection will allow full control of the entire database.

This is something that is often overlooked. And the solution is simple. The application should not use a general account with full privileges. The application should use the database account of the user accessing the application. All actions performed by this user are thus limited by the privileges of this database account. The impact of SQL-injection would be significantly reduced.

The public part of a website is still using an application account, but the privileges of this account can be significantly reduced. To obtain elevated privileges, a user must first authenticate against the application and thus the database.

Please understand another benefit: it is not required to store username/password combinations of privileged accounts on the application server. The configuration file will only contain the credentials of the unprivileged account. An attacker compromising the application server with limited privileges, won't have access to the database with elevated privileges.

I understand that this solution requires a bit more work to setup at the start, but once implemented, it reduces complexity and improves security so much.

Of course, the security of your data is as good as the hardening of your database server. But that's another story.

Do not buy a hardware RAID controller for home use

2010-11-17T13:00:00.001-08:00

Hardware RAID controllers are considered 'the best' solution for high performance and high availability. However, this is not entirely true. Using a hardware RAID controller might even endanger your precious data.

For enterprise environments, where performance is critical, it is more important that the arrays keeps on delivering data at a high speed. Professional RAID controllers use TLER with TLER-enabled disks to limit the time spend on recovering bad sectors. If a disk encounters a bad sector, there is no time to pause and try to fix it. The disk is just dropped out of the RAID array after just a couple of seconds. At that moment, the array still performes relatively well, but there is no redundancy. If another disk fails (another bad sector?) the array is lost, with all its data.

More people are building NAS boxes for centralized storage of data, for private home use. Since disks are cheap, it is possible to create lots of storage capacity for little money. Creating backups of terabytes of data is however not cheap. Or you have to create two NAS boxes. But that is very expensive and not worth the effort.

People seem to spend lots of money on expensive enterprise level hardware RAID cards, not understanding that the whole TLER-mechanism causes an increased risk for their data. In enterprise environments, budgets are relatively big, and data is always backed up. They can afford to take the risk of losing a RAID array due to these backups. But consumers often don't have the money to spend on creating backups of terabytes of data. They just go for RAID 5 or RAID 6 and hope for the best.

For consumers, if the RAID array goes, all data is lost.

So consumers should choose a RAID solution that will do its best to recover from hardware failure. Performance is not so much an issue. Reliability is. So consumers do want disks to spend 'ages' on recovering bad sectors if this means that the RAID array will survive.

Linux software RAID and ZFS do not use TLER and therefore are a safer choice for your data then regular hardware RAID controllers. You may still use such controllers (but please test them properly) but only to provide SATA ports with individual disks, the RAID part should be handled by Linux.

So in my opinion, hardware RAID controllers are more expensive, require more expensive (enterprise) disks and are less safe for your data.

Linux network interface bonding / trunking or how to get beyond 1 Gb/s

2010-11-11T14:40:00.000-08:00

This article discusses Linux bonding and how to achieve 2 Gb/s transfer speeds with a single TCP/UDP connection.

A gigabit network card provides about 110 MB/s (megabytes) of bandwidth. If you want to go faster, the options are:

buy infiniband stuff: I have no experience with it, may be smart thing to do but seems expensive.
buy 10Gigabit network cards: very very expensive compared to other solutions.
strap multiple network interfaces together to get 2 Gb/s or more with more cards.

This article is discussing the third option. Teaming or bonding two network cards to a single virtual card that provides twice the bandwidth will provide you with that extra performance that you where looking for.

But the 64000 dollar question is:

How to obtain 2 Gb/s with a single transfer? Thus with a single TCP connection?

The trick is to use Linux network bonding.

Most bonding options only provide an accumulated performance of 2 Gb/s, by balancing different network connections over different interfaces. Individual transfers will never reach beyond 1 Gbit/s but it is possible to have two 1 Gb/s transfers going on at the same time.

That is not what I was looking for. I want to copy a file using NFS and just get more than just 120 MB/s.

The only bonding mode that supports single TCP or UDP connections to go beyond 1 Gb/s is mode 0: Round Robin. This bonding mode is kinda like RAID 0 over two or more network interfaces.

However, you cannot use Round Robin with a standard switch. You need an advanced switch that is capable of creating "trunks". A trunk is a virtual network interface, that consists of individual ports that are grouped together". So you cannot use Round Robin mode with an average unmanaged switch. The only other option is to use direct cables between two hosts, although I didn't tested this.

Now the results: I was able to obtain a transferspeed (read) of 155 MB/s with a file copy using NFS. Normal transfers capped at 109 MB/s. To be honest: I had hoped to achieve way more, like 180MB/s. However, the actual transfer speeds that will be obtained will depend on the hardware used. I recommend using Intel or Broadcom hardware for this purpose.

Also, I was not able to obtain write speed that surpasses the 1 Gb/s. Since I used a fast RAID array to write the data to, the underlying storage subsystem was not the bottleneck.

So the bottom line is that it is possible to get more than 1 Gb/s but the performance gain is not as high as you may want to.

Configuration:

Client:

modprobe bonding mode=0

ifconfig bond0 up

ifenslave bond0 eth0 eth1

ifconfig bond0 10.0.0.1 netmask 255.255.255.0

Server:

modprobe bonding mode=4 lacp_rate=0 xmit_hash_policy=layer3+4

ifconfig bond0 up

ifenslave bond0 eth0 eth1

ifconfig bond0 10.0.0.2 netmask 255.255.255.0

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation

Transmit Hash Policy: layer3+4 (1)

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

802.3ad info

LACP rate: slow

Active Aggregator Info:

Aggregator ID: 2

Number of ports: 2

Actor Key: 9

Partner Key: 26

Partner Mac Address: 00:de:ad:be:ef:90

Slave Interface: eth0

MII Status: up

Link Failure Count: 0

Permanent HW addr: 00:co:ff:ee:aa:00

Aggregator ID: 2

Slave Interface: eth1

MII Status: up

Link Failure Count: 0

Permanent HW addr: 00:de:ca:fe:b1:7d

Aggregator ID: 2

The iPhone, iPad and iOS are powering a revolution

2010-11-05T18:12:00.000-07:00

Most people just don't understand computers. Are these people dumb? Some may be dumb, but the people who make them are maybe even dumber. Because they can't seem to figure out how to create a computer that the majority of people understand.

When the original macintosh arrived at the stage back in the eighties, computers became a bit more human-friendly, but it was limited to the constraints of the then available hardware. It put away the text-based interface and introduced the graphic interface. It used the desktop metaphor to create this graphic environment. But this metaphor has had its day.

Many people don't understand the desktop metaphor since they don't have a desktop and have never used one. Also, it is a metaphor, it's to translate the computer environment to something humans understand. But what if they don't understand the metaphor? For example, many people just don't 'get' the Windows Explorer or the Mac OS X finder. The desktop metaphor does not seem to fit in how people think.

Every time you see a person enter a URL like www.youtube.com in the google search field, you will realize that we still have a long way to go.

Most people did not seem to realize back then that the release of the iPhone wasn't that important, but the release of iOS. The iPhone was the first smartphone (a word most people are not familiar with) that did away with a stylus or hardware keyboard. It uses what is closest to us: our fingers. A totally new user interface, one that is very natural and close to us, is now available.

Using touch as input required a total redesign of the entire user interface. All other interfaces were designed around hardware keyboard and mouse devices. Fingers are big, and are obstructing the view. But it allows for a more direct interaction with a device. And now all new smart phones sport a touch interface.

Rumors of an Apple tabled existed for long, but it was very clear when the iPhone was released that if Apple would release a tablet, it would run this new iOS operating system.

When the iPad was released, it became an instant hit. As of today, there is no device on the market that can be truly called a competitor. But why is this so? The ground work has been done by the iPhone. Most people with an iPhone will notice that aside from some performance issues in the past, the device just always worked. It was instantly available to sent an email, look something up on wikipedia or find the nearest Starbucks. An iPhone just always works. No boot. Very reliable. And an interface that makes you happy.

Why does iOS make people happy? Because it provides a user interface that is human. People understand it instinctively. Any person of any age or background will be able to use an iOS device within minutes. The interface doesn't make you look like you are dumb because you just don't understand how it works. It not only works, it is easy to use and you are not afraid to break anything.

The iPhone and the iPad are learning a lot of people not to fear computers.

The iOS does away with the old desktop metaphor, but so does Symbian or similar interfaces. It is the combination with touch and the well thought out interface that sets it apart from other mobile operating systems. Even when the iOS platform did not have native applications, people still bought it and not only because Apple released a new shiny toy.

However, the app store on iOS has created a very special and important environment. People can finally install and remove applications in an extremely simple way. They don't need to be scared that some program will crash your computer either while installing it, using it, or removing it. The whole iOS ecosystem creates an environment wherein people don't need any help any longer from other people. They are finally in control. They don't need to be afraid of their computer.

This trend will affect the old-school user interfaces such as Mac OS X. How it will turn out is anybodies guess. But there is at least a small trend to 'eradicate' the finder as much as possible. iPhoto stores your photos. iTunes stores your music. If you want to include a photo or song within an application, you pick the photo or song in question from a miniature iPhoto or iTunes interface. There is no finder anymore. The finder is disappearing from the workflow. And why not? If programs are written well, why bother with it? The finder should be abstracted away, as is the case on iOS, where you don't have a finder.

Another thing is multitasking, you know, that stuf we like to do, but cant. We can only do one thing at a time. What we do want is fast task switching, not multitasking. Sure, some programs must be running in de background, to continue to operate, such as a chat program, but that is not the point. Most people are just going crazy if you show how multitasking works, with different windows. Again, iOS shows how 'multitasking' should be implemented. It is implemented as fast application switching, allowing these applications to register services that must continue to run, while the application itself freezes when the user switches to another application. People tend to use one application at a time and especially on mobile devices, every single bit of screen real estate counts, so they are always running full screen. This full screen notion will also be incorporated in the next Mac OS X release, Lion. People switch, but do one thing at a time.

Computer nerds tend to feel superior to people who don't have much skill using a computer. This feeling of superiority is totally misplaced. They should be really humble. because up until the advent of iOS, nobody was able to create a human friendly computer interface. It is not the lack of understanding on the side of computer users, it is the lack of understanding on the part of the computer nerds on how normal humans think and act.

Simple, human friendly computer interfaces will liberate humanity from those pesky computer nerds. And that will cause a bit less suffering in the world I hope.

Apple is killing off the optical drive just like the floppy disk

2010-10-23T04:05:00.000-07:00

With the release of the new MacBook Air we are one step closer to killing off the cd-rom and the dvd. As with the previous MacBook Air, this device has no optical drive. And that is a good thing. People do not need an optical drive. You have the network and you have USB disks. They are faster, more reliable and have more capacity.

I may expect that in the upcoming years this trend may continue with the other laptops. Just as Apple killed the floppy disk, it is killing the optical drive, one step at a time. I'd rather have a smaller and lighter laptop or more disk or battery capacity, than an optical drive. So I hope this is a trend that will continue and all other manufacturers will follow.

Linux Software RAID benchmarking script

2010-09-29T13:58:00.000-07:00

Just a small post.

To benchmark your Linux software RAID array as setup with MDADM, please use my new benchmark script. I used this script to create these results.

You may need to configure some values within the header of this file to make it fit your enviroment.

DEVICES="/dev/sd[a-f]"
NO_OF_DEVICES=6
ARRAY=/dev/md5
CHUNKS="4 8 16 32 64 128 256 512 1024"
MOUNT=/storage
LOG=/var/log/raid-test.log
LOGDEBUG=/var/log/raid-test-debug.log
LEVEL="0 5 6 10"
TESTFILE=$MOUNT/test.bin
TESTFILESIZE=10000 (IN MB, thus this is 10 GB)
TRIES=5 (how many times to run a benchmark.)

By default, the script wil format the array using XFS, feel free to format it with another filesystem such as EXT4 or EXT3 or whatever you want to test.

Zabbix Security: client-server communication seems insecure

2010-09-27T15:35:00.000-07:00

Zabbix is a populair tool for monitoring servers, services and network equipment. For monitoring hosts, Zabbix provides an agent that can be installed on the hosts that must be monitored.

Based on the supplied documentation and some remarks on the internets, the 'security' of Zabbix agents seems to rely on an IP-filter. It only accepts traffic from a specific IP-address. However, the protocol that is used between the Zabbix server and agents is unencrypted and does not seem to employ any additional authentication.

With a man-in-the-middle attack, pretending to be the Zabbix server, you would be able to compromise all servers running Zabbix. If remote commands are enabled on these hosts, the damage that could be done may be something you don't want to think about. Or maybe you do. Although it is true that for such an attack to be possible, as an attacker you need access to a system within the same network (VLAN) as the server, but none the less, it is just not secure.

Personally I don't think that Zabbix is suitable for high-security environments, due to the lack of encryption of sensitive data and the weak authentication mechanism.

Zabbix should employ at least SSL as a means for encrypted transport and use a password or shared secret for authentication. Even better would be the use of client-side certificates such as implemented by the system management tool Puppet.

[update]
Please note that Nagios agents also seem to work this way, but I have no experience with Nagios so I can't say for sure.

And Nagios is widely deployed in the enterprise...

Debian Lenny and Dell R410 network card not supported

2010-08-20T12:53:00.000-07:00

For those who are running Debian Lenny and want to order the new Dell R410 server, beware!

There is no safe solution to get Debian Lenny working with the on-board Broadcom network cards. A fairly recent kernel is required. Basically, you will have to install back-ported kernels, more recent modules and thus must violate the reasons why you were running Debian Lenny in the first place.

There is one solution, although it may not be an option: run VMware on the hardware and run Debian Lenny in a virtual machine. I think that in most cases, this will be sufficient for many cases and it has all the benefits of virtualisation and the stability of Debian Lenny.

It is unfortunate that Broadcom or Dell do not support Debian.

If you do have an easy and quite safe solution to get the Broadcom network cards working with Debian Lenny, please drop a note.

Beware though: the H200 and H700 RAID controllers are also not supported by Debian Lenny.

Solaris is an obsolete platform

2010-08-14T15:55:00.000-07:00

Assuming that the rumor is true and OpenSolaris will be slain by Oracle, we must conclude that the Solaris operating system is obsolete. Solaris can be considered legacy. Sun was a hardware shop and to sell their hardware, they needed a great operating system.

Sun had a great operating system. And the Solaris platform was popular for a long time. And I think that was for the right reasons, at that time. If you or your company is still running on a Solaris platform, it may be time to rethink this strategy.

I do not understand why Oracle bought Sun. Oracle sells software. Sun sells hardware. Sun had some great products, like Java, so I can see some reasons. In the past, Solaris and Oracle had a tight relationship. But the only thing Oracle may be doing right now is a vendor lock-in strategy, where you are totally dependent on both hardware and software from Oracle.

But people don't seem to buy this, literally. People do want to continue to run Solaris, because thats the platform the've invested in. But they don't want to pay for those exotic Solaris Sparc systems, often way more expensive than commodity x86 hardware.

Oracle invested bilions in Sun assets. How are they going to make money of it? Squeeze out existing Sun Solaris customers who are depending on their platform?

If you are setting up a new business or if you think you can pull this off: stay away from this legacy platform. Migrate away from Solaris. Use an open platform that does not lock you in.

And this is also a very interesting read.

The future of ZFS now that OpenSolaris is dead

2010-08-14T03:51:00.000-07:00

With the probable loss of OpenSolaris, there may be another, maybe more devastating loss.

The very popular and very advanced Zetabyte File System (ZFS)

The only open source platform that actively supports ZFS is FreeBSD. And they just 'copied' the code from OpenSolaris. Are they able to maintain and further develop ZFS on their own? I don't think the community can handle a thing like that. Development on ZFS will severely be hampered and will not continue in the pace it did.

It is also clear that Oracle doesn't give a shit about open source or open operating systems. That is ok with me, everyone is entitled to their own opinion. But keep this in mind when you decide to use any Oracle product whatsoever.

It's not that I'm suggesting that you should not buy Oracle stuff. I have no grudge against Oracle in any way, it is just an objective observation, just be aware of this issue.

From the perspective of Oracle: what is their benefit regarding OpenSolaris? I understand their decision, but its sad nevertheless. And I'm really scared for the future of ZFS.

RAID 5 vs. RAID 6 or do you care about your data?

2010-08-13T08:06:00.000-07:00

Storage is cheap. Lots of storage with 10+ hard drives is still cheap. Running 10 drives increases the risk of a drive failure tenfold. So often RAID 5 is used to keep your data up and running if one single disks fails.

But disks are so cheap and storage arrays are getting so vast that RAID 5 does not cut it anymore. With larger arrays, the risk of a second drive failure while your failed array is in a degraded state (a drive already failed and the array is rebuilding or waiting for a replacement), is serious.

RAID 6 uses two parity disks, so you loose two disks of capacity, but the rewards in terms of availability are very large. Especially regarding larger arrays.

I found a blog posting that showed the results on a big simulation run on the reliability of various RAID setups. One picture of this post is important and it is shown below. This picture shows the risk of the entire RAID array failing before 3 years.

From this picture, the difference between RAID 5 and RAID 6 regarding reliability (availability) is astounding. There is a strong relation with the size of the array (number of drives) and the increased risk that more than one drive fails, thus destroying the array. Notice the strong contrast with RAID 6.

Even with a small RAID 5 array of 6 disks, there is already a 1 : 10 chance that the array will fail within 3 years. Even with 60+ drives, a RAID 6 array never comes close to a risk like that.

Creating larger RAID 5 arrays beyond 8 to 10 disks means there is a 1 : 8 to 1 : 5 chance that you will have to recreate the array and restore the contents from backup (which you have of course).

I have a 20 disk RAID 6 running at home. Even with 20 disks, the risk that the entire array fails due to failure of more than 2 disks is very small. It is more likely that I lose my data due to failure of a RAID controller, motherboard or PSU than dying drives.

There are more graphs that are worth viewing, so take a look at this excelent blog post.

A Moment Of Silence: OpenSolaris Is Dead

2010-08-04T08:39:00.000-07:00

The last release of OpenSolaris dates back to July 2009. The next release was scheduled for March 2010, but Oracle did not release anything. It is dead silent around OpenSolaris.

On 12 July 2010, the OpenSolaris Government Board sent out an ultimatum to Oracle: "please communicate with us or we will resign and hand over the little power we have back to Oracle".

These are all signs that OpenSolaris has no real future. And think about it: what is Oracle's interest in OpenSolaris? Maintaining an operating system and develop new features is extremely expensive. What is their benefit? I can't see any. Sun was a hardware manufacturer and promoting OpenSolaris made sure that people got experience with the operating system that powers their hardware.

Unless your environment already consists of Solaris-based systems, as of 2010, there is no reason to use Solaris any more. OpenSolaris was, until recent, the only operating system that supported ZFS natively, so the only choice if you really wanted to use ZFS. Now FreeBSD has native support for ZFS too.

For the people who are fanatical about ZFS this is great, because it would be a shame if ZFS could only be used in combination with a dead operating system that supports less hardware than Mac OS X.

But I see no future in OpenSolaris and you should think twice about running it, either as a production platform or at home. Use a platform that has a large user-base and a big community.

I think that if you are running OpenSolaris, you must start thinking about migrating to FreeBSD or Linux, however painful that may be.

Compiling Handbrake CLI on Debian Lenny

2010-08-03T12:31:00.000-07:00

In this post I will show you how to compile Handbrake for Debian Lenny. Please note that although the Handbrake GUI version does compile on Lenny, it crashes with a segmentation fault like this:

Gtk: gtk_widget_size_allocate(): attempt to allocate widget with width -5 and height 17

(ghb:1053): GStreamer-CRITICAL **: gst_element_set_state: assertion `GST_IS_ELEMENT (element)' failed

(ghb:1053): GLib-GObject-CRITICAL **: g_object_get: assertion `G_IS_OBJECT (object)' failed

Segmentation fault

So this post only describes how to compile the command-line version of Handbrake: HandBrakeCLI.

Issue the following apt-get commando to install all required libraries and software:

apt-get install subversion yasm build-essential autoconf libtool zlib1g-dev libbz2-dev intltool libglib2.0-dev libpthread-stubs0-dev

Download the source code at http://sourceforge.net/projects/handbrake/files/
Extract the source code and cd into the new handbrake directory.
Compile handbrake like this:

./configure --disable-gtk --launch --force --launch-jobs=2

The --launch-jobs parameter determines how many parallel threads are used for compiling Handbrake, based on the number of CPU cores of your system. If you have a quad-core CPU you should set this value to 4.

The resulting binary is called HandBrakeCLI and can be found in the ./build directory. Issue a 'make install' to install this binary onto your system.

Debian Lenny and Dell S300 H200 and H700 RAID controllers

2010-07-10T14:50:00.000-07:00

update november 2010: it is reported in the comments that the next release will support these controllers. However, which controllers are supported is not clear. Also, we may have to wait for quite some time before squeeze will be released.

Just a quick note: it seems that the new Dell RAID controllers are not supported by the current stable version of Debian: Lenny. The S300, H200 and H700 controllers as supplied with the R310, R410 and 'higher' systems, are thus not a great choice if you want to keep things 'stock'. You may have to run backported kernels and installer images and I couldn't figure out if these controllers actually work with these latest images.

The Perc 6/i controller is supported by Debian. It seems that it can only be supplied with the R410 and higher systems. If you have any additional information, like actual experience with these controllers, please report this.

I installed VMware ESX 4.1 and installed DEbian Lenny om top of that without problems.

Please note that the network cards of the R410 are still not supported and will cause a headache!

Lustre and the risk of Serious Data Loss

2010-07-03T15:53:00.000-07:00

Personally I have a weakness for big-ass storage. Say 'petabyte' and I'm interested. So I was thinking about how you would setup a large, scalable storage infrastructure. How should such a thing work?

Very simple: you should be able just to add hosts with some bad-ass huge RAID arrays attached to them. Maybe even not that huge, say 8 TB RAID 6 arrays or maybe bigger. You use these systems as building blocks to create a single and very large storage space. And then there is one additional requirement: as the number of these building blocks increase, you must be able to loose some and not loose data or availability. You should be able to continue operations without one or two of those storage building blocks before you would loose data and/or availability. Like RAID 5 or 6 but then over server systems instead of hard drives.

The hard part is in connecting all this separate storage to one virtual environment. A solution to this problem is Lustre.

Lustre is a network clustering filesystem. What does that mean? You can use Lustre to create a scalable storage platform. A single filesystem that can grow to multiple Petabytes. Lustre is deployed within production environments at large scale sites involving some of the fastest and largest computer clusters. Luster is thus something to take seriously.

Lustre stores all metadata about files on a separate MetaDataServer (MDS). Al actual file data is stored on Object Storage Targets (OSTs). These are just machines with one or more big RAID arrays (or simple disks) attached to them. The OSTs are not directly accessible by clients, but through an Object Storage Server (OSS). The data stored within a file can be striped over multiple OSTs for performance reasons. A sort of network RAID 0.

Lustre does not only allow scaling up to Petabytes of storage, but allows also a parallel file transfer performance in excess of 100 GB/s. How you like them apples? That is just wicked sick.

Just take a look at this diagram about how Lustre operates:

I'm not going into the details about Lustre. I want to discuss a shortcoming that may pose a serious risk of data loss: if you loose a single OST with any attached storage, you will lose all data stored on that OST.

Lustre cannot cope with the loss of a single OST! Even if you buy fully redundant hardware, with double RAID controllers, ECC memory, double PSU, etc, even then, if the motherboard gets fried, you will loose data. Surely not everything, but let's say 'just' 8 TB maybe?

I guess the risk is assumed to be low, because of the wide scale deployment of Lustre. Deployed by people who actually use it and have way more experience and knowledge than me about this whole stuff. So maybe I'm pointing out risks that are just very small. But I have seen server systems fail this bad as described. I don't think the risk, especially at this scale, is not that small.

I am certainly not the first to point out this risk.

The solution for Lustre to became truly awesome is to implement some kind of network based RAID 6 striping so you could loose one or even two OSTs and not have any impact on availability except maybe for performance. But it doesn't (yet).

This implies that you have to create your OSTs super-reliable, which would be very expensive (does not scale). Or have some very high-capacity backup solution, which would be able to restore some data. But you would have downtime.

So my question to you is: is there an actual scalable filesystem as Lustre that actually is capable of withstanding the failure of a single storage building block? If you have something to point out, please do.

BTW: please note that the loss of an OSS can be overcome because another OSS can take over the OSTs of a failed OSS.

Secure caching DNS server on Linux with DJBDNS

2010-06-12T09:37:00.000-07:00

The most commonly used DNS server software is ISC BIND, the "Berkeley Internet Name Daemon". However, this software has a bad security track record and is in my opinion a pain to configure.

Mr. D.J. Bernstein developed "djbdns", which comes with a guarantee: if anyone finds a security vulnerability within djbdns, you will get one thousand dollars. This price has been claimed once. But djbdns has a far better track record than BIND.

Well, attaching your own name to your DNS implementation and tying a price to it if someone finds a vulnerability in it, does show some confidence. But there is more to it. D.J. Bernstein already pointed out some important security risks regarding DNS and made djbdns immune against them, even before it became a serious world-wide security issue. However, djbdns is to this day vulnerable to a variant of this type of attack and the dbndns package is as of 2010 still not patched. Although the risk is small, you must be aware of this. I still think that djbdns is less of a security risk, especially regarding buffer overflows, but it is up to you to decide which risk you want to take.

The nice thing about djbdns is that it consists of several separate programs, that each perform a dedicated task. This is in stark contrast with BIND, which is one single program that performs all DNS functionality. One can argue that djbdns is far more simpler and easy to use.

So this post is about setting up djbdns on a Debian Linux host as a forwarding server, thus a 'DNS cache'. This is often used to speed up DNS queries. Clients do not have to connect to the DNS server of your ISP but can use your local DNS server. This server will also cache the results of queries, so it will reduce the number of DNS queries that will be sent out to your ISP DNS server or the Internet.

Debian Lenny has a patched version of djbdns in its repository. The applied patch adds IPV6 support to djbdns. This is how you install it:

  apt-get install dbndns

The dbndns package is actually a fork of the original djbdns software. Now the program we need to configure is called 'dnscache', which only does one thing: performing recursive DNS queries. This is exactly what we want.

To keep things secure, the djbdns software must not be run with superuser (root) privileges, so two accounts must be made: one for the service, and one for logging.

  groupadd dnscache
  useradd -g dnscache dnscache
  useradd -g dnscache dnscachelog

The next step is to configure the dnscache software like this:

  dnscache-conf dnscache dnscachelog /etc/dnscache 192.168.0.10

The first two options tell dnscache which system user accounts to use for this service. The /etc/dnscache directory stores the dnscache configuration. The last option specifies which IP address to listen on. If you don't specify an IP address, localhost (127.0.0.1) is used. If you want to run a forwarding DNS server for your local network, you need to make dnscache listen on the IP address on your local network, as in the example.

Djbdns relies on daemontools and in order to be started by daemontools we need to perform one last step:

  ln -s /etc/dnscace /etc/service/

Within a couple of seconds, the dnscache software will be started by the daemontools software. You can check it out like this:

  svstat /etc/service/dnscache

A positive result will look like this:

  /etc/service/dnscache: up (pid 6560) 159 seconds

However, the cache cannot be used just yet. Dnscache is governed by some text-based configuration files in the /etc/dnscache directory. For example, the ./env/IP file contains the IP address that we configured previously on which the service will listen.

By default, only localhost will be able to access the dnscache. To allow access to all clients on the local network you have to create a file with the name of the network in ./root/ip/. If your network is 192.168.0.0/24 (thus 254 hosts), create a file named 192.168.0:

  Mini:/etc/dnscache/root/ip# pwd
  /etc/dnscache/root/ip
  Mini:/etc/dnscache/root/ip# ls
  192.168.0

Now clients will be able to use the dnscache. Now you are running a simple forwarding DNS server and it probably took you under ten minutes to configure it. Although djbdns is not very well maintained in Debian Lenny, there is currently not a really good alternative for BIND. PowerDNS is not very secure  (buffer overflows) and djbdns / dbndns has in more than 10 years never been affected by this type of vulnerability.

Linux RAID level and chunk size: the benchmarks

2010-05-23T12:11:00.001-07:00

Introduction

When configuring a Linux RAID array, the chunk size needs to get chosen. But what is the chunk size?

When you write data to a RAID array that implements striping (level 0, 5, 6, 10 and so on), the chunk of data sent to the array is broken down in to pieces, each part written to a single drive in the array. This is how striping improves performance. The data is written in parallel to the drive.

The chunk size determines how large such a piece will be for a single drive. For example: if you choose a chunk size of 64 KB, a 256 KB file will use four chunks. Assuming that you have setup a 4 drive RAID 0 array, the four chunks are each written to a separate drive, exactly what we want.

This also makes clear that when choosing the wrong chunk size, performance may suffer. If the chunk size would be 256 KB, the file would be written to a single drive, thus the RAID striping wouldn't provide any benefit, unless manny of such files would be written to the array, in which case the different drives would handle different files.

In this article, I will provide some benchmarks that focus on sequential read and write performance. Thus, these benchmarks won't be of much importance if the array must sustain a random IO workload and needs high random iops.

Test setup

All benchmarks are performed with a consumer grade system consisting of these parts:

Processor: AMD Athlon X2 BE-2300, running at 1.9 GHz.

RAM: 2 GB

Disks: SAMSUNG HD501LJ (500GB, 7200 RPM)

SATA controller: Highpoint RocketRaid 2320 (non-raid mode)

Tests are performed with an array of 4 and an array of 6 drives.

All drives are attached to the Highpoint controller. The controller is not used for RAID, only to supply sufficient SATA ports. Linux software RAID with mdadm is used.
A single drive provides a read speed of 85 MB/s and a write speed of 88 MB/s
The RAID levels 0, 5, 6 and 10 are tested.
Chunk sizes starting from 4K to 1024K are tested.
XFS is used as the test file system.
Data is read from/written to a 10 GB file.
The theoretical max through put of a 4 drive array is 340 MB/s. A 6 drive array should be able to sustain 510 MB/s.

About the data:

All tests have been performed by a Bash shell script that accumulated all data, there was no human intervention when acquiring data.
All values are based on the average of five runs. After each run, the RAID array is destroyed, re-created and formatted.
For every RAID level + chunk size, five tests are performed and averaged.
Data transfer speed is measured using the 'dd' utility with the option bs=1M.

Test results

Results of the tests performed with four drives:

Test results with six drives:

Analysis and conclusion

Based on the test results, several observations can be made. The first one is that RAID levels with parity, such as RAID 5 and 6, seem to favor a smaller chunk size of 64 KB.

The RAID levels that only perform striping, such as RAID 0 and 10, prefer a larger chunk size, with an optimum of 256 KB or even 512 KB.

It is also noteworthy that RAID 5 and RAID 6 performance don't differ that much.

Furthermore, the theoretical transfer rates that should be achieved based on the performance of a single drive, are not met. The cause is unknown to me, but overhead and the relatively weak CPU may have a part in this. Also, the XFS file system may play a role in this. Overall, it seems that on this system, software RAID does not seem to scale well. Since my big storage monster (as seen on the left) is able to perform way better, I suspect that it is a hardware issue.

Update: the main reason there is no higher transfer rate beyond 350 MB/s is because the M2A-VM consumer-grade motherboard can't go any faster.

Tool of the month: iftop - advanced bandwidth monitoring

2010-04-20T13:56:00.000-07:00

The utility iftop allows you to monitor bandwidth usage. It is in some sense similar to tools like iptraf, dstat and bwm-ng. Iftop is more special than those. Because iftop lets you monitor the speed of individual TCP / UDP connections. Basically, you will be able to determine how much traffic is flowing between two hosts.

It is definitely worth a try.

WFS - WAN Failover Script now available

2010-04-10T11:05:00.000-07:00

Since I could not find a WAN failover script for Linux to my likening, I wrote one myself. If you have any use for it: I put it on a Google code project.

WFS tests the availability of your primary WAN connection and switches to your secondary / backup connection when a failure is detected. When in failover mode, WFS continues to monitor the availability of the primary WAN connection and once it becomes available again, it switches back.

For more information and a download, please take a look at the project:

http://code.google.com/p/wanfailoverscript/

HP Procurve "auto DoS" feature causing network problems

2010-04-07T11:51:00.001-07:00

A feature on more recent HP Procurve models (18xx series, such as 1810G etc.) is called "Auto DoS". You can find it in the section "Security" and then "Advanced security".

If you enable the Auto DoS feature, traffic is blocked based on one of these conditions:

the source port (TCP / UDP) is identical to the destination port (NTP, SYSLOG, etc)
the source port (TCP / UDP) is 'privileged' thus in the range of 1 -1023.

This will cause all kinds of problems, but first this: "Why on earth is a Layer 2 device filtering on Layer 3?". This is just insane.

NTP does not work any more. Syslog traffic will not arive. VPN traffic may not arrive.

This issue cost me a lot of time to solve. I first blamed our Firewall, but the actual traffic arrived on the tagged trunk port on the affected switch. The traffic somehow was not sent to the switch port on which the destination device was connected.

Affected products:

HP ProCurve 1810G - J9449A ( 8 ports ) and J9450A ( 24 ports )

RAID array size and rebuild speed

2010-04-03T16:02:00.000-07:00

When a disk fails of a RAID 5 array, you are no longer protected against (another) disk failure and thus data loss. During this rebuild time, you are vulnerable. The longer it takes to rebuild your array, the longer you are vulnerable. Especially during a disk-intensive period, because the array must be reconstructed.

When one disk fails of a RAID 6 array, you are still protected against data loss because the array can take a second disk failure. So RAID 6 is almost always the better choice. Especially with large disks (1+ TB), because the rebuild time largely depends on the size of a single disk, not on the size of the entire RAID array.

However, there is one catch. The size of the RAID array matters when it becomes big, 10+ drives or more. No matter if you use hardware- or software-based RAID, the processor must read the contents of all drives simultaneously and use that information to rebuild the replaced drive. When creating a large RAID array, such as in my storage array, with 20 disks, the check and rebuild of the array becomes CPU-bound.

This is because the CPU must process 1,1 GB/s (as in gigabyte!) of data and use that data stream to rebuild that single drive. Using 1 TB drives, it checks or rebuilds the array at about 50 MB/s, which is less than half what the drives are capable of (100+ MB/s). Top shows that indeed the CPU is almost saturated (95%). Please note that a check or rebuild of my storage server takes about 5 hours currently, but that could be way shorter if the CPU was not saturated.

My array is not for professional use and fast rebuild times are not that of an issue. But if you're more serious about your setup, it may be advised to create more smaller RAID vollumes and glue them together using LVM or some similar solution.

Linux: monitor a directory for files

2010-03-22T10:01:00.000-07:00

Inotify is a mechanism in the Linux kernel that reports when a file system event occurs.

The inotifywait comand line utility can be used in shell scripts to monitor directories for new files. It can also be used to monitor files for changes. Inotifywait must be installed and is often not part of the base installation of your Linux distro. However, if you need to monitor a directory or some task like that, it is worth the effort. (apt-get install inotify-tools).

Here is how you use it:

inotifywait -m -r -e close_write /tmp/ | while read LINE; do echo $LINE | awk '{ print $1 $3 }'; done

Let's dissect this example one part at a time. The most interesting part is this:

inotifywait -m -r -e close_write /tmp/

What happens here? First, inotifywait monitors the /tmp directory. The monitoring mode is specified with the -m option, otherwise inotifywait would exit after the first event. The -r option specifies recursion, beware of large directory trees. The -e option is the most important part. You only want to be notified of new files if they are complete. So only after a close_write event should your script be notified of an event. A 'create' event for example, should not cause your script to perform any action, because the file would not be ready yet.

The remaining part of the example is just to get output like this:

/tmp/test1234/blablaf
/tmp/test123
/tmp/random.bin

This output can be used to use as an argument to other scripts or functions, in order to perform some kind of action on this file.

This mechanism is specific to Linux. So it is not a OS independent solution.

Syslog: the hidden security risk

2010-03-18T01:23:00.000-07:00

People sometimes forget that there are also a number of UDP-based services that may pose a threat to the security of your systems. SNMP is a well-known service, notorious for being configured with a default password (or community string).

But there is another service that is often not seen as a risk. This is the syslog service. Syslog is used on virtually all UNIX-like platform for logging messages of the system to one or more log files to disk. The syslog service often listens on the network, on UDP-port 514. Please note that syslog does not perform any authentication of data that is sent to it.

So what does this mean?

An attacker can:

Create a denial-of-service condition (DoS) by sending large amounts of data to the syslog service, filling up disk space.
Once the disk is full, logs can no longer be saved, thus any attack that would leave a trail within the logs would go unnoticed.
by sending large amounts of specially crafted messages, an attacker can cause chaos if logs are monitored by intrusion detection systems or other systems that create alerts.

How to attack? Just use netcat:

nc -u [IP-address] 514

Once you are connected, anything you type will be logged in a log file.

How to mitigate this issue?

Firewall access to UDP-port 514
Make sure that the syslog service does not listen on the network if not required, only on localhost.

Ubuntu and full disk encryption (FDE)

2010-02-22T14:50:00.000-08:00

Ubuntu is based on Debian Linux. As part of a regular Debian installation, you can choose to create an encrypted disk volume based on LUKS. This is different from the option within the Ubuntu installation to encrypt home directories. To be able to install Ubuntu and use full disk encryption, you need to download the alternate install CD / DVD. Only this version of Ubuntu supports LUKS as an installation option.

You will have either two options:

use the default choice, creating a swap partition, boot partition and the encrypted root file system on top of LVM;
create separate crypted partitions yourself manualy.

Personaly I don't care for separate partitions and use the provided automatic option. If you do care, please read this blog for more info.

Scanning many hosts in parallel with Nmap using PPSS

2010-02-18T13:41:00.000-08:00

Scanning a large number of hosts using Nmap often takes a lot of time. During this time, no output is written to a file or disk. Only when Nmap is finished, is all output written to the output file. Often, I want to start processing results of hosts that have already been scanned. Often, the trick is to split the input file with all the hosts and start multiple Nmap instances by hand using the different input files. This is rather cumbersome. Now what I really want is that I get the results of a scan of a particular host immediately available as soon as it's finished. Here is where PPSS comes in. PPSS can start Nmap scans and proces a list of hosts as contained in an input file. PPSS will only start a predefined max number of simultaneous scans in parallel, as not to overwhelm the scanner, network or target hosts. This is an example on how PPSS can be used to obtain results immediately:

./ppss -f hosts.txt -c 'nmap -n -v -sS -A -p- -oN "$ITEM" "$ITEM"' -p 4

Where hosts.txt contains IP-addresses, networks or domain names like:

192.168.0.1
192.168.0.2
192.168.0.3
192.168.1.1-254
www.google.nl

The 'ITEM' part is the fun bit. In this example, multiple instances of Nmap will scan a single hosts. The output is written to a file called "$ITEM", which is of course substituted for the IP-address or domain name as read from hosts.txt. The second "$ITEM" is the argument to Nmap which tells which host to scan. The -p 4 option tells PPSS to run 4 nmap scans simultaneously at all times.

You will end up with a large number of output files, one per host. As soon as a scan is finished on one host, you can start processing the results, instead of waiting for that big scan to finish.

Corsair CM PSU-750HX seems ok

2010-02-10T15:59:00.000-08:00

I had to replace my Coolermaster PSU and after some searching on the interweb, I chose the Corsair CMPSU-750HX. One of the reasons is that Corsair states that this PSU can withstand 50 degree Celcius in continuous operation on full load.

The package is very, clean, with all the cables in some neat pouch and the PSU itself also in a neat bag. You pay quite some money, but it at least suggest quality. The modular design with the modular kables is excellent.

Personally, I think that PSUs with multiple 12v rails are not that usefull. This particular PSU has just one single 12v rail that can be loaded up to full capacity of the PSU. I think this makes the design easier and if you are running heavy systems that have short spikes (starting al drives) this is ideal.

I hope this one lasts longer.

Smoking coolermaster Silent Pro M 600W TN6M50

2010-02-09T16:46:00.000-08:00

So I was just messing around on my work station, when suddenly I smelled the smell any person familiar with electronics fears. The smell of some electrical component burning.

The whole upper floor smelled like something was smoldering. So I shut down both my storage servers. After half an hour I still hadn't found the source of the smell, but it definitely was my 18 TB NAS.

So I switched it back on to see if I could determine the cause. The smell became stronger and within the strike light of my flash light, I saw small strands of grey smoke slowly twirling out of the Coolermaster Silent Pro M 600W power supply.

That was the moment I decided to turn the system off immediately. I know this kind of thing can happen. But it is still very bad for such a component to crap out on me like that.

Needless to say that I don't want a replacement of this model in my server. I will now try my luck with the Corsair CMPSU-750HX 750Watt PSU.

Based on a single experience it cannot be concluded that this Coolermaster PSU is a bad one that should be avoided. But the only moving part of a PSU is the (big) fan. What must be wrong with it that it decided to start smoking?

For reasons of availability, a redundant power supply should be used, but that is just too expensive for my taste ($500+).

Is the iPhone OS a threat to our freedom?

2010-01-31T11:55:00.000-08:00

There is one fundamental flaw with both the iPhone and the iPad. As a user, you do not have full control over your device. You can only install or run software that is approved by Apple.

This is something that is unprecedented. All major platforms, Windows, Linux, Mac OS X do provide the user with full control over what programs can be run or not.

Is this a bad thing? Are you trading your freedom for some luxurious gadget such as the iPhone or the iPad?

I say no. People are over reacting.

You would trade in your freedom and become 'owned' by Apple, if an iPhone or iPad is your sole computing device.

However, everybody also has a computer at home, that is still under your full control.Since the iPhone and iPad is just an auxiliary device (although quite powerful), you are still free.

Most importantly, I think it is about the content, not the device. Do you retain control over your own content, such as music, files and e-mail? I think so. I think that on this level, which is most crucial, Apple does not restrict people on iPhones or iPads in any way. And that is what matters most I think.

Why the iPad will be a breakthrough in human computing

2010-01-30T02:50:00.000-08:00

The fact is that computers aren't made for humans. Computers are just made, and humans have to adjust to them.

The problem is that most people that are not into technology just don't understand how computers work. Should I single click of double click? Click left or right?

The only people that can work with some ease on computers are people with a more than average interest in technology. Ordinary people often have to fight great battles to get the job done. The therm 'ordinary' does not justice to these people, because technologists seem to forget that they are the people that actually do something useful.

The iPad is the first personal computer that is actually made for humans that do not have a primary interest in computers. The entire computer is abstracted away to focus on the things you really want to do. People do not need the help of technologists anymore to do the things they want.

The iPhone was the first in this new kind of computers, but due to it's size it's still a bit of a gimmick. But it was a revolution within the world of smart phones. It was the first device that was actually usable for non-technologists. And the iPad will be the first 'regular' computer that (hopefully) won't be such a nuisance as current home computers are.

The iPad will be the death of Flash

2010-01-30T02:16:00.000-08:00

So Apple finally released their tablet computer: the iPad. One of the most debated drawbacks is that it lacks support for Adobe Flash. The iPhone does not support Flash either, and since the iPad is based on the iPhone OS, this should not come as a surprise.

Now many people see this as a serious problem. There are a slew of websites that cannot be viewed on an iPad for this reason.

However, the thing is this: Flash sucks. It is a proprietary technology under the control of Adobe and the only reason it is currently popular is that it provides a kind of standard for video playback across all browsers and operating systems. But it still sucks. It crashes. It is shit.

The iPhone is immensely popular, despite the lack of flash. The iPhone user base is huge. By using Flash in your website, you are excluding a large population of potential visitors. So this is what will happen: websites will start dumping Flash. The iPhone and the iPad will together kill Flash.

And that is a good thing for everybody, especially for proponents of open standards and open formats. Oh irony, that it will take a closed proprietary platform to do so.

Mounting a file system or partition from a disk image

2010-01-23T02:13:00.000-08:00

You cannot just make a disk copy with dd and then just mount it as a regular disk. You must know where the partition starts on the disk. So first, you need to get the partition table with sfdisk:

sfdisk -l -uS image_file.dd

The output is something like:

Disk /mnt/image/image_file.dd: 9729 cylinders, 255 heads, 63 sectors/track
Warning: The partition table looks like it was made
for C/H/S=*/240/63 (instead of 9729/255/63).
For this listing I'll assume that geometry.
Units = sectors of 512 bytes, counting from 0

Device Boot Start End #sectors Id System
/mnt/image/simon-besmet.img1 63 8497439 8497377 b W95 FAT32
/mnt/image/simon-besmet.img2 * 8497440 156280319 147782880 7 HPFS/NTFS
/mnt/image/simon-besmet.img3 0 - 0 0 Empty
/mnt/image/simon-besmet.img4 0 - 0 0 Empty

Next, we need to calculate the actual starting point of the partition. The #sector number is in sectors, which contain 512 bytes. So in this case, the starting point of the NTFS partition is 8497440 x 512 = 4350689280.

To mount the image, enter the following command, using the calculated offset:

mount -o loop,offset=4350689280 /mnt/image/disk_image.dd /mnt/disk

Source:

http://lists.samba.org/archive/linux/2005-April/013444.html

Linux: show graphical layout of disk temperatures

2010-01-02T19:12:00.000-08:00

To get a visual representation of hard drive temperatures, I wrote a small script. The output of this script looks like this:

This output is tailored to the exact disk lay-out of my storage server. However, it is also usable for other servers. You have to edit the lay-out depending on the system.

This script assumes that you have smartmontools installed on your system. It uses smartctl to obtain the drive temperatures.

The script can be found here or here in tgz format.

Recovering a lost partition using gpart

2009-12-22T11:12:00.000-08:00

Even today people do not understand how important it is for the safety of your data to make backups. I was asked to perform some data recovery on a hard drive of an old computer, which still contained important documents and photo's.

The first thing I did was to make a disk image with ddrescue. I always work with the image and not with the original drive, to prevent any risk of accidentally messing things up for good.

Example:

ddrescue -r 2 -v if=/dev/sdf of=/storage/image/diskofperson.dd bs=1M

Next, I tried using gparted on this file but got this error:

Welcome to GNU Parted! Type 'help' to view a list of commands.

(parted) p

Error: /storage/image/diskofperson.dd: unrecognised disk label

(parted) quit

Also fdisk -l didn't work:

Disk /storage/image/diskofperson.dd doesn't contain a valid partition table

It seemed that the partition table was gone. I used the utility testdisk to recover this partition, to no avail. Why this tool didn't work is beyond me.

I found a very old utility called 'gpart' that just searches a disk for existing partitions. I just want to know the starting offset of the relevant partition.

So I ran:

gpart -g /storage/image/diskofperson.dd

And I got nothing useful, although a partition was found:

Begin scan...

Possible partition(DOS FAT), size(57255mb), offset(0mb)

End scan.

So I ran the command again with more verbosity:

gpart -g /storage/image/diskofperson.dd

...

Begin scan...

Possible partition(DOS FAT), size(57255mb), offset(0mb)

type: 011(0x0B)(DOS or Windows 95 with 32 bit FAT)

size: 57255mb #s(117258372) s(63-117258434)

chs: (1023/255/0)-(1023/255/0)d (0/0/0)-(0/0/0)r

hex: 00 FF C0 FF 0B FF C0 FF 3F 00 00 00 84 38 FD 06

End scan.

...

This time I got something useful. The s(63-117258434) part shows the starting sector, which is 63. A sector is 512 bytes, so the exact starting offset of the partition is 32256.

So to mount this partition, just issue:

mount -o loop,ro,offset=32256 /storage/image/diskofperson.dd /mnt/recovery

And voilá, access to the filesystem has been obtained.

/storage/image/jdiskofperson.dd on /mnt/recovery type vfat (ro,loop=/dev/loop0,offset=32256)

Wake On Lan not working with realtek r8168 card

2009-12-20T07:18:00.000-08:00

After messing around with different kernels on Debian Lenny, I noticed that my system did no longer respond to WOL packets. The system is using an on-board Realtek R8168 chip which supports wake on lan.

After searching and reading i found the problem. The system is using an r8168 chip, but the driver that is loaded by Linux is the r8169 driver. If the r8168 driver is loaded, as it should be, then WOL is working again.

Shunit2, unit testing for shell scripts

2009-12-17T08:47:00.001-08:00

This may be of interest to people who are as stupid as I am and write elaborate shell scripts instead of using a proper scripting language such as Python or Ruby. No I am deliberately not mentioning Perl here.

Anyway, testing is always an issue. With PPSS, I encountered many times that some change broke something in another place. Testing if some change screwed things up manually is just tedious and a pain.

As any person who is semi-serious about programming should have done already, I started to write some unit tests. For bash scripts, there is this unit test framework called Sunit2. It takes some effort to write tests. But once you have written some tests and witness how the tests that passed before now fail after you changed some code is priceless. Because often, you would not have found the issue before a long time.

Google DNS - what to think of it?

2009-12-03T10:42:00.001-08:00

Google now provide an open DNS service. At first I was scared that they use their service to get information about users.

However, their privacy statement tells us that no personal identifiable information is stored for longer than 48 hours. The permanent logs do not contain such information. The most notable information that is stored permanently is the requested domain. The other thing is that they record the location of the request on the city level. And even that info is only stored for two weeks. After that, only a small random sample of info is stored for permanent storage.

The main question is now: who do you trust more: Google or your ISP.

To be honest, I think I trust Google more than my ISP. I don't think my ISP adheres to the same privacy policy.

Convert dos file to linux format wit vim

2009-12-03T09:12:00.000-08:00

If you have some file that has been saved on a (Win)do(w)s host you may notice that all lines end with ^M.

To fix this, you can use the tool dos2unix. If however this tool is not at your disposal, you may have a problem. If vim is available, you do not have a problem:

The trick is to delete all occurances of ^M with this line:

:%s/ctrl-v ctrl-m//g

This will look like:

:%s/^M//g

Just save the file and you're done.

Which file system for a large storage array under Linux?

2009-11-30T05:03:00.000-08:00

There are many file systems available under Linux, however only a few of them can be used for a large storage array.

I am assuming that you want to create a single file system. I don't care if you use LVM or other layers beneath, this is about which file system to use.

I will discuss each file system in short.

- EXT3

Old and trusted. but slow, wastes tremendous amounts of free space and has an 16 TB limit. Not recommended. At 16 TB, you will lose significant amounts of free space.

- JFS

Developed by IBM, it seems an ok file system and it supports 16+ TB file systems. However it is abandoned by IBM and does not seem to have a future. The reason I did not choose JFS is because it does not seem to be in wide spread use and therefore, and I do not want to take any risk in this regard. I might encounter some rare bug because I would be one of the few Linux jfs users that creates 16+ TB file systems.

- XFS

A file systems that works. Supports 16+ TB file systems and is widely used. It has some drawbacks, such as data loss (files are zeroed) when something goes wrong (powerloss) however, nobody that is interested in creating a large filesystem is going to run the box without a UPS.

- Reiserfs

The guy who designed and wrote it is in jail for killing his wife. Not much future here, i suppose. Also, it does not support file systems bigger than 16 TB.

- Reiser4

Some new version of Reiserfs which is still not available as part of a stable Linux kernel.

- EXT4

Ext3 upgraded to current days need, although with a draw back. On paper, it supports 16 TB+

file system, but currently, in practice, it does not. I found out the hard way and had to reformat my array with another file system.

My choice:

Honestly, there is not much of an option currently under Linux other than XFS. In the past, I had some quirks with file names that contained strange characters causing trouble. However, I never lost any data whatsoever.

Currently, what I'm really hoping for is ZFS for Linux or a stable version of btrfs. These are truly modern day file systems with support for snapshots, etc. But this is still a dream. Until then, I will stick with XFS.

Tip of the day: Scrolling in GNU screen

2009-11-30T03:44:00.000-08:00

Just the answer:

By default, on Debian, the scroll back buffer is about 1K lines. This can be changed in the .screenrc file in your home directory. The following example sets the scroll back buffer to 5K lines.

defscrollback 5000

Set the scroll back buffer on the fly with:

First enter the command line mode:

ctrl + a, :

Then enter:

scrollback 2000

To set the scroll back buffer to 2000 lines.

To actually scroll back, the actuall stuff why you may be reading this post:

First enter copy mode:

ctrl + a, [

Use standard VI controls to navigate through the lines (h j k l).

Use ctrl+b to scroll a full page up

Use ctrl+f to scroll a full page down.

For more details, please visit this link:

Gzip with parallel compression support: pigz

2009-11-22T10:29:00.001-08:00

The speed at which files are compressed with gzip is currently almost always determined by the speed of the CPU. However, standard unix gzip is single-threaded and only uses a single CPU (core).

However, the maintainer of the zlib library has released 'pigz' or 'pig-zee' whichs adds just that: support for parallel compression. This dramatically improves the speed at which a file can be gzipped.

In this example, a 3 GB compressable file is gzipped:

Gzip:

root@Core7i:~# time gzip pigz.bin

real 1m58.994s

user 1m56.480s

sys 0m1.820s

Pigz:

root@Core7i:~# time pigz pigz.bin

real 0m31.524s

user 2m54.890s

sys 0m2.900s

This simple and a bit unscientific example shows a 400% speed improvement. Since the Core i7 has four real cores, this shows that pigz scales nicely.

How to determine which process causes IO ?

2009-11-22T07:19:00.000-08:00

There is a nifty little program called 'iotop'. Iotop is part of Debian or Ubuntu and can be installed with a simple apt-get.

Once you have determined with 'top' that the system is waiting on IO-access, It is nice to know

which process is responsibe for this IO. Therefore, you want a list of processes, just like top provides, but instead of CPU or RAM usage, it shows the IO a process is generating. This is exactly what 'iotop' provides.

This is ideal when troubleshooting performance problems caused by heavy IO.

http://guichaz.free.fr/iotop/

Making cowpatty recognize a four-way handshake

2009-11-22T04:52:00.000-08:00

I was unable to get cowpatty working with a packet capture that actually

contains a four-way handshake of a WPA session.

I got it working like this:

First, download cowpatty 4.6 right here, within the source directory of cowpatty.

Extract cowpatty and apply this patch using these instructions

Then build couwpatty just with 'make' and 'make install'.

I created a test setup with a known password. However, this patched

version did not find the passphrase using a dictionary file.

I then used genpmk to create a precomputed hash database like this:

genpmk -f ./aircrack-ng-1.0/test/password.lst -d hashes -s default

Please note that I added the correct passphrase to this password list to

make sure that cowpatty works.

Finally, using the -d option on the hash file, cowpatty

managed to crack the PSK.

Monitor power usage with your UPS

2009-11-18T13:09:00.000-08:00

If a system is connected to a UPS (Uninterruptible Power Supply), it is possible to determine how much power it consumes. For this purpose, I wrote a small script:

Host:~# ./ups.sh
---------------------------------
UPS model: Back-UPS RS 1200 LCD
APC model: Back-UPS RS 1200 LC
----------------------------------
Capacity: 720 Watt
Load: 18 Percent
Usage: 129 Watt
Time left: 33 Minutes
Status: ONLINE
----------------------------------
Host:~# ./ups.sh
----------------------------------
UPS model: Back-UPS RS 1200 LCD
APC model: Back-UPS RS 1200 LC
----------------------------------
Capacity: 720 Watt
Load: 19 Percent
Usage: 136 Watt
Time left: 22 Minutes
Status: ONBATT
----------------------------------

This script assumes that:

You are running a unix
You run apcupsd

The script can be downloaded here:

24 TB based on Norco RPC-4020 and Linux Software RAID

2009-11-14T05:07:00.000-08:00

Just a quick link:

Some person build basically the same setup, including identical controller, providing 28 TB of storage: Take a look here

The main difference is that this person uses 1.5 TB disk, thus achieving more storage.

Debian Linux Preseeding - an installation framework

2009-11-11T13:11:00.000-08:00

Look Ma!

No hands!

just watching a system PXE-boot and install itself into a fully operational system is fun. PXE boot and preseeding is not enough however to accomplish this.

By itself, preseeding only installs a basic operating system. Next, you need to configure all sorts of things and services, install additional software, create accounts. etc.

So there is a whole post-install process that you also need to automate. Preseeding by itself, as I understand it, does not seem to be able to do that for you. But you can execute commands at the end of the installation.

First, I'd like to point out that the alternative to Debian preseeding is, in my opinion, FAI or Fully Automated Installation. I want to make clear that this is a product with a long history that may better suit your needs. I do not use FAI because I find it way to complicated. It seems very powerful to me, but I don't understand it.

I understand preseeding and shell scripts. Putting commands that I would otherwise execute manually are now performed by some shell script. And I think every administrator understands shell scripts and will be able to automate installation with almost no effort by just executing a bunch of shell scripts that install various additional components.

I wrote a small installation 'framework' that can be used in conjunction with preseeding to truly fully automate your installation, without the requirement of learning complex stuff like FAI.

Unfortunately, it is not available yet on-line, but as soon as I find the time, a new project will be started and this small installation framework will be available with some example preseed configuration.

In day-to-day operations I use this script to install servers and laptops. Once you see it you will like it.

Linux Mac Mini - temperature monitoring with lm-sensors

2009-11-08T17:21:00.000-08:00

This post is about getting temperature monitoring to work with a Mac Mini running Linux.

Using Debian Lenny, out of the box, lm-sensors is not working. No sensors can be found. This is how temperature monitoring and fan speed monitoring can be made to work:

modprobe applesmc

If you run "sensors-detect" after this, and do a:

modprobe coretemp

Then "sensors" will give you ouput like this:

Mini:/sys/devices/platform# sensors
applesmc-isa-0300
Adapter: ISA adapter
Master : 2151 RPM (min = 1500 RPM)
temp1: +83.2°C
temp2: +68.0°C

Some software to control the fan speed:

http://stargate.solsys.org/mod.php?mod=faq&op=extlist&topicid=27&expand=yes#118

Linux on Mac Mini - boot after power failure

2009-11-08T12:57:00.000-08:00

When using a Mac Mini as a server or router, it is very nice if the machine automatically boots if a power failure has occurred.

User chirhoxi on the ubuntu forum found out how this can be achieved:

http://ubuntuforums.org/showthread.php?t=1209576

Basically you need one of these commands:

Original Mac Mini:

setpci -s 00:03.0 0xa4.b=0

Newer Mac Mini:

setpci -s 00:03.0 0x7b.b=19

These commands might not work with the latest Mac Minis but the thread discusses how to determine for yourself what the appropriate command is.

How to run Debian Linux on an Intel based Mac Mini

2009-11-01T15:22:00.000-08:00

The Mac Mini is just a gorgeous device. It is beautiful, small, silent, powerfull yet energy efficient. When idle, it uses around 20 watts. I'm using one of the first Intel-based Minis with an Intel Core Duo chip, running at 1.6 Ghz.

I want to use this mini as an expensive router and download host. I could have used something embedded, such as one of those router boxes that costs about 70 euros, but no, I want to do some more with my router, such as downloading, etc. It is the only device in my house that is allowed to run 24/7 so it has to be a bit more powerful if I want more than just routing. I know that this mini was like 600 euros or something back in the days, and that is quite some money to spend on something that is now only a router. However, when I was still running Mac OS X on it, I didn't do much more with it than I will now, it will actually do more.

I am assuming that you want to run Linux exclusively on the Mac and that Mac OS X will be wiped off.

To get this puppy running Debian Linux (Lenny), you need to first boot the Mac with the (Snow) Leopard OS X boot CD and startup the diskutilily.

You need to create at least two partitions: one for the root file system and one for swap. The most important step is to select 'options' under the partition layout screen, and select Master Boot Record partitioning instead of the other 2 options. Do NOT use GUID or Apple Partition Map.

Now, boot your regular Debian Linux boot CD, I use the regular network installation CD. When you get to the partitioning screen, do NOT auto-partition the hard disk. Just reconfigure the existing partitions you just made using Diskutility. So the large partition will be configured as "/" and made bootable. The small partition must be configured as swap.

After the installation finishes, just install GRUB in the MBR and reboot. If all went alright, you will see a non-blinking folder on a gray background for a couple of seconds, after which Linux will boot. If you get a blinking gray folder with a question mark, something went wrong.

It seems that if configured properly, after the EFI boot mechanism fails to find a system folder on some Mac partition, the legacy BIOS emulation seems to kick in, and star to search for something to boot.

The Mini has only one network card, so another one is necessary to run it as a router. I bought some no brand USB2 to 100 MBIT NIC (Bus 005 Device 003: ID 9710:7830 MosChip Semiconductor MCS7830 Ethernet) which seems to run smoothly.

UPDATE: According to Campr, it is possible to replace the Wireless mini PCI-e card with a dual gigabit card.

I guess you will need to mod the Mini but it will allow true gigabit speeds on all interfaces.

The security risk of vendor-supplied default SSL certificates

2009-10-30T13:09:00.001-07:00

Often, software comes supplied with some default SSL certificate, for testing purposes, such as those 'snake oil' certificates (they are called snake oil certificates for a reason). In practice, I often encounter usage of such certificates. People may seem to think that as long SSL is used, authentication and thus credentials are safe, but nothing could be further from the truth.

If you encounter a service that uses a default vendor-supplied SSL certificate, decryption of communication is trivial. Just obtain a copy of this vendor software and grab the private key. This private key can be loaded into Wireshark to decrypt any captured SSL traffic that has been encrypted with this certificate. Please read this link about decrypting SSL with Wireshark.

So it is important to always replace default SSL certificates with a freshly generated, no matter if it is self-signed or not.

Blu Ray is dead

2009-10-05T12:12:00.000-07:00

HD-DVD may be dead, but Blu Ray is just as dead. The whole concept of optical media is dead. Honestly, who is still burning CD's or DVD's nowadays? (If you are, why for Christ sake? I can't think of a single good reason) And at 10 euros ($100) for a single Blu Ray disk, you must be totally bonkers to buy one of them recorders.

I mean, let's face it, CD's were really cool in an age where 650 MB was way more than the 40 or 80 MB hard drive in your computer. It made a difference. That was already less so with a DVD, with a capacity of 'only' 4 GB. However, in the early years of the DVD, you could backup your entire hard drive on two, maybe three disks, since a hard drive averaged around 4 to 10 GB at that time.

Then finally came Blu Ray and HD-DVD. A whopping 25 GB on a disk. No shit!. You mean, like I need no less than 40 Blu Ray disks (400 euros) and an eon of burning disks to backup one of my 1 TB hard drives that cost me like 70 euros?

What must they have been thinking when they developed Blu Ray? As a backup medium it is useless, but it was ofcourse intended as a carrier for movies, I know. But why should I go outside, through the cold and the rain to go to some shitty video store that only has last years block busters? Why bother with 40 Mbit downstream and the Internet at your disposal? Downloading a movie will take me as much time as going to the video store and selecting something so dreadful even the DVD player will refuse to play it. What else is there to choose from. The Internet provides us with the most rare and obscure but most beautifull movies you ever saw. And the more main stream movies can be obtained in full HD 1080p.

Current Internet connections are of such quality, that physical media such as Blu Ray disk are becomming irrelevant. A normal DVD is downloaded within 20 minutes at 4 MB/s. And when Internet connections will reach 100 Mbit or 12 MB/s, even a HD 1080p movie will be downloaded within the hour.

Storage is not a problem. If you can store 40 HD movies on a single 1 TB disk, then you will pay 1,5 euros for each movie. Beats any Blu Ray disk in price and time. If you even care about them that much, buy or build a NAS and store them on some redundant storage.

In time people will do with DVDs and CDs what most people already do with CDs: rip them to some format your computer understands (to transfer it to your MP3 player) and get rid of that CD that will become scratched and useless even if you don't touch it. No, you don't want to transcode your HD movies to some iPod or something, but you may want to stream it to your media player in the living room? A jukebox full of films, just as you have a jukebox full of music.

Give the CD, DVD and Blu Ray a little push, and let them fall into the grave. It is an outdated technology for a problem that existed 10 years ago. The Internet has made it irrelevant. Be done with it.

PPSS version 2.30 now operates asynchronous

2009-09-26T12:32:00.000-07:00

If you background a bash or shell process, how do you determine if it has finished? Since inter process communication is not possible using shell scripts, people often refer to while loops or other polling mechanisms to determine if some process has stopped.

However, the one player that knows best if a process has finished is the process itself. So if only this process could tell the parent or other process about this...

The solution is using a FIFO or 'pipe'. A listener process reads the pipe and executes a command for every message received through this pipe. This was already build-in into PPSS. However, PPSS had this dirty while loop that polls every x seconds to determine if there are still running workers. If not, PPSS finishes itself.

However, while loops and polling mechanisms are evil, dirty and bad. The nicest solution is to make PPSS fully asynchronous. To achieve this, every job must tell PPSS that is has finished. PPSS already has this listening process that listens to the pipe for commands. If this channel is used by workers to communicate that a worker is finished, PPSS will know when all workers are finished.

This makes PPSS a lot faster and responsive.

Laptop or netbook as router?

2009-09-20T04:19:00.000-07:00

If you want a router for distribution of internet to your computers at home, there are several options.

1. buy some embedded device from Linksys, Draytek, Asus, 3com, ZyXtel or netgear

This type of hardware is cheap, economical, and gets you up and running in a few minutes. The downside is that you can't do much else with these things. Yes, there are many custom firmwares, which allow you more freedom, but the hardware is often the limiting factor.

2. convert a regular PC into a router

If you want more than just routing, building your own router using a(n) (old) PC is the preferred course of action. The downside is that a PC often uses more 'juice' than an embedded device.

However, those new Atom-based PC's may be a very nice option. Just add a second network card, though an USB-port and you have something far more flexible than an embedded router.

3. convert a laptop into a router

It sounds a bit strange and silly at first to use a laptop as a router, but it makes sense when you think of it.

a. It is economical in terms of power usage

b. It has a build-in UPS called a 'battery'

c. It has a build-in screen and keyboard

All these things are an advantage regarding option 2.

Nowadays you can have a netbook for only 300 euros. It is more expensive than an embedded device, but almost as economical and provides much more performance and flexibility.

I've been running an old laptop as a router for 6 months without problems. Unfortunately, the disk died due to old age, but that can happen to any computer.

How to build an energy efficient computer for home use

2009-09-19T12:17:00.000-07:00

In short:

1. Buy whatever you fucking want.

2. Turn the fucking thing off when you're not using it.

Long:

People are spending a lot of time building an energy efficient home computer, that can act as an HTPC, NAS, or whatever. It must consume as little power as possible, because it it will be on 24/7.

Why the fuck do you want to leave a system on 24/7 at home?

Unless you're unemployed you will be at work most of the time. And during that time, this machine is doing absolutely nothing.

There may only be one system that stays on 24/7 and that is your router, either some embedded router thingy or something based on a low-power PC, but that's about it. Turn every thing else off.

A machine that does 200 Watt idle that is only turned on when necessary will be more energy efficient than your specially build 40 watt NAS or whatever it is that is running 24/7. Nothing can beat a system that is turned off.

If you need a system, just use wake-on-lan or WOL to turn the damn thing on. It will be ready in about 2 or 3 minutes and then you can do whatever you want.

Fine if you leave a system on that is downloading some stuff during the night or day, but after the download has finished, turn the damn thing off.

If you're honest with yourself and really think about it, there is no need to keep a system on for 24/7. I know you may come up with excuses, but remember that you still can use your router for that.

By the way, if you're searching for a really energy efficient computer, buy a Mac Mini. Although quite expensive if you ask me, the're doing 30 watt in idle and almost nothing when sleeping. And if a mac is asleep, it can be woken with WOL and is up in seconds. They make an excelent download server.

[EDIT]

There is also the option of S3 or S4 under Linux: suspend to ram or disk. However, your mileage may vary. If it works for you your system will be down and up in seconds. However, my experience is that it very much depends on your hardware if this will work. My Highpoint cards do not seem to like it, my array does not awake and the screen of the system stays blank.

If it works, it is a faster solution than just to turn the system off and on with WOL. My experience is that is seems not that robust.

Automatic conversion of DVD to iPhone and ipod touch format

2009-09-14T13:45:00.000-07:00

If you have some DVD's that you want to convert to your iPhone or iPod touch, there is already a nice solution called 'Handbrake'.

However, if you want to convert multiple DVD's and don't want to spend time manually adding each movie or episode to the queue, I might have some solution for you. I've written a script that uses HandBrakeCLI, the command-line version of Handbrake. This script is called "dvd2iphone" and can be downloaded here.

This script uses the VIDEO_TS folder or even the ISO image as input, and then processes all movies or episodes that are found, which are automatically named.

To be able to use this script, you need the command-line (CLI) version of Handbrake, which can be found here. Place this binary on your system and place the dvd2iphone script in the same directory.

Usage:

./dvd2iphone.sh (INPUT)

for example:

./dvd2iphone.sh /cdrom/VIDEO_TS

./dvd2iphone.sh casablanca.iso

This script can only process a single VIDEO_TS folder or single ISO at a time. However, you can create a text file containing a list of to-be-processed files and use PPSS on this list, in combination with the dvd2iphone script to automatically process all the DVD's.

Warning; you might want to edit the HandBrake command in the script to select the correct audio track and to add subtitles.

Please note that Handbrake can only use up to four cores to encode a single movie. If you have more then four cores available, it may be worthwile to run two instances of Handbrake simultaneously. PPSS is ideal for such a setup. Since my Core 7i 920 emulates 8 logical cores, I can encode two regular DVD's to the iPhone format at a framerate of arount 100 fps, for each DVD, which is 200 fps combined!

I will also create another version of this script that will allow you to transcode your DVD's to mp4 format at the same resolution. This allows you to reduce the size of your video collection on disk considerably, without loss of quality.

Although Handbrake is available for Windows, you have to figure out yourself how to run a Bash script under Windows.

Visual representation of hard drives and their temperature

2009-09-12T16:53:00.000-07:00

If you build a NAS with many drives, it may be of interest to you which drives get hot and where they are located in the chassis. My Norco 4020 case has twenty drives in RAID 6, plus two operating system drives in RAID 1. I wrote a script that shows me the temperature of each drive, positioned in such a way that it represents the actual physical location of the drive in the chassis:

-- 30 -- | -- 27 --
| 26 | 27 | 28 | 25 |
| 30 | 32 | 30 | 29 |
| 31 | 33 | 33 | 29 |
| 33 | 32 | 35 | 30 |
| 32 | 34 | 35 | 31 |

In this example, you can determine that the top drives seem to stay the coolest. The center and lower drives get hotter.

If you want to use this script, you will have to change it for your own specific setup. You can get it here.

This visual representation would also be nice to identify which drive has failed and where it is located in the chassis.

Linux: obtain motherboard model / type and vendor

2009-08-30T14:16:00.000-07:00

If you want to know what motherboard is installed in a system, use the tool dmidecode:

dmidecode | grep -e "Manufacturer\|Product" | head -n 4 | tail -n 2

The result might be something like:

Manufacturer: ASUSTeK Computer INC.

Product Name: P5Q-EM DO

Manufacturer: ASUSTeK Computer INC.

Product Name: P6T DELUXE

Manufacturer: ASUSTeK Computer INC.

Product Name: M2A-VM

Manufacturer: Apple Computer, Inc.

Product Name: Mac-F4208EC8

Manufacturer: Compaq

Product Name: 0688h

lm-sensors: hardware monitoring with the w83627ehf module

2009-08-30T09:26:00.000-07:00

I have two systems that use the w83627ehf driver for hardware monitoring. However, if this driver is installed with a regular modprobe like:

modprobe w83627ehf

The result will be:

FATAL: Error inserting w83627ehf (/lib/modules/2.6.28-1-amd64/kernel/drivers/hwmon/w83627ehf.ko): No such device

I had this issue with the following mainboards:

Asus P5Q-EM DO (Core 2 duo)

Asus: P6T deluxe (Core 7i)

The solution is simple, as I found after googling for some time:

modprobe w83627ehf force_id=0x8860

The result:

w83627ehf-isa-0290

Adapter: ISA adapter

VCore: +0.88 V (min = +0.00 V, max = +1.74 V)

in1: +11.67 V (min = +0.00 V, max = +5.49 V) ALARM

AVCC: +3.34 V (min = +0.40 V, max = +0.03 V) ALARM

3VCC: +3.31 V (min = +1.15 V, max = +0.58 V) ALARM

in4: +1.65 V (min = +0.54 V, max = +0.22 V) ALARM

in5: +2.04 V (min = +0.71 V, max = +0.11 V) ALARM

in6: +3.79 V (min = +0.10 V, max = +2.30 V) ALARM

VSB: +3.38 V (min = +0.45 V, max = +2.43 V) ALARM

VBAT: +3.30 V (min = +0.67 V, max = +0.77 V) ALARM

in9: +0.00 V (min = +1.10 V, max = +0.57 V) ALARM

Case Fan: 0 RPM (min = 104 RPM, div = 128) ALARM

CPU Fan: 0 RPM (min = 42187 RPM, div = 32) ALARM

Aux Fan: 0 RPM (min = 162 RPM, div = 128) ALARM

fan4: 0 RPM (min = 42187 RPM, div = 32) ALARM

fan5: 0 RPM (min = 42187 RPM, div = 32) ALARM

Sys Temp: +40.0°C (high = +4.0°C, hyst = +0.0°C) ALARM sensor = thermistor

CPU Temp: +42.0°C (high = +80.0°C, hyst = +75.0°C) sensor = diode

AUX Temp: +30.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor

cpu0_vid: +0.000 V

Using ncat to provide SSL-support to non-ssl capable software

2009-08-21T08:49:00.000-07:00

Sometimes, people are using software that does not support encrypted connections using SSL. To provide SSL-support to such a client, ncat can be used. Ncat is part of nmap, the famous port-scanner.

The main principle is that the non-ssl capable software does not connect to the SSL-based service, but to the local host. Ncat will be listening on the localhost and will setup an SSL-connection with the SSL-based service on behalf of the non-ssl capable software.

This simple command allows an application to browse to port 80, and perform regular HTTP-request, while in fact, they are encapsulated within a SSL-connection:

ncat -l 80 -c "ncat (ip-address) (port) --ssl"

The -l option specifies the local port on which the SSL-tunnel will be listening. The ip-address and port refer to the SSL-based service.

So if the client connects to 127.0.0.1 on port 80 it will actually connect through the SSL-tunnel to the external service.

Often stunnel is used for this job but this software craps out on debian Etch with some error like:

SSL routines:SSL3_GET_RECORD:bad decompression

But ncat is an excellent alternative.

Howto get the hard disk size under Linux?

2009-08-18T14:19:00.000-07:00

Q: How do I obtain the capacity of a hard drive under Linux?

A: There is no single tool for this job, but it seems that Fdisk is just fine:

server:~# fdisk -l 2> /dev/null | grep Disk | grep -v identifier

Disk /dev/sda: 500.0 GB, 500028145664 bytes

Disk /dev/sdb: 500.0 GB, 500028145664 bytes

Disk /dev/sdc: 1000.1 GB, 1000123400192 bytes

Disk /dev/sdd: 500.0 GB, 500028145664 bytes

Disk /dev/sde: 500.0 GB, 500028145664 bytes

Disk /dev/hda: 80.0 GB, 80026361856 bytes

Disk /dev/md5: 1500.0 GB, 1500084240384 bytes

The nice thing about using fdisk is that it automatically lists the size of all block devices.

Here is a list of my NAS, mentioned earlier.

Beast:~# fdisk -l 2> /dev/null | grep Disk | grep -v identifier

Disk /dev/sda: 60.0 GB, 60011642880 bytes

Disk /dev/sdb: 60.0 GB, 60011642880 bytes

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes

Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes

Disk /dev/sde: 1000.2 GB, 1000204886016 bytes

Disk /dev/sdf: 1000.2 GB, 1000204886016 bytes

Disk /dev/md0: 57.9 GB, 57996345344 bytes

Disk /dev/md1: 2015 MB, 2015100928 bytes

Disk /dev/sdg: 1000.1 GB, 1000123400192 bytes

Disk /dev/sdh: 1000.1 GB, 1000123400192 bytes

Disk /dev/sdi: 1000.1 GB, 1000123400192 bytes

Disk /dev/sdj: 1000.1 GB, 1000123400192 bytes

Disk /dev/sdk: 1000.1 GB, 1000123400192 bytes

Disk /dev/sdl: 1000.1 GB, 1000123400192 bytes

Disk /dev/sdm: 1000.1 GB, 1000123400192 bytes

Disk /dev/sdn: 1000.1 GB, 1000123400192 bytes

Disk /dev/sdo: 1000.1 GB, 1000123400192 bytes

Disk /dev/sdp: 1000.1 GB, 1000123400192 bytes

Disk /dev/sdq: 1000.1 GB, 1000123400192 bytes

Disk /dev/sdr: 1000.1 GB, 1000123400192 bytes

Disk /dev/sds: 1000.1 GB, 1000123400192 bytes

Disk /dev/sdt: 1000.1 GB, 1000123400192 bytes

Disk /dev/sdu: 1000.1 GB, 1000123400192 bytes

Disk /dev/sdv: 1000.1 GB, 1000123400192 bytes

Disk /dev/md5: 18002.2 GB, 18002220023808 bytes

20 disk 18 TB RAID 6 storage based on Debian Linux

2009-07-21T14:08:00.001-07:00

This blog has been relocated to louwrentius.com

This is my NAS storage server based on Debian Linux, software RAID and 20 one terrabyte hard drives. It provides a total of 18 terrabytes of storage in one single array.

Case:	Norco RPC-4020
Processor:	Core 2 duo E7400 @ 2.8GHz
RAM:	4 GB
Motherboard:	Asus P5Q-EM DO
LAN:	Intel Gigabit
PSU:	~~Coolermaster 600 Watt~~ Corsair CMPSU-750HX 750 Watt (Coolermaster died)
Controller:	HighPoint RocketRAID 2340 (16) and on-board controller (6).
Disks:	20 x Samsung Spinpoint F1, 1 TB(array) and 2 x FUJITSU MHY2060BH (60 GB)
Arrays:	Boot 2x 60 GB RAID 1 and 20 x 1 TB RAID 6
RAID setup:	Linux software RAID using MDADM.
RAM:	4 GB
Read performance:	1.2 GB/s
Write performance:	350 MB/s.
OS:	Linux Debian Lenny 64-bit (backported)
Filesystem:	XFS (can handle > 16 TB partitions.)
Rebuild time:	about 5 hours.
UPS:	Back-UPS RS 1200 LCD using Apcupsd
Idle power usage:	about 220 Watt

RAID 6 array details:

/dev/md0:
Version : 00.90
Creation Time : Sat Jul 18 14:01:24 2009
Raid Level : raid6
Array Size : 17580292992 (16765.87 GiB 18002.22 GB)
Used Dev Size : 976682944 (931.44 GiB 1000.12 GB)
Raid Devices : 20
Total Devices : 20

HighPoint RocketRAID and staggered spinup with Samsung F1

2009-07-21T13:13:00.000-07:00

I have experienced problems using a HighPoint RocketRaid 2320 and 2340 when they are using 'staggered spinup' in combination with Samsung Spinpoint F1, 1 (one) terrabyte disks.

The problem is that the F1 disks spinup very slowly and often seem to 'hang' while making ticking noises that will scare any computer user to death. When, after ages, you have the luck that no disks keeps hanging during startup, the first thing to do is to disable staggered spinup.

However, if staggered spinup is not used and all disks will spinup together, please note that you will need a strong PSU to handle the short peak load. For example, starting up 20 (twenty) disks will briefly generate a load of 550 Watt on my APC ups, according to its display.

I estimate that during startup, each disk is roughly consuming 20 Watt. Take this all into account when deciding which type of disks and controllers you will be using.

8 TB RAID 6 Linux software RAID using EXT4

2009-07-05T15:45:00.000-07:00

Case: Norco RPC-4020

Mobo: Asus P5Q-EM DO ( 6x sata)
CPU: Core 2 Duo E7400
RAM: 4 GB
Controller; HighPoint RocketRAID 2314 (16x poorten)
PSU: CooMas Silent Pro M 600W ATX2 (definitely not overkill)
HD: Samsung 1 TB SAT2 HD103UJ (10x)
OS: Debian Lenny with custom kernel (backported)
FS: EXT4

RAID config: 10 disk RAID 6 based on Linux software RAID.
Performance: Read: 850 MB/s Write: 300 MB/s
Capacity: 7,5 GiB or 8 GB.

I am very pleased with the result. Points that should be improved are the temperature and noise. The PSU is a major overkill since this puppy draws less than 200 watt.

Bunny:~# ./show-hdd-temp.sh

Device Temperature

----------------------------

/dev/sda 32

/dev/sdb 34

/dev/sdc 33

/dev/sdd 36

/dev/sde 34

/dev/sdf 33

/dev/sdg 38

/dev/sdh 39

/dev/sdi 37

/dev/sdj 36

/dev/sdk 35

Personalities : [raid6] [raid5] [raid4]

md0 : active raid6 sdj[10] sdb[0] sdl[9] sdk[8] sdg[7] sdf[6] sdi[5] sdh[4] sde[3] sdd[2] sdc[1]

7813463552 blocks super 0.91 level 6, 64k chunk, algorithm 2 [11/11] [UUUUUUUUUUU]

[======>..............] reshape = 30.1% (294904524/976682944) finish=585.3min speed=19413K/sec

Script that shows ETA of RAID rebuild / reshape

2009-06-28T15:43:00.000-07:00

I made a small script that converts the output of `cat /proc/mdstat` to an actual date and time telling you when the RAID rebuild / reshape is finished.

This is the link to the correct version of the script.

Example:

debian:~# ./raid-rebuild-eta.sh

----------------------------------------------

Estimated time of finishing rebuild / reshape:

Mon Jun 29 00:45:07 CEST 2009

----------------------------------------------

Linux RAID 6 performance using software RAID

2009-06-28T12:32:00.001-07:00

So after toying around with RAID 0 just for fun, time to get serious. I created a RAID 6 of 10 x 1 TB disks. This gives me raw device read speeds of 850 MB/s and write speeds of 300 MB/s. I think this is exactly what should be expected, but boy it is damn fast. Especially the write speed surprises me.

Still, if I format the device using EXT4, write speed stays the same. However, read speed drop to about 350 MB/s. I don't understand why. Maybe some misalignment? I don't know.

UPDATE:

The issue with the dropped read speed is due to an incorrect chunk size of the array. The array now consists of 20 drives, providing 18 TB of storage using XFS. Read speads exceed 1.1 GB/s (that is not a typo) and write speeds are about 350 MB/s.

1.0 GB/s using Linux software RAID

2009-06-26T16:58:00.001-07:00

I filled the Norco case with hardware. It is now up and running, based on Debian Linux (Lenny).

I immediately performed some initial tests with software RAID 0. The results are just astounding.

debian:~# dd if=/dev/md0 of=/dev/null bs=1M count=50000

50000+0 records in

50000+0 records out

52428800000 bytes (52 GB) copied, 50.9222 s, 1.0 GB/s

This is correct. 1 Gigabyte per second using 10 Samsung Spinpoint F1's connected to the motherboard (4) and to the Highpoint Rocketraid 2340 (6).

Got myself a Norco RPC-4020

2009-06-16T12:47:00.001-07:00

I've got this fetish for storage. So I bought a case that gives me some room for future expantion. The current 6 TB RAID 6 storage server does not have any room for expantion.

This Norco RPC-4020 case with 20 hot swap drive bays does however. Don't know what to fill it with yet.

Bash Shell Function Library (BSFL) released.

2009-05-31T13:22:00.000-07:00

The Bash Shell Function Library (BSFL) is a small Bash script that acts as a library for bash scripts. It provides a couple of functions that makes the lives of most people using shell scripts a bit easier.

The purpose of this library is to provide pre-build functions for actions that often need to be performed, focussing on error-checking and logging. In this example, a test-script is written that demonstrates the functions the library provides.

The project can be found at this location.

Tip of the day for every Linux or Unix user: brace expantion

2009-05-02T15:24:00.001-07:00

Searching the web I discovered some really nice feature of the unix shell, which I didn't know about.

Try this:

touch foobar.conf

Now try this:

cp foobar.conf{,.bak}

It is equivalent to:

cp foobar.conf foobar.conf.bak

This is also the easiest way to create sequences. Do not use 'seq' since you cannot rely on it being installed.

bash-3.2$ echo {1..10}
1 2 3 4 5 6 7 8 9 10

Please visit this location for additional examples.

The Dirtiest Computer In The World

2009-04-30T11:20:00.001-07:00

I helped a family the other day with a malfunctioning computer. The system had a tendency to shutdown at random during games.

Hmm...

It didn't take long to discover why.

I think this is probably one of the filthiest computer in the world.

More photos at http://members.multiweb.nl/nan1/img/dirty/

Smoking doesn't only kill people, it seems.

Compatibility Highpoint RocketRAID 2320 and Samsung Spinpoint F1

2009-04-26T15:38:00.000-07:00

There are some reports about incompatibility between RAID controllers and Samsung Spinpoint F1 drives. I have no troubles with my 0.5 and 1.0 TB drives from Samsung using mentioned controller. See below:

Controller 1: RocketRAID 232x SATA Controller

------------------------------------------------

1/1/1 SAMSUNG HD103UJ 1000123MB, Normal

1/2/1 SAMSUNG HD103UJ 1000123MB, Normal

1/3/1 SAMSUNG HD103UJ 1000123MB, Normal

1/4/1 SAMSUNG HD103UJ 1000123MB, Normal

1/5/1 SAMSUNG HD501LJ 500028MB, Normal

1/6/1 SAMSUNG HD501LJ 500028MB, Normal

1/7/1 SAMSUNG HD501LJ 500028MB, Normal

1/8/1 SAMSUNG HD501LJ 500028MB, Normal

Automated install of Debian Linux based on PXE net booting

2009-04-25T09:30:00.000-07:00

Every honest and good system administrator is continue bussy with automating his work. For two reasons:

Repeating the same task over and over again is friggin boring. A system administrator has better things to do, such as drinking coffee.
Humans make mistakes, especially if boring. Computers do not.

If a computer can do a certain job, it wil do it always faster and better than a human. Automating system installation is both more time efficient and allows you to deliver a constant quality.

Netbooting or PXE booting

Regarding the installation of hosts, the holy grail of automated installation is netbooting or PXE booting. Almost every system today contains a network interface card that supports booting over the network. A system obtains instructions from the local DHCP server where to obtain an operating system kernel. This kernel is obtained using TFTP and then loaded. From then on, the operating system takes over and the installation continues, for example based on Debian preseeding and/or FAI.

How to prepare for netbooting

The following requirements must be met:

a DHCP server must be available
a TFTP server must be avaialble
the correct files for netbooting must be in place

Configuring the DHCP server

The following two lines must be added to the 'subnet' section of your DHCP server configuration.

filename "pxelinux.0";

next-server 10.0.0.1;

The 'next-server' section specifies the IP-address of the system that is running the TFTP server, so change it based on your configuration, this is just an example.

Don't forget to restart the DHCP server daemon.

Configuring the TFTP server

First, make sure you install "tftpd-hpa" since the standard "tftpd" server does not seem to support the "tsize" option. Then, edit /etc/defaults/tftpd-hpa like this:

RUN_DAEMON="yes"

OPTIONS="-l -a -R 30000:30100 -s /var/lib/tftpboot"

Do not run the TFTP server from inetd because the above lines provide more control over how the server behaves, especially in regard to firewalls.

The -R option specifies the port-range used for data transfers. This port range should also be configured within your firewall configuration. Watch out! Do not allow TFTP access from the Internet. TFTP requires NO authentication and is very insecure.

Start the TFTPD server with:

/etc/init.d/tftpd-hpa start

Install the files required for netbooting

The fun thing is that Debian provides a complete package for netbooting. So cd to /var/lib/tftpboot and enter:

wget http://ftp.debian.org/debian/dists/lenny/main/installer-i386/current/images/netboot/netboot.tar.gz

Then extract the contents of netboot.tar.tz like:

tar xzf netboot.tar.gz

That is all there is to it. If you start a host and make it boot using PXE, it will show you the regular installation menu that is also shown when a system is booted from a regular Debian installation CD-ROM.

However, if you want automated installation and not use this boot menu, first cd to:

/var/lib/tftpboot/debian-installer/i386/boot-screens

Then edit syslinux.cfg and comment this rule out:

default debian-installer/i386/boot-screens/vesamenu.c32

If you want to use preseeding, first edit adtxt.cfg and goto label auto. Edit label auto like this:

label auto

menu label ^Automated install

kernel debian-installer/i386/linux

append auto=true priority=critical vga=normal initrd=debian-installer/i386/initrd.gz url=http://(IP-address)/preseed/preseed.cfg -- quiet

The IP-address section should point towards the preseed server that is hosting the preseed configuration file.

Last, edit txt.cfg. Change 'default install' to:

default auto

I always though that PXE booting was a pain to setup. However, I got it working within 60 minutes using this howto.

FFmpeg performance on a Core i7 920 @ 3.6 Ghz

2009-04-22T16:10:00.000-07:00

The system i'm running is a Core i7 920 @ 3.6 Ghz.

I am transcoding a DVD (Grave Of The Fire Flies) to iPod format (640x480 x264).

Thread support is enabled, to FFmpeg uses about 250% CPU. That's 2.5 of the 4 cores available. If possible, I would have liked to see it use all four to the max.

Any way. I use these settings:

FFmpeg version SVN-r18628

ffmpeg -i $1 -pass 1 -acodec libfaac -ab 128k -ac 2 -vcodec libx264 -vpre normal -vpre ipod640 -s 640x480 -b 512k -bt 512k -threads 0 -f mp4 $2

With these settings, I get an encoding speed of fps=193.

I don't know how that stacks up against other systems. It seems fast to me though. In effect, the system is playing the DVD at 7.6 times the speed of the movie (25 fps?). So encoding is 88 minutes / 7.6 = 11.45 minutes for encoding a DVD to iPod x264.

How to escape file names in bash shell scripts

2009-03-29T14:49:00.000-07:00

After fighting with Bash for quite some time, I found out that the following code provides a nice basis for escaping special characters. Ofcource it is not complete, but the most important characters are filtered.

If anybody has a better solution, please let me know. It works and it is readable but not pretty.

FILE_ESCAPED=`echo "$FILE" | \
sed s/\\ /\\\\\\\\\\\\\\ /g | \
sed s/\\'/\\\\\\\\\\\\\\'/g | \
sed s/\&/\\\\\\\\\\\\\\&/g | \
sed s/\;/\\\\\\\\\\\\\\;/g | \
sed s/$/\\\\\\\\\\(/g | \
sed s/$/\\\\\\\\\\)/g `

Distributed Parallel Processing Shell Script (PPSS) released

2009-03-12T15:05:00.000-07:00

I'd like to announce the release of the distributed version of the Parallel Processing Shell Script (PPSS). PPSS is a bash script that allows you to run commands in parallel. It is written to make use of current multi-core CPUs.

The new distributed version of PPSS allows you to run PPSS on multiple hosts, that simultaneously work on a single group of files or other items. A central server is used for file locking. This way, nodes know which files are 'locked' by other nodes and/or have already been processed.

PPSS is written to be very easy to use. You can be up and running on a single host within 5 minutes. Distributed usage requires the (one-time) setup of some SSH accounts on your nodes and server, but that's about it. Please take a look at distributed PPSS for yourself at:

http://code.google.com/p/ppss

I used distributed PPSS to encode 400 GB of WAV files to MP3 using 4 computer systems. A total of 14 cores where available to PPSS to encode those files using Lame. However, PPSS is not limited to encoding WAV files. To the contrary, it is written to execute whatever command you want to execute. Gzip a bunch of files in parallel? No problem. It is totally up to your imagination.

Core i7 920 @ 3,6 Ghz is a true beast!

2009-03-04T13:19:00.000-08:00

Even today, Core 2 Duo processors clocked at 2 ghz are no slugs. However, the Core i7 920 is of a different kind. First, it is not only clocked at a higher speed (default 2,8 Ghz), it is also a quad-core processor. Thanks to the re-introduction of hyperthreading, this processor can handle 8 parallel proceses simultaneously.

Just how fast a Core i7 can be, especially if overclocked to 3,6 Ghz shows this diagram:

Using my still in development version of PPSS, four systems processed 400 GB of WAV files and converted them to MP3. This simple pie-chart shows that the Core i7 on it's own, using 8 parallel processes, managed to process 2/3 of the files. The Core i7 was way faster than the other 3 systems combined! This is marvelous, I think. And it seems all due to Hyperthreading. If an additional duo core system would have been added, the other systems combined would have also 8 parallel threads available and would have processed roughly 50% of the items. However, please note that the Core i7 is a quad core processor and has 'only' four physical cores...

Linux: unattended installation with Debian preseeding

2009-02-22T11:38:00.000-08:00

Debian Linux provides a mechanism to install the operating system without user intervention. This mechanism is called 'preseeding' and is similar to Red Hat Kick Start and Sun Solaris Jump Start.

The basic idea is that the installer is fed a recipe, according to which the system is installed. This recipe can be fed by a floppy, usb stick, cdrom, or through a web server over the network. To use such a recipe, just boot from a Debian CD-rom and issue the following command:

Floppy based: (you really shouldn't be using those anymore)

Boot: auto file=/floppy/preseed.cfg

USB stick based:

Boot: auto file=/hd-media/preseed.cfg

Network based:

Boot: auto url=http://internal.web.server.com/preseed.cfg

The only work you have to do is to create a preseed configuration file. This is really simple, since preseeding is well-documented and preseed configuration files are easy to understand.

d-i debian-installer/country string US

d-i debian-installer/locale string en_US.UTF-8

d-i mirror/country string manual

d-i mirror/http/hostname string ftp.uk.debian.org

d-i mirror/http/directory string /debian

base-config apt-setup/hostname string ftp.uk.debian.org

base-config apt-setup/directory string /debian

As you can see, it is just a text-based file that configures some variables that are used during installation. It is basically an answer file. Questions that are asked by the installer during installation are answered with the preseed file.

For a full example, take a look here.

Very extensive documentation can be found here.

A minimal debian installation without support for X can be installed within 2.5 minutes, assuming a network-based installation (tested in VMware Workstation).

Please note that if your company uses Debian Linux not only for servers but also for desktops / laptops, preseeding is an ideal solution to provide your users with a new and fresh installation whenever they want. Users or sysadmins shouldn't be bussy manually installing these systems.

I have implemented Debian Preseeding to create a fully unattended and automated installation of laptops, based on LUKS full disk encryption, which is supported by the Debian installer (!), with all required software installed. All additional software is installed with a custom installation framework based on shell-scripts. The installation framework makes sure that if anything goes wrong during installation, it is noticed.

Unattended installation allows system administrators to quickly deploy new installations and guarantee that such installations are 100% correct. They rule out the human factor, which tends to introduce random errors. So take a look at Debian Preseeding and decide for yourself how useful it is.

Why Debian/Ubuntu Linux is to be preferred

2009-02-16T11:38:00.000-08:00

There are many Linux distributions around. However, I always come back to just one: Debian. The reason why so many people use Debian is the same reason I like it so much: software management. With good old apt-get or the new aptitude, software is installed within minutes. Due to the vast amount of software available even the most obscure software can be installed without resorting to manually downloading and compiling.

But the most important aspect of Debian is it's mantra of stability. It is build for servers. For people who don't want to take risks and prefer stability and security above anything else. This is also the main gripe most people have about Debian: it is often not very up-to-date regarding drivers or the latest software versions. If that is a problem, there is still the possibility to run the testing branch of Debian, exchanging the risk on things getting broken or unstable for the availability of newer software.

As a part-time system administrator, one of the most ideal components of Debian is its installer. Especially the "preseeding" bit. Preseeding is for Debian what Kickstart is for Red Hat and Jump start is for Sun Solaris. It allows a full unattended installation of Debian Linux on any hardware without ever touching your keyboard. This isn't new, but it is much more user friendly as opposed to, for example, kick start.

Debian Preseeding is very well documented and can easily be extended to run your own scripts after installation for some post-configuration.

I currently use it to install hosts by booting them with an USB stick and using a network install. Not only are network installs often the fastest solution, assuming that a local Debian mirror is available, the system is also direct up-to-date.

Abount preseeding:

http://d-i.alioth.debian.org/manual/en.i386/apb.html

http://wiki.debian.org/DebianInstaller/Preseed

About setting up a local Debian mirror (requires about 50 GB of storage space on a web server)

http://www.howtoforge.com/local_debian_ubuntu_mirror

Why I still won't switch to Linux and keep my Mac

2009-02-01T14:49:00.000-08:00

The current state of Linux is amazing. If we take a look at, for example, Ubuntu Linux, we have to admit that the Linux desktop is really becoming a nice, user-friendly environment. I'm truly starting to like what I see. I considered whiping Mac OS X from my macbook, but there are some reasons why I won't.

- I like to mess around in Adobe Photoshop now and then. There is no serious Linux alternative.

- I've got an iPhone, so now i'm 'stuck' with iTunes. I really like my iPhone btw.

- I really like iPhoto, it is very easy to use and archive my photo's.

- My entire music collection is in iTunes -> especially the song ratings are important.

Maybe those reasons can be overcome. But for day-to-day usage, I still prefer Mac OS X above Linux.

Unattended automatic installation of Linux nvidia binary driver

2009-01-13T02:35:00.000-08:00

As part of an unattended installation, it was necessary to install a binary nvidia graphics driver. This is a manual proces by default. However, it can be done fully automatic:

*prerequisite:* install xserver-xorg-dev package or similar xorg development package.

sh NVIDIA-.run -q -a -n -X -s

That's all there is to it. The -q option means quiet, the -a option means accept licence, the -n action suppresses questions, the -X option updates the xorg.conf file and the -s option disables the ncurses interface.

Release of PPSS - the Parallel Processing Shell Script

2009-01-02T17:51:00.000-08:00

PPSS is a shell script that processess files or other items in parallel. It is designed to make use of the current multi-core CPUs. It will detect the number of available CPUs and start a thread for each CPU core.

This script is build with the goal to be very easy to use. Also, it must be robust (atomic), and portable. It employs a locking mechanism and runs on Linux and Mac OS X. It should work on other Unix-like operating system that support the bash shell.

Please visit http://code.google.com/p/ppss/ for a download.

Benefits of hyperthreading on a Core 7i processor

2008-12-27T16:26:00.001-08:00

I had some wave files that I wanted to encode to mp3. I wrote a small parallel processing framework for this job. It executes parallel jobs so I can benefit from the 4 cores + 4 virtual cores of my new Core 7i 920 processor.

The framework is based on some shell scripts ru

nning on Linux (or probably any other unix) and Lame.

I wanted to test how much impact hyperthreading has on parallel performance. Hyperthreading has a bit of a bad reputation, dating back from the P4 erra. Lots of 'hype' but no significant performance gain.

I ran some test with hyperthreading enabled and disabled (within the bios).

In this test I used 16 wav files and encoded them to mp3 with Lame under Linux. The data in the table are raw measurements.

As you can see, the benefit of HT is there and can improve processing speed with up to 24 percent.

As more threads are run simultaneously, processing speed is significantly improved comparted to non-HT.

Rebooting results in degraded RAID array using Debian Lenny

2008-12-24T08:49:00.000-08:00

As described earlier, I setup a RAID 6 array consisting of physical 1 TB disk and 'virtual' 1 TB disks that are in fact two 0.5 TB disks in RAID 0.

I wanted to upgrade to Lenny because the new kernel that ships with Lenny supports growing a RAID 6 array. After installing Lenny the RAID 0 devices were running smootly, but not recognised as part of the RAID 6.

So the array was running in degraded mode. That is bad.

In Lenny, a new version of mdadm is used that requires the presense of the mdadm.conf file. The mdadm.conf file contains these lines:

#DEVICE partitions

#DEVICE /dev/md*

After I uncommented the "DEVICE /dev/md*" line and generated a new initramfs file with:

update-initramfs -u

The RAID 0 drives were recognised as part of a RAID array and everything was OK again. So mdadm must be instructed to check if /dev/md? devices are a member of a RAID array.

I guess this is also relevant if you are running a RAID 10 based on a mirrored stripe or a striped mirror.

Calculating EXT2 EXT3 EXT4 stride size when using RAID

2008-12-20T14:15:00.000-08:00

When formatting a RAID device with an EXT filesystem, it is always advised to specify a stride size. The format utility will take this stride size into account when formatting a device. The stride size is the number you get when you divide the 'chunck' size, as specified with MDADM by the filesystem block size (almost always 4K).

So a 128 KB chunck size gives you a stride of 32. A nice and simple utility to calculate your stride can be found here:

http://busybox.net/~aldot/mkfs_stride.html

Please note that the stripe-with option does not seem to work on Debian Etch. Maybe because that option is too old or too new.

Did you know that RDP is as insecure as telnet?

2008-11-21T01:47:00.000-08:00

Microsoft developed the remote desktop protocol in order to allow remote GUI-based access to hosts. It is easy to pick on Microsoft and I do respect what they have accomplished, however, it may not come as a surprise that they did it all wrong with RDP in terms of security.

By default, out-of-the-box, a server or desktop supports RDP with RC4 encryption. Most recent servers and clients support 128-bits encryption. Sounds secure, right? Well, it isn't unfortunately. There are two major security weaknesses that affect the remote desktop protocol.

1. Keystroke vulnerability

The original , RDP version 4.0, is based on the T.128 protocol. Input events, such as keystrokes, are sent to the server with a unique timestamp as specified in the T.128 standard. This means that each message containing an input event is unique and so will be the checksum generated from this input event data. In other words: if two RDP messages contain the same data, they will still be unique because of the timestamp. Microsoft did it right in RDP version 4.0, but there was one drawback: the timestamp generated overhead, making RDP messages relatively large.

So in RDP version 5.0, Microsoft decided to drop the timestamp. Thus, if two RDP messages contain the same data, their checksum will be identical. So if the same key is pressed twice, the two packets that are generated will contain different encrypted data segments but the checksums will be identical. Although it is not clear which key is pressed, if a larger data set is analyzed, patterns can be discovered and an accurate guess can be made to wich keys are pressed.

In a nuttshell an attacker that has access to the data stream of an RDP session will be able to decode keystrokes and thus be able to:

- obtain your username and password, allowing access to the system!
- read what you type.

Please visit the following location as it provides a more detailed explanation of this vulnerability:

http://www.xatrix.org/article.php?s=1943

2. Man-in-the-middle-attack

At first, back in 2003, RDP-clients did not verify the identity of the RDP server. So an attacker can easily spoof an identity and perform a man-in-the-middle-attack. Microsoft attempted to fix this vulnerability but did not succeed. Although more recent RDP-clients do verify the identify of the server, it is still possible to spoof this identity. Microsoft uses a fixed hard-coded key to identify the server with. This key resided in a system DLL that is available to the general public. An attacker can easily obtain this key, create a fake identity and still prove to be legitimate since the identity is signed with a trusted key.

More detailed information:
http://www.oxid.it/downloads/rdp-gbu.pdf

Conclusion

Given these vulnerabilities, RDP is, in its default configuration, virtually as insecure as plain old telnet. Think about that the next time you logon with administrator credentials using RDP.

Tool that implements the attacks

The well-known hack-tool Cain & Abel has a build-in RDP man-in-the-midlle implementation. It automatically decrypts (parts of) RDP data packets and reveals the key strokes that are send to a server. So if you don't believe this story and just want to see for yourself how easy such an attack really is, download this tool and test it for yourself.

To defend against the attacks

To protect against this attack, it is strongly advised to implement RDP based on TLS/SSL encryption. Ofcourse, you'll have to install a server-side SSL-certificate and it may be necessary to obtain an official (as in not self-) signed certificate.

http://technet2.microsoft.com/windowsserver/en/library/a92d8eb9-f53d-4e86-ac9b-29fd6146977b1033.mspx?mfr=true

So please understand the risks of using RDP, especially over the Internet. If TLS is for some reason not an option, use an encrypted VPN-connection to tunnel the RDP-session in, as an added layer of protection. Never use RDP in its default configuration over the interne without additional protection.

EDIT: IN RDP version 6.0 the man-in-the-middle-attack is no longer possible!

EDIT2: Configure your terminal server to use FIPS compliant encryption. It will use Triple DES encryption even if TLS is not enabled.

After you enable this setting on a Windows Server 2003-based computer, the following is true:
The RDP channel is encrypted by using the 3DES algorithm in Cipher Block Chaining (CBC) mode with a 168-bit key length.
The SHA-1 algorithm is used to create message digests.
Clients must use the RDP 5.2 client program or a later version to connect.
So you don't need to use TLS if you want to use a more secure algorithm than RC4.

The kind of sound you never want to hear from your computer

2008-11-14T13:16:00.000-08:00

The sounds of hard drives that have perished

These sounds are only frightening or scary if you imagine your precious data only exists on that failing drive. If you make consistent and frequent backups and/or you run a fault-tolerant RAID flavour, it is but a nuisance.

Is storage really that cheap?

2008-10-05T14:47:00.000-07:00

Nowadays you can buy a 1 TB harddrive for less than 100 euro's . So I build myself a 4 TB NAS box, which is already 50% full. However, although it is to some degree fault-tollerant by using RAID 6, one mistake or catastrophic hardware faillure and all data is lost.

And that's where the 'problem' start. For every € spend on storage, you may need another € to secure that storage.

You can choose to take and accept the risk outlined earlier and not to make backups. However, if you do want to make backups of terrabytes of storage, how are you going to pull that off without too much cost?

In my opinion, the only reliable and usable solution is to build a second NAS box and sync the two. Ideally, the machines reside at different locations, but hey, I'm talking about a home solution, not a professional environment. Although you might ask why on earth you need 4 TB of space at home.

Anyway, the point I'm trying to make is that although storage in itself is cheap, if you want to make backups and really keep your data safe, storage is far more expensive than you might think.

Highpoint RocketRAID 2320 on Debian howto

2008-09-28T01:06:00.000-07:00

- Get the 'open source' driver from www.highpoint-tech.com. (It's not open source, it uses a closed binary driver.)

http://www.highpoint-tech.com/USA/bios_rr2320.htm

- Install the kernel headers if you haven't already

apt-get install linux-headers-2.6.18-6-k7

- extract the downloaded driver with tar xzf

- cd into the product/rr232x/linux directory

- type 'make'

As of 28 september 2008, this gives no errors.

However, a 'make install' will fail. This is cause by the fact that the installation script uses mkinitrd, which is obsolete: the mkinitramfs command should be used. So we have to edit a script.

Update Dec 2009: the latest drivers do not need the following steps anymore.

- vi osm/linux/install.sh

- replace all occurrences of mkinitrd with mkinitramfs ( press : and type %s/mkinitrd/mkinitramfs/g and then :wq)

Continue here:

- Go back to the product/rr232x/linux directory and type 'make install'.

That's it.

Server status on your iPhone

2008-09-12T05:19:00.000-07:00

Totally useless but fun nonetheless. I made an iPhone-compatible webpage showing the current status of my storage server. It is based on some shell scripts and some python, using an iPhone-specific website template.

Building a RAID 6 array of mixed drives

2008-08-10T04:36:00.000-07:00

To be honest, 4 TB of storage isn't really necessary for home usage. However, I like to collect movies in full DVD or HD quality and so I need some storage.

I decided to build myself a NAS box based on Debian Etch. Samba is used to allow clients to access the data. The machine itself was initially based on 4 x 0.5 TB disks using the four SATA ports on the mainboard. With Linux build-in support for software RAID, I created a RAID 5 array, giving me 1.5 TB of storage space. Since a single movie is around 4 GB, the 1.5 TB turned out to become rather tight.

So I bought 4 x 1 TB disks and a Highpoint RocketRaid 2320 controller (SATA 4x). I put all 8 disks on this controller.

I wanted to create a single RAID 6 array using both the 1 TB disks and the 0.5 TB disks. I didn't want to create two separate array's because although it would have provided additional space, it wouldn't have given me the same safety level as RAID 6 does.

I mainly chose for RAID 6 since I cannot afford a backup solution for this amount of data. I'm aware that RAID is no substitute for a proper backup, but it's an accepted risk for me.

Using both 1 TB disks and 0.5 TB disks, how to create a RAID 6 array using different drive sizes? The solution is fairly simple. Just put two 0.5 TB disks together in one RAID 0 volume and you'll have a 'virtual' 1 TB disk. Since I had four 0.5 TB disks, I could create 2 'virtual' 1 TB disks.

The only downside is that I had to skim a little bit of storage capacity of the native 1 TB drives, because 2 x 0.5 TB provides slightly less storage space than a single 1 TB disk. We're talking about something like 50 MB here, so It's not a big deal in my opinion.

The funny thing is that this array actually performs rather well. The disks are connected using a HighPoint RocketRaid 2320 controller. This controller is used just for it's SATA-ports, the on-board RAID functionality is not used. For RAID, I use Linux software RAID, using mdadm. This is how the RAID 6 array looks like:


server:~# mdadm --detail /dev/md5
/dev/md5:
    Version : 00.90.03
Creation Time : Thu Jul 24 22:40:26 2008
 Raid Level : raid6
 Array Size : 3906359808 (3725.40 GiB 4000.11 GB)
Device Size : 976589952 (931.35 GiB 1000.03 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 5
Persistence : Superblock is persistent

Update Time : Sun Aug 10 15:36:18 2008
      State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

 Chunk Size : 128K

       UUID : 0442e8fa:acd9278e:01f9e43d:ac30fbff (local to host server)
     Events : 0.14170

Number   Major   Minor   RaidDevice State
   0       8        1        0      active sync   /dev/sda1
   1       8       17        1      active sync   /dev/sdb1
   2       8       33        2      active sync   /dev/sdc1
   3       8       49        3      active sync   /dev/sdd1
   4       9        0        4      active sync   /dev/md0
   5       9        1        5      active sync   /dev/md1

And this is how this array performs:

server:~# dd if=/storage/test.bin of=/dev/null bs=1M
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 45.7107 seconds, 229 MB/s
server:~# dd if=/dev/zero of=/storage/test.bin bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 81.0798 seconds, 129 MB/s

With 229 MB/s read performance and 129 MB/s write performance using RAID 6, I think I should be content.

Why RAID 1, 5 and 10 will kill you some day

2008-08-02T01:55:00.000-07:00

How important is availability of an information system to you and your company? What are the costs of, let's say, a couple of hours downtime and maybe the loss of all the work since the last backup?

Depending on the information system, the impact can be quite grave, I presume. So what are the biggest risks regarding the availability of your systems? Human error is probably number one. Number two might be the hardware.

One of the most unreliable components of the hardware on which your precious information systems run is the good old hard drive. There are not two but three certainties in life: death, taxes and that sooner or later hard drives will fail.

So back in the eighties (some patent was already awarded back in 1978) some smart people invented RAID. Using RAID, your information system can tollerate a disk faillure, and still continue to operate.

There are many different tastes of RAID, so called RAID levels. One of the most populair RAID levels is RAID 1. Two disks acting like 1. If one fails, the other takes over. For performance, you can stack them together and you get RAID 10 arrays. However, 50% of your storage space is waisted because for every n of storage, you need (n / c ) * 2 disks, where c represents the capacity of a single drive.

RAID level 5 is a more efficient solution. Using this RAID level, only the capacity of one disk is lost in order to provide redundancy. So for every n of storage, you need ( n / c ) + 1 disks. It is easy to see that for larger arrays with more disks, RAID 5 is much more efficient. The downside of RAID 5 is mainly (write) performance, if compared to RAID 10. However, if it is sufficient, that is often not an issue. Hence the popularity of RAID 5.

This story is all about risk vs. costs. And there is a risk using RAID 1 and 5 that can not be neglected that should be pointed out. If a drive fails, redundancy is lost. At that moment, until the faulty drive is replaced you will run the risk of losing the entire RAID array and all data if another disk would fail.

How big is that risk? Well, that is the weakest point of this article. I honestly don't know. There is some anecdotal "evidence" that it occurs occasionally. And it is not that surprising: restoring an array puts extra stress on all de disks involved, which might be fatal for a second drive.

Today, RAID arrays of 10+ disks are not a rarity. With that amount of drives, it wouldn't be surprising if, during recovery, a second drive would fail. It's easy: with a 10-disk array the chance that a disk fails is twice that of a 5 disk array.

The most common solution is to revert back to RAID 10. RAID 10 consists of disk pairs concatenated to one big virtual disk. RAID 10 can tollerate up to 50% loss of drives if one member of every pair would fail. The caveat is obvious: if a disk fails and the other drive of that pair will fail during recovery, the whole array will be lost. However, compared with RAID 5, the risk is reduced. In degraded mode (non-redundant) any drive failure will destroy a RAID 5 array. RAID 10 can tollerate additional drive failures as long as it is not the drive of the pair that just already lost one.

So, although the risk that a second drive failure might destroy your array is greatly reduced using RAID 10 (compared to RAID 5), there is still a risk that the array is lost is the 'wrong' drive fails.

So the solution should be that redundancy is not lost if a single drive failure occurs. RAID 6 provides that solution. RAID 6 is in nature identical to RAID 5. However, an additional drive is sacrified for additional redundancy. So for every n of storage, you need ( n / c ) + 2 disks. If you need 10 TB of storage using 1 TB disks, you need 12 disks. If a disk fails, the array is still redundant. Even a second drive can fail and the array will still continue to operate. I think that the chance that a third drive would fail is so low that it is an accepted risk.

For smaller arrays, the risk of a double drive failure might not that high to justify RAID 6, but with larger arrays (more drives) RAID 6 might become a necessity.

So there you have it. With current costs of hard drives and the wide support for RAID 6, it is an option that should be taken into account when designing the hardware platform for an information system.

Aftertought: this article is mainly about considering RAID 6 in stead of RAID 5. Raid 5 or 6 may often not be a solution if performance in terms of IO (input/output) is an issue. Please note that when running in degraded mode (a drive failure occurred) the performance penalty on RAID 5 and RAID 6 will be severe (may be 80%). RAID 10 will suffer far less in that regard.