This blog has been moved to louwrentius.com

I decided to move away from blogger and host my own blog at another location. The new location for my blog is:

http://louwrentius.com

Visit my new blog for a motivation.

All comments have also been migrated.

LFS - Linux Firewall Script released

I started a small new Google project for a new script I wrote called LFS. It stands for Linux Firewall Script.

I run a small Linux box as an internet router that doubles as a firewall. The firewall is configured using iptables. In my opinion, iptables is not the easiest tool to use and may have a steep learning curve for people new to it.

The goal of LFS is to provide an easier interface to iptables. It also adds some features that by default are not or difficult to setup using only iptables. The most important additional feature is the use of objects and  groups. Object groups can be used to make a single rule affect multiple hosts, networks or services.

LFS uses a single configuration file which contains the firewall rules. Rules look like this:

nat "$INTERNAL_NETWORK" "$EXTERNAL_IP" "$NAT_INTERFACE"
port_forward "$EXTERNAL_IP"  "$INTERNAL_HTTP_SERVER" "80/tcp" "8080/tcp"

Please visit the project page for some examples.

http://code.google.com/p/lfs/

Why filtering DHCP traffic is not always possible with iptables

When configuring my  new firewall using iptables, I noticed something very peculiar. Even if all input, forward and output traffic was dropped, DHCP traffic to and from my DHCP server was not blocked even if there were no rules permitting this traffic. 

I even flushed all rules, put a drop all rule on all chains and only allowed SSH to the box. It did not matter. The DHCP server received the DHCP requests and happily answered back. 

How on earth is this possible? In my opinion, a firewall should block all traffic no matter what. 

But at least I found out the cause of this peculiar behaviour.  The ISC DHCP daemon does not use the TCP/UDP/IP stack of the kernel. It uses RAW sockets. Raw sockets bypass the whole netfilter mechanism and thus the firewall. 

So remember: applications using RAW sockets cannot be fire walled by default. Applications need root privileges to use RAW sockets, so RAW sockets thankfully cannot be used by arbitrary unprivileged users on a system, but never the less. Be aware of this issue. 

Please understand that if a serious security vulnerability is found in the ISC DHCP daemon, you cannot protect your daemon with a local firewall on your system. Patching or disabling would then be the only solution.

Belkin Gigabit USB 2.0 adapter works perfectly with Linux

My ISP upgraded my internet connection speed to a whopping 120 Mbit. I am using a mac mini as my internet router. As you may be aware, the mini has only one network interface, so I added a second interface using a USB to ethernet adapter. This adapter was limited to 100 Mbit, so to make full use of the 120 Mbit connection, I had to upgrade this adapter.

I took the gamble and bought the Belkin Gigabit USB 2.0 adapter. I could not figure out if it would work with Linux, but on the box it officially supports Mac OS X, which is always a good sign.

This adapter is recognized by Debian Linux without a hitch:

Mini:~# ethtool -i eth0

driver: asix

version: 14-Jun-2006

firmware-version: ASIX AX88178 USB 2.0 Ethernet

bus-info: usb-0000:00:1d.7-5

lsusb output:


Bus 005 Device 004: ID 050d:5055 Belkin Components F5D5055 Gigabit Network Adapter [AX88xxx]

I did not test the actuall performance of this adapter, but it at least goes beyond the 100 Mbit (it does 120 Mbit at least). I expect it to be limited at say max 300 Mbit, being constrained by the maximum speed of  USB 2.0. 

The minimum requirements for a secure system

The most secure server system is a system that is not connected to a network and turned off. However, little work seems to be getting done this way. So we want to turn systems on and connect them to a network, or even (God forbid) the internet.

The thing is this. A system connected to a network without any running services is almost as as secure as a system that is turned off. They also share a common property: they are useless. A system starts to get useful if you start running services on them. And make these services accessible from the network for clients.

Services

Security on a technical level is all about securing those services. Every service that you enable is an opportunity for an attacker to compromise your system. If a service is not installed or running on your system, it cannot be used to compromise your server. 

If a service is enabled and accessible through the network, it is logically of vital importance that you know:

1. what does this service do?
2. what can it be used for?
3. what steps needs to be taken to properly secure it?

If you know what a service does, you can understand the potential security risks. If you understand the product you are using, you can secure it properly. Security is all about understanding. If you don't understand what you are running, then it can't be secure.

Firewalls

So if you only run required services, why do you need to run a firewall? You don't. Yes that's right. Think about it. A firewall protects services that should not be accessible and allows access to services that should be accessible. If you just disable those services that should not be accessible from the outside, why use a local firewall? You don't want the Internet to access the SNMP-service on your system, you say? But then why not bind it only to the management interface instead of the production interface? You have a separate management network, right?

Of course, firewalls are a good thing. They are an ADDITIONAL line of defense. They mostly protect you against yourself. If you make a mistake and, by accident, enable some vulnerable service on a system, a properly configured firewall will prevent access to it and save your behind. That is the purpose of a firewall.

People often wrongly see the firewall as the first line of defense. If you do, you are wrong. The first line of defense is to secure your services. 

The whole point is that there are holes in your firewalls. Those holes allow access to services. Those services may be necessary, like a web server, but nevertheless holes. You are exposing services to the Internet. 

Web applications (or web-based back doors?)

We are now mostly running web-based applications on the services that we make accessible for the network or the internet. Those applications run on application servers. Yes, these application servers, like Apache Tomcat or IIS ASP.NET need to be secured, but nowadays, they are almost secure by default. 

All security depends on the level of security of the application you are running on your application server. Is your application written well, with security principles in mind? Does it protect against SQL-injection or cross-site scripting? Are sessions predictable? Can a user access data of another user?

Firewalls don't protect against vulnerabilities in your web applications. You need to do it right at the core level: the application itself. Just like how you harden a system. You must run secure code.

And be aware that if you run third-party code, watch out for security news. There have been many worms exploiting vulnerable commodity software such as phpBB, Wordpress or similar products.

This is the really hard part. Deploying secure software and keeping it secure during the development life cycle.

Patches
The last fundamental principle of keeping systems secure is keeping up with security patches. Many security vulnerabilities are often only exploitable under specific conditions and may not be that important. But the most important thing is to be aware of vulnerabilities and available patches. Then you can decide for yourself how to act.

There is always a risk that a security patch breaks functionality. But that's not a real problem, because you have this test environment so you can check first, right?

Keep up with security patches and non-security patches. If you first have to install 100+ patches to be able to install the latest high-risk security patch, something might break. So then it's choosing between staying vulnerable or going off-line until you have fixed everything.

Conclusion

So what are the most basic ingredients for secure systems?

1. only run required services
2. harden those required services
3. deploy a firewall as an additional defense layer
4. deploy secure application code
5. keep up-to-date with security patches
6. Audit and review your systems and application code on a regular basis.

With this small number of steps, you will be able to protect against a lot of security threats. I don't say this is everything that is necessary. But it is a good foundation to build on. You still have to identify risks that may apply to your particular situation. These risks may require you to take (additional) measures not discussed here. 

Linux: using disk labels to counter storage device name changes

My router decided to change the device name for some USB storage devices.  So /dev/sdc was swapped for /dev/sdd and vice versa. The result was some file system corruption on /dev/sdc, because it was used on a remote system through iSCSI, using a different file system from /dev/sdd.

With regular internal disks, attached with PATA, SATA or SAS, the chances are very small that such an event will occur, but it is possible, especially if you start adding/subtracting disks. With USB devices the risk is substantially bigger.

To prevent your system from mixing up drives because there device names change, use file system labels.  All information that follows have been stolen from this location. Since this blog is also my personal notepad, the relevant bits are reproduced here.

There are three steps involved, the third being optional:

1. add a label to the file system
2. add the label to /etc/fstab
3. update grub boot manager (optional)

1. Add a label to the file system


Setting a label when the file system is created

mkfs.ext3 -L ROOT /dev/sda1
mkfs.xfs -L BIGRAID /dev/sde

Set label for existing file system

EXT3:
e2label /dev/sda1 PRIMARY_ROOT
e2label /dev/sda1


XFS:
xfs_admin -L DATA1 /dev/sdf
xfs_admin /dev/sdf

Set label for swap partition

mkswap -L SWAP0 /dev/sdb5

2. add the label to fstab

Example of contents of fstab:

LABEL=ROOT          /         ext3    defaults        1 1
LABEL=BOOT          /boot     ext3    defaults        1 2
LABEL=SWAP          swap      swap    defaults        0 0
LABEL=HOME          /home     ext3    nosuid,auto     1 2

3. Update the grub boot manager

title server
root (hd0,0)
  kernel (hd0,0)/vmlinuz ro root=LABEL=SERVER_ROOT0 rhgb quiet
  initrd (hd0,0)/initrd.img

Secure programming: how to implement user account management

Most web applications work like this:

The application uses a single database account to perform all actions. Users are just some records in a table. Account privileges and roles are part of this table, or separate tables.

This implies that all security must be designed and build by the application developer. I think this is entirely wrong. There is a big risk:

In such applications, SQL-injection will allow full control of the entire database.

This is something that is often overlooked. And the solution is simple. The application should not use a general account with full privileges. The application should use the database account of the user accessing the application. All actions performed by this user are thus limited by the privileges of this database account. The impact of SQL-injection would be significantly reduced.

The public part of a website is still using an application account, but the privileges of this account can be significantly reduced. To obtain elevated privileges, a user must first authenticate against the application and thus the database.

Please understand another benefit: it is not required to store username/password combinations of privileged accounts on the application server. The configuration file will only contain the credentials of the unprivileged account. An attacker compromising the application server with limited privileges, won't have access to the database with elevated privileges.

I understand that this solution requires a bit more work to setup at the start, but once implemented, it reduces complexity and improves security so much.

Of course, the security of your data is as good as the hardening of your database server. But that's another story.

Do not buy a hardware RAID controller for home use

Hardware RAID controllers are considered 'the best' solution for high performance and high availability. However, this is not entirely true. Using a hardware RAID controller might even endanger your precious data.

For enterprise environments, where performance is critical, it is more important that the arrays keeps on delivering data at a high speed. Professional RAID controllers use TLER with TLER-enabled disks to limit the time spend on recovering bad sectors. If a disk encounters a bad sector, there is no time to pause and try to fix it. The disk is just dropped out of the RAID array after just a couple of seconds. At that moment, the array still performes relatively well, but there is no redundancy. If another disk fails (another bad sector?) the array is lost, with all its data.

More people are building NAS boxes for centralized storage of data, for private home use. Since disks are cheap, it is possible to create lots of storage capacity for little money. Creating backups of terabytes of data is however not cheap. Or you have to create two NAS boxes. But that is very expensive and not worth the effort.

People seem to spend lots of money on expensive enterprise level hardware RAID cards, not understanding that the whole TLER-mechanism causes an increased risk for their data. In enterprise environments, budgets are relatively big, and data is always backed up. They can afford to take the risk of losing a RAID array due to these backups. But consumers often don't have the money to spend on creating backups of terabytes of data. They just go for RAID 5 or RAID 6 and hope for the best.

For consumers, if the RAID array goes, all data is lost.

So consumers should choose a RAID solution that will do its best to recover from hardware failure. Performance is not so much an issue. Reliability is. So consumers do want disks to spend 'ages' on recovering bad sectors if this means that the RAID array will survive.

Linux software RAID and ZFS do not use TLER and therefore are a safer choice for your data then regular hardware RAID controllers. You may still use such controllers (but please test them properly) but only to provide SATA ports with individual disks, the RAID part should be handled by Linux.

So in my opinion, hardware RAID controllers are more expensive, require more expensive (enterprise) disks and are less safe for your data.

Linux network interface bonding / trunking or how to get beyond 1 Gb/s

This article discusses Linux bonding and how to achieve 2 Gb/s transfer speeds with a single TCP/UDP connection.

A gigabit network card provides about 110 MB/s (megabytes) of bandwidth. If you want to go faster, the options are:

1. buy infiniband stuff: I have no experience with it, may be smart thing to do but seems expensive.
2. buy 10Gigabit network cards: very very expensive compared to other solutions.
3. strap multiple network interfaces together to get 2 Gb/s or more with more cards.

This article is discussing the third option. Teaming or bonding two network cards to a single virtual card that provides twice the bandwidth will provide you with that extra performance that you where looking for.

But the 64000 dollar question is:

How to obtain 2 Gb/s with a single transfer? Thus with a single TCP connection?

The trick is to use Linux network bonding.

Most bonding options only provide an accumulated performance of 2 Gb/s, by balancing different network connections over different interfaces. Individual transfers will never reach beyond 1 Gbit/s but it is possible to have two 1 Gb/s transfers going on at the same time.

That is not what I was looking for. I want to copy a file using NFS and just get more than just 120 MB/s.

The only bonding mode that supports single TCP or UDP connections to go beyond 1 Gb/s is mode 0: Round Robin. This bonding mode is kinda like RAID 0 over two or more network interfaces.

However, you cannot use Round Robin with a standard switch. You need an advanced switch that is capable of creating "trunks". A trunk is a virtual network interface, that consists of individual ports that are grouped together". So you cannot use Round Robin mode with an average unmanaged switch. The only other option is to use direct cables between two hosts, although I didn't tested this.

Now the results: I was able to obtain a transferspeed (read) of 155 MB/s with a file copy using NFS. Normal transfers capped at 109 MB/s. To be honest: I had hoped to achieve way more, like 180MB/s. However, the actual transfer speeds that will be obtained will depend on the hardware used. I recommend using Intel or Broadcom hardware for this purpose.

Also, I was not able to obtain write speed that surpasses the 1 Gb/s. Since I used a fast RAID array to write the data to, the underlying storage subsystem was not the bottleneck.

So the bottom line is that it is possible to get more than 1 Gb/s but the performance gain is not as high as you may want to.

Configuration:

Client:

modprobe bonding mode=0

ifconfig bond0 up

ifenslave bond0 eth0 eth1

ifconfig bond0 10.0.0.1 netmask 255.255.255.0

Server:

modprobe bonding mode=4 lacp_rate=0 xmit_hash_policy=layer3+4

ifconfig bond0 up

ifenslave bond0 eth0 eth1

ifconfig bond0 10.0.0.2 netmask 255.255.255.0

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation

Transmit Hash Policy: layer3+4 (1)

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

802.3ad info

LACP rate: slow

Active Aggregator Info:

Aggregator ID: 2

Number of ports: 2

Actor Key: 9

Partner Key: 26

Partner Mac Address: 00:de:ad:be:ef:90

Slave Interface: eth0

MII Status: up

Link Failure Count: 0

Permanent HW addr: 00:co:ff:ee:aa:00

Aggregator ID: 2

Slave Interface: eth1

MII Status: up

Link Failure Count: 0

Permanent HW addr: 00:de:ca:fe:b1:7d

Aggregator ID: 2

The iPhone, iPad and iOS are powering a revolution

Most people just don't understand computers. Are these people dumb? Some may be dumb, but the people who make them are maybe even dumber. Because they can't seem to figure out how to create a computer that the majority of people understand.

When the original macintosh arrived at the stage back in the eighties, computers became a bit more human-friendly, but it was limited to the constraints of the then available hardware. It put away the text-based interface and introduced the graphic interface. It used the desktop metaphor to create this graphic environment. But this metaphor has had its day.

Many people don't understand the desktop metaphor since they don't have a desktop and have never used one. Also, it is a metaphor, it's to translate the computer environment to something humans understand. But what if they don't understand the metaphor? For example, many people just don't 'get' the Windows Explorer or the Mac OS X finder. The desktop metaphor does not seem to fit in how people think. 

Every time you see a person enter a URL like www.youtube.com in the google search field, you will realize that we still have a long way to go. 

Most people did not seem to realize back then that the release of the iPhone wasn't that important, but the release of iOS. The iPhone was the first smartphone (a word most people are not familiar with) that did away with a stylus or hardware keyboard. It uses what is closest to us: our fingers. A totally new user interface, one that is very natural and close to us, is now available.

Using touch as input required a total redesign of the entire user interface. All other interfaces were designed around hardware keyboard and mouse devices. Fingers are big, and are obstructing the view. But it allows for a more direct interaction with a device. And now all new smart phones sport a touch interface.

Rumors of an Apple tabled existed for long, but it was very clear when the iPhone was released that if Apple would release a tablet, it would run this new iOS operating system.

When the iPad was released, it became an instant hit. As of today, there is no device on the market that can be truly called a competitor. But why is this so? The ground work has been done by the iPhone. Most people with an iPhone will notice that aside from some performance issues in the past, the device just always worked. It was instantly available to sent an email, look something up on wikipedia or find the nearest Starbucks. An iPhone just always works. No boot. Very reliable. And an interface that makes you happy.

Why does iOS make people happy? Because it provides a user interface that is human. People understand it instinctively. Any person of any age or background will be able to use an iOS device within minutes. The interface doesn't make you look like you are dumb because you just don't understand how it works. It not only works, it is easy to use and you are not afraid to break anything. 

The iPhone and the iPad are learning a lot of people not to fear computers.

The iOS does away with the old desktop metaphor, but so does Symbian or similar interfaces. It is the combination with touch and the well thought out interface that sets it apart from other mobile operating systems. Even when the iOS platform did not have native applications, people still bought it and not only because Apple released a new shiny toy. 

However, the app store on iOS has created a very special and important environment. People can finally install and remove applications in an extremely simple way. They don't need to be scared that some program will crash your computer either while installing it, using it, or removing it. The whole iOS ecosystem creates an environment wherein people don't need any help any longer from other people. They are finally in control. They don't need to be afraid of their computer.

This trend will affect the old-school user interfaces such as Mac OS X. How it will turn out is anybodies guess. But there is at least a small trend to 'eradicate' the finder as much as possible. iPhoto stores your photos. iTunes stores your music. If you want to include a photo or song within an application, you pick the photo or song in question from a miniature iPhoto or iTunes interface. There is no finder anymore. The finder is disappearing from the workflow. And why not? If programs are written well, why bother with it? The finder should be abstracted away, as is the case on iOS, where you don't have a finder. 

Another thing is multitasking, you know, that stuf we like to do, but cant. We can only do one thing at a time. What we do want is fast task switching, not multitasking. Sure, some programs must be running in de background, to continue to operate, such as a chat program, but that is not the point. Most people are just going crazy if you show how multitasking works, with different windows. Again, iOS shows how 'multitasking' should be implemented. It is implemented as fast application switching, allowing these applications to register services that must continue to run, while the application itself freezes when the user switches to another application. People tend to use one application at a time and especially on mobile devices, every single bit of screen real estate counts, so they are always running full screen. This full screen notion will also be incorporated in the next Mac OS X release, Lion. People switch, but do one thing at a time. 

Computer nerds tend to feel superior to people who don't have much skill using a computer. This feeling of superiority is totally misplaced. They should be really humble. because up until the advent of iOS, nobody was able to create a human friendly computer interface. It is not the lack of understanding on the side of computer users, it is the lack of  understanding on the part of the computer nerds on how normal humans think and act.

Simple, human friendly computer interfaces will liberate humanity from those pesky computer nerds. And that will cause a bit less suffering in the world I hope.