Using ccache to speed up kernel compilation

We are going to use "ccache" to speed up our Linux kernel compilation. This neat tool caches the output of C/C++ compilation so the next time the results can be taken from the cache. It can result in 5-10 times speed improvements.


Update : Recorded a short video on it !

1. First we need to install ccache which is already available in Ubuntu repositories

$sudo apt-get install ccache

2. Now let us configure our system to use ccache. Edit your ~/.bashrc

$vim ~/.bashrc

And add the following lines to the end of the file

# ccache
export CCACHE_DIR="/home/prashants/.ccache"
export CC="ccache gcc"
export CXX="ccache g++"
export PATH="/usr/lib/ccache:$PATH"

Replace the /home/prashant/.ccache folder to your own home folder. It is a empty folder that ccache will use to store the result of the compilation. You can use any other folder name of your choice.

The /usr/lib/ccache folder is automatically created by ccache and it contains symlinks to ccache for gcc/g++ compiler names. By adding it to our PATH environment variable, ccache will be called instead of gcc compiler.

$ls /usr/lib/ccache

Now to apply the changes in the ~/.bashrc file

$source ~/.bashrc

3. Next we need to setup the maximum cache size

$ccache -M 2G

This limits the cache size to 2GB. You can change it to something more if you have enough free space.

4. You are done ! Next time you run gcc/g++/make commands it will use ccache. You can monitor the current usage of ccache by executing

$watch -n1 -d ccache -s

5. Lets try it out ! Download the latest linux kernel source from http://kernel.org and compile it. Simultaneously keep a terminal open with the above command running and you will see this

Combining ccache along with distributed kernel compilation can give really amazing results ! Read about the distributed kernel compilation in my previous article here.

Debugging Linux kernel over serial port

We are going to see how to output the kernel messages over the serial port. We need two machines for this but it can also be done with a single machine and simulating the second machine in Virtualbox.

The primary machine is called the HOST machine where we will receive the debug messages. It is running Ubuntu Linux 11.10. We will install Virtualbox Open Source Edition (OSE) on this machine and create another machine called the TARGET machine inside Virtualbox. We will install Ubuntu Linux 11.10 inside this TARGET machine. This TARGET machine is the one we will want to debug over the serial port.

In this guide I have assumed that you have install Ubuntu Linux in the Virtualbox TARGET machine.

1. Let us first configure the Virtualbox TARGET machine. We need to enable the serial port for the TARGET machine. Open the settings for this machine in Virtualbox. Go to "Serial ports". In the "Port 1" tab enable the "Enable Serial Port" option. Use the following settings :

Port Number : COM1
Port Mode : Host Pipe
Create Pipe (Enabled)
Port/File Path : /tmp/vserial

2. Now start the TARGET machine and edit the entry in the Grub2 selection menu

- Click on 'e' key to edit the entry
- Navigate to the line

linux /boot/vmlinuz...

- Press the "End" key to append to the end of this line and type the following at the end of the line :

console=tty console=ttyS0,9600

This is how it should look like

linux /boot/vmlinuz-3.0.0-12-generic root=UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx ro quiet splash vt.handoff=7 console=tty console=ttyS0,9600

Incase if you make some mistake press the "Esc" key to go back the previous menu and again click on 'e' to edit it.

Once done, wait at this screen and follow the next step. We will come back to this screen later.

3. Let us switch to our HOST machine. We are now going to setup our HOST machine to connect to the above Virtualbox TARGET machine using the serial port. We will use minicom terminal program for this

$sudo apt-get install minicom

Since we have already started the Virtualbox TARGET machine in the above step it will automatically create a /tmp/vserial file in our HOST machine. Remember the HOST machine is our main machine and the TARGET machine is running inside Virtualbox. Let us check in our host machine if this file is created.

$ls -al /tmp/vserial
srwxrwxr-x 1 prashants prashants 0 2012-05-05 19:47 /tmp/vserial

The file is created and the 's' in the first character means its a socket file. We are now going to setup minicom to use that file.

$minicom -s

Select "Serial port setup". Now, Press "A" to setup "Serial device". Delete whatever the current entry is, and type

unix#/tmp/vserial

Press "Enter" when done.

Now, press "E" and then press "C" to select 9600 baud rate. Then press "L" to select "No Parity" and then press "V" to select 8-data bit. When done press "Enter".

This is how it should look like finally

Now press "Enter" again to come out of the Serial port setup and back to the main configuration menu.

Once on this screen press "Esc" and it will connect to the socket file. This is how it should looks.

Check the status in the bottom - it says "unix-socket" and "Online". Also check the socket filename in the serial terminal it should be "unix#/tmp/vserial"

Our HOST machine is now ready to receive the kernel debugging message.

4. Let us now switch back to the TARGET machine inside the Virtualbox which is still at the screen



Now, press "Ctrl-X" or F10 inside this screen to boot into this and also watch the minicom screen in the host machine and you will see the kernel messages.

5. The changes we made to the Grub2 entry in above Step 1 are temporary so you need to do this after every reboot of the Virtualbox TARGET machine. To make these entry in Grub2 permanent you need to update the Grub2 configuration files for the TARGET machine which is beyond the scope of this tutorial. One short-cut method is to directly edit the "/boot/grub/grub.cfg" file but this method is not recommended - do it at your own risk. An invalid entry can render the system unbootable.

Also in some cases you might not see the Grub2 menu during boot - You can either press the "Shift" key during boot or else edit the "/etc/default/grub" file and change the following entry

GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10

Also you need to update Grub2 after changing the "/etc/default/grub" file

$sudo update-grub