Building LLVM 5.x from source on FreeBSD

A short guide on how to compile LLVM 5.0.0 a low level compiler infrastructure with Clang on FreeBSD.

1. Install required dependencies. You need to be logged in as root to do this.

#pkg install devel/ninja devel/swig13

2. Replace the FreeBSD default linker (ld) with the gold linker. This is important since the default linker (ld) takes up too much memory and usually crashes the building process if you don't have minimum 16GB DRAM and large swap space. You need to be logged in as root to do this.

#cp /usr/local/bin/ld /usr/local/bin/ld.ORIGINAL # backup the original ld linker
#cp /usr/local/bin/ld.gold /usr/local/bin/ld # replace with gold linker

After this step you can logout as root user.
3. Create a directory for LLVM

$mkdir ~/llvm
$cd ~/llvm

4. Download the latest version of LLVM sources including clang (C frontend) and compiler RT from http://releases.llvm.org/download.html#5.0.0

$wget http://releases.llvm.org/5.0.0/llvm-5.0.0.src.tar.xz
$wget http://releases.llvm.org/5.0.0/cfe-5.0.0.src.tar.xz
$wget http://releases.llvm.org/5.0.0/compiler-rt-5.0.0.src.tar.xz
$wget http://releases.llvm.org/5.0.0/libcxx-5.0.0.src.tar.xz
$wget http://releases.llvm.org/5.0.0/libcxxabi-5.0.0.src.tar.xz
$wget http://releases.llvm.org/5.0.0/libunwind-5.0.0.src.tar.xz
$wget http://releases.llvm.org/5.0.0/lld-5.0.0.src.tar.xz
$wget http://releases.llvm.org/5.0.0/lldb-5.0.0.src.tar.xz
$wget http://releases.llvm.org/5.0.0/openmp-5.0.0.src.tar.xz
$wget http://releases.llvm.org/5.0.0/polly-5.0.0.src.tar.xz
$wget http://releases.llvm.org/5.0.0/clang-tools-extra-5.0.0.src.tar.xz
$wget http://releases.llvm.org/5.0.0/test-suite-5.0.0.src.tar.xz

5. Extract the downloaded sources

$tar xJf ./llvm-5.0.0.src.tar.xz
$tar xJf ./cfe-5.0.0.src.tar.xz
$tar xJf ./compiler-rt-5.0.0.src.tar.xz
$tar xJf ./libcxx-5.0.0.src.tar.xz
$tar xJf ./libcxxabi-5.0.0.src.tar.xz
$tar xJf ./libunwind-5.0.0.src.tar.xz
$tar xJf ./lld-5.0.0.src.tar.xz
$tar xJf ./lldb-5.0.0.src.tar.xz
$tar xJf ./openmp-5.0.0.src.tar.xz
$tar xJf ./polly-5.0.0.src.tar.xz
$tar xJf ./clang-tools-extra-5.0.0.src.tar.xz
$tar xJf ./test-suite-5.0.0.src.tar.xz

6. Rename and move folders to correct location where LLVM expects them to be.

$mv ./llvm-5.0.0.src ./llvm-5.0.0
$mv ./cfe-5.0.0.src ./clang
$mv ./clang ./llvm-5.0.0/tools/
$mv ./compiler-rt-5.0.0.src ./compiler-rt
$mv ./compiler-rt ./llvm-5.0.0/projects/
$mv ./libcxx-5.0.0.src ./libcxx
$mv ./libcxx ./llvm-5.0.0/projects/
$mv ./libcxxabi-5.0.0.src ./libcxxabi
$mv ./libcxxabi ./llvm-5.0.0/projects/
$mv ./libunwind-5.0.0.src ./libunwind
$mv ./libunwind ./llvm-5.0.0/projects/
$mv ./lld-5.0.0.src ./lld
$mv ./lld ./llvm-5.0.0/tools/
$mv ./lldb-5.0.0.src ./lldb
$mv ./lldb ./llvm-5.0.0/tools/
$mv ./openmp-5.0.0.src ./openmp
$mv ./openmp ./llvm-5.0.0/projects/
$mv ./polly-5.0.0.src ./polly
$mv ./polly ./llvm-5.0.0/tools/
$mv ./clang-tools-extra-5.0.0.src ./extra
$mv ./extra ./llvm-5.0.0/tools/clang/tools/
$mv ./test-suite-5.0.0.src ./test-suite
$mv ./test-suite ./llvm-5.0.0/projects/

7. This is how it should look like

LLVM source code  => ./llvm
Clang source code  => ./llvm/tools/clang
compiler-rt source code  => ./llvm/projects/compiler-rt
libc++ source code  => ./lvm/projects/libcxx
libc++abi source code  => ./lvm/projects/libcxxabi
libunwind source code  => ./lvm/projects/libunwind
LLD Source code   => ./llvm/tools/lld
LLDB Source code  => ./llvm/tools/lldb
OpenMP Source code  => ./llvm/projects/openmp
Polly Source code  => ./llvm/tools/polly
clang-tools-extra  => ./llvm/tools/clang/tools/extra
LLVM Test Suite   => ./llvm/projects/test-suite

8. Once everything is in place we create a separate folder for the build and installation process

$mkdir ./llvm-5.0.0-build
$mkdir llvm-5.0.0-install
$cd ./llvm-5.0.0-build

9. Now we start the actual configuration and compilation of LLVM inside the 'llvm-5.0.0-build' folder outside of the main source directory so as to keep the main source tree clean

$cmake -G Ninja -DBUILD_SHARED_LIBS=true -DCMAKE_INSTALL_PREFIX=~/llvm/llvm-5.0.0-install -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD="X86" ~/llvm/llvm-5.0.0

$ninja

10. If everything is good it will start compiling. Once its done you can install it

$ninja install

11. After everything is completed all binaries are available in the '~/llvm/llvm-5.0.0-install/bin' folder and libraries are available in the '~/llvm/llvm-5.0.0-install/lib' folder.
12. To start using LLVM we have to include the LLVM binaries and libraries in our path.

$gedit ~/.bashrc

Add this line to the end of the file

export PATH=$PATH:$HOME/llvm/llvm-5.0.0-install/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/llvm/llvm-5.0.0-install/lib

To apply the new settings do

$source ~/.bashrc

13. Test a sample program

$cd ..
$mkdir test
$cd test
$gedit test.c

Add the following lines of a simple C program and save it

#include <stdio.h>
int main(void)
{
 printf("Hello World from LLVM!\n");
 return 0;
}

Compile it using clang the C frontend to LLVM

$clang test.c -o test

Run it

$./test
Hello World from LLVM!

Check the version of LLVM

clang --version
clang version 5.0.0 (tags/RELEASE_500/final)
Target: x86_64-unknown-freebsd11.0
Thread model: posix
InstalledDir: /home/USERNAME/llvm/llvm-5.0.0-install/bin

Building LLVM 3.4.2 from source

A short guide on how to compile LLVM 3.4 a low level compiler infrastructure with Clang on Ubuntu 12.04 (64bit).

NOTE : This is a update to the guide on building LLVM 3.2 already posted here. Clang is also known as Clang CFE (C Front End).

1. Create a directory for LLVM

$mkdir ~/llvm
$cd ~/llvm

Install required dependencies (Note : my system is already setup for development so I cannot find out the exact packages required, if you know them post them as comments)

$sudo apt-get install build-essential

2. Download the latest version of LLVM sources including clang (C frontend) and compiler RT from http://llvm.org/releases/download.html#3.4.2

$wget http://llvm.org/releases/3.4.2/llvm-3.4.2.src.tar.gz
$wget http://llvm.org/releases/3.4.2/cfe-3.4.2.src.tar.gz
$wget http://llvm.org/releases/3.4/compiler-rt-3.4.src.tar.gz

3. Extract the downloaded sources

$tar zxvf ./llvm-3.4.2.src.tar.gz
$tar zxvf ./cfe-3.4.2.src.tar.gz
$tar zxvf ./compiler-rt-3.4.src.tar.gz

4. Move folders to correct location. We start by renaming 'llvm-3.4.2.src' to 'llvm-3.4.2' and moving 'clang' inside the LLVM 'tools' folder and compiler-rt under the LLVM 'projects' folder. This is where LLVM expects them to be.

$mv ./llvm-3.4.2.src ./llvm-3.4.2
$mv ./cfe-3.4.2.src ./clang
$mv ./clang ./llvm-3.4.2/tools/
$mv ./compiler-rt-3.4 ./compiler-rt
$mv ./compiler-rt ./llvm-3.4.2/projects/

5. Once everything is in place we create a separate folder for the build process

$mkdir ./build
$cd ./build

This is how the folder structure looks like

|-- build (currently we are here)
|-- llvm-3.4.2
|   |-- projects
|   |   |-- compiler-rt
|   |-- tools
|   |   |-- clang
|-- cfe-3.4.2.src.tar.gz
|-- compiler-rt-3.4.src.tar.gz
|-- llvm-3.4.2.src.tar.gz

6. Now we start the actual configuration and compilation of LLVM inside the 'build' folder outside of the main source directory so as to keep the main source tree clean

$../llvm-3.4.2/configure --enable-shared --enable-targets=host-only

Note : There are various configuration flags for CPU architecture, optimize builds, threads, etc. Check them with the '--help' option.

$../llvm-3.4.2/configure --help

If there are any missing packages required to compile LLVM it will ask you here.
7. If everything is good we can go ahead and compile it and also keep track of the time taken. After compilation all binaries are available in the 'build/Release+Asserts/bin' folder and libraries are available in the 'build/Release+Asserts/lib' folder.

$time make -j 3
.....
llvm[0]: ***** Completed Release+Asserts Build
real 64m49.849s
user 61m49.232s
sys 2m46.721s

Great job so far, few more steps to go :)

8. Its good to run some test suites that comes along with LLVM to verify everything is working.

$make check-all

9. To start using LLVM we have to include the LLVM binaries and libraries in our path.
To add the LLVM binaries

$gedit ~/.bashrc

Add this line to the end of the file

export PATH=$PATH:$HOME/llvm/build/Release+Asserts/bin

To add the LLVM libraries

sudo gedit /etc/ld.so.conf.d/llvm.conf

Add this line to the file (replace <your_user_name> with your current username)

/home/<your_user_name>/llvm/build/Release+Asserts/lib

To apply the new settings do

$source ~/.bashrc
$sudo ldconfig

10. To check whether the LLVM binaries and libraries are correctly set

$which clang
~/llvm/build/Release+Asserts/bin/clang
$sudo ldconfig -p | grep LLVM
libLLVM-3.4.so (libc6,x86-64) => /home/prashants/llvm/build/Release+Asserts/lib/libLLVM-3.4.so

11. Test a sample program

$cd ..
$mkdir test
$cd test
$gedit test.c

Add the following lines of a simple C program and save it

#include <stdio.h>
int main(void)
{
 printf("Hello World from LLVM!\n");
 return 0;
}

Compile it using clang the C frontend to LLVM

$clang test.c -o test

Run it

$./test
Hello World from LLVM!

Git commands

git log -p --author="Linus Torvalds"

ZFS Commands

Pool related :

#zpool create [name] [type = mirror, raidz, log, cache, sparse] [disk, file]

#zpool status

#zpool destroy

#zpool scrub

#zpool import | export

#zpool online | offline [device]

#zpool attach | detach [mirror]

#zpool cache

#zpool add pool [type]

#zpool get all

#zpool set [property=value] [pool]

#zpool history

Dataset (Volumn / Filesystem) related :

#zfs create [pool]/[dataset]

#zfs get all [pool]/[dataset]

#zfs set [property=value] [pool]/[dataset]

#zfs list

#zfs snapshot [pool]/[dataset]@[name]

#zfs rollback

#zfs clone

#zfs send | receive

Debugging :

#zdb

Using LLVM based Go compiler - llgo

We are going to use llgo - a compiler for Go, written in Go, and using the LLVM compiler infrastructure. The project is hosted at https://github.com/axw/llgo



1. Install LLVM from source using this guide http://linuxdeveloper.blogspot.in/2012/12/building-llvm-32-from-source.html

2. Download Go 1.0.3 binary packages from http://code.google.com/p/go/downloads/list. The file is named "go1.0.3.linux-amd64.tar.gz" and you have to download it in the "~/Downloads" folder.

3. Extract Go in your home directory

$cd ~
$tar zxvf ~/Downloads/go1.0.3.linux-amd64.tar.gz

4. Add the Go binaries to the path.

$gedit ~/.bashrc

Add the following lines

export GOROOT=$HOME/go
export PATH=$PATH:$GOROOT/bin

Reload the shell environment

$source ~/.bashrc

5. Check if Go is available

$go
Go is a tool for managing Go source code.
Usage:
 go command [arguments]
The commands are:...

Good !

6. Now to build llgo we are going to use "go get" command. These steps are also listed in the llgo's README.md file on their website https://github.com/axw/llgo. Type the following in the shell

$export CGO_CFLAGS="`llvm-config --cflags`"
$export CGO_LDFLAGS="`llvm-config --ldflags` -Wl,-L`llvm-config --libdir` -lLLVM-`llvm-config --version`"
$go get github.com/axw/llgo/llgo

It will automatically build and install the llgo binaries in the "~/go/bin" folder.

7. Check if the "llgo" has been build

$ls ~/go/bin
go  godoc  gofmt  llgo
$llgo
No Go source files were specified

There it is :)

8. Lets test it

$gedit ~/test.go

Add these lines

package main
import "fmt"
func main() {
  fmt.Println("Hello World")
}

Now run on the terminal

$llgo -dump test.go

; ModuleID = 'main'
target datalayout = "e-p:64:64:64..."
target triple = "x86_64-unknown-linux"
%0 = type { i8*, i8* }
....

That's the LLVM assembly code which is then given to LLVM backend to generate the actual code.

Building LLVM 3.2 from source

A short guide on how to compile LLVM 3.2 a low level compiler infrastructure with Clang on Ubuntu 12.04 (64bit).



1. Create a directory for LLVM

$mkdir ~/llvm
$cd ~/llvm

Install required dependencies (Note : my system is already setup for development so I cannot find out the exact packages required, if you know them post them as comments)

$sudo apt-get install build-essential

2. Download the latest version of LLVM sources including clang (C frontend) and compiler RT from http://llvm.org/releases/download.html#3.2

$wget http://llvm.org/releases/3.2/llvm-3.2.src.tar.gz
$wget http://llvm.org/releases/3.2/clang-3.2.src.tar.gz
$wget http://llvm.org/releases/3.2/compiler-rt-3.2.src.tar.gz

3. Extract the downloaded sources

$tar zxvf ./llvm-3.2.src.tar.gz
$tar zxvf ./clang-3.2.src.tar.gz
$tar zxvf ./compiler-rt-3.2.src.tar.gz

4. Move folders to correct location. We start by renaming 'llvm-3.2.src' to 'llvm-3.2' and moving 'clang' inside the LLVM 'tools' folder and compiler-rt under the LLVM 'projects' folder. This is where LLVM expects them to be.

$mv ./llvm-3.2.src ./llvm-3.2
$mv ./clang-3.2.src ./clang
$mv ./clang ./llvm-3.2/tools/
$mv ./compiler-rt-3.2.src ./compiler-rt
$mv ./compiler-rt ./llvm-3.2/projects/

5. Once everything is in place we create a separate folder for the build process

$mkdir ./build
$cd ./build

This is how the folder structure looks like

|-- build (currently we are here)
|-- llvm-3.2
|   |-- projects
|   |   |-- compiler-rt
|   |-- tools
|   |   |-- clang
|-- clang-3.2.src.tar.gz
|-- compiler-rt-3.2.src.tar.gz
|-- llvm-3.2.src.tar.gz

6. Now we start the actual configuration and compilation of LLVM inside the 'build' folder outside of the main source directory so as to keep the main source tree clean

$../llvm-3.2/configure --enable-shared

Note : There are various configuration flags for CPU architecture, optimize builds, threads, etc. Check them with the '--help' option.

$../llvm-3.2/configure --help

If there are any missing packages required to compile LLVM it will ask you here.
7. If everything is good we can go ahead and compile it and also keep track of the time taken. After compilation all binaries are available in the 'build/Release+Asserts/bin' folder and libraries are available in the 'build/Release+Asserts/lib' folder.

$time make -j 3
.....
llvm[0]: ***** Completed Release+Asserts Build
real 31m58.051s
user 52m50.842s
sys 2m37.610s

Great job so far, few more steps to go :)

8. Its good to run some test suites that comes along with LLVM to verify everything is working.

$make check-all

9. To start using LLVM we have to include the LLVM binaries and libraries in our path.
To add the LLVM binaries

$gedit ~/.bashrc

Add this line to the end of the file

export PATH=$PATH:~/llvm/build/Release+Asserts/bin

To add the LLVM libraries

sudo gedit /etc/ld.so.conf.d/llvm.conf

Add this line to the file (replace <your_user_name> with your current username)

/home/<your_user_name>/llvm/build/Release+Asserts/lib

To apply the new settings do

$source ~/.bashrc
$sudo ldconfig

10. To check whether the LLVM binaries and libraries are correctly set

$which clang
~/llvm/build/Release+Asserts/bin/clang
$sudo ldconfig -p | grep LLVM
libLLVM-3.2svn.so (libc6,x86-64) => ~/llvm/build/Release+Asserts/lib/libLLVM-3.2svn.so

11. Test a sample program

$cd ..
$mkdir test
$cd test
$gedit test.c

Add the following lines of a simple C program and save it

#include <stdio.h>
int main(void)
{
 printf("Hello World from LLVM!\n");
 return 0;
}

Compile it using clang the C frontend to LLVM

$clang test.c -o test

Run it

$./test
Hello World from LLVM!

Building Linux kernel the clean way

I like to keep my kernel sources clean and have the kernel build in a separate directory. Here is how I do it.



1. Lets create a "~/kernel" directory for holding everything

$cd ~
$mkdir ~/kernel
$cd ~/kernel

2. Download and extract the kernel source

$wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.6.tar.bz2
$tar jxvf linux-3.6.tar.bz2

3. Create a build directory

$mkdir ~/kernel/build

Now our directory structure looks like:

~/
~/kernel/
~/kernel/build/
~/kernel/linux-3.6.tar.bz2
~/kernel/linux-3.6/

4. We will create a bash function that will automatically create the build folder for us based on the folder name of the kernel source. Add the following to the ~/.bashrc file. Once this is done open a new bash terminal so that the function is available for use.

function kb {
    mkdir -p ~/kernel/build/$(basename $(pwd))
    echo ~/kernel/build/$(basename $(pwd))
}

Running $kb in your shell will create the build folder and return the path of the build folder which we will pass it to "make" in the next step.

5. To build the kernel in the "build" folder follow your normal step except...

$cd linux-3.6
$make distclean
$make defconfig O=`kb`
$make O=`kb`

You can see that a folder called "~/kernel/build/linux-3.6" has been created and all the object files are created in the build folder and our original kernel source folder is kept clean. After the compiling is done the kernel is available in the ~/kernel/build/arch/x86/boot/bzImage or whatever is your architecture folder.

Note :
The keys surrounding the `kb` in the above steps are the backtick keys which are located along with the "~" key on your keyboard.

Extra :
If you download and extract the linux-3.5.tar.bz2 and repeat the above 5th step it will create a ~/kernel/build/linux-3.5 folder and build everything there.

Next time I am going to post how you can fully automate the build with git.