To utilize the github.com/cilium/ebpf package, commonly known as ebpf-go, you must be operating within a Linux environment. Unfortunately, this package is not compatible with macOS or Windows. Fear not, though! You can always fire up a Linux virtual machine. For instance, you could use Ubuntu as your development environment for ebpf projects.
For Mac users, I recommend Parallels Desktop as a virtual machine software—it’s a breeze to set up, though it comes at a cost.
This guide is based on an experiment conducted using the Ubuntu 22.04 system with the ARM64 CPU architecture.
Development Environment Setup
First and foremost, we need to prepare our development environment according to the compilation environment listed in the official ebpf-go documentation.
Official Documentation: https://ebpf-go.dev/guides/getting-started/#whats-next
Here’s a summary of the requirements:
- Your system’s Linux kernel version must be >= 5.7.
- You need to have
clanginstalled, with a version >= 11. If the installedclangdoes not includellvm, you will also need to installllvm. - If you are using a Debian/Ubuntu system, you will need to install
libbpf-dev. - For Debian/Ubuntu systems, you need to create a symbolic link with
sudo ln -sf /usr/include/asm-generic/ /usr/include/asm, or you may encounter errors about missingasm/*.hheader files during compilation. - Your Go language version must be >= the Go version declared in
ebpf-go’sgo.mod.
Once your Linux virtual machine is ready and you’ve installed Go (I recommend directly installing the latest version of Go), you can proceed to install the required tools.
First, check and confirm that the kernel version is greater than 5.7:
$ uname -r
Install clang and llvm:
sudo apt install clang
sudo apt install llvm
Verify the clang version:
$ clang --version
Install libbpf-dev:
sudo apt install libbpf-dev
Create a symbolic link as instructed to run the official examples:
sudo ln -sf /usr/include/asm-generic/ /usr/include/asm
Start Implementing a Demo
First, organize your requirements and write the ebpf C program code. For instance, let’s implement an ebpf-go probe to intercept the vfs_read system function call, capture the file name, and print it out.
C code (vfs-trace.c):
//go:build ignore
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <linux/ptrace.h>
char __license[] SEC("license") = "Dual MIT/GPL"; // This line is mandatory
SEC("kprobe/vfs_read")
int kprobe_vfs_read(struct pt_regs *ctx) {
struct file *f = (struct file *)PT_REGS_PARM1(ctx); // PT_REGS_PARM1 is a method defined in the bpf/bpf_tracing.h header file for capturing the first argument of the intercepted method. If you want the second argument, use PT_REGS_PARM2, and so on.
struct path path;
bpf_probe_read(&path, sizeof(struct path), &f->f_path);
struct dentry dentry;
bpf_probe_read(&dentry, sizeof(struct dentry), path.dentry);
char filename[256];
bpf_probe_read(filename, sizeof(filename), dentry.d_name.name);
bpf_printk("file name: %s \n", filename);
return 0;
}
The
PT_REGS_PARM1macro is defined in thebpf_tracing.hheader file.
Initialize the project by creating a Go project directory, for example: vfs-ebpf-trace. Place the generated C code file in this directory as well.
Navigate to the directory and execute the following commands in order to initialize the Go project with ebpf-go.
cd vfs-ebpf-trace
go mod init vfs-ebpf-trace
go mod tidy
## Make sure the installed Go version is as new as the one used by ebpf-go
go get github.com/cilium/ebpf/cmd/bpf2go
Write the gen.go file:
Modify
-target arm64according to the CPU architecture, and specify the root directory of all header files with-I/usr/include. The code uses#include <bpf/bpf_tracing.h>to find the file at/usr/include/bpf/bpf_tracing.h.
package main
//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cc clang -cflags $BPF_CFLAGS -target arm64 vfstrace vfs-trace.c -- -I/usr/include
Compile the C program to generate Go code:
go generate
You may encounter compilation errors:
In file included from ~/wujiuye/goprojects/src/vfs-ebpf-trace/vfs-trace.c:3:
/usr/include/linux/ip.h:21:10: fatal error: 'asm/byteorder.h' file not found
#include <asm/byteorder.h>
^~~~~~~~~~~~~~~~~
1 error generated.
Error: can't execute clang: exit status 1
exit status 1
gen.go:3: running "go": exit status 1
Upon checking the /usr/include/asm-generic/ directory, you’ll find that there is indeed no byteorder.h header file, but I found it in another directory, which is aarch64-linux-gnu.
Referencing the documentation’s sudo ln -sf /usr/include/asm-generic/ /usr/include/asm, I changed it to sudo ln -sf /usr/include/aarch64-linux-gnu/ /usr/include/asm, and it no longer reports that it can’t find byteorder.h, but there are still many other header files that can’t be found.
Remember to
rm -rf /usr/include/asmbefore making the change.
/usr/include/string.h:26:10: fatal error: 'bits/libc-header-start.h' file not found
#include <bits/libc-header-start.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
Error: can't execute clang: exit status 1
exit status 1
gen.go:3: running "go": exit status 1
/usr/include/features.h:461:12: fatal error: 'sys/cdefs.h' file not found
# include <sys/cdefs.h>
^~~~~~~~~~~~~
1 error generated.
Error: can't execute clang: exit status 1
exit status 1
gen.go:3: running "go": exit status 1
/usr/include/features.h:485:10: fatal error: 'gnu/stubs.h' file not found
#include <gnu/stubs.h>
^~~~~~~~~~~~~
1 error generated.
Error: can't execute clang: exit status 1
exit status 1
gen.go:3: running "go": exit status 1
I created symbolic links for these directories to aarch64-linux-gnu in the same manner.
sudo ln -sf /usr/include/aarch64-linux-gnu/asm /usr/include/asm
sudo ln -sf /usr/include/aarch64-linux-gnu/bits /usr/include/bits
sudo ln -sf /usr/include/aarch64-linux-gnu/gnu /usr/include/gnu
Following this plan, the compilation should proceed without issues. If errors persist, check the error messages to see if they are caused by mistakes in your C code; if so, they should be easy to fix.
If you are using an Ubuntu with X86_64 (amd64) architecture, you should also encounter similar issues. Simply replace aarch64-linux-gnu with x86_64-linux-gnu:
sudo ln -sf /usr/include/x86_64-linux-gnu/asm /usr/include/asm
sudo ln -sf /usr/include/x86_64-linux-gnu/bits /usr/include/bits
sudo ln -sf /usr/include/x86_64-linux-gnu/gnu /usr/include/gnu
eBPF C code compiled and the Go scaffolding generated, all that remains is to write the Go code responsible for loading the program and attaching it to the Linux kernel’s hooks.
The implementation of main.go:
package main
import (
"github.com/cilium/ebpf/link"
"github.com/cilium/ebpf/rlimit"
"log"
"os"
"os/signal"
"time"
)
func main() {
// Remove kernel resource limits for kernels < 5.11.
if err := rlimit.RemoveMemlock(); err != nil {
log.Fatal("Removing memlock:", err)
}
// Load the compiled eBPF ELF and load it into the kernel.
var objs fusetraceObjects
if err := loadFusetraceObjects(&objs, nil); err != nil {
log.Fatal("Loading eBPF objects:", err)
}
defer objs.Close()
// Use kprobe to mount.
kp, err := link.Kprobe("vfs_read", objs.KprobeVfsRead, nil)
if err != nil {
log.Fatalf("opening kprobe: %s", err)
}
defer kp.Close()
// Listen for interrupt signals to exit the process.
stop := make(chan os.Signal, 5)
signal.Notify(stop, os.Interrupt)
for {
select {
case <-stop:
log.Print("Received signal, exiting..")
return
}
}
}
Compile and run:
go build
./vfs-ebpf-trace
Chain the three commands:
go generate && go build && ./vfs-ebpf-trace
View the log output:
The logs printed by bpf_printk are not output to the console but are written to the /sys/kernel/debug/tracing/trace_pipe file.
sudo cat /sys/kernel/debug/tracing/trace_pipe
If you encounter an error when viewing the logs: cat: /sys/kernel/debug/tracing/trace_pipe: Device or resource busy, it’s because the file is being used by another process, possibly because our last SSH session to view it wasn’t closed, and then SSH timed out. We opened a new window, but the original process still has the file, hence the error.
Use the lsof command to find out which process is using this file, then kill the process.
root@vultr:~# sudo lsof /sys/kernel/debug/tracing/trace_pipe
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
cat 122966 root 3r REG 0,12 0 12385 /sys/kernel/debug/tracing/trace_pipe
root@vultr:~# kill -9 122966
Pitfalls
Header File Not Found Issues
For compilation errors related to missing header files, the aforementioned approach only suits the case in the article. If we use other headers, such as <linux/fs.h>, we may still encounter various issues.
We can use this powerful trick.
Generate a header file that includes all structures based on the current kernel version using the bpftool command.
bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
Then place the vmlinux.h file in the /usr/include directory:
mv ./vmlinux.h /usr/include
Finally, modify the import section of the C program code:
#include <vmlinux.h>
//#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
//#include <linux/ptrace.h>
Headers starting with bpf/ are obtained from the installation of libbpf-dev and are not system headers, so they cannot be replaced by vmlinux.h. Headers starting with linux/ are provided by the system, and after including vmlinux.h, they are no longer needed, so they are commented out.
Reference: https://www.strickland.cloud/post/1
Data Reading Optimization
In the example, we need to read f->path.dentry->d_name.name. Directly calling like this won’t work, so we used bpf_probe_read several times to read the file name.
SEC("kprobe/vfs_read")
int kprobe_vfs_read(struct pt_regs *ctx) {
struct file *f = (struct file *)PT_REGS_PARM1(ctx);
struct path path;
bpf_probe_read(&path, sizeof(struct path), &f->f_path);
struct dentry dentry;
bpf_probe_read(&dentry, sizeof(struct dentry), path.dentry);
char filename[256];
bpf_probe_read(filename, sizeof(filename), dentry.d_name.name);
bpf_printk("file name: %s \n", filename);
}
We can use the BPF_CORE_READ macro defined in the bpf_core_read.h header file to support chained reading.
After installing libbpf-dev, this header file is located at /usr/include/bpf/bpf_core_read.h.
....
#include <bpf/bpf_core_read.h> // Include this header file
SEC("kprobe/vfs_read")
int kprobe_vfs_read(struct pt_regs *ctx) {
struct file *f = (struct file *)PT_REGS_PARM1(ctx);
const unsigned char *filename;
filename = BPF_CORE_READ(&f->f_path, dentry, d_name.name);
bpf_printk("file name: %s \n", filename);
}
Here we use BPF_CORE_READ to read the file name in one go, solving the problem of needing to call bpf_probe_read multiple times.
In the example, BPF_CORE_READ(&f->f_path, dentry, d_name.name) reads the dentry (structure) from f_path (structure), then reads the d_name (structure) from dentry, and obtains the name field of d_name (const unsigned char *).
Other Issues
1
To verify that the probe points are correct, you can use the bpftrace tool.
For example:
sudo bpftrace -e 'kprobe:vfs_open {printf("hello")}'
2
If you encounter the error Removing memlock: failed to set memlock rlimit: operation not permitted when running, you need to start with root privileges.
If you are using a development tool like GoLand in a virtual machine to run the project, you need to start GoLand with root privileges.
sudo ./goland.sh
3
If you encounter this error when running:
Loading eBPF objects: field KprobeFuseFileWrite: program kprobe_fuse_file_write: load program: invalid argument: cannot call GPL-restricted function from non-GPL compatible program (13 lines omitted)
Refer to the example and add the following line to our C program code:
char __license[] SEC("license") = "Dual MIT/GPL";
You need to re-execute go generate.
4
For Linux system functions, you can search on this website and even search according to your own system’s kernel version. It can be used to find which parameters a function has and what the types of the parameters are. https://elixir.bootlin.com/linux/v5.10.218/source/fs/open.c#L928