Skip to content

Logging Packets

In the previous chapter, our XDP application ran until Ctrl-C was hit and permitted all the traffic. Each time a packet was received, the BPF program created a log entry. Let's expand this program to log the traffic that is being permitted in the user-space application instead of the BPF program.

Source Code

Full code for the example in this chapter is available here

Getting Data to User-Space

Sharing Data

To get data from kernel-space to user-space we use an eBPF map. There are numerous types of maps to chose from, but in this example we'll be using a PerfEventArray.

While we could go all out and extract data all the way up to L7, we'll constrain our firewall to L3, and to make things easier, IPv4 only. The data structure that we'll need to send information to user-space will need to hold an IPv4 address and an action for Permit/Deny, we'll encode both as a u32.

xdp-log-common/src/lib.rs
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
#![no_std]

#[repr(C)]
#[derive(Clone, Copy)]
pub struct PacketLog {
    pub ipv4_address: u32,
    pub action: u32,
}

#[cfg(feature = "user")]
unsafe impl aya::Pod for PacketLog {} // (1)
  1. We implement the aya::Pod trait for our struct since it is Plain Old Data as can be safely converted to a byte-slice and back.

Alignment, padding and verifier errors

At program load time, the eBPF verifier checks that all the memory used is properly initialized. This can be a problem if - to ensure alignment - the compiler inserts padding bytes between fields in your types.

Example:

#[repr(C)]
struct SourceInfo {
    source_port: u16,
    source_ip: u32,
}

let port = ...;
let ip = ...;
let si = SourceInfo { source_port: port, source_ip: ip };

In the example above, the compiler will insert two extra bytes between the struct fields source_port and source_ip to make sure that source_ip is correctly aligned to a 4 bytes address (assuming mem::align_of::<u32>() == 4). Since padding bytes are typically not initialized by the compiler, this will result in the infamous invalid indirect read from stack verifier error.

To avoid the error, you can either manually ensure that all the fields in your types are correctly aligned (eg by explicitly adding padding or by making field types larger to enforce alignment) or use #[repr(packed)]. Since the latter comes with its own footguns and can perform less efficiently, explicitly adding padding or tweaking alignment is recommended.

Solution ensuring alignment using larger types:

#[repr(C)]
struct SourceInfo {
    source_port: u32,
    source_ip: u32,
}

let port = ...;
let ip = ...;
let si = SourceInfo { source_port: port, source_ip: ip };

Solution with explicit padding:

#[repr(C)]
struct SourceInfo {
    source_port: u16,
    padding: u16,
    source_ip: u32,
}

let port = ...;
let ip = ...;
let si = SourceInfo { source_port: port, padding: 0, source_ip: ip };

Writing Data

Using Kernel Network Types

To get useful data to add to our maps, we first need some useful data structures to populate with data from the XdpContext. We want to log the Source IP Address of incoming traffic, so we'll need to:

  1. Read the Ethernet Header to determine if this is an IPv4 Packet
  2. Read the Source IP Address from the IPv4 Header

The two structs in the kernel for this are ethhdr from uapi/linux/if_ether.h and iphdr from uapi/linux/ip.h. Rust equivalents of those structures (EthHdr and Ipv4Hdr) are provided by the network-types crate.

Let's add it to our eBPF crate by adding a dependency on network-types in our xdp-log-ebpf/Cargo.toml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[package]
name = "xdp-log-ebpf"
version = "0.1.0"
edition = "2021"

[dependencies]
aya-bpf = { git = "https://github.com/aya-rs/aya", branch = "main" }
aya-log-ebpf = { git = "https://github.com/aya-rs/aya", branch = "main" }
xdp-log-common = { path = "../xdp-log-common" }
network-types = "0.0.4"

[[bin]]
name = "xdp-log"
path = "src/main.rs"

[profile.dev]
opt-level = 3
debug = false
debug-assertions = false
overflow-checks = false
lto = true
panic = "abort"
incremental = false
codegen-units = 1
rpath = false

[profile.release]
lto = true
panic = "abort"
codegen-units = 1

[workspace]
members = []

Getting Packet Data From The Context And Into the Map

The XdpContext contains two fields, data and data_end. data is a pointer to the start of the data in kernel memory and data_end, a pointer to the end of the data in kernel memory. In order to access this data and ensure that the eBPF verifier is happy, we'll introduce a helper function called ptr_at. This function will ensure that before we access any data, we check that it's contained between data and data_end. It is marked as unsafe because when calling the function, you must ensure that there is a valid T at that location or there will be undefined behaviour.

With our helper function in place, we can:

  1. Read the Ethertype field to check if we have an IPv4 packet.
  2. Read the IPv4 Source Address from the IP header

To do this efficiently we'll add a dependency on memoffset = "0.8" in our myapp-ebpf/Cargo.toml

Reading Fields Using offset_of!

As there is limited stack space, it's more memory efficient to use the offset_of! macro to read a single field from a struct, rather than reading the whole struct and accessing the field by name.

Once we have our IPv4 source address, we can create a PacketLog struct and output this to our PerfEventArray

The resulting code looks like this:

xdp-log-ebpf/src/main.rs
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
#![no_std]
#![no_main]

use aya_bpf::{bindings::xdp_action, macros::xdp, programs::XdpContext};
use aya_log_ebpf::info;

use core::mem;
use network_types::{
    eth::{EthHdr, EtherType},
    ip::{Ipv4Hdr, IpProto},
    tcp::TcpHdr,
    udp::UdpHdr,
};


#[panic_handler]
fn panic(_info: &core::panic::PanicInfo) -> ! {
    unsafe { core::hint::unreachable_unchecked() }
}

#[xdp]
pub fn xdp_firewall(ctx: XdpContext) -> u32 {
    match try_xdp_firewall(ctx) {
        Ok(ret) => ret,
        Err(_) => xdp_action::XDP_ABORTED,
    }
}

#[inline(always)] // (2)
unsafe fn ptr_at<T>(ctx: &XdpContext, offset: usize) -> Result<*const T, ()> {
    let start = ctx.data();
    let end = ctx.data_end();
    let len = mem::size_of::<T>();

    if start + offset + len > end {
        return Err(());
    }

    Ok((start + offset) as *const T)
}

fn try_xdp_firewall(ctx: XdpContext) -> Result<u32, ()> {
    let ethhdr: *const EthHdr = unsafe { ptr_at(&ctx, 0)? };
    match unsafe { (*ethhdr).ether_type } {
        EtherType::Ipv4 => {}
        _ => return Ok(xdp_action::XDP_PASS),
    }

    let ipv4hdr: *const Ipv4Hdr = unsafe { ptr_at(&ctx, EthHdr::LEN)? };
    let source_addr = u32::from_be(unsafe { (*ipv4hdr).src_addr });

    let source_port = match unsafe { (*ipv4hdr).proto } {
        IpProto::Tcp => {
            let tcphdr: *const TcpHdr =
                unsafe { ptr_at(&ctx, EthHdr::LEN + Ipv4Hdr::LEN) }?;
            u16::from_be(unsafe { (*tcphdr).source })
        }
        IpProto::Udp => {
            let udphdr: *const UdpHdr =
                unsafe { ptr_at(&ctx, EthHdr::LEN + Ipv4Hdr::LEN) }?;
            u16::from_be(unsafe { (*udphdr).source })
        }
        _ => return Err(()),
    };

    info!(
        &ctx,
        "SRC IP: {:ipv4}, SRC PORT: {}", source_addr, source_port
    );

    Ok(xdp_action::XDP_PASS)
}
  1. Create our map
  2. Here's ptr_at, which gives ensures packet access is bounds checked
  3. Using ptr_at to read our ethernet header
  4. Outputting the event to the PerfEventArray

Don't forget to rebuild your eBPF program!

Reading Data

In order to read from the AsyncPerfEventArray, we have to call AsyncPerfEventArray::open() for each online CPU, then we have to poll the file descriptor for events. While this is do-able using PerfEventArray and mio or epoll, the code is much less easy to follow. Instead, we'll use tokio, which was added to our template for us.

We'll need to add a dependency on bytes = "1" to xdp-log/Cargo.toml since this will make it easier to deal with the chunks of bytes yielded by the AsyncPerfEventArray.

Here's the code:

xdp-log/src/main.rs
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
use aya::{include_bytes_aligned, Bpf};
use anyhow::Context;
use aya::programs::{Xdp, XdpFlags};
use aya_log::BpfLogger;
use clap::Parser;
use log::{info, warn};
use tokio::signal;

#[derive(Debug, Parser)]
struct Opt {
    #[clap(short, long, default_value = "eth0")]
    iface: String,
}

#[tokio::main]
async fn main() -> Result<(), anyhow::Error> {
    let opt = Opt::parse();

    env_logger::init();

    // This will include your eBPF object file as raw bytes at compile-time and load it at
    // runtime. This approach is recommended for most real-world use cases. If you would
    // like to specify the eBPF program at runtime rather than at compile-time, you can
    // reach for `Bpf::load_file` instead.
    #[cfg(debug_assertions)]
    let mut bpf = Bpf::load(include_bytes_aligned!(
        "../../target/bpfel-unknown-none/debug/xdp-log"
    ))?;
    #[cfg(not(debug_assertions))]
    let mut bpf = Bpf::load(include_bytes_aligned!(
        "../../target/bpfel-unknown-none/release/xdp-log"
    ))?;
    if let Err(e) = BpfLogger::init(&mut bpf) {
        // This can happen if you remove all log statements from your eBPF program.
        warn!("failed to initialize eBPF logger: {}", e);
    }
    // (1)
    let program: &mut Xdp = bpf.program_mut("xdp").unwrap().try_into()?;
    program.load()?;
    program.attach(&opt.iface, XdpFlags::default())
        .context("failed to attach the XDP program with default flags - try changing XdpFlags::default() to XdpFlags::SKB_MODE")?;

    info!("Waiting for Ctrl-C...");
    signal::ctrl_c().await?;
    info!("Exiting...");

    Ok(())
}
  1. Name was not defined in xdp-log-ebpf/src/main.rs, so use xdp
  2. Define our map
  3. Call open() for each online CPU
  4. Spawn a tokio::task
  5. Create buffers
  6. Read events in to buffers
  7. Use read_unaligned to read our data into a PacketLog.
  8. Log the event to the console.

Running the program

As before, the interface can be overwritten by providing the interface name as a parameter, for example, RUST_LOG=info cargo xtask run -- --iface wlp2s0.

$ RUST_LOG=info cargo xtask run
[2022-12-22T11:32:21Z INFO  xdp_log] SRC IP: 172.52.22.104, SRC PORT: 443
[2022-12-22T11:32:21Z INFO  xdp_log] SRC IP: 172.52.22.104, SRC PORT: 443
[2022-12-22T11:32:21Z INFO  xdp_log] SRC IP: 172.52.22.104, SRC PORT: 443
[2022-12-22T11:32:21Z INFO  xdp_log] SRC IP: 172.52.22.104, SRC PORT: 443
[2022-12-22T11:32:21Z INFO  xdp_log] SRC IP: 234.130.159.162, SRC PORT: 443