General

While preparing the Xillinux distribution for Cyclone V SoC, it turned out more difficult than expected to build an SD card image from scratch. This post outlines the essentials for preparing a custom U-boot based preloader and framework for loading Linux (and possibly other images). This covers the “HPS first” type of boot from an SD (MMC) device on a SocKit board.

The details are related to working on a Linux machine (Fedora Core 12) using Quartus II 13.0sp1. The Quartus version is significant — the U-boot version that comes with e.g. 13.1 is different, even regarding issues like the baud rate on the UART port.

The boot process

First, some background on the boot stages:

  • The ARM processor loads a hardcoded boot routine from an on-chip ROM, and runs it. There is of course no way to change this code.
  • The SD card’s partition table is scanned for a partition with the partition type field having the value 0xa2. Most partition tools will consider this an unknown type.
  • The 0xa2 partition is expected to contain raw boot images of the preloader, as explained below. Since there’s a 60 kB limit on this stage, the full U-boot loader can’t fit. Rather, the SPL (”Secondary Program Loader”) component of U-boot is loaded into the processor.
  • The U-boot SPL, which functions as the preloader, contains board-specific initialization code, which sets up the processor’s registers to reflect the settings made in Qsys: That the correct UART is used, the DDR memory becomes usable and the pins designated as GPIO start to behave like such, etc. One side-effect is that the four leftmost LEDs on a SocKit board, which are connected directly to the HPS, are turned off. This is a simple visible indication that the SPL has loaded.
  • The SPL loads the “full U-boot” image into memory, and runs it. The image resides in the 0xa2 partition, immediately after the SPL’s boot images (details below).
  • U-boot launches, counts down for autoboot, and executes its default boot command (unless a key is pressed on the console, allowing an alternative boot through the shell). See below on how to change this and other environment variables.
  • In a typical setting, U-boot loads three files from the first partition of the SD device, which is expected to be FAT: The kernel image as uImage (in U-boot image format), the device tree as  socfpga.dtb, and the FPGA bitstream as soc_system.rbf.
  • The kernel is launched.

So how can one tell if something is going on? Well, as mentioned above, there’s the four LEDs going off when the SPL has loaded. The following output (or similar)  is expected on the console by the SPL (UART at 57600 8N1):

U-Boot SPL 2012.10 (Nov 04 2013 - 19:29:22)
SDRAM: Initializing MMR registers
SDRAM: Calibrating PHY
SEQ.C: Preparing to start memory calibration
SEQ.C: CALIBRATION PASSED
DESIGNWARE SD/MMC: 0

If it stops at this point, or this appears more than once, there’s a problem with loading and launching the full U-boot software.

A split second later, the following (or similar) should appear:

U-Boot 2012.10 (Nov 04 2013 - 19:29:32)

CPU   : Altera SOCFPGA Platform
BOARD : Altera SOCFPGA Cyclone 5 Board
DRAM:  1 GiB
MMC:   DESIGNWARE SD/MMC: 0
In:    serial
Out:   serial
Err:   serial
Net:   mii0
Hit any key to stop autoboot:  5

(and counting down to zero)

This short video clip shows what it looks like when booting Xillinux. As mentioned earlier, the four LEDs to the left go off when the SPL is loaded. The four other LEDs go off when the FPGA is loaded. The one LED that blinks is toggled by FPGA logic (a "heartbeat" LED).

In fact, the LEDs aren't really lit to begin with -- they reflect a floating condition. What makes them go off is that a clear logic level is set. It's like someone takes control.

Note that in this clip there is nothing connected to the UART USB port. The two LEDs to the right of the power button, that flash a few times when the power goes on, are related to the UART and will show considerable activity if something is connected to the UART port.

What’s where

The partition table of a functioning system may look as following:

# fdisk -lu /dev/sda

Disk /dev/sda: 7948 MB, 7948206080 bytes
245 heads, 62 sectors/track, 1021 cylinders, total 15523840 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1              62      167089       83514    b  W95 FAT32
/dev/sda2          167090      182279        7595   a2  Unknown
/dev/sda3          182280    15508989     7663355   83  Linux

Note the Id of /dev/sda2, which is 0xa2. This turns /dev/sda2 into the initial boot partition. Its content is described in Appendix A of the Cyclone V handbook, volume 3, but there’s no need to know the structure of the preloader. There are four preloader images, located as 4x64 kB raw blocks, making the first segment 256 kB long. In all known settings, it’s the same 64 kB image, repeated four times.

One can recognize a preloader segment by its signature:

# hexdump -v -C -n 192 /dev/sda2
00000000  1a 00 00 ea 14 f0 9f e5  14 f0 9f e5 14 f0 9f e5  |................|
00000010  14 f0 9f e5 14 f0 9f e5  14 f0 9f e5 14 f0 9f e5  |................|
00000020  20 00 ff ff 24 00 ff ff  28 00 ff ff 2c 00 ff ff  | ...$...(...,...|
00000030  30 00 ff ff 00 01 ff ff  38 00 ff ff 78 56 34 12  |0.......8...xV4.|
00000040  41 53 30 31 00 00 88 29  00 00 a6 01 07 00 00 ea  |AS01...)........|
00000050  40 00 00 01 20 a6 00 00  1c a6 00 00 b8 a7 00 00  |@... ...........|
00000060  1c a6 00 00 de c0 ad 0b  de c0 ad 0b de c0 ad 0b  |................|
00000070  40 00 00 eb 00 00 0f e1  1f 00 c0 e3 d3 00 80 e3  |@...............|
00000080  00 f0 29 e1 10 0f 11 ee  02 0a c0 e3 10 0f 01 ee  |..).............|
00000090  a8 00 9f e5 10 0f 0c ee  05 00 00 eb 12 00 00 eb  |................|
000000a0  9c d0 9f e5 07 d0 cd e3  00 00 a0 e3 08 06 00 eb  |................|
000000b0  1e ff 2f e1 00 00 a0 e3  17 0f 08 ee 15 0f 07 ee  |../.............|

The “AS01″ signature at offset 0x40 indicates a preloader boot section. It appears three more times, with 64 kB intervals.

The “full U-boot” image is found at offset 0x40000 in the partition. This is the reason some variables (e.g. spl.boot.SDMMC_NEXT_BOOT_IMAGE and CONFIG_PRELOADER_SDMMC_NEXT_BOOT_IMAGE) in the sources are given the value 0x40000. The SPL loader is told to start reading the next step this far into the partition with the 0xa2 ID (the SPL component of U-boot doesn’t know where it was loaded from, so it rescans the partition table).

And indeed,

# hexdump -v -C -s 0x40000 -n 192 /dev/sda2
00040000  27 05 19 56 7a 6a 81 22  52 77 d9 93 00 03 c9 cc  |'..Vzj."Rw......|
00040010  01 00 00 40 00 00 00 00  8f c6 72 88 11 02 05 00  |...@......r.....|
00040020  55 2d 42 6f 6f 74 20 32  30 31 32 2e 31 30 20 66  |U-Boot 2012.10 f|
00040030  6f 72 20 73 6f 63 66 70  67 61 5f 63 79 63 6c 6f  |or socfpga_cyclo|
00040040  16 00 00 ea 14 f0 9f e5  14 f0 9f e5 14 f0 9f e5  |................|
00040050  14 f0 9f e5 14 f0 9f e5  14 f0 9f e5 14 f0 9f e5  |................|
00040060  40 02 00 01 a0 02 00 01  00 03 00 01 60 03 00 01  |@...........`...|
00040070  c0 03 00 01 20 04 00 01  60 04 00 01 78 56 34 12  |.... ...`...xV4.|
00040080  40 00 00 01 d4 6c 03 00  d4 6c 03 00 50 b4 06 00  |@....l...l..P...|
00040090  cc c9 03 00 de c0 ad 0b  de c0 ad 0b de c0 ad 0b  |................|
000400a0  4f 04 00 eb 00 00 0f e1  1f 00 c0 e3 d3 00 80 e3  |O...............|
000400b0  00 f0 29 e1 10 0f 11 ee  02 0a c0 e3 10 0f 01 ee  |..).............|

The 0x27, 0x05, 0x19, 0x56 sequence is U-boot’s native image signature. Those curious to know what the first 32 bytes mean, can look in include/image.h in U-boot’s sources.  Or use U-boot’s own tool, which resides in the sources, and is built automatically when U-boot is compiled:

$ uboot-socfpga/tools/mkimage -l uboot-socfpga/u-boot.img
Image Name:   U-Boot 2013.01.01 for socfpga bo
Created:      Mon Nov  4 19:39:59 2013
Image Type:   ARM U-Boot Firmware (uncompressed)
Data Size:    257216 Bytes = 251.19 kB = 0.25 MB
Load Address: 01000040
Entry Point:  00000000

Building the preloader / U-boot

One of the possible motivations to build your own U-boot binary is to change the environment variables by editing CONFIG_BOOTCOMMAND and/or CONFIG_EXTRA_ENV_SETTINGS in U-boot’s sources (uboot-socfpga/include/configs/socfpga_common.h, or socfpga_cyclone5.h in the same directory). This is however not necessary in most cases, since U-boot reads its defaults from its raw storage beginning at sectors 1 (offset 0x200), immediately after the MBR.

Using the “setenv” and “saveenv” commands at U-boot prompt, it’s easy to change the power-up defaults to anything desirable.

To revert to the hardcoded defaults, go (at U-boot’s prompt):

SOCFPGA_CYCLONE5 # env default -a
SOCFPGA_CYCLONE5 # saveenv

The process for setting up the boot loader is somewhat tangled, mainly because the SPL’s initialization phase depends on the processor’s settings made within Qsys. To make things somewhat more exotic, some C sources and their headers are generated as a byproduct of fully implementing the FPGA project which includes the processor (i.e. running quartus_asm for obtaining a .sof).

In other words, it’s not enough to “generate” the Qsys project, but it must be used in an FPGA project, which in turn refers to the .qip file representing the top-level module of the processor. In fact, the end-to-end flow is

  • “Generate” the Qsys project, from the Qsys GUI or with ip-generate
  • Compile the Quartus project using the Qsys element all the way
  • Building the preloader (that is, the U-boot project)

On the bright side, most of the really tangled parts are handled automagically with a simple Makefile.

The first thing to do is invoking an Altera Embedded shell, so that the execution paths are set up properly. This is done by issuing a command like

$ /my/software/altera/13.0sp1/embedded/embedded_command_shell.sh

Now go

bsp-create-settings --type spl --bsp-dir build \
  --preloader-settings-dir ../../../synth/hps_isw_handoff/soc_system_hps_0/ \
  --settings build/settings.bsp --set spl.boot.WATCHDOG_ENABLE false

This creates a new directory, “build/”, and populates it with some files. Most notably,  a Makefile.

The –preloader-settings-dir must point at a directory containing the handoff sources. It should be found in the same directory to which the .sof file was written. If you didn’t generate a .sof file, don’t be surprised if you can’t find the handoff files. See above.

Note that these handoff files are the link between the processor settings made in Qsys and the initialization routine that sets the processor’s registers, so that these settings actually happen. In other words, if you’ve changed the function of one of the processor’s direct I/O pins, you’ll need to reimplement the Qsys project, then the Quartus project, and then the U-boot project (the SPL part in particular).

Now to the magic: To implement the preloader (U-boot SPL) just go

$ make -C build

which means “change directory to build/ and run the ‘all’ target in the Makefile”. This compiles the U-boot SPL component, and generates the 256 kB worth of preloader image as build/preloader-mkpimage.bin. And also compiles U-boot’s utitlies under tools/, including mkimage (which is, by the way, necessary to compile the Linux kernel as an uImage).

As a matter of fact, so much happens as a result of this command, that one may forget to ask: Where did the sources for U-boot come from? Surely some esoteric git repo? Well, no. Among the peculiarities of this build process, a tarball is opened from /path/to/altera/13.0sp1/embedded/host_tools/altera/preloader/uboot-socfpga.tar.gz. It’s not as bad as it sounds, because:

  • This U-boot version is good for use with SocKit. Well, almost. The environment variables need to be changed as explained below.
  • This happens once, and the sources may be edited and recompiled
  • A custom tarball can be used instead by issuing
$ make -C build TGZ=/path/to/my/uboot-socfpga.tar.gz

instead. Note that the tarball's name should be consistent with the directory it generates, so don't just rename the tarball.

Another little peculiarity is that the ‘all’ target in the relevant Makefile stands for ’spl’ and ‘mkpimage-spl’. So the “full U-boot” image isn’t built. To achieve this, simply go

$ make -C build uboot

which builds uboot-socfpga/u-boot.img.

Did it work? Great. Now edit uboot-socfpga/include/configs/socfpga_cyclone5.h so its environment variables reflect the following:

fpgaload=fatload mmc 0:${mmcloadpart} 0x2000000 soc_system.rbf; fpga load 0 ${fpgadata} ${filesize}
filesize=6AEBD0
bootcmd=run mmcload; run fpgaload; run mmcboot

And issue “make uboot” again. (”filesize” is the hexadecimal number for the file length of the .rbf file).

To wrap this up, verify that build/preloader-mkpimage.bin is exactly 262144 bytes long (a.k.a. 256 kB) and concatenate the two files into a partition image:

$ cat build/preloader-mkpimage.bin build/uboot-socfpga/u-boot.img > build/boot-partition.img

And finally, write this to the 0xa2 partition.

# dd of=/dev/sda2 bs=512 if=boot-partition.img ; sync

997+1 records in
997+1 records out
510476 bytes (510 kB) copied, 0.3039 seconds, 1.7 MB/s

The numbers may vary, of course. The “sync” command in the end makes sure that the data is flushed to the SD device.

Note: /dev/sda2 is given here for illustration. If you’re not sure which device file to write to, don’t. Picking the wrong device file can wipe your computer’s hard disk.

Please keep in mind, that if the SD card has been used with U-boot previously, U-boot may remember the previous environment variables. It’s therefore recommended to use the short procedure described above for reverting to the default environment.