Scope
This page describes how to change the settings of a CPM-based PCIe block in a Xilinx' Versal ACAP FPGA by manipulating the process of setting up this block's registers during the FPGA's configuration. The intervention is done just before the generation of the PDI file by bootgen, so this method is useful for quick turnaround tests of different options, as well as for creating slightly different PCIe blocks (e.g. giving each board a different DSN identification number, or trying different settings of the equalizers).
The use case shown in this page is a workaround of a bug in Vivado (present in Vivado 2022.1 and earlier, at least), which causes the PCIe device to announce that it supports the D1 low-power state, as well as supporting PME to wakeup from low-power states, even if the user has opted these out in Vivado.
It's a good idea to read through this page first, as it gives some background.
Other (non-)possibilities
Before jumping to the hackiest way for changing a block's settings, it's worth mentioning two other theoretical possibilities:
- Many functional blocks have an APB3 interface, which is in principle the same as the well-known DRP ports, only with a different API.
- The PCI configuration space registers can be accessed directly through the PCIe block's cfg_mgmt_* ports (this interface needs to be enabled explicitly for a CPM-based PCIe block). This allows reading the configuration registers, as well as writing to some of them.
None of these methods are useful for turning off the D1 capability in the relevant case, however: The CPM-based PCIe block doesn't have an APB3 interface. As for the cfg_mgmt_* ports, the ability to change the registers of the configuration space is very limited. In fact, it appears to be possible to modify only the same bits in the registers that can be altered by the host. So the registers that present low-power capabilities don't change, even if written to via this interface.
Introduction to CDOs
Unlike classic FPGAs, which are configured by virtue of loading a bitstream, Versal's bringup process is performed by a processor.
Those who have worked with Zynq devices may be familiar with a boot process where U-boot is loaded into the embedded processor, and this utility software is responsible for reading the bitstream image from a flash memory (typically an SD card or a QSPI flash) and load this bitstream into the FPGA's logic fabric. Versal's bringup sequence is made in the same spirit, but instead of U-boot, there's a Xilinx-specific boot loader that controls the procedure.
This change in the boot process is the reason why Versal devices are loaded with a PDI file, and not with the well-known bitstream. This file contains several different sections, that are listed in the .bif file in the impl_1 directory.
Among others, this .bif file requests some .cdo files, which consist of register write command that are performed during the FPGA's bringup process. The syntax of the CDO file is straightforward, making it fairly easy to add commands to this file. The bootgen utility, which is responsible for generating the PDI file (in every Vivado implementation for Versal) reads the files that are listed the .bif files, and translates the commands in the CDO files into simple data structures. The write commands in these data structures are scanned and executed by the processor that performs the bringup process.
bootgen takes about a second to run on a computer, so it's suitable for quick tweaking attempts.
For the sake of setting up PCIe blocks, the relevant file is impl_1/gen_files/cpm_data.cdo. A CPM-based PCIe block's parameters are set up by virtue of hundreds of register write operations.
The base address for the first PCIe block (which is most commonly used) is 0xfca50000. A full reference of these registers can be viewed or downloaded as a ZIP file from the Versal ACAP Register Reference (am012). The PCIe block's registers can be found under the "CPM4_PCIE0_ATTR Module" section.
The same information is also listed in XML format in a file that is installed along with Vivado, e.g. Vivado/2022.1/data/versal/ps_pmc/reg_db/PCIEA_ATTRIB_0.xml. The registers for other modules are listed in other XML files in the same directory.
Non-CPM PCIe blocks
It's important to note the difference between a CPM-based PCIe block, which is part of the CIPS unit, and a PCIe block that was generated the old-school way, as a standalone IP with Versal ACAP Integrated Block for PCI Express IP. The crucial difference is that the CPM's parameters are set by virtue of the register write commands in the said CDO file, but with the standalone Integrated Block, the PCIe block's attributes are part of the logic design. Hence no cpm_data.cdo file is generated at all when the old-school PCIe block is used.
Vivado's bug, which causes the undesired announcement of low-power capabilities, is in fact a faulty setting of the relevant registers in the CDO file. Accordingly, this bug doesn't affect the old-school PCIe block, which handles the low-power parameters correctly.
Manipulating the CDO file
For the sake of working around Vivado's bug which causes it to ignore the CPM block's settings, and announce that the PCIe block supports D1 and other power management capabilities, changes in impl_1/gen_files/cpm_data.cdo should be made, followed by a re-run the bootgen utility in order to produce an updated PDI file. Note that this CDO file is generated in the last stage of the implementation, i.e. "Generate Device Image".
The problematic line in this file is very long, and looks like this:
write 0xfca50030 0x1 0x2 0 0 0 0 0 0 0 0 0 0 0 0 [ ... ] 0x60 0x60 0x60 0x1 0x1 0x1 0x1 0x1 0x3 0x1 0 0 0 [ ... ]
This line writes to a range of registers, starting from 0xfca50030 and ending at 0xfca50bfc (756 registers). The address increments by 4 for each word written.
The five values that are marked in red above are written to the registers between 0xfca506f8 and 0xfca50708. These registers are, in rising address order:
- PF0_PM_CAP_PMESUPPORT_D3HOT: PME Support for D3hot State
- PF0_PM_CAP_PMESUPPORT_D3COLD: PME Support for D3hot State
- PF0_PM_CAP_PMESUPPORT_D1: PME Support for D1 State
- PF0_PM_CAP_PMESUPPORT_D0: PME Support for D0 State
- PF0_PM_CAP_SUPP_D1_STATE: D1_Support for D0 State
When these registers are assigned the value 1, the relevant feature is enabled.
Once again, Vivado's bug is that all five are assigned 1, no matter the configuration of the CPM unit.
The trivial workaround is to add the following line in cpm_data.cdo immediately after the long line above:
write 0xfca506f8 0 0 0 0 0
This writes five zeros to the said registers, and by doing so all PME features as well as the D1 support are turned off. Almost needless to say, the reason that this line must be after the long write command to 0xfca50030 is that the write operations take place in the same order as they appear in the CDO file, so the added line overwrites the previous values.
After adding this line, it's recommended to delete the existing PDI to avoid confusion, and re-generate the PDI with
bootgen -arch versal -image theproject.bif -w -o ./theproject.pdi'
(replace "theproject" with the relevant name, of course).
The "bootgen" utility resides in the bin/ directory of Vivado's installation, which is in the execution path if the environment has been set up with e.g. settings64.sh.
Note that cpm_data.cdo is overwritten each time "Generate Device Image" is executed, so the extra line must be inserted each time, and bootgen must be executed after that as well. This is quite annoying to carry out every time manually, so it's natural to solve this with a script which inserts the change into cpm_data.cdo after each execution, e.g. by assigning the STEPS.WRITE_DEVICE_IMAGE.TCL.POST property with a Tcl script. This can also be done in Vivado's GUI, with Project Settings. It may also be a good idea to add the "-no_pdi" flag to write_device_image, so it doesn't call bootgen, and let the script do that instead. Just to prevent generation of an uncorrected PDI file, and possibly cause confusions.
