Andrew B. Wright, S. M. ’88, Ph. D.
6/7/2021
Previous Blog: Robotics using the Beaglebone Black
Next Blog: Beaglebone: Toggling a bit with the PRU (remoteproc version)
I am in process of bringing this into function with the latest kernel. Of course, there were significant changes!
The TI page describing remoteproc is here. This provides bread crumbs to see how to get started.
CODE: (warning! I’m making a couple of minor updates to this code. The current code needs an edit to the makefile to add $(PRU_CGT_ROOT)lib/libc.a instead of libc.a in the LDFLAGS definition.
Also, this code does not correctly initialize when /dev/rpmsg_pru31 is not ready. I’m pretty sure I fixed this in later code, but did not back-port the updates. When you run this code once, it will end with segmentation fault, but the driver will be installed. Subsequent access to the code work until you deinstall the driver or reboot the beaglebone. This has been fixed in later code and will be fixed in this code eventually.
The code, including the Makefile, is packed up in this little archive.
You should create a subdirectory under your $(HOME) called hello (e.g., /root/hello) and put your code files and the Makefile in that directory. You will have to create subdirectories pru0 and pru1 in your code directory for this example to work.
You should set the variable, CODE_ROOT, in the Makefile to the directory containing the code. The archive (below) is set for /root/code/hello.
Download the archive to a directory (/root/code if you want the code to work out of the box) and unpack it using the command “tar -xzf ‘archive name’ “. This will create an extra directory /root/code/hello in which the files will be located. The archive should create the pru0 and pru1 subdirectories as well. If you go to the directory, typing “make hello” and “make install” will create the executable files. Typing “./hello” should give you the output (10 iterations of “PRU1 Responding: Hello, world!” echoed back from PRU1).
I put the communication routines in arm_comm.c (for the ARM processor code, hello.c) and pru_comm.c (for the PRU side processor code, main.c). On both sides, rpmsg_init is used to initialize the communication layer. These routines were set up as a state machine, which is designed to work in a loop. So, as each set up step is completed, the state advances to the next set up step. This allows the ARM side to be interrupted and allows the PRU side to insert code that runs every cycle.
On the ARM side, rpmsg_init is located before the ARM starts to do anything. When the state reaches Initialized, execution exits the loop and the rest of the code can begin. If the initialization fails, then the user can interrupt with ctrl-c. In the future, when the ARM side code is running other processes, this initialization should be inserted into the main loop, as it is done in the PRU.
On the PRU side, rpmsg_init is checked every loop in the real time cycle. This allows code to be run in the real time cycle to control motors and read sensors. It is not desirable on a processor that is ensuring the state of a device, such as a robot, to wait until it can talk to the host processor before making sure that motors are in a safe state.
The routine, listen, monitors incoming traffic on the rpmsg channel, and places the received data into a buffer (pbuffer) of length, plen. A routine must be written to drain the buffer. In this example, this function is coded in main. However, the routine, parse_the_message, is provided to both drain the buffer and parse command codes passed between the ARM and PRU. That routine will be described in the next blog.
There were some big changes in the latest image after November when I started this. Type uname -a. I’m updating this for 4.14.49-ti-r54. There have been a lot of iterations since I started this project. With each iteration, the code gets more stable and usable; however, some modifications are usually required with each update.
If you’re going to use the archive, check your version. If the version is different from the above and the code doesn’t work, wait a bit. I’ll get to the updates eventually.
With the latest beaglebone debian kernel, the method of communicating with the pru is the remoteproc system. There is an overview of the structure here.
Here’s the debian project page, from which you can link to the source code. Carry on from here and you get to the kernel source. If you look in the drivers sub-directory, you find a folder remoteproc and another folder rpmsg. These are the pieces of the kernel that interact with the pru. In remoteproc, you find a few files (pru_rproc.c, pru_rproc.h, pruss_intc.c, pruss.c, pruss.h) that directly interact with the pru. The good news is that this development is yielding an easier to understand, easier to use remoteproc system. When this is all done, this will be a great system.
If you perform a lsmod, you should see the following drivers running (pru_rproc, pruss_intc, pruss, rpmsg_pru, rpmsg_core, virtio_rpmsg_bus). These are the drivers which implement ARM:PRU communication.
The header files for the code have been located in /usr/share/ti/cgt-pru/include. This directory includes some standard header files, so the normal /usr/include is not included in the search path.
Important variables that you need to know:
maximum number of messages (32)
size of vring (512)
The hand’s on training from TI does have some useful information in conjunction with the code in /opt/source/…/usr/lib/ti/pru-software-support-package. And, while the labs are useful, the place to dig is the “examples/am335x” directory. This contains completed code that can be tested and adapted quickly. The programs that follow in this blog are adapted from the code in PRU_Halt, PRU_RPMsg_Echo_Interrupt0, PRU_RPMsg_Echo_Interrupt1. If those files are not present, then “apt-get install pru-software-support-package”.
Other functions that look interesting for exploration are PRU_Direct_Connect0, PRU_Direct_Connect1, PruArm_to_Pru_Interrupt, and Pru_toPruArm_Interrupt. This should give a full suite of Arm-Pru0-Pru1 interconnects. With that enabled, debugging and interaction across all devices should be possible.
In the latest kernel, there is an entry in /sys/class called remoteproc which allows the pru to be stopped and started by writing to the file /sys/class/remoteproc/remoteprocX/state, where X=1 will talk to PRU0 and X=2 will talk to PRU1.
The removes the need to load/unload the rpmsg module (which was a horrible hack FWIW).
The Makefile in the examples gives a hint at how to compile for the PRU. The archive is a stripped down version for the “Hello, world!” example given in the examples. I have set it up so that both pru0 and pru1 are involved, so adaptions can be made relative to either unit.
On the latest debian, the PRU compiler, clpru, is installed by default. The compiler is located in /usr/bin.
The libc-dev header files are located in /usr/share/ti/cgt-pru/include (PRU_CGT_ROOT in the Makefile).
Communication with the remoteproc kernel driver from the pru requires that pru-code is linked with the library, rpmsg_lib.lib. The header files are located in /usr/lib/ti/pru-software-support-package/include/ (PRU_RPMSG_ROOT in the Makefile).
The structure of communications using INTC can be seen in the Technical Reference Manual, Figure 4-17.
PRU0 is connected to Host-0 using R31 bit 30.
PRU1 is connected to Host-1 using R31 bit 31.
System events (SYS_EVT) are connected to one of the ten channels (0..9). This channel connection is done through the resource table.
Figure 4-21 demonstrates the Interrupt number and connection to a pin. Interrupt number 16, 17, 18, and 19 are connected to signals pr1_pru_mst_intr[0]_intr_req, pr1_pru_mst_intr[1]_intr_req, pr1_pru_mst_intr[2]_intr_req, and pr1_pru_mst_intr[3]_intr_req.
So, the numbers for TO_ARM_HOST and FROM_ARM_HOST correspond to these interrupts and pins. When the INTC controller is initialized, the information in the resource_table is loaded into the registers in the INTC controller that allow it to establish the appropriate interrupt connections.
The registers where this is stored are Channel Map Registers (CMR0, CMR1, CMR2 ,… CMR15) and Host Map Registers (HRM0, HRM1, HRM2) in the PRU_ICSS_INTC (see Table 4-102 of the TRM). If you’re using the remoteproc driver, you really, really, really do not want to write anything to these registers.
As an aside, the PRU is not interruptible. This is part of its charm. So, the INTC ‘interrupt’ is not really an interrupt from the PRU side of things. In order to see if the ARM has placed information in the buffer, the PRU has to know to check. To tell the PRU that it’s done something, the ARM sets a bit in R31. The PRU has to poll this bit to determine if something has happened.
This allows the PRU to run in a very deterministic manner and provide an extremely repeatable timed loop.
Unless you code an infinite loop (see below) …
PRU side responses (from pru_rpmsg):
PRU_RPMSG_INVALID_EVENT (pru_rpmsg_init) – means that to_arm_event or from_arm_event is not within MIN_VALID_EVENT or MAX_VALID_EVENT
PRU_RPMSG_SUCCESS (pru_rpmsg_init, pru_rpmsg_send, pru_rpmsg_receive) – function returned successfully
PRU_RPMSG_NO_BUF_AVAILABLE (pru_rpmsg_send, pru_rpmsg_receive) – response from pru_virtqueue_get_avail_buf was less than zero, which occurs if vq->last_avail_idx == avail->idx.
PRU_RPMSG_BUF_TOO_SMALL (pru_rpmsg_send) – the size of the data + header is greater than RPMSG_BUF_SIZE (512)
PRU_RPMSG_INVALID_HEAD (pru_rpmsg_send, pru_rpmsg_receive) – the result of pru_virtqueue_add_used_buf was less than zero, which occurs if head > vq->vring.num.
PRU_RPMSG_NO_KICK (pru_virtqueue_kick) means that no kick was sent (the VRING_AVAIL_F_NO_INTERRUPT bit is set in vq->vring.avail->flags).
ARM side responses (available through dev_err -> dmesg) obtained from kernel source rpmsg_pru.c driver code
Message length table is full – MAX_FIFO_MSG (32) has been exceeded
Unable to allocate fifo for the rpmsg_pru device – MAX_FIFO_MSG * FIFO_MSG_SIZE bytes are not available in rpmsg_pru_probe
Device already open – the PRU is locked (mutex_lock)
Data too large for RPMsg Buffer (rpmsg_pru_write) – same as PRU_RPMSG_BUF_TOO_SMALL, but from the ARM side
Error copying buffer from user space – not sure what gives rise to this, but, it’s not good.
rpmsg_send failed – what it says. Not sure where rpmsg_send is defined or why it would fail.
Failed to get a minor number for the rpmsg_pru device
Unable to add cdev for the rpmsg_pru device
Unable to create the rpmsg_pru device
In the Makefile, the compiler directive, -D PRU0 and -D PRU1, works with the #ifdef statements in the resource_table file to provide the different sys_evt -> channel -> host definitions required by rpmsg_pru30 and rpmsg_pru31 devices.
There are two places in main.c that would be undesirable in an embedded, real-time application. They’re fine in a test like this.
The first,
while (!(*status & VIRTIO_CONFIG_S_DRIVER_OK));
could lock up the PRU forever, waiting until a system resource becomes available. It would be better to check this status and then make sure that all the PRU controlled resources remain in a safe holding pattern state. Imagine if the PRU were controlling motors. Ideally, you would want the system to make sure those motors were in the ‘safe’ state while awaiting the ARM to PRU communication channel to start up, and to keep making sure they stayed in that state.
The second line is
while (pru_rpmsg_channel(RPMSG_NS_CREATE, &transport, CHAN_NAME, CHAN_DESC, CHAN_PORT) != PRU_RPMSG_SUCCESS);
The problem is the same as with the first line; however, this one could really stall for a while if all the buffers had been taken by another process.
There are two instructions in the linker command file, hello.cmd, and in the header files, pru_cfg.h and pru_intc.h, which interact with the linker allocations and symbol definitions. The header files are located in /usr/lib/ti/pru-software-support-package/include.
PRU_INTC : org = 0x00020000 len = 0x00001504 CREGISTER=0
volatile __far pruIntc CT_INTC __attribute__((cregister(“PRU_INTC”, far), peripheral));
PRU_CFG : org = 0x00026000 len = 0x00000044 CREGISTER=4
volatile __far pruCfg CT_CFG __attribute__((cregister(“PRU_CFG”, near), peripheral));
The statement, cregister (“PRU_INTC” … , causes the linker to look for a symbol definition, PRU_INTC … CREGISTER=0. There is a section on cregisters in the Technical Reference Manual which documents all the PRU registers. Access is faster using this method. The __far keywords in this example make sure that 32 bit pointers are used.
The volatile and peripheral keywords are also necessary. From the PRU C/C++ Language Implementation Guide:
“Any variable which might be modified by something external to the obvious control flow of the program must be declared volatile. This tells the compiler that a function might modify the value at any time, so the compiler should not perform optimizations which will change the number or order of accesses of that variable. This is the primary purpose of the volatile keyword.”
“The peripheral attribute can only be used with the cregister attribute and has two effects. First, it puts the object in a section that will not be loaded onto the device. This prevents initialization of the peripheral at runtime. Second, it allows the same object to be defined in multiple source files without a linker error. The intent is that peripherals can be completely described in a header file.”
The near and far keywords in the cregister statements mean that the compiler will use either 16 bit or 32 bit offsets relative to the cregister.
Clearing STANDBY_INIT in pruCfg initializes the OCP controller, which is necessary for communicating with L3 and L4 interconnects.
The ARM-side code to communicate with the PRU is called hello.c.
Once the executable file has been compiled, the executable must be copied to the /lib/firmware directory.
The name of the firmware (for instance, am335x-pru0-fw) must be placed into /sys/class/remoteproc/remoteprocX/firmware (X is either 1 or 2 depending on whether the processor is pru0 or pru1) file.
The convention is to use the names am335x-pru0-fw and am335x-pru1-fw, although those names are no longer mandatory.
Remoteproc will load those files into the appropriate pru (0 or 1) when you send ‘start’ to /sys/class/remoteproc/remoteprocX/state.
In the latest debian, remoteproc should be enabled by default. Check by running:
lsmod | grep pru
If pru_rproc shows up, then remoteproc is actively communicating with the pru.
The Makefile and hello.c have all of these steps automated. Once a program has been compiled, type make install and the steps will be executed in sequence.
