Banked registres ARM v7

Bug in Undefined Instruction Handler ARM

Problem Statement:

Multimedia team reported random user space crash.
The issue was tough to reproduce. If you run monkey test on, say 7 devices for around 5-10 hours.
They also indicated about a pattern, that SP is shifted by 8bytes in tombstone.

Most of the time crash was reproduced while returning from a bionic libc function strtoimax().
When objdump of strotimax was investigated, found few instruction which may cause corruption.
Out of them the most important one was vpush {d8} and vpop {d8}. This is basically a floating point instruction.

To confirm that vpush and vpop is causing the issue, added a new function called strtoimax_debug in bionic libc. This was a dummy function which does noting, but check for any stack corruption. We added those suspected instruction here,
Like stmdb sp!, {r4, r5, r6, r7, r8, r9, sl, fp, lr}, vpush and vpos
and check for the expected SP, if not generate an intentional data abort doing an ldr r0 [0]

From the result of this experiment, confirmed that the issue is with vpop instruction.

Now did bit study on how vfp instruction are executed. By default vfp engine is turned off druing a context switch. When a process executes floating point instruction, a undefined exception will be generated.
exception handler enables the vfp engine and jump back to the same instruction which caused the exception.
code looks okey in that perspective.

Tracing __und_usr(arch/arm/kernel/entry-armv.S) revealed that control came out of exception handler without performing do_vfp which is necessary to handle any vfp instruction when the engine is off.
This gave an indication that there can be only one possibility(for deviating the path) and that is an exception has triggered while executing the path.

Looking at the code further, saw that und_user is reading the instruction which cause the undefined exception. And comment says that this can cause a fault. Now the question is why it can falut?. Onepossible option is the code page might have reclaimed by other core. A fixup handler is register in the exception table. This is not the normal exception table. Here what I m talking is about kernel's fixup exception table.
But the fixup handler was not proper. It was calling ret_from_execption which retrun back to the next instruction. The problem is present in latest kernel aswell, but this problem is very rarely hit.

Now the fix is to return to the same instruction which cause fault instead of next instruction.

Some Notes:

The NEON/VFP register file is managed using lazy preserve (on UP systems) and
lazy restore (on both SMP and UP systems). This means that the register file is
kept 'live', and is only preserved and restored when multiple tasks are
contending for the NEON/VFP unit (or, in the SMP case, when a task migrates to
another core). Lazy restore is implemented by disabling the NEON/VFP unit after
every context switch, resulting in a trap when subsequently a NEON/VFP
instruction is issued, allowing the kernel to step in and perform the restore if
necessary.

Difference btw Neon and VFP
Neon donot support double precision
no complex operations like square root and divide.

Managed to push this fix upstream, [Link to kenrel.org]

Stack in ARM

Different types of Stack:

Descending and Ascending: The stack grows downwards, starting with higher address and progressing to lower one(a descending stack). or upwards, starting with lower address and progressing to higher one(a ascending stack).

Empty or Full Stack:
The stack pointer can either point to the last item in the stack(a full stack), or the next free space on the stack(an empty stack).

To make it easier for the programmer, stack-oriented suffixes can be used instead of the
increment or decrement, and before or after suffixes

For example:
STMFD sp!, {r0-r5} ; Push onto a Full Descending Stack
LDMFD sp!, {r0-r5} ; Pop from a Full Descending Stack

! signifies, final address is written back to sp.

The Procedure Call Standard for the ARM Architecture (AAPCS), and ARM and Thumb C and C++ compilers always use a full descending stack. The PUSH and POP instructions assume a full descending stack.

ARM Linux do not use TTBR1

Russell King - ARM Linux linux@arm.linux.org.uk via vger.kernel.org	6/26/13
to Will, Yalin, linux-arch, linux-kernel, linux-arm-kern.

As I don't have the original mail (because it wasn't copied to the right

list) I can't reply to the original author, so I'll do it like this

instead.

On Wed, Jun 26, 2013 at 06:16:49PM +0100, Will Deacon wrote:
> [adding the ARM list -- please try and remember to do that in future]
>
> On Wed, Jun 26, 2013 at 03:41:40AM +0100, Wang, Yalin wrote:
> > Hi  Will,
>
> Hello,
>
> > I have a question about  arm pagetable setting in Linux .
> >
> > From armV6,  there is TTBR0 and TTBR1  translation base address registers  in mmu .
> > But I  found linux only use TTBR0 for translation base address ,
> > Could we use TTBR0 and TTBR1 to split user task and kernel pagetables (swapper_pg_dir)？

We don't use TTBR1 because the configurable page table splits between
TTBR0 and TTBR1 are not appropriate for Linux kernels.  The common
configuration is to have 3GB of userspace and 1GB of kernel space.

However, the TTBR splits supported are 2GB, 1GB, 512MB etc.  As I had
prior knowledge of ARMv6 before it was released, I raised this point
with ARM Ltd because I knew that it would not be appropriate for Linux.
Unfortunately, the response was basically that they didn't want to know.
So, as the hardware provided support mismatches what we want, we don't
use the feature.

It's as simple as that; had we been listened to and the architecture
altered to do what we required, then we'd be using it...

> > 1. Because we don’t need copy kernel first –level pagetables into every
> > User task’s pagetables and flush tlb (for example fork() a new process).
>
> Well, you still need the TLB maintenance for setting up CoW, so this win is
> probably not very big.
>
> > 2. And don’t need handle kernel page fault because that user task’s kernel
> > Pagetable when it is not set up , need copy again( for example vmalloc()  ioremap()  kmap()  will change
> > Kernel pagetables and need update to every task pagetables ) .
>
> Is that really a fastpath?

No it isn't, because for all of the above cases we're talking about copying
L1 page table entries, not the individual L2 page table entries between
threads.

Every page table above TASK_SIZE gets shared between processes, and once
it's been shared to a process, any new process forked from that gets its
own pointer to that 2nd level page table immediately.

So, during the initial boot there will be a number of the L1 copies, but
the system will stabilize and there will be no further L1 faulted copies
needed.

using printascii and printhex8 for debugging during very early init

Situation:
Porting one of brcm board to kernel 3.13-rc3

Problem:
kernel was hanging at one do_one_initcall
To figure out which init call, put prints in do_one_initcal function

diff --git a/init/main.c b/init/main.c
index febc511..dd9f5c1 100644
--- a/init/main.c
+++ b/init/main.c
@@ -88,6 +88,8 @@
 #include
 #endif

+extern void printascii(const char *);
+extern void printhex8(unsigned int);
 static int kernel_init(void *);

@@ -693,6 +701,9 @@ int __init_or_module do_one_initcall(initcall_t fn)

        int ret;
        char msgbuf[64];

+       printascii("func:");
+       printhex8(fn);
+       printascii("\n");
        if (initcall_debug)

Output:
func:c0415524

System.map shows
c0415524 t customize_machine

arch/arm/kernel/setup.c:782:arch_initcall(customize_machine);
Function was calling machine_init.

Fixed in machine_init. It was an smc call to init L2 cache controller.

SMP bit affects L1 cache operation on Cortex-A9 MPCore

Q) We have a region which innercacheble with Write back write allocate and SMP bit is set. So if we update this region with some data it's updating in cache, main memory is not updated, which is as expected. 

But if we clear the SMP bit and update this region then the data is getting updated to external memory not getting updated in Cache. Please not all these operation is from core0, core1 is in WFI during this and it's SMP bit is not set.

Ans) This information is contained in the ARM Architecture Reference Manual with the TRM for the Cortex-A9 MPCore and the Cortex-A9 TRM offering additional information.

When an area of memory is declared as Normal, Write-back and shareable with the processor operating in SMP mode, the data will be cached.  At this point the SCU will effectively ensure that the data shared between the processors in SMP mode is managed to maintain coherency.

However, when you leave SMP the shareable attribute tells the MMU that this data is shared amongst devices and this overrides the cache attributes and is treated as non-cacheable.  This is explained in section 4.3.10, Auxiliary Control Register of the Cortex-A9 TRM.  This is consistent with the behaviour that you're observing.

GNU assembler - Linker Scripts

GNU assembler - Linker Scripts,
http://www.bravegnu.org/gnu-eprog/index.html

Arm gcc inline cook book,
http://www.ethernut.de/en/documents/arm-inline-asm.html

The fight for PTM Traces

PTM(Programme trace Marcocell)

This is an interesting feature provided by ARM CoreSight. This piece of hardware monitors a ARM core. When every there is a deviation from the normal execution of the core, PTM generates traces. These deviations can be interrupts or a simple branching.

There is already a linux driver for this at arch/arm/kernel/etm.c. But few patches are missing compared to linaro tree as of this writing. Apart from what driver provides, you have to do few more things.

 1. Register a AMBA device. This has to be done to get the probe of etm driver to be called. You can see how omap guys have done this here,
AMBA_APB_DEVICE(omap3_etm, "etm", 0x102bb921, ETM_BASE, { }, NULL);


2. Configure the funnels. This will require a bigger explanation.
    Can be found from CoreSight documention in arm infocenter.
    In short there will a lot of trace sources(PTM is one amoung them), you need to configure those funnels so the PTM traces will reach ETB or TPIU.
   

Now I got it working. Working here can be divided into two parts.
1. Configure the traces sources(PTM here) and funnels and get the data to ETB.
2. Collect the data from the ETB and decode it to meaningful information.

Some how after two week , PTM stopped working. Actually it is partially working. It is working for single core traces but not for dual core. Either configuring the PTM or decoding the PTM can go wrong.

After lot of debugging, I figure out that this is because of arm going to dormant. When ever arm goes to dormant  it cuts off the power to the processor which results in loosing values in PTM registers.

The reason it worked two weeks before was because dormant was disabled in our kernel. I m just wondering how difficutl it should have been if I have used this driver (PTM) after the dormant is enabled. It would have been a long journey to figure out for me that dormant was the culprit.

VIVT, VIPT and PIPT caches

https://patchwork.kernel.org/patch/1046072/

Uncompressing Linux... done, booting the kernel.

[    0.000000] Booting Linux on physical CPU 0

[    0.000000] Initializing cgroup subsys cpuset

[    0.000000] Initializing cgroup subsys cpu

[    0.000000] Linux version 3.4.5+ (arunks@xl-blr-02) (gcc version 4.4.3 (GCC) ) #19 PREEMPT Mon Oct 1 10:42:05 IST 2012

[    0.000000] CPU: ARMv7 Processor [412fc099] revision 9 (ARMv7), cr=10c53c7d

[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache

[    0.000000] Machine: rheastone

[    0.000000] Kona dt @0xc00001ac, size of 0x284 words, rooted @ 0xc00001ec

Related links,
http://blogs.arm.com/software-enablement/716-page-colouring-on-armv6-and-a-bit-on-armv7/#MMAP

Discussion in linux-arm, good explantion from Catalin Marinas,
sub: ARM caches variants,
http://lists.infradead.org/pipermail/linux-arm-kernel/2010-March/011900.html

http://www.linuxjournal.com/article/7105?page=0,0