crosscompiling and gdb

As an example I m trying to see how division is implemented on arm and arm64.

created a test.c

int main(void)
{
        int a = 10;
        int b = 20;
        return (a/b);
}

Make sure to comple with -o option always because a.out doesn't has symbols

To compile for arm64
/usr2/arunks/workspace/LA.BR.1.3.1/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin/aarch64-linux-android-gccc -o test64 test.c --sysroot=/usr2/arunks/workspace/LA.BR.1.3.1/prebuilts/ndk/9/platforms/android-17/arch-arm64/usr/

(gdb) disassemble main
Dump of assembler code for function main:
   0x00000000004005a8 <+0>:     sub     sp, sp, #0x10
   0x00000000004005ac <+4>:     mov     w0, #0xa                        // #10
   0x00000000004005b0 <+8>:     str     w0, [sp,#12]
   0x00000000004005b4 <+12>:    mov     w0, #0x14                       // #20
   0x00000000004005b8 <+16>:    str     w0, [sp,#8]
   0x00000000004005bc <+20>:    ldr     w1, [sp,#12]
   0x00000000004005c0 <+24>:    ldr     w0, [sp,#8]
   0x00000000004005c4 <+28>:    sdiv    w0, w1, w0
   0x00000000004005c8 <+32>:    add     sp, sp, #0x10
   0x00000000004005cc <+36>:    ret
End of assembler dump.

start gdb
/usr2/arunks/workspace/LA.BR.1.3.1/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.8/bin/arm-linux-androideabi-gcc -o test test.c --sysroot=/usr2/arunks/workspace/LA.BR.1.3.1/prebuilts/ndk/9/platforms/android-17/arch-arm/usr/

To compile for arm
/usr2/arunks/workspace/LA.BR.1.3.1/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.8/bin/arm-linux-androideabi-gdb test

start gdb
/usr2/arunks/workspace/LA.BR.1.3.1/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.8/bin/arm-linux-androideabi-gdb test

Dump of assembler code for function main:
   0x00008460 <+0>:     push    {r11, lr}
   0x00008464 <+4>:     add     r11, sp, #4
   0x00008468 <+8>:     sub     sp, sp, #8
   0x0000846c <+12>:    mov     r3, #10
   0x00008470 <+16>:    str     r3, [r11, #-8]
   0x00008474 <+20>:    mov     r3, #20
   0x00008478 <+24>:    str     r3, [r11, #-12]
   0x0000847c <+28>:    ldr     r0, [r11, #-8]
   0x00008480 <+32>:    ldr     r1, [r11, #-12]
   0x00008484 <+36>:    bl      0x8498 <__divsi3>
   0x00008488 <+40>:    mov     r3, r0
   0x0000848c <+44>:    mov     r0, r3
   0x00008490 <+48>:    sub     sp, r11, #4
   0x00008494 <+52>:    pop     {r11, pc}
End of assembler dump.

So arm32 implements division in software whereas arm64 has a dedicated instruction to do this.

Banked registres ARM v7

Interpreting values in stack using crash utility

This post shows how to figure out the values in stack using crash utility.

Let me explain a scenario:
Speculation: Process named Binder_5 is in the runqueue holding cgroup_mutex and looping there infinitely.

Below is the do while code which I m suspecting,
File: kerne/cgroups.c
Function: cgroup_attach_task
kernel version:3.10.17

                                                                    Figure 1:

Backtrace of Binder_5 task is as follows,

                                                                      Figure 2:

So from the do while, task_cgroup_from_root is called and this process(binder_5) was preempted out. It was because of some other high priority task might have come.

To confirm that we are in this loop for ever, we need to check the if condition(marked in red box in figure 1). Lets take the objdump of vmlinux and see how to figure out the value of ent.cgrp and cgrp. Both will be stored in stack.

Figure 3 show the partial objdump of function cgroup_attach_task(corresponding c function is shown in figure 1). It is taken using the following command.
$ arm-none-linux-gnueabi-objdump -D vmlinux > vmlinux_objdump.txt

                                                                      Figure 3:

Now we need to find the value of r5 and r0. And if they are same, then my speculation of infinite loop is correct. Tough job ahead :-). Lets have fun.

Finding the value of r5:
If we are lucky the value of r5 will be pushed in to stack during entry of function task_cgroup_from_root.
Here is the objdump:

                                                                     Figure 4:

Yes it is pushed to the stack.
Now figure 5 explains below how to find out the value of r5.
The left part of the figure 5 is created by printing the frame of task binder_5 using crash tool.
The command used,
crash> bt -f 2077
whree 2077 is the pid of process binder_5

                                                                                     Figure :5

push    {r4, r5, fp, ip, lr, pc}

In above instruction, pc will be stored at highest address in stack, followed by lr. And so on.
This is because arm linux stack type is full descending.

So value of r5 = 0xed306c00.

Finding the value of r0:
Now from figure 3, r0 is also stored at fp - 84 th location(marked in red box). so if we look at that location we could get what was the value of r0 when task_cgroup_from_root returned last time.

So for this, we need to find the value of fp first. This will be stored in the frame of task_cgroup_from_root. Same way as our r5 was stored.

Below figure shows how we found the value of fp.
The left side of figure 6 show the backtrace with frames same as figure 5. This time I used the notepad++.

                                                                                     Figure :6

So value of r0 and r5 are same and the speculation of of infinite loop is proved.
This was a bug in the kernel 3.10 and is fixed later. commit

Thanks,
Arun

RB Tree parsing in crash utility

Requirement: see all the virual address allocated to a process.

This is just for the demo purpose of tree command. If you want to do this you can do using command
crash>vm -p

from the ps output,

we pick the process ".android.chrome" whose task_struct address is 0xe22f6200

crash> struct task_struct 0xe22f6200
----strip-------
 mm = 0xedab1500,
----strip-------

we got the value of mm.

 crash> struct mm_struct 0xe22f6200
 struct mm_struct {          
  mmap = 0x1,                
  mm_rb = {                  
    rb_node = 0xc2b00000    
  },                        
  mmap_cache = 0x2,          

Now to list all vm_area(vm_start and vm_end)
crash> tree -t rbtree -r mm_struct.mm_rb 0xedab1500 -o 16 -s vm_area_struct.vm_start,vm_end

the value passed to -o is calulate as follows,
 crash> struct vm_area_struct.vm_rb  
struct vm_area_struct {              
  [16] struct rb_node vm_rb;        
}  

Crash Utility: How to view user space back trace

Step 1) Download gcore extension from,

http://people.redhat.com/anderson/extensions.html

Step 2) To build the module from the top-level crash- directory, enter:
  $ tar xzf crash-gcore-command-1.2.2.tar.gz
  $ mv crash-gcore-command-1.2.2/* extensions
  $ make extensions

Step 3) Extend gcore command as follows from crash command prompt,

crash> extend /projects/mobcom_andrwks_ext8/users/arunks/crashutility/src/crash_utility/extensions/gcore.so

Step 4) Use gcore command to dump the user core dump file

            $gcore 1291

            Where 1291 is the pid of the process. This creates a file core.1291.init

Step 5) Use gdb to view the back trace. Pass elf and the core dump file to gdb as follows

            $ arm-none-linux-gnueabi-gdb /projects/mobcom_andrwks_ext8/users/arunks/bringup_eos2b/android/out/target/product/amethyst/obj/EXECUTABLES/init_intermediates/LINKED/init core.1291.init

Step 6) view back trace by bt command.

(gdb) bt

#0  umount2 () at bionic/libc/arch-arm/syscalls/umount2.S:10

#1  0x0000e0a0 in check_fs (blk_device=0x648b0 "/dev/block/platform/e6bd0000.mmcif/by-name/MODEMLOG", fs_type=0x64900 "ext4", target=0x648e8 "/mnt/modemlog") at system/core/fs_mgr/fs_mgr.c:512

#2  0x0000e7ea in fs_mgr_mount_all (fstab=0x645c0) at system/core/fs_mgr/fs_mgr.c:512

#3  0x0000873a in do_mount_all (nargs=Unhandled dwarf expression opcode 0xf3

) at bionic/libc/include/string.h:217

#4  0x0000992a in execute_one_command () at system/core/init/init.c:939

#5  0x00009e56 in main (argc=Unhandled dwarf expression opcode 0xf3

) at system/core/init/init.c:939

(gdb)

Setting up Tmux

For anyone else who wants to install tmux to their home folder (as non root) here is what you need to do.

1) Download libevent and ncurses.
2) Compile them to $HOME/local (ie ./configure --prefix=$HOME/local, then make install)
3) Install tmux by the following:

cd tmux
./configure
CPPFLAGS="-I$HOME/local/include" LDFLAGS="-static -L$HOME/local/include -L$HOME/local/include/ncurses -L$HOME/local/lib" make
make install

4)./tmux

My .tmux.conf file can be found here,
https://github.com/getarunks/linux_config/blob/master/.tmux.conf

Bug in Undefined Instruction Handler ARM

Problem Statement:

Multimedia team reported random user space crash.
The issue was tough to reproduce. If you run monkey test on, say 7 devices for around 5-10 hours.
They also indicated about a pattern, that SP is shifted by 8bytes in tombstone.

Most of the time crash was reproduced while returning from a bionic libc function strtoimax().
When objdump of strotimax was investigated, found few instruction which may cause corruption.
Out of them the most important one was vpush {d8} and vpop {d8}. This is basically a floating point instruction.

To confirm that vpush and vpop is causing the issue, added a new function called strtoimax_debug in bionic libc. This was a dummy function which does noting, but check for any stack corruption. We added those suspected instruction here,
Like stmdb sp!, {r4, r5, r6, r7, r8, r9, sl, fp, lr}, vpush and vpos
and check for the expected SP, if not generate an intentional data abort doing an ldr r0 [0]

From the result of this experiment, confirmed that the issue is with vpop instruction.

Now did bit study on how vfp instruction are executed. By default vfp engine is turned off druing a context switch. When a process executes floating point instruction, a undefined exception will be generated.
exception handler enables the vfp engine and jump back to the same instruction which caused the exception.
code looks okey in that perspective.

Tracing __und_usr(arch/arm/kernel/entry-armv.S) revealed that control came out of exception handler without performing do_vfp which is necessary to handle any vfp instruction when the engine is off.
This gave an indication that there can be only one possibility(for deviating the path) and that is an exception has triggered while executing the path.

Looking at the code further, saw that und_user is reading the instruction which cause the undefined exception. And comment says that this can cause a fault. Now the question is why it can falut?. Onepossible option is the code page might have reclaimed by other core. A fixup handler is register in the exception table. This is not the normal exception table. Here what I m talking is about kernel's fixup exception table.
But the fixup handler was not proper. It was calling ret_from_execption which retrun back to the next instruction. The problem is present in latest kernel aswell, but this problem is very rarely hit.

Now the fix is to return to the same instruction which cause fault instead of next instruction.

Some Notes:

The NEON/VFP register file is managed using lazy preserve (on UP systems) and
lazy restore (on both SMP and UP systems). This means that the register file is
kept 'live', and is only preserved and restored when multiple tasks are
contending for the NEON/VFP unit (or, in the SMP case, when a task migrates to
another core). Lazy restore is implemented by disabling the NEON/VFP unit after
every context switch, resulting in a trap when subsequently a NEON/VFP
instruction is issued, allowing the kernel to step in and perform the restore if
necessary.

Difference btw Neon and VFP
Neon donot support double precision
no complex operations like square root and divide.

Managed to push this fix upstream, [Link to kenrel.org]