Enabling core dump on Ubuntu

type command,

ulimit -c unlimited

Still not present, monitor the logs here,
/var/log/apport.log

crosscompiling and gdb

As an example I m trying to see how division is implemented on arm and arm64.

created a test.c

int main(void)
{
        int a = 10;
        int b = 20;
        return (a/b);
}

Make sure to comple with -o option always because a.out doesn't has symbols

To compile for arm64
/usr2/arunks/workspace/LA.BR.1.3.1/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin/aarch64-linux-android-gccc -o test64 test.c --sysroot=/usr2/arunks/workspace/LA.BR.1.3.1/prebuilts/ndk/9/platforms/android-17/arch-arm64/usr/

(gdb) disassemble main
Dump of assembler code for function main:
   0x00000000004005a8 <+0>:     sub     sp, sp, #0x10
   0x00000000004005ac <+4>:     mov     w0, #0xa                        // #10
   0x00000000004005b0 <+8>:     str     w0, [sp,#12]
   0x00000000004005b4 <+12>:    mov     w0, #0x14                       // #20
   0x00000000004005b8 <+16>:    str     w0, [sp,#8]
   0x00000000004005bc <+20>:    ldr     w1, [sp,#12]
   0x00000000004005c0 <+24>:    ldr     w0, [sp,#8]
   0x00000000004005c4 <+28>:    sdiv    w0, w1, w0
   0x00000000004005c8 <+32>:    add     sp, sp, #0x10
   0x00000000004005cc <+36>:    ret
End of assembler dump.

start gdb
/usr2/arunks/workspace/LA.BR.1.3.1/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.8/bin/arm-linux-androideabi-gcc -o test test.c --sysroot=/usr2/arunks/workspace/LA.BR.1.3.1/prebuilts/ndk/9/platforms/android-17/arch-arm/usr/

To compile for arm
/usr2/arunks/workspace/LA.BR.1.3.1/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.8/bin/arm-linux-androideabi-gdb test

start gdb
/usr2/arunks/workspace/LA.BR.1.3.1/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.8/bin/arm-linux-androideabi-gdb test

Dump of assembler code for function main:
   0x00008460 <+0>:     push    {r11, lr}
   0x00008464 <+4>:     add     r11, sp, #4
   0x00008468 <+8>:     sub     sp, sp, #8
   0x0000846c <+12>:    mov     r3, #10
   0x00008470 <+16>:    str     r3, [r11, #-8]
   0x00008474 <+20>:    mov     r3, #20
   0x00008478 <+24>:    str     r3, [r11, #-12]
   0x0000847c <+28>:    ldr     r0, [r11, #-8]
   0x00008480 <+32>:    ldr     r1, [r11, #-12]
   0x00008484 <+36>:    bl      0x8498 <__divsi3>
   0x00008488 <+40>:    mov     r3, r0
   0x0000848c <+44>:    mov     r0, r3
   0x00008490 <+48>:    sub     sp, r11, #4
   0x00008494 <+52>:    pop     {r11, pc}
End of assembler dump.

So arm32 implements division in software whereas arm64 has a dedicated instruction to do this.

Banked registres ARM v7

Interpreting values in stack using crash utility

This post shows how to figure out the values in stack using crash utility.

Let me explain a scenario:
Speculation: Process named Binder_5 is in the runqueue holding cgroup_mutex and looping there infinitely.

Below is the do while code which I m suspecting,
File: kerne/cgroups.c
Function: cgroup_attach_task
kernel version:3.10.17

                                                                    Figure 1:

Backtrace of Binder_5 task is as follows,

                                                                      Figure 2:

So from the do while, task_cgroup_from_root is called and this process(binder_5) was preempted out. It was because of some other high priority task might have come.

To confirm that we are in this loop for ever, we need to check the if condition(marked in red box in figure 1). Lets take the objdump of vmlinux and see how to figure out the value of ent.cgrp and cgrp. Both will be stored in stack.

Figure 3 show the partial objdump of function cgroup_attach_task(corresponding c function is shown in figure 1). It is taken using the following command.
$ arm-none-linux-gnueabi-objdump -D vmlinux > vmlinux_objdump.txt

                                                                      Figure 3:

Now we need to find the value of r5 and r0. And if they are same, then my speculation of infinite loop is correct. Tough job ahead :-). Lets have fun.

Finding the value of r5:
If we are lucky the value of r5 will be pushed in to stack during entry of function task_cgroup_from_root.
Here is the objdump:

                                                                     Figure 4:

Yes it is pushed to the stack.
Now figure 5 explains below how to find out the value of r5.
The left part of the figure 5 is created by printing the frame of task binder_5 using crash tool.
The command used,
crash> bt -f 2077
whree 2077 is the pid of process binder_5

                                                                                     Figure :5

push    {r4, r5, fp, ip, lr, pc}

In above instruction, pc will be stored at highest address in stack, followed by lr. And so on.
This is because arm linux stack type is full descending.

So value of r5 = 0xed306c00.

Finding the value of r0:
Now from figure 3, r0 is also stored at fp - 84 th location(marked in red box). so if we look at that location we could get what was the value of r0 when task_cgroup_from_root returned last time.

So for this, we need to find the value of fp first. This will be stored in the frame of task_cgroup_from_root. Same way as our r5 was stored.

Below figure shows how we found the value of fp.
The left side of figure 6 show the backtrace with frames same as figure 5. This time I used the notepad++.

                                                                                     Figure :6

So value of r0 and r5 are same and the speculation of of infinite loop is proved.
This was a bug in the kernel 3.10 and is fixed later. commit

Thanks,
Arun

RB Tree parsing in crash utility

Requirement: see all the virual address allocated to a process.

This is just for the demo purpose of tree command. If you want to do this you can do using command
crash>vm -p

from the ps output,

we pick the process ".android.chrome" whose task_struct address is 0xe22f6200

crash> struct task_struct 0xe22f6200
----strip-------
 mm = 0xedab1500,
----strip-------

we got the value of mm.

 crash> struct mm_struct 0xe22f6200
 struct mm_struct {          
  mmap = 0x1,                
  mm_rb = {                  
    rb_node = 0xc2b00000    
  },                        
  mmap_cache = 0x2,          

Now to list all vm_area(vm_start and vm_end)
crash> tree -t rbtree -r mm_struct.mm_rb 0xedab1500 -o 16 -s vm_area_struct.vm_start,vm_end

the value passed to -o is calulate as follows,
 crash> struct vm_area_struct.vm_rb  
struct vm_area_struct {              
  [16] struct rb_node vm_rb;        
}  

Crash Utility: How to view user space back trace

Step 1) Download gcore extension from,

http://people.redhat.com/anderson/extensions.html

Step 2) To build the module from the top-level crash- directory, enter:
  $ tar xzf crash-gcore-command-1.2.2.tar.gz
  $ mv crash-gcore-command-1.2.2/* extensions
  $ make extensions

Step 3) Extend gcore command as follows from crash command prompt,

crash> extend /projects/mobcom_andrwks_ext8/users/arunks/crashutility/src/crash_utility/extensions/gcore.so

Step 4) Use gcore command to dump the user core dump file

            $gcore 1291

            Where 1291 is the pid of the process. This creates a file core.1291.init

Step 5) Use gdb to view the back trace. Pass elf and the core dump file to gdb as follows

            $ arm-none-linux-gnueabi-gdb /projects/mobcom_andrwks_ext8/users/arunks/bringup_eos2b/android/out/target/product/amethyst/obj/EXECUTABLES/init_intermediates/LINKED/init core.1291.init

Step 6) view back trace by bt command.

(gdb) bt

#0  umount2 () at bionic/libc/arch-arm/syscalls/umount2.S:10

#1  0x0000e0a0 in check_fs (blk_device=0x648b0 "/dev/block/platform/e6bd0000.mmcif/by-name/MODEMLOG", fs_type=0x64900 "ext4", target=0x648e8 "/mnt/modemlog") at system/core/fs_mgr/fs_mgr.c:512

#2  0x0000e7ea in fs_mgr_mount_all (fstab=0x645c0) at system/core/fs_mgr/fs_mgr.c:512

#3  0x0000873a in do_mount_all (nargs=Unhandled dwarf expression opcode 0xf3

) at bionic/libc/include/string.h:217

#4  0x0000992a in execute_one_command () at system/core/init/init.c:939

#5  0x00009e56 in main (argc=Unhandled dwarf expression opcode 0xf3

) at system/core/init/init.c:939

(gdb)

Setting up Tmux

For anyone else who wants to install tmux to their home folder (as non root) here is what you need to do.

1) Download libevent and ncurses.
2) Compile them to $HOME/local (ie ./configure --prefix=$HOME/local, then make install)
3) Install tmux by the following:

cd tmux
./configure
CPPFLAGS="-I$HOME/local/include" LDFLAGS="-static -L$HOME/local/include -L$HOME/local/include/ncurses -L$HOME/local/lib" make
make install

4)./tmux

My .tmux.conf file can be found here,
https://github.com/getarunks/linux_config/blob/master/.tmux.conf