Kloudless Blog

Tutorials, case studies and how-tos from our experts

A primer on debugging Native Client code in Chrome

This isn’t your father’s your average client-side app.

Chromium NaCl

Native what?

Yesterday, I was faced with an unfamiliar 10k line C program that did custom image manipulation. It takes in two arguments: an input image file and a destination for the resulting output file.
This is straightforward to run on the server-side, but I don’t want to maintain varying compute capacity just for an infrequently run, on-demand image conversion script! Before you mention it, no, I am not about to rewrite the entire program in JavaScript, no matter how fun that sounds.
Enter Chromium Native Client, an “open-source technology for running native compiled code in the browser”. Combined with Pepper.JS, I can run binaries in a browser window! Before you bring out the pitchforks, consider that the C program is almost a perfect fit for this! As long as I write a little code to manage that pesky file I/O, it should be smooth sailing from here, right? Not exactly. Turns out the Pepper.JS docs weren’t kidding about getting your hands dirty.

Moving from JS on the backend to C on the frontend

Downloading the SDK and playing with the File I/O example to get a feel for it showed that simple file I/O operations such as fseek were failing and I had no clue why. Error messages were useless of course, as is often the case:
NaCl I/O Demo
Firing up Developer Tools in Chrome shows that I am faced with a binary .nexe blob to debug, as expectedxkcd. A little searching reveals a few ways to debug Native Client apps, but for the purpose of this post, I am going to detail what I found to be the most straightforward and fruitful approach.

What I would give for a breakpoint

Let’s start with the prerequisites. Instructions on how to set up the Native Client SDK will get you set up with a working environment to build NaCl modules in. I am focusing on NaCl I/O, and working in a Mac OS X environment with Chrome 31, which has PNaCl enabled by default.
[code language=”bash” light=”true”]
$ cd nacl_sdk/pepper_31/examples/demo/nacl_io
First let’s start up our server. I’ve found that the documentation isn’t very clear regarding the various toolchains and config options, but here is the simplest way to get up and running:
[code language=”bash” light=”true”]
$ make CONFIG=Debug TOOLCHAIN=newlib
$ make serve CONFIG=Debug
make -C /Users/vinod/nacl_sdk/pepper_31/src/nacl_io STAMPDIR=/Users/vinod/nacl_sdk/pepper_31/examples/demo/nacl_io/newlib/Debug /Users/vinod/nacl_sdk/pepper_31/examples/demo/nacl_io/newlib/Debug/nacl_io.stamp
STAMP /Users/vinod/nacl_sdk/pepper_31/examples/demo/nacl_io/newlib/Debug/nacl_io.stamp
LINK newlib/Debug/nacl_io_x86_32.nexe
LINK newlib/Debug/nacl_io_x86_64.nexe
LINK newlib/Debug/nacl_io_arm.nexe
CREATE_NMF newlib/Debug/nacl_io.nmf
python /Users/vinod/nacl_sdk/pepper_31/tools/httpd.py -C /Users/vinod/nacl_sdk/pepper_31/examples/demo/nacl_io
Serving /Users/vinod/nacl_sdk/pepper_31/examples/demo/nacl_io on http://localhost:5103/…
This should start a http server at http://localhost:5103/ that you can navigate to and break, as I did above. Now we just need to get a grasp of what exactly broke, causing that vague error.
Enter GDB. GDB is by far the most powerful tool we have in our belt. If you are unfamiliar with how GDB works, I recommend glancing through an overview before proceeding further. As always, the first step is to modify our current Makefile to add debugging symbols to our executable:
[code language=”diff” title=”~/nacl_sdk/pepper_31/examples/demo/nacl_io/Makefile” light=”true”]
— Makefile    2013-11-18 04:45:56.000000000 -0800
+++ Makefile    2013-11-18 00:35:21.000000000 -0800
@@ -14,7 +14,7 @@
DEPS = nacl_io
LIBS = $(DEPS) ppapi pthread
-CFLAGS = -Wall
+CFLAGS = -Wall -g -O0
SOURCES = handlers.c  nacl_io_demo.c queue.c
We quit out of the server with Ctrl-C and run make serve CONFIG=Debug again.
The next step is to enable GDB debugging in Chome. Although the docs point you to running chrome off the command line with various flags, I’ve found that navigating to chrome:flags, enabling “Native Client GDB-based debugging” and re-launching Chrome is enough.
Navigating to  http://localhost:5103/ should now show the application waiting for us to connect via nacl-gdb.
NaCl I/O Demo
nacl-gdb is located in our SDK:
[code language=”bash” light=”true”]
$ find ~/nacl_sdk -name “*-nacl-gdb”
Recall we are using newlib from earlier. Architecture varies, but uname -a should be able to give you a clue as to what to choose. In my case I am going to run nacl_sdk/pepper_31/toolchain/mac_x86_newlib/bin/x86_64-nacl-gdb.
In addition, there are a couple of GDB commands we need to run first. We will put them all in a file, for easy repeated debugging:
[code title=”~/nacl_sdk/pepper_31/examples/demo/nacl_io/nacl_gdb_debug”]
nacl-manifest “newlib/Debug/nacl_io.nmf”
nacl-irt “/Applications/Google Chrome.app/Contents/Versions/31.0.1650.57/Google Chrome Framework.framework/Internet Plug-Ins/nacl_irt_x86_32.nexe”
target remote localhost:4014

  • nacl-manifest points to our NaCl application’s manifest file. I use a relative path since I will be running gdb from the demo application directory.
  • nacl-irt points to the Native Client Integrated Runtime. You can find it easily:
    [code language=”bash” light=”true”]$ find “/Applications/Google Chrome.app” -name “*irt*”
    /Applications/Google Chrome.app/Contents/Versions/30.0.1599.101/Google Chrome Framework.framework/Internet Plug-Ins/nacl_irt_x86_32.nexe
    /Applications/Google Chrome.app/Contents/Versions/31.0.1650.48/Google Chrome Framework.framework/Internet Plug-Ins/nacl_irt_x86_32.nexe
    /Applications/Google Chrome.app/Contents/Versions/31.0.1650.57/Google Chrome Framework.framework/Internet Plug-Ins/nacl_irt_x86_32.nexe
    Just choose the one that corresponds to the version of Chrome you’re running. It’s usually the highest version listed.
  • target points to a TCP port, 4014, being listened on by Chrome that we connect to to debug.

Now we can invoke GDB. Make sure you first have the NaCl application server running in a separate terminal, and have navigated to localhost:5103. The order is important here. If all goes well, you should be presented with a prompt.
[code language=”bash” light=”true” autolinks=”false”]
$ ~/nacl_sdk/pepper_31/toolchain/mac_x86_newlib/bin/x86_64-nacl-gdb -x nacl_gdb_debug
GNU gdb (GDB) 7.5.1 20130827 (Native Client r12067, Git Commit aeafd16b903de784108b15534ccdab8b1a83f8bb)
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type “show copying”
and “show warranty” for details.
This GDB was configured as “–host=i386-apple-darwin10.8.0 –target=x86_64-nacl”.
For bug reporting instructions, please see:
0x0fa00200 in ?? ()
(gdb) b handlers.c:197
Breakpoint 1 at 0x206e6: file handlers.c, line 197.
(gdb) c
Above, I set a breakpoint at line 197 in handlers.c, where an fopen is performed in my application code. I then continue so the process can keep running. Navigating back to the web page, we see that the application is running now:
NaCl I/O Demo
I am going to try out the http mount point. Entering in “http/index.html” and clicking fopen results in the break point being triggered.
NaCl I/O Demo
[code language=”cpp” light=”true”]
Breakpoint 1, HandleFopen (num_params=2, params=0x3ec3ff8c, output=0x3ec3ff88) at handlers.c:197
197      file = fopen(filename, mode);
(gdb) s
198      if (!file) {
(gdb) l
194      filename = params[0];
195      mode = params[1];
197      file = fopen(filename, mode);
198      if (!file) {
199     *output = PrintfToNewString(“Error: fopen returned a NULL FILE*.”);
200     return 2;
201      }
(gdb) c
But wait! Trying to step in took us right past. Unfortunately, this appears to my untrained eye to be a kernel-level syscall, which is magically intercepted and handled by the SDK. But how do we get in there?

Stepping through all the things

The main application file, nacl_io_demo.c, contains headers pointing to a nacl_io library:
[code language=”cpp” title=”nacl_io_demo.c” firstline=”18″]

#include “ppapi/c/ppp_messaging.h”
#include “nacl_io/nacl_io.h”

That looks promising. The NaCl SDK source code is located at src/ under the root SDK directory. For me that would be ~/nacl_sdk/pepper_31/src/, with headers in include/. Opening nacl_io.cc reveals more clues, which eventually leads us to a file named suspiciously relevant to our hunt: mount_http.cc. This contains a MountHttp::Open method that appears like a great place to set a breakpoint. But how do we get there?
Just like we did for our application, we need to include debugging symbols for the Native Client SDK. We modify the Makefile to add the same flags we did above:
[code language=”bash” title=”~/nacl_sdk/pepper_31/src/nacl_io/Makefile
” firstline=”10″]

CFLAGS = -Wall -g -O0

and then compile:
[code language=”bash” light=”true”]
$ make TOOLCHAIN=newlib CONFIG=Debug
With 4 am fast approaching, we delve back into our debugging process:

  • Kill GDB with Ctrl-C and then Ctrl-D, and quit.
  • Stop the application server, and start it back up again.
  • Refresh the web page to reload the application.
  • Start GDB back up again.

This time, we set a break point in mount_http.cc:
[code language=”cpp” light=”true”]
(gdb) b mount_http.cc:83
Breakpoint 1 at 0x49dc0: file mount_http.cc, line 83.
(gdb) c
Clicking fopen() on the web page brings us to it:
[code language=”cpp” light=”true”]
Breakpoint 1, nacl_io::MountHttp::Open (this=0x3eee1c60, path=…, mode=0, out_node=0x3ec3fdd0) at mount_http.cc:83
83        error = node->GetStat(NULL);
(gdb) bt
#0  nacl_io::MountHttp::Open (this=0x3eee1c60, path=…, mode=0, out_node=0x3ec3fdd0) at mount_http.cc:83
#1  0x00084ee0 in nacl_io::KernelObject::AcquireMountAndNode (this=0x3eee04d8, path=…, oflags=0, out_mount=0x3ec3fdd4, out_node=0x3ec3fdd0) at kernel_object.cc:102
#2  0x00028fa0 in nacl_io::KernelProxy::open (this=0x3eee04d8, path=0x3eee185e “http/index.html”, oflags=0) at kernel_proxy.cc:161
#3  0x00024900 in ki_open (path=0x3eee185e “http/index.html”, oflag=0) at kernel_intercept.cc:131
#4  0x00043260 in __nacl_irt_open_wrap (pathname=0x3eee185e “http/index.html”, oflag=0, cmode=0, newfd=0x3ec3fe6c) at kernel_wrap_newlib.cc:114
#5  0x00176100 in open (pathname=0x3eee185e “http/index.html”, flags=0) at open.c:34
#6  0x00149580 in _open_r ()
#7  0x00141e80 in _fopen_r ()
#8  0x00142060 in fopen ()
#9  0x00020760 in HandleFopen (num_params=2, params=0x3ec3ff8c, output=0x3ec3ff88) at handlers.c:197
#10 0x00022420 in HandleMessage (message=0x3eee1858 “fopen”) at nacl_io_demo.c:246
#11 0x00022660 in HandleMessageThread (user_data=0x0) at nacl_io_demo.c:280
#12 0x00092840 in nc_thread_starter () at nc_thread.c:118
#13 0x0fa00700 in ?? ()
#14 0x00000000 in ?? ()
By typing in bt, I obtained a backtrace of how in the world we got here. This clears up some of the ambiguity about the black magic that makes NaCl apps work.
From here on, it’s just a matter of stepping through the code, setting breakpoints and, god forbid, print statements if you prefer the back-to-basics approach, till you know enough to solve the problem. Be aware that unless you turn off the debugging flag at chrome:flags, your NaCl application will always wait for nacl-gdb to connect. So be sure to turn it off after you’re done debugging.
In my case, the bug turned out to be in the SDK after all and I filed a bug report (here, if you’re curious). A workaround was easy enough to implement. More on what the hell I’m doing with my C script in a future blog post.
So that was my first day with NaCl… mostly debugging, although I hope to eventually be able to port over legacy desktop apps to the browser for fun and profit.

Does this kind of silliness interest you? Kloudless is hiring Front-End Engineers! If you know when to get your C++ JavaScript stirred rather than shaken, check out https://kloudless.com/jobs#frontend and shoot us your CoffeeScript/TypeScript/Dart preference at work@kloudless.com.

Published By

Vinod Chandru