OLD | NEW |
(Empty) | |
| 1 #summary tips for debugging on Linux |
| 2 #labels Linux |
| 3 |
| 4 This page is for Chromium-specific debugging tips; learning how to run gdb is ou
t of scope. |
| 5 |
| 6 |
| 7 |
| 8 ## Symbolized stack trace |
| 9 |
| 10 The sandbox can interfere with the internal symbolizer. Use --no-sandbox (but ke
ep this temporary) or an external symbolizer (see tools/valgrind/asan/asan\_symb
olize.py). |
| 11 |
| 12 Generally, do not use --no-sandbox on waterfall bots, sandbox testing is needed.
Talk to security@chromium.org. |
| 13 |
| 14 ## GDB |
| 15 **GDB-7.7 is required in order to debug Chrome on Linux.** |
| 16 |
| 17 Any prior version will fail to resolve symbols or segfault. |
| 18 |
| 19 ### Basic browser process debugging |
| 20 |
| 21 ``` |
| 22 gdb -tui -ex=r --args out/Debug/chrome --disable-seccomp-sandbox http://google.c
om |
| 23 ``` |
| 24 |
| 25 ### Allowing attaching to foreign processes |
| 26 On distributions that use the [Yama LSM](https://www.kernel.org/doc/Documentatio
n/security/Yama.txt) (that includes Ubuntu and Chrome OS), process A can attach
to process B only if A is an ancestor of B. |
| 27 |
| 28 You will probably want to disable this feature by using |
| 29 ``` |
| 30 echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope |
| 31 ``` |
| 32 |
| 33 If you don't you'll get an error message such as "Could not attach to process". |
| 34 |
| 35 Note that you'll also probably want to use --no-sandbox, as explained below. |
| 36 |
| 37 ### Multiprocess Tricks |
| 38 #### Getting renderer subprocesses into gdb |
| 39 Since Chromium itself spawns the renderers, it can be tricky to grab a particula
r with gdb. This command does the trick: |
| 40 ``` |
| 41 chrome --no-sandbox --renderer-cmd-prefix='xterm -title renderer -e gdb --args' |
| 42 ``` |
| 43 The "--no-sandbox" flag is needed because otherwise the seccomp sandbox will kil
l the renderer process on startup, or the setuid sandbox will prevent xterm's ex
ecution. The "xterm" is necessary or gdb will run in the current terminal, whic
h can get particularly confusing since it's running in the background, and if yo
u're also running the main process in gdb, won't work at all (the two instances
will fight over the terminal). To auto-start the renderers in the debugger, send
the "run" command to the debugger: |
| 44 ``` |
| 45 chrome --no-sandbox --renderer-cmd-prefix='xterm -title renderer -e gdb -ex run
--args' |
| 46 ``` |
| 47 If you're using Emacs and `M-x gdb`, you can do |
| 48 ``` |
| 49 chrome "--renderer-cmd-prefix=gdb --args" |
| 50 ``` |
| 51 |
| 52 Note: using the `--renderer-cmd-prefix` option bypasses the zygote launcher, so
the renderers won't be sandboxed. It is generally not an issue, except when you
are trying to debug interactions with the sandbox. If that's what you are doing,
you will need to attach your debugger to a running renderer process (see below)
. |
| 53 |
| 54 You may also want to pass `--disable-hang-monitor` to suppress the hang monitor,
which is rather annoying. |
| 55 |
| 56 You can also use "--renderer-startup-dialog" and attach to the process in order
to debug the renderer code. Go to http://www.chromium.org/blink/getting-started-
with-blink-debugging for more information on how this can be done. |
| 57 |
| 58 #### Choosing which renderers to debug |
| 59 If you are starting multiple renderers then the above means that multiple gdb's
start and fight over the console. Instead, you can set the prefix to point to th
is shell script: |
| 60 |
| 61 ``` |
| 62 #!/bin/sh |
| 63 |
| 64 echo "**** Child $$ starting: y to debug" |
| 65 read input |
| 66 if [ "$input" = "y" ] ; then |
| 67 gdb --args $* |
| 68 else |
| 69 $* |
| 70 fi |
| 71 ``` |
| 72 |
| 73 #### Selective breakpoints |
| 74 When debugging both the browser and renderer process, you might want to have sep
arate set of breakpoints to hit. You can use gdb's command files to accomplish t
his by putting breakpoints in separate files and instructing gdb to load them. |
| 75 |
| 76 ``` |
| 77 gdb -x ~/debug/browser --args chrome --no-sandbox --disable-hang-monitor --rende
rer-cmd-prefix='xterm -title renderer -e gdb -x ~/debug/renderer --args ' |
| 78 ``` |
| 79 |
| 80 Also, instead of running gdb, you can use the script above, which let's you sele
ct which renderer process to debug. Note: you might need to use the full path to
the script and avoid $HOME or ~/. |
| 81 |
| 82 #### Connecting to a running renderer |
| 83 |
| 84 Usually `ps aux | grep chrome` will not give very helpful output. Try `pstree -p
| grep chrome` to get something like |
| 85 |
| 86 ``` |
| 87 | |-bash(21969)---chrome(672)-+-chrome(694) |
| 88 | | |-chrome(695)---chrom
e(696)-+-{chrome}(697) |
| 89 | | |
\-{chrome}(709) |
| 90 | | |-{chrome}(675) |
| 91 | | |-{chrome}(678) |
| 92 | | |-{chrome}(679) |
| 93 | | |-{chrome}(680) |
| 94 | | |-{chrome}(681) |
| 95 | | |-{chrome}(682) |
| 96 | | |-{chrome}(684) |
| 97 | | |-{chrome}(685) |
| 98 | | |-{chrome}(705) |
| 99 | | \-{chrome}(717) |
| 100 ``` |
| 101 |
| 102 Most of those are threads. In this case the browser process would be 672 and the
(sole) renderer process is 696. You can use `gdb -p 696` to attach. Alternativ
ely, you might find out the process ID from Chrome's built-in Task Manager (unde
r the Tools menu). Right-click on the Task Manager, and enable "Process ID" in
the list of columns. |
| 103 |
| 104 Note: by default, sandboxed processes can't be attached by a debugger. To be abl
e to do so, you will need to pass the `--allow-sandbox-debugging` option. |
| 105 |
| 106 If the problem only occurs with the seccomp sandbox enabled (and the previous tr
icks don't help), you could try enabling core-dumps (see the **Core files** sect
ion). That would allow you to get a backtrace and see some local variables, tho
ugh you won't be able to step through the running program. |
| 107 |
| 108 Note: If you're interested in debugging LinuxSandboxIPC process, you can attach
to 694 in the above diagram. The LinuxSandboxIPC process has the same command li
ne flag as the browser process so that it's easy to identify it if you run `pstr
ee -pa`. |
| 109 |
| 110 #### Getting GPU subprocesses into gdb |
| 111 Use `--gpu-launcher` flag instead of `--renderer-cmd-prefix` in the instructions
for renderer above. |
| 112 |
| 113 #### Getting browser\_tests launched browsers into gdb |
| 114 Use environment variable `BROWSER_WRAPPER` instead of `--renderer-cmd-prefix` sw
itch in the instructions above. |
| 115 |
| 116 Example: |
| 117 $ BROWSER\_WRAPPER='xterm -title renderer -e gdb --eval-command=run --eval-comma
nd=quit --args' out/Debug/browser\_tests --gtest\_filter=Print |
| 118 |
| 119 #### Plugin Processes |
| 120 Same strategies as renderers above, but the flag is called `--plugin-launcher`: |
| 121 ``` |
| 122 chrome --plugin-launcher='xterm -e gdb --args' |
| 123 ``` |
| 124 |
| 125 _Note: For now, this does not currently apply to PPAPI plugins because they curr
ently run in the renderer process._ |
| 126 |
| 127 #### Single-Process mode |
| 128 Depending on whether it's relevant to the problem, it's often easier to just run
in "single process" mode where the renderer threads are in-process. Then you c
an just run gdb on the main process. |
| 129 ``` |
| 130 gdb --args chrome --single-process |
| 131 ``` |
| 132 |
| 133 Currently, the --disable-gpu flag is also required, as there are known crashes t
hat occur under TextureImageTransportSurface without it. The crash described in
http://crbug.com/361689 can also sometimes occur, but that crash can be continu
ed from without harm. |
| 134 |
| 135 Note that for technical reasons plugins cannot be in-process, so `--single-proce
ss` only puts the renderers in the browser process. The flag is still useful fo
r debugging plugins (since it's only two processes instead of three) but you'll
still need to use `--plugin-launcher` or another approach. |
| 136 |
| 137 ### Printing Chromium types |
| 138 gdb 7 lets us use Python to write pretty-printers for Chromium types. The direc
tory `tools/gdb/` contains a Python gdb scripts useful for Chromium code. There
are similar scripts [in WebKit](http://trac.webkit.org/wiki/GDB) (in fact, the
Chromium script relies on using it with the WebKit one). |
| 139 |
| 140 To include these pretty-printers with your gdb, put the following into `~/.gdbin
it`: |
| 141 ``` |
| 142 python |
| 143 import sys |
| 144 sys.path.insert(0, "<path/to/chromium/src>/third_party/WebKit/Tools/gdb/") |
| 145 import webkit |
| 146 sys.path.insert(0, "<path/to/chromium/src>/tools/gdb/") |
| 147 import gdb_chrome |
| 148 ``` |
| 149 |
| 150 Pretty printers for std types shouldn't be necessary in gdb 7, but they're provi
ded here in case you're using an older gdb. Put the following into `~/.gdbinit`
: |
| 151 ``` |
| 152 # Print a C++ string. |
| 153 define ps |
| 154 print $arg0.c_str() |
| 155 end |
| 156 |
| 157 # Print a C++ wstring or wchar_t*. |
| 158 define pws |
| 159 printf "\"" |
| 160 set $c = (wchar_t*)$arg0 |
| 161 while ( *$c ) |
| 162 if ( *$c > 0x7f ) |
| 163 printf "[%x]", *$c |
| 164 else |
| 165 printf "%c", *$c |
| 166 end |
| 167 set $c++ |
| 168 end |
| 169 printf "\"\n" |
| 170 end |
| 171 ``` |
| 172 |
| 173 [More STL GDB macros](http://www.yolinux.com/TUTORIALS/src/dbinit_stl_views-1.01
.txt) |
| 174 |
| 175 ### Graphical Debugging Aid for Chromium Views |
| 176 |
| 177 The following link describes a tool that can be used on Linux, Windows and Mac u
nder GDB. |
| 178 |
| 179 http://code.google.com/p/chromium/wiki/GraphicalDebuggingAidChromiumViews |
| 180 |
| 181 ### Faster startup |
| 182 |
| 183 Use the gdb-add-index script (e.g. build/gdb-add-index out/Debug/browser\_tests) |
| 184 |
| 185 Only makes sense if you run the binary multiple times or maybe if you use the co
mponent build since most .so files won't require reindexing on a rebuild. |
| 186 |
| 187 See https://groups.google.com/a/chromium.org/forum/#!searchin/chromium-dev/gdb-a
dd-index/chromium-dev/ELRuj1BDCL4/5Ki4LGx41CcJ for more info. |
| 188 |
| 189 Alternatively, specify: |
| 190 ``` |
| 191 linux_use_debug_fission=0 |
| 192 ``` |
| 193 |
| 194 in GYP\_DEFINES. This improves load time of gdb significantly at the cost of lin
k time. |
| 195 |
| 196 ## Core files |
| 197 `ulimit -c unlimited` should cause all Chrome processes (run from that shell) to
dump cores, with the possible exception of some sandboxed processes. |
| 198 |
| 199 Some sandboxed subprocesses might not dump cores unless you pass the `--allow-sa
ndbox-debugging` flag. |
| 200 |
| 201 If the problem is a freeze rather than a crash, you may be able to trigger a cor
e-dump by sending SIGABRT to the relevant process: |
| 202 ``` |
| 203 kill -6 [process id] |
| 204 ``` |
| 205 |
| 206 ## Breakpad minidump files |
| 207 |
| 208 See LinuxMinidumpToCore |
| 209 |
| 210 ## Running Tests |
| 211 Many of our tests bring up windows on screen. This can be annoying (they steal
your focus) and hard to debug (they receive extra events as you mouse over them)
. Instead, use `Xvfb` or `Xephyr` to run a nested X session to debug them, as o
utlined on LayoutTestsLinux. |
| 212 |
| 213 ### Browser tests |
| 214 By default the browser\_tests forks a new browser for each test. To debug the b
rowser side of a single test, use a command like |
| 215 ``` |
| 216 gdb --args out/Debug/browser_tests --single_process --gtest_filter=MyTestName |
| 217 ``` |
| 218 **note the underscore in single\_process** -- this makes the test harness and br
owser process share the outermost process. |
| 219 |
| 220 |
| 221 To debug a renderer process in this case, use the tips above about renderers. |
| 222 |
| 223 ### Layout tests |
| 224 See LayoutTestsLinux for some tips. In particular, note that it's possible to d
ebug a layout test via `ssh`ing to a Linux box; you don't need anything on scree
n if you use `Xvfb`. |
| 225 |
| 226 ### UI tests |
| 227 UI tests are run in forked browsers. Unlike browser tests, you cannot do any sin
gle process tricks here to debug the browser. See below about `BROWSER_WRAPPER`
. |
| 228 |
| 229 To pass flags to the browser, use a command line like `--extra-chrome-flags="--f
oo --bar"`. |
| 230 |
| 231 ### Timeouts |
| 232 UI tests have a confusing array of timeouts in place. (Pawel is working on redu
cing the number of timeouts.) To disable them while you debug, set the timeout
flags to a large value: |
| 233 * `--test-timeout=100000000` |
| 234 * `--ui-test-action-timeout=100000000` |
| 235 * `--ui-test-terminate-timeout=100000000` |
| 236 |
| 237 ### To replicate Window Manager setup on the bots |
| 238 Chromium try bots and main waterfall's bots run tests under Xvfb&openbox combina
tion. Xvfb is an X11 server that redirects the graphical output to the memeory,
and openbox is a simple window manager that is running on top of Xvfb. The behav
ior of openbox is markedly different when it comes to focus management and other
window tasks, so test that runs fine locally may fail or be flaky on try bots.
To run the tests on a local machine as on a bot, follow these steps: |
| 239 |
| 240 Make sure you have openbox: |
| 241 ``` |
| 242 apt-get install openbox |
| 243 ``` |
| 244 Start Xvfb and openbox on a particular display: |
| 245 ``` |
| 246 Xvfb :6.0 -screen 0 1280x1024x24 & DISPLAY=:6.0 openbox & |
| 247 ``` |
| 248 Run your tests with graphics output redirected to that display: |
| 249 ``` |
| 250 DISPLAY=:6.0 out/Debug/browser_tests --gtest_filter="MyBrowserTest.MyActivateWin
dowTest" |
| 251 ``` |
| 252 You can look at a snapshot of the output by: |
| 253 ``` |
| 254 xwd -display :6.0 -root | xwud |
| 255 ``` |
| 256 |
| 257 Alternatively, you can use testing/xvfb.py to set up your environment for you: |
| 258 ``` |
| 259 testing/xvfb.py out/Debug out/Debug/browser_tests --gtest_filter="MyBrowserTest.
MyActivateWindowTest" |
| 260 ``` |
| 261 |
| 262 ### BROWSER\_WRAPPER |
| 263 You can also get the browser under a debugger by setting the `BROWSER_WRAPPER` e
nvironment variable. (You can use this for `browser_tests` too, but see above f
or discussion of a simpler way.) |
| 264 |
| 265 ``` |
| 266 BROWSER_WRAPPER='xterm -e gdb --args' out/Debug/browser_tests |
| 267 ``` |
| 268 |
| 269 ### Replicating Trybot Slowness |
| 270 |
| 271 Trybots are pretty stressed, and can sometimes expose timing issues you can't no
rmally reproduce locally. |
| 272 |
| 273 You can simulate this by shutting down all but one of the CPUs (http://www.cyber
citi.biz/faq/debian-rhel-centos-redhat-suse-hotplug-cpu/) and running a CPU load
ing tool (e.g., http://www.devin.com/lookbusy/). Now run your test. It will run
slowly, but any flakiness found by the trybot should replicate locally now - and
often nearly 100% of the time. |
| 274 |
| 275 ## Logging |
| 276 ### Seeing all LOG(foo) messages |
| 277 Default log level hides `LOG(INFO)`. Run with `--log-level=0` and `--enable-log
ging=stderr` flags. |
| 278 |
| 279 Newer versions of chromium with VLOG may need --v=1 too. For more VLOG tips, see
the chromium-dev thread: http://groups.google.com/a/chromium.org/group/chromium
-dev/browse_thread/thread/dcd0cd7752b35de6?pli=1 |
| 280 |
| 281 ### Seeing IPC debug messages |
| 282 Run with CHROME\_IPC\_LOGGING=1 eg. |
| 283 ``` |
| 284 CHROME_IPC_LOGGING=1 out/Debug/chrome |
| 285 ``` |
| 286 or within gdb: |
| 287 ``` |
| 288 set environment CHROME_IPC_LOGGING 1 |
| 289 ``` |
| 290 |
| 291 If some messages show as unknown, check if the list of IPC message headers in ch
rome/common/logging\_chrome.cc is up-to-date. In case this file reference goes o
ut of date, try looking for usage of macros like IPC\_MESSAGE\_LOG\_ENABLED or I
PC\_MESSAGE\_MACROS\_LOG\_ENABLED. |
| 292 |
| 293 ## Using valgrind |
| 294 |
| 295 To run valgrind on the browser and renderer processes, with our suppression file
and flags: |
| 296 ``` |
| 297 $ cd $CHROMIUM_ROOT/src |
| 298 $ tools/valgrind/valgrind.sh out/Debug/chrome |
| 299 ``` |
| 300 |
| 301 You can use valgrind on chrome and/or on the renderers e.g |
| 302 `valgrind --smc-check=all ../sconsbuild/Debug/chrome` |
| 303 or by passing valgrind as the argument to `--render-cmd-prefix`. |
| 304 |
| 305 Beware that there are several valgrind "false positives" e.g. pickle, sqlite and
some instances in webkit that are ignorable. On systems with prelink and addres
s space randomization (e.g. Fedora), you may also see valgrind errors in libstdc
++ on startup and in gnome-breakpad. |
| 306 |
| 307 Valgrind doesn't seem to play nice with tcmalloc. To disable tcmalloc run GYP |
| 308 ``` |
| 309 $ cd $CHROMIUM_ROOT/src |
| 310 $ build/gyp_chromium -Duse_allocator=none |
| 311 ``` |
| 312 and rebuild. |
| 313 |
| 314 ## Profiling |
| 315 See https://sites.google.com/a/chromium.org/dev/developers/profiling-chromium-an
d-webkit and http://code.google.com/p/chromium/wiki/LinuxProfiling |
| 316 |
| 317 ## i18n |
| 318 We obey your system locale. Try something like: |
| 319 ``` |
| 320 LANG=ja_JP.UTF-8 out/Debug/chrome |
| 321 ``` |
| 322 If this doesn't work, make sure that the LANGUAGE, LC\_ALL and LC\_MESSAGE envir
onment variables aren't set -- they have higher priority than LANG in the order
listed. Alternatively, just do this: |
| 323 |
| 324 ``` |
| 325 LANGUAGE=fr out/Debug/chrome |
| 326 ``` |
| 327 |
| 328 Note that because we use GTK, some locale data comes from the system -- for exam
ple, file save boxes and whether the current language is considered RTL. Withou
t all the language data available, Chrome will use a mixture of your system lang
uage and the language you run Chrome in. |
| 329 |
| 330 Here's how to install the Arabic (ar) and Hebrew (he) language packs: |
| 331 ``` |
| 332 sudo apt-get install language-pack-ar language-pack-he language-pack-gnome-ar la
nguage-pack-gnome-he |
| 333 ``` |
| 334 Note that the `--lang` flag does **not** work properly for this. |
| 335 |
| 336 On non-Debian systems, you need the `gtk20.mo` files. (Please update these docs
with the appropriate instructions if you know what they are.) |
| 337 |
| 338 ## Breakpad |
| 339 See the last section of LinuxCrashDumping; you need to set a gyp variable and an
environment variable for the crash dump tests to work. |
| 340 |
| 341 ## Drag and Drop |
| 342 If you break in a debugger during a drag, Chrome will have grabbed your mouse an
d keyboard so you won't be able to interact with the debugger! To work around t
his, run via `Xephyr`. Instructions for how to use `Xephyr` are on the LayoutTes
tsLinux page. |
| 343 |
| 344 ## Tracking Down Bugs |
| 345 |
| 346 ### Isolating Regressions |
| 347 Old builds are archived here: http://build.chromium.org/buildbot/snapshots/chrom
ium-rel-linux/ |
| 348 |
| 349 `tools/bisect-builds.py` in the tree automates bisecting through the archived bu
ilds. Despite a computer science education, I am still amazed how quickly binar
y search will find its target. |
| 350 |
| 351 ### Screen recording for bug reports |
| 352 `sudo apt-get install gtk-recordmydesktop` |
| 353 |
| 354 ## Version-specific issues |
| 355 |
| 356 ### Google Chrome |
| 357 Google Chrome binaries don't include symbols. Googlers can read where to get sy
mbols from [the Google-internal wiki](http://wiki/Main/ChromeOfficialBuildLinux#
The_Build_Archive). |
| 358 |
| 359 ### Ubuntu Chromium |
| 360 Since we don't build the Ubuntu packages (Ubuntu does) we can't get useful backt
races from them. Direct users to https://wiki.ubuntu.com/Chromium/Debugging . |
| 361 |
| 362 ### Fedora's Chromium |
| 363 Like Ubuntu, but direct users to https://fedoraproject.org/wiki/TomCallaway/Chro
mium_Debug . |
| 364 |
| 365 ### Xlib |
| 366 If you're trying to track down X errors like: |
| 367 ``` |
| 368 The program 'chrome' received an X Window System error. |
| 369 This probably reflects a bug in the program. |
| 370 The error was 'BadDrawable (invalid Pixmap or Window parameter)'. |
| 371 ``` |
| 372 Some strategies are: |
| 373 * pass `--sync` on the command line to make all X calls synchronous |
| 374 * run chrome via [xtrace](http://xtrace.alioth.debian.org/) |
| 375 * turn on IPC debugging (see above section) |
| 376 |
| 377 ### Window Managers |
| 378 To test on various window managers, you can use a nested X server like `Xephyr`.
Instructions for how to use `Xephyr` are on the LayoutTestsLinux page. |
| 379 |
| 380 If you need to test something with hardware accelerated compositing (e.g., compi
z), you can use `Xgl` (`sudo apt-get install xserver-xgl`). E.g.: |
| 381 ``` |
| 382 Xgl :1 -ac -accel glx:pbuffer -accel xv:pbuffer -screen 1024x768 |
| 383 ``` |
| 384 ## Mozilla Tips |
| 385 https://developer.mozilla.org/en/Debugging_Mozilla_on_Linux_FAQ |
OLD | NEW |