Artikel getaggt mit Apport

Apport 2.7: ARM retracing support, better hook debugging

I just released Apport 2.7.

The main new feature is supporting foreign architectures in apport-retrace. If apport-retrace works in sandbox mode and works on a crash that was not produced on the same architecture as apport-retrace is running on, it will now build a sandbox for the report’s architecture and invoke gdb with the necessary magic options to produce a proper stack trace (and the other gdb information).
Right now this works for i386, x86_64, and ARMv7, but if someone is interested in making this work for other architectures, please ping me.

This is rolled out to the Launchpad retracers, see for example Bug #1088428. So from now on you can report your armhf crashes to Launchpad and they ought to be processed. Note that I did a mass-cleanup of old armhf crash bugs this morning, as the existing ones were way too old to be retraced.

For those who are running their own retracers for their project: You need to add an armhf specific apt sources list your per-release configuration directory, e. g. Ubuntu 12.04/armhf/sources.list as armhf is on instead of Also, you need to add an armhf crash database to your crashdb.conf and add a cron job for the new architecture. You can see how all this looks like in the configuration files for the Launchpad retracers.

The other improvement concerns package hooks. So far, when a package hook crashed the exception was only printed to stderr, where most people would never see them when using the GTK or KDE frontend. With 2.7 these exceptions are also added to the report itself (HookError_filename), so that they appear in the bug reports.

The release also fixes a couple of bugs, see the release notes for details.

Tags: , , , , ,

Apport 2.5: Better support for third-party and PPA packages

I just released Apport 2.5 with a bunch of new features and some bug fixes.

By default you cannot report bugs and crashes to packages from PPAs, as they are not Ubuntu packages. Some packages like Unity or UbuntuOne define their own crash database which reports bugs against the project instead. This has been a bit cumbersome in the past, as these packages needed to ship a /etc/apport/crashdb.conf.d/ snippet. This has become much easier, package hooks can define a new crash database directly now (#551330):

def add_info(report, ui):
   if determine_whether_to_report_to_upstream:
       report['CrashDB'] = '{ "impl": "launchpad", "project": "picsaw" }'

(Documented in package-hooks.txt)

Apport now also looks for package hooks in /opt (#1020503) if the executable path or a file in the package is somewhere below /opt (it tries all intermediate directories).

With these two, we should have much better support for filing bugs against ARB packages.

This version also finally drops the usage of gksu and moves to PolicyKit. Now we only have one package left in the default install (update-notifier) which uses it. Almost there!

Tags: , , , ,

Apport API users: Watch your data types / Python 3 porting

I just uploaded Apport 2.1 to Quantal. A big change in that version is that the whole code now works with both Python 2 and 3, except for the launchpadlib crash database backend (as we do not yet have a python3-launchpadlib package).

I took some care that apport report objects get along with both strings (unicode type in Python 2) and byte arrays (str type in Python 2) in values, so most package hooks should still work. However, now is the time to check whether they also work with Python 3, to make the impending transition to Python 3 easier.

However, you need to watch out if you use projects or scripts which directly use python-apport to process reports: The open(), write(), and write_mime() methods now require the passed file descriptors to be open in binary mode. You will get an exception otherwise.

A common pattern so far has been code like

  report = apport.Report()

This needs to be changed to

  report = apport.Report()
  with open('myfile.crash', 'rb') as f:

The “with” context is not strictly required, but it takes care of timely closing the files again. This avoids ResourceWarning spew when you run this in test suites or enable warnings.

Tags: , , ,

Apport 1.90: Client-side duplicate checking

Apport and the retracer bot in the Canonical data center have provided server-side automatic closing of duplicate crash report bugs for quite a long time. As we have only kept Apport crash detection enabled in the development release, we got away with this as bugs usually did not get so many duplicates that they became unmanageable. Also, the number of duplicates provided a nice hint to how urgent and widespread a crash actually was.

However, it’s time to end that era and provide something better now:

  • This probably caused a lot of frustration when a reporter of the crash spent time, bandwidth, and creativity to upload the crash data and create a description for it, only to find that it got closed as a duplicate 20 minutes later.
  • Some highly visible crashes sometimes generated up to a hundred duplicates in Launchpad, which was prone to timeouts, and needless catch-up by the retracers.
  • We plan to have a real crash database soon, and eventually want to keep Apport enabled in stable releases. This will raise the number of duplicates that we get by several magnitudes.
  • For common crashes we had to write manual bug patterns to avoid getting even more duplicates.

So with the just released Apport 1.90 we introduce client-side duplicate checking. So from now, when you report a crash, you are likely to see “We already know about this” right away, without having to upload or type anything, and you will get directed to the bug page. You should mark yourself as affected and/or subscribe to the bug, both to get a notification when it gets fixed, and also to properly raise the “hotness” of the bug to bubble up to developer attention.

For the technically interested, this is how we detect duplicates for the “signal” crashes like SIGSEGV (as opposed to e. g. Python crashes, where we always have a fully symbolic stack trace):
As we cannot rely on symbolic stack traces, and do not want to force every user to download tons of debug symbols, Apport now falls back to generating a “crash address signature” which combines the absolute addresses of the (non-symbolic) stack trace and the /proc/pid/maps mapping to a stack of libraries and the relative offsets within those, which is stable under ASLR for a given set of dependency versions. As the offsets are specific to the architecture, we form the signature as combination of the executable name, the signal number, the architecture, and the offset list. For example, the i386 signature of bug looks like this:


As library dependencies can change, we have more than one architecture, and the faulty function can be called from different entry points, there can be many address signatures for a bug, so the database maintains an N:1 mapping. In its current form the signatures are taken as-is, which is much more strict than it needs to be. Once this works in principle, we can refine the matching to also detect duplicates from different entry points by reducing the part that needs to match to the common prefix of several signatures which were proven to be a duplicate by the retracer (which gets a fully symbolic stack trace).

The retracer bots now exports the current duplicate/address signature database to in an indexed text format from where Apport clients can quickly check whether a bug is known.

For the Launchpad crash database implementation we actually check if the bug is readable by the reporter, i. e. it is private and the reporter is in a subscribed team, or the bug is public; if not, we let him report the bug anyway and duplicate it later through the existing server-side retracer, so that the reporter has a chance of getting subscribed to the bug. We also let the bug be filed if the currently existing symbolic stack trace is bad (tagged as apport-failed-retrace) or if a developer wants a new symbolic stack trace with the current libraries (tagged as apport-request-retrace).

As this is a major new feature, I decided that it’s time to call this Apport 2.0. This is the first public beta towards it, thus called 1.90. With Apport’s test driven and agile development the version numbers do not mean much anyway (the retracer bots in the data center always just run trunk, for example), so this is as good time as any to reset the rather large “.26″ minor version that we are at right now.

Tags: , , , , ,

Apport: debug symbol retrieval now in GUI

On a rather calm ten-hour flight to Orlando I once again did some pygobject, udisks, and Apport hacking (It’s scary how productive one can be when not constantly being interrupted by IRC, email, etc). One more visible change amongst these was finally fixing a five year old five-digit bug to integrate apport-retrace into the GUI, now that it does not potentially wreck your installation any more.

If the apport-retrace package is installed, the crash detail dialog will show a new “Examine locally” button:

Apport crash detail dialog

After clicking this, you can choose what do do exactly:

Retrace action dialog

I know this dialog is not a beauty, as it’s implemented using the ui_question_choice() API which is used by package hooks. That makes it work for all available UIs (GTK, KDE, CLI), though, and can easily be extended to have more actions. And if you get this far and want to stack traces, you are used to looking at eye-bleeding gibberish anyway..

Presumably the most useful (and default) action is to download all the debug symbols, open a Terminal, and put you into a GDB session with all these, and the core dump loaded, so that you can poke around the crashed program state with all symbols available.

But you can also run gdb without downloading debug symbols, or just update the .crash report file with a fully symbolic stack trace.

This works just as well in apport-cli, but not yet in the KDE version: Someone needs to implement the equivalent of the apport-gtk implementation to apport-kde and kde/bugreport.ui, i. e. show an “Examine locally” button if self.can_examine_locally() is true, and add an appropriate ui_run_terminal() method (which should be fairly similar to the GTK one, just with Qt/KDEish terminal emulators). But as Kubuntu does not currently use Apport (and also because I didn’t have all the dependencies installed on my laptop) I did not yet do this. Please catch me on IRC/mail/merge proposal if you want to work on this. If you look at above commit, the changes to the GtkBuilder file look huge, but that’s only because I haven’t touched it for ages and the current Glade shuffled the elements quite a bit; it just adds the button to the dialog.

For now this is all sitting in trunk, I’ll do a new upstream release and Ubuntu precise upload soon.

Happy debugging!

Tags: , , , , , ,

apport-retrace made useful

The tool to reprocess an Apport crash report to produce a symbolic stack trace, apport-retrace, has been pretty hard to use on a developer system so far: It either installed the packages from the crash report, plus its debug symbol packages (“ddebs”) into the running system (which frequently caused problems like broken dependencies), or it required setting up a chroot and using apport-chroot with fakechroot and fakeroot.

I’m happy to announce that with Apport 1.22, which landed in Oneiric yesterday, this has now become much easier: In the default mode it just calls gdb on the report’s coredump, i. e. expects that all the necessary packages are already installed and will complain about the missing ones. But with the new --sandbox/-Smode, it will just create a temporary directory, download and unpack packages there, and run gdb with some magic options to consider that directory a “virtual root”. These options haven’t been available back when this stuff was written the first time, which is why it used to be so complicated with fakechroots, etc. Now this does not need any root privileges, chroot() calls, etc.

As it only downloads and installs the bare minimum, and does not involve any of the dpkg/apt overhead (maintainer scripts, etc.), it has also become quite a lot faster. That’s how the apport retracers were able to dig through a backlog of about a thousand bugs in just a couple of hours.

So now, if you locally want to retrace or investigate a crash, you can do

   $ apport-retrace -s -S system /var/crash/_usr_bin_gedit.1000.crash

to get the stack traces on stdout, or

   $ apport-retrace -g -S system /var/crash/_usr_bin_gedit.1000.crash

to be put into a gdb session.

If you do this regularly, it’s highly recommended to use a permanent cache dir, where apt can store its indexes and downloaded packages: Use -C ~/.cache/apport-retrace for this (or the long version --cache).

You can also use this to reprocess crashes for a different release than the one you are currently running, by creating a config directory with an appropriate apt sources.list.

The manpage has all the details. (Note that at the time of this writing, still has the old version — use the local one instead.)

Enjoy, and let me know how this works for you!

Tags: , , , , ,

New Apport feature: custom bug duplicate identification

Apport has provided built-in support for automatically identifying and marking duplicate bug reports for normal signal as well as Python crashes. However, we have more kinds of bug reports submitted through Apport which could benefit from automatic duplication: GPU freezes, package installation failures, kernel oopses, or gcc internal compiler errors, i. e. pretty much everything that gets reported automatically these days.

The latest Apport 1.20 (which also just hit current Ubuntu Natty) now allows package hooks to set a special field DuplicateSignature, which abstracts the concept for other kinds of bug reports where Apport doesn’t do automatic duplication. This field should both uniquely identify the problem class (e. g. “XorgGPUFreeze”) as well as the particular problem, i. e. variables which tell this instance apart from different problems. Aside from these requirements, the value can be any free-form string, Apport only treats it as an opaque value. It doesn’t even need to be ASCII only or only be one line, but for better human inspection I recommend this.

So your report could do something like

   report['DuplicateSignature'] = 'XorgGPUFreeze: instruction %s regs:%s:%s:%s' % (
                     current_instruction, regs[0], regs[1], regs[2])


    report['DuplicateSignature'] = 'PackageFailure: ' + log.splitlines()[-1]

This is integrated into Apport’s already existing CrashDatabase class, which maintains a signature →master bug mapping in a SQLite database. So far these contained the crash signatures (built from executable name, signal number, and the topmost 5 stack trace names). As usual, if an incoming report defines a duplicate signature (from the crash stack trace or from DuplicateSignature), the first one will become the master bug, and all subsequent reports will automatically get closed as a duplicate in Launchpad.

Thanks to Bryce Harrington, who already came up with using this in the latest Intel graphics driver for GPU hangs!

Tags: , , , , , ,

GTK 3.0/GIR application porting: Successes and problems

GNOME 3.0 and Ubuntu Natty are currently undergoing a major architectural shift from GTK 2.0 to 3.0. Part of this is that the previous set of manually maintained language bindings, such as PyGTK, are being deprecated in favor of GObject Introspection, a really cool technology!

For us this means that we have to port all our PyGTK applications from PyGTK 2 to gobject-introspection and GTK 3.0 at the same time. I started with that for my own projects (Apport and Jockey) a few days ago, and along the way encountered a number of problems. They are being fixed (particular thanks to the quick responsiveness of John Palmieri!), so I guess after a few of those iterations, porting should actually become straight forward and solid.

So now I’m proud to announce Apport 1.16 which is now fully working with GTK 3.0 and pygobject-introspection. I just uploaded it to Ubuntu Natty, where it can get some wider testing. I also have a pygi/GTK3.0 branch for Jockey, which is also mostly working now, but it’s blocked on the availability of a GIR for AppIndicator. Once that lands, I’ll release and upload the Jockey as well.

For other people working on porting, these are the bugs and problems I’ve encountered:

  • [pygobject] GtkMessageDialog constructors did not work at all. This was fixed in pygobject 2.27, so it works fine in Natty, but there is little to no chance of making these work in Ubuntu 10.10.
  • [pygobject] Gtk.init_check() crashes (upstream bug). It’s not strictly necessary, though, just avoids crashes (and crash reports) when calling it without a $DISPLAY. I’ll put that back once that gets fixed.
  • [pygobject] You can’t pass unicode objects as method arguments or property values (upstream bug). Method arguments were recently fixed in upstream git head, and I backported it to the Natty package, so these work now. Property values are still outstanding. The workaround for both is to convert unicode values to bytes with .encode('UTF-8') everywhere.
  • [pygobject] is a great help which automates most of the mechanical rewriting work and thus gets you 90% of the porting done automatically. It’s currently missing MessageDialog constants, and is a bit inconvenient to call. I sent two patches upstream (upstream bug) which improve this.
  • [GIR availability] libnotify did not build a GIR yet (upstream bug). This was fixed in upstream git head with some contributions of mine, and I uploaded it to Natty. This works quite well now, although add_action() crashes on passing the callback. This is still to be investigated.
  • [GIR availability] There is no Application Indicator GIR, as already mentioned. Our DX team and Ken VanDine are working on this, so this should get fixed soon.
  • [GTK] Many dialogs now scale in a very ugly manner: Instead of resizing the contents, the dialog just grows huge outer padding. This is due to a change of the default “fill” property, and apparently is not fully understood yet (upstream bug). As a workaround I explicitly added the fill property to the top-level GTKBox in my GTKBuilder .ui files. (Note that this happens in C as well, it’s not a GI specific bug.)
  • [GTK] Building GtkRadioButton groups is currently a bit inconvenient and requires special-casing for the first group member, due to some missing annotations. I sent a patch upstream which fixes that (upstream bug).
  • [GTK] After initial porting, a lot of my dialogs got way too narrow with wrapped labels and text, because GTK 3 changed the behaviour and handling of widget and window sizes. This is intended, not a bug, but it does require some adaptions in the GtkBuilder files and also in the code. The GTK 2 → 3 migration guide has a section about it.

Despite those, I’m impressed how well gobject-introspection already works. It was relatively painless for me to generate a GIR from a libnotify (thanks to the annotations already being mostly correct for building the documentation) and use it from Python. In another couple of months this will be rock solid, and porting our bigger pygtk projects like ubiquity and software-center will hopefully become feasible.

Tags: , , , , , , , , , ,

What I do

It’s been a decade ago when I did my first steps with contributing to Free Software, about seven years when I joined Debian, and about 6 with Canonical and Ubuntu. Time for some reflection what I have done over these years!

Distribution Packaging and Maintenance

My first sponsored Debian upload ever was cracklib2, which seriously needed some love and was looking for a new maintainer. So in that upload I managed to close all outstanding bugs. Thanks to my mentor Martin Godisch about providing a lot of guidance for this!

Since then I’ve maintained various packages, where the most popular ones are certainly the free database server “PostgreSQL” (see next section) and the e-book management software “Calibre”.

“Maintaining” by and large means “making it really easy to get and use this software”. This decomposes to:

  • Packaging it in a way that a simple apt-get install makes the software work out of the box (as far as possible)
  • Provide a default configuration/customizations so that it integrates and plays well with the rest of the system; this includes the paths and permissions for log files, log rotation, debconf, configuration file standards, etc.
  • Be the front line for bug reports from users, sort, answer, and de-duplicate them, and either fix them myself, or forward useful bugs to upstream.
  • Providing security updates for stable releases
  • To some extent, help with the development of the software; this gets mostly driven by user demand, and of course my personal interests.

In August 2004 I got employed by Canonical to work full time on Ubuntu, which pretty much turned a hobby into a profession. I never regretted this in the past years, it’s an awesome job to do!

In principle I’m doing the same thing in Ubuntu as well: Bring stuff from developers to the people out there. Except with a different focus, in Ubuntu my daily bread and butter is the GNOME desktop and stuff around (and immediately below) it. And even though after a long day of bug triaging and debugging I feel a bit low-hearted (“50.000 bugs away from perfection”), when I take a step back and see how much the usage of Free Software in the world has grown since 2004, I am very proud of being part of Ubuntu, which certainly has its fair contribution and share in this growth. So what seemed like a crazy idea from Mark back in 2004 actually has made a remarkable progress.

In the beginning of Ubuntu I mostly sent back patches to the Debian bug tracker, but this evolved quite a bit on both sides: These days I try to keep “my” packages in sync and commit stuff directly to Debian, which works very well with e. g. the pkg-utopia team, which is responsible for HAL, udisks, upower, PolicyKit, and related packages. At this point I want to thank Michael Biebl for being such an awesome guy on the Debian side! Also, it seems that Debian has moved a fair bit away from the strong “Big Maintainer Lock” towards team based maintenance, so these days it is easier than ever to commit stuff directly to Debian for a lot of packages, without much fuss.


I have done a handful of changes to PostgreSQL, but these mostly concerned easy packaging and crash fixes, nothing out of the ordinary. I’m not really a PostgreSQL upstream developer.

The thing I am really proud of is the postgresql-common package, which is a very nice example what a distro can provide on top of upstream: If you install the upstream tarball, you have to manually care for creating clusters, providing a sensible configuration for them, set up SSL, set up log rotation, etc. With postgresql-common, this is all done automatically. The biggest feature it provides is a robust and automatic way of upgrading between major releases with the pg_upgradecluster tool, which takes care of a dozen corner cases and the nontrivial process of dumping the old cluster and reloading the new one. Also, you can effortlessy run several instances of the same version in parallel, so that you can e. g. have a production and a development instance, or try the new 9.0 RC1 while still running the 8.4 production one. (more details)

Crash Reporting with Apport

This has been a pet peeve of mine pretty much from day 1. Back in the old days, crashes in software were a pain to track down: many crashes are hard to reproduce, it takes ages to get useful information from bug reporters, and a lot of data cannot be recovered any more when you try to reproduce and analyze a crash after the fact.

With the growing demand for QA from both Canonical and Ubuntu, in 2006 I finally got some time to start Apport, which would make all this a lot easier: It intercepts crashes as they happen, collects the data that we as a developer need, and makes it very easy for the user to submit them as a bug report. This is accompanied by a backend service (called “retracers”) which would reprocess the bug reports by taking the core dump, reproducing a chroot with the packages and versions that the reporter had, installing the debugging symbols, and re-running gdb, to produce a fully symbolic stack trace.

See this bug report for how this looks like. Since then, tons of crashes were fixed, way more than we could ever have done “the old way” with asking users to rebuild with “-g -O0″, running gdb, etc.

By today, Apport has grown quite a bit: rich bug reports, per-package hooks, automatic duplication of crashes, interactive GUI elements in hooks, etc.

Plumbing Development

Handling hotpluggable hardware has always interested me, since the day when I got my first USB stick and it was ridiculously hard (from an user perspective) to use it:

    $ su -
    # mount -t vfat -o uid=1000 /dev/sda1 /mnt

My first go at this was to write pmount which would allow normal users to mount hotpluggable storage without root privileges and worrying about mount paths and options, and then integrate it into GNOME and HAL. Personally I abandoned it years ago, but it seems other people still use it, so I’m glad that Vincent Fourmond took over the maintenance now.

Since then, the entire stack evolved quite a bit: HAL grew to something useful and rather secure, and finally into some monstrous unmaintainable beast, which is why it was declared dead in 2008, and replaced with the “U” stack: udev, udisks, upower, etc. I enjoy hacking on that a lot, and since it’s part of my Desktop Team Tech Lead/Developer role in Canonical, I can spend some company time on it. so far I worked on bug fixes, small new features, and writing a rather comprehensive test suite for udisks (see my udisks commits so far). I also did a fair share of porting stuff away from HAL to the new stack, including some permanent commitments like maintaining the keymaps in udev.


Before I started with Canonical I was never much of a GUI person: I was fully content with fvwm and a few xterms around it. But as an Ubuntu developer I do dogfooding, and thus I switched to GNOME for my day to day work. It didn’t take long before I really fell in love with it!

Similar to my Debian packages, my upstream involvement with GNOME is mostly integration and bug fixing. As already explained, we get a looooot of bug reports, so my focus is mostly on bug fixing. To date, I sent 93 patches to bugzilla. Since January 2010 I became a committer, so that it’s easier for me to get patches upstream.

Debugging problems and fixing bugs is a pretty tedious task, but I still enjoy the rewarding nice feeling when you finally tracked down something and can close a bug with 50 duplicates, and you have made people’s life a little bit easier from now on.

Tags: , , , , , , ,

Apport crash processing now enabled for Maverick

The Debian import freeze is settled, the first rush of major changes went into Maverick, and the dust now has settled a bit. Thus it’s time to turn back some attention to crashes and quality in general.

This morning I created maverick chroots for the Apport retracers, and they are currently processing the backlog. I also uploaded a new Apport package which now enables crash reporting by default again.

Happy segfaulting!

Tags: , , , , , , ,