Artikel getaggt mit ubuntu

Apport 1.90: Client-side duplicate checking

Apport and the retracer bot in the Canonical data center have provided server-side automatic closing of duplicate crash report bugs for quite a long time. As we have only kept Apport crash detection enabled in the development release, we got away with this as bugs usually did not get so many duplicates that they became unmanageable. Also, the number of duplicates provided a nice hint to how urgent and widespread a crash actually was.

However, it’s time to end that era and provide something better now:

  • This probably caused a lot of frustration when a reporter of the crash spent time, bandwidth, and creativity to upload the crash data and create a description for it, only to find that it got closed as a duplicate 20 minutes later.
  • Some highly visible crashes sometimes generated up to a hundred duplicates in Launchpad, which was prone to timeouts, and needless catch-up by the retracers.
  • We plan to have a real crash database soon, and eventually want to keep Apport enabled in stable releases. This will raise the number of duplicates that we get by several magnitudes.
  • For common crashes we had to write manual bug patterns to avoid getting even more duplicates.

So with the just released Apport 1.90 we introduce client-side duplicate checking. So from now, when you report a crash, you are likely to see “We already know about this” right away, without having to upload or type anything, and you will get directed to the bug page. You should mark yourself as affected and/or subscribe to the bug, both to get a notification when it gets fixed, and also to properly raise the “hotness” of the bug to bubble up to developer attention.

For the technically interested, this is how we detect duplicates for the “signal” crashes like SIGSEGV (as opposed to e. g. Python crashes, where we always have a fully symbolic stack trace):
As we cannot rely on symbolic stack traces, and do not want to force every user to download tons of debug symbols, Apport now falls back to generating a “crash address signature” which combines the absolute addresses of the (non-symbolic) stack trace and the /proc/pid/maps mapping to a stack of libraries and the relative offsets within those, which is stable under ASLR for a given set of dependency versions. As the offsets are specific to the architecture, we form the signature as combination of the executable name, the signal number, the architecture, and the offset list. For example, the i386 signature of bug looks like this:


As library dependencies can change, we have more than one architecture, and the faulty function can be called from different entry points, there can be many address signatures for a bug, so the database maintains an N:1 mapping. In its current form the signatures are taken as-is, which is much more strict than it needs to be. Once this works in principle, we can refine the matching to also detect duplicates from different entry points by reducing the part that needs to match to the common prefix of several signatures which were proven to be a duplicate by the retracer (which gets a fully symbolic stack trace).

The retracer bots now exports the current duplicate/address signature database to in an indexed text format from where Apport clients can quickly check whether a bug is known.

For the Launchpad crash database implementation we actually check if the bug is readable by the reporter, i. e. it is private and the reporter is in a subscribed team, or the bug is public; if not, we let him report the bug anyway and duplicate it later through the existing server-side retracer, so that the reporter has a chance of getting subscribed to the bug. We also let the bug be filed if the currently existing symbolic stack trace is bad (tagged as apport-failed-retrace) or if a developer wants a new symbolic stack trace with the current libraries (tagged as apport-request-retrace).

As this is a major new feature, I decided that it’s time to call this Apport 2.0. This is the first public beta towards it, thus called 1.90. With Apport’s test driven and agile development the version numbers do not mean much anyway (the retracer bots in the data center always just run trunk, for example), so this is as good time as any to reset the rather large “.26” minor version that we are at right now.

Tags: , , , , ,

12.04: Testing FTW

I arrived back home in Augsburg, from last week’s Ubuntu Developer Summit in Orlando, FL. As this is a quality/LTS cycle, we pretty much already knew in advance what to do (bug fixing, bug fixing, some boot speed, and did I mention bug fixing?), but still we had many highly interesting and exciting sessions this time, not so much about what we are going to do, but how we are going to build 12.04.

So far our common practice has been to toss everything new into the development release until Feature Freeze and then try and clean up most of the fallout. Me and many other developers have always cried for having more time for fixing long-standing bugs and not introducing breakage in the first place. It seems that now with 12.04, Ubuntu/Canonical are actually getting serious about it.

(Any resemblance to that postcard from the Kennedy Space Center which I went to last Sunday is of course absolutely unintended and purely coincidental 🙂 ).

The mission statement is now to have working ISOs, stable → development, and daily intra-development upgrades every day, quick and regular cleanup of uninstallable packages, component-mismatches, NBS etc., backed by a new “stable +1” team backed by three people on a rotational shift.

QA team is now setting up daily automatic smoketesting of the installer and other packages which have tests. For the latter we’ll convert some packages to the DEP-8, the proposed format for running autopkgtest on (I’ll do udisks, postgresql-common, pygobject, apport, and jockey soon).

We’ll try do put uploads which might break something (like new libraries) to a staging area first, against which we can run test suites of reverse dependencies before it lands in the new release. As doing this on a large scale still requires infrastructure to be created, we’ll only exercise it for a few packages by uploading to precise-proposed first, but this has a high potential for extension.

We want to commit to fixing major breakage within 3 hours of development time, or otherwise revert the faulty package to the previous version (unless that aggravates problems, such as file conflicts).

Finally, for Canonical upstreams we are introducing “acceptance criteria”, which will hopefully significantly raise the quality and lower the regressions of each Unity etc. release.

So, the mission is clear. In practice we’ll probably have to make some real-life concessions, and Murphy’s law dictates that there still will be some breakage, but we can learn from that as we go.

Let’s build 12.04 LTS!

Tags: , , , ,

Riding the Pangolin

Just took the plunge, using the excellent bandwidth and local mirror at UDS:

$ lsb_release -irc
Distributor ID: Ubuntu
Release: 12.04
Codename: precise

Nothing blew up in my face, so it seems today is a good day to die^Wupgrade.

Tags: , , ,

Happy Birthday Ubuntu!

7 years ago, The Ubuntu 4.10 “The Warty Warthog” was announced. A huge congrats to the community, Canonical, and especially Mark for getting so far from “there” to “here”.

This brings back old memories of my first conference in Oxford in August, the great-great-grandfather to what is UDS these days. Back then, there was no company, no Launchpad, no Blueprints, no work items, no detailled plans, just a bunch of ideas, BoFs, and this was a third of the entire crowd:

Warty Hack Room

Back then we worked on the famous TRLS technology (“Totally Rad Laptop Support”) and were proud when we got the ThinkPads to suspend once. During that conference I wrote pmount to provide automatic mounting of USB sticks in a safe manner. Those were the days… 🙂

But I can also safely say that there are some things that haven’t changed. Even though both the community and the company (which changed away from recently) grew by two magnitudes since then, we still have the same serious attitude, stern look, and formal attire as we had back then:

We are professionals, really!

We are professionals, really!

Tags: , , , , , ,

Top ideas on Ubuntu Brainstorm (March 2011)

Update at 13:06 UTC: Corrected NetworkManager description, thanks Mathieu for pointing out.

A few months ago, Matt Zimmerman kicked offa new tradition of a quarterly review of the most popular Ubuntu Brainstorm ideas. He did the December review, now it was my turn to coordinate the March review.

7zip desktop support (#26504)

The 7zip compression format becomes increasingly more popular these days; Ubuntu releases up to 10.10 did not support it on the desktop support as well as older formats like zip or bzip2.

Ubuntu developer Sebastien Bacher responds:

The 7z format has in fact been supported by file-roller for quite some time but it does require the installation of the command lines utilities to work. The issue is pretty much addressed in Ubuntu 11.04 (Natty) though since file-roller […] will ask you if you want to install “p7zip” when you try open an archive using that format.

The other part of the brainstorm request is to also add support for it to gvfs, i. e. that you can browse a 7zip archive as a virtual storage device.This can’t be supported, as the library which is used for this (libarchive) only supports streamable format for efficiency. 7zip is not streamable, and thus would provide a very poor performance.

Empty directories in the Nautilus file manager (#26335)

In tree view mode, nautilus currently displays an expander symbol even if a directory is empty. This looks slightly confusing and makes it harder to see which directories actually have content.

This is indeed a long-standing known problem (the upstream bug is almost ten years old!). Rodrigo Moya, one of the GNOME maintainers in the Ubuntu desktop team, explains why fixing this is actually a lot harder than it might seem initially: Checking each folder to see if it’s got children or not might be time and CPU consuming when displaying lots of subfolders; it gets worse if you are browsing a directory on a remote or slow virtual file system like gphoto cameras or compressed tarballs.

One possible improvement would be to do the test asynchronously and display/hide the expander arrow as the subfolders are checked, and possibly restrict this to local file systems with a maximum number of directory entries. This would create an inconsistency, though.

So unfortunately it is not very realistic to see this being addressed soon.

Login screen (gdm) improvements (#26482)

This item suggests adding features to gdm which make it more useful, such as adding a clock, widgets, or a guest session without requiring an already existing running user session.

Ubuntu and GNOME developer Robert Ancell has a lot of experience with both gdm as well as his own LightDM project.

He points out that in GNOME 3 a clock was added to the login screen and looks similar to the proposed design. So we will get that in Ubuntu 11.10. Other changes to gdm should be discussed and proposed in the upstream bug tracker.

For 11.10 there is an existing proposal to use LightDM by default. LightDM offers a a lot more and easier possibilities for customization and theming, so any contributions for writing widgets or other improvements will be welcome.

Easy side-by-side window arrangement (#26152)

With nowaday’s modern big screens it often is too wasteful or even impractical to run applications fullscreen. A common case is to arrange two applications (such as a web browser and a document editor) side by side. This hasn’t had any particular support up to Ubuntu 10.10, aside from moving and resizing windows manually to fit.

John Lea of the Canonical Design Team explains how the main use case has been implemented in Ubuntu 11.04:

Windows can be opened into semi-maximised state where they occupy 50% of the screen width simply by dragging the window to the left or right border of the screen. A preview shadow informs the user that if they drop the window in this location the window will be resized. This interaction provides a simple, clean solution to the problem without introducing any additional window chrome.

Note that the remaining part of the request, resizing two adjacent windows at the same time, is not currently provided. It is quite a complex interaction which can also trigger false positives, and probably also requires some deeper design studies to get the user experience and definition of “adjacent” right. There are currently no plans to implement this.

man usability (#25975)

First-time users of the man utility often wonder how to quit the program again after they are done reading. Neither the manpage itself nor –help explain that, or other keys for navigation.

Colin Watson is one of the man-db upstream developers. He responds:

I’ve made a change upstream for man-db 2.6.0 which will address this, by adding “(press h for help or q to quit)” to the default prompt string which is displayed on the bottom line of the screen when reading manual pages. I think this is a reasonable balance between providing guidance and taking up too much screen space, and people who get fed up of seeing it can always follow the documentation in man(1) for customising the prompt.

[…] It will definitely be in Ubuntu 11.10.

Naming of Ethernet connections in the UI (#27250)

When connecting to a wired network, it automatically gets assigned a name like “Auto eth0”. Many people will not know what this is, or even if they do, distinguishing between one or another is difficult.

Our NetworkManager maintainer Mathieu Trudel-Lapierre adopted this problem, and wrote a detailled blog entry about how connection naming will be done in Ubuntu 11.10. In particular, network-manager will make the meaning of the default profiles clearer, and notifications will contain “Wired network” in addition to “eth0”. We still need to keep the actual interface name for more experienced users who want to customize their network configuration.

For the case of telling apart multiple ethernet adapters, Ubuntu 11.04 already layed the foundation for integrating biosdevname, which will provide more meaningful names to Ethernet ports than just enumerating them in an arbitrary order, provided that the BIOS provides names for these. It is not enabled by default yet, but might be in 11.10.

Save dialogs should have the three most recently used folders (#26471)

When saving files you often choose the same couple of folders to store your data. Sadly, the drop down menu for the save-as-dialog box only shows the last folder where you have saved a file. Another common use case is to save a document in e. g. Firefox somewhere, and wanting to open it in another application again.

The desktop world is moving towards better tracking of what the user did most recently, so we asked the Zeitgeist developers about the feasibility of this. Manish Sinha discussed the idea within the project and also with the GTK developers, and summarized the possible options in an email to the technical board list.

We don’t currently know about any developer who wants to work on this. GTK developer Federico Mena Quintero said that it is not too difficult to do, and that he would be happy to guide someone who wants to pick this up. So if this interests you, please give him a ping.

Configure auto-mounting of internal drives (#26946)

Ubuntu (and GNOME in general) does not automount internal hard drive partitions in general, as this might cause unwanted data disruption on e. g. Windows system partitions, and also has a performance impact. However, in some use cases it would actually be practical to do so for selected partitions.

David Zeuten and Martin Pitt, the current udisks upstream maintainers, discussed options how this should be integrated and found an agreement (see the response in brainstorm for details). In short, the gnome-disk-utility program will grow some options which allow you to configure individual partitions similar to this:

Automatically mount this drive:
( ) Never
(X) When I log in
( ) On computer startup

(note that this is in no way a finished design or even user fiendly strings).

The current timeline for this is to implement this for GNOME 3.4, which would be in time for Ubuntu 12.04.

Tags: , , , ,

PyGTK is dead, long live PyGI! – App Developer Week Talk

On next Monday this cycle’s Ubuntu Application Developer Week classes will start.

The topic that kept me busy most in this cycle was Python gobject-introspection, and porting pygtk2 apps to PyGI (see my initial steps and my report from the PyGI hackfest.)

To spread the love, there will be two talks about this next week: On Monday 17:00 UTC the very Tomeu Vizoso himself will explain what gobject-introspection (“GI”) is, why we need it, and how library developers use it to ship a good and useful GI binding (“typelib”) for application developers. I will then follow up on Tuesday 16:00 UTC about the app developer side, in particular how to use the GI typelibs in Python, and how to port PyGTK2 applications to PyGI.

For the most part these sessions are distribution neutral (we don’t have any special sauce for this in Debian/Ubuntu, it all happened right upstream :-)); only a very small fraction of it (where I explain package names, etc.) will be specific to Debian/Ubuntu, but shouldn’t be hard to apply to other distributions as well.

So please feel invited to join, and bombard us with questions!

Tags: , , , , , , ,

New Apport feature: custom bug duplicate identification

Apport has provided built-in support for automatically identifying and marking duplicate bug reports for normal signal as well as Python crashes. However, we have more kinds of bug reports submitted through Apport which could benefit from automatic duplication: GPU freezes, package installation failures, kernel oopses, or gcc internal compiler errors, i. e. pretty much everything that gets reported automatically these days.

The latest Apport 1.20 (which also just hit current Ubuntu Natty) now allows package hooks to set a special field DuplicateSignature, which abstracts the concept for other kinds of bug reports where Apport doesn’t do automatic duplication. This field should both uniquely identify the problem class (e. g. “XorgGPUFreeze”) as well as the particular problem, i. e. variables which tell this instance apart from different problems. Aside from these requirements, the value can be any free-form string, Apport only treats it as an opaque value. It doesn’t even need to be ASCII only or only be one line, but for better human inspection I recommend this.

So your report could do something like

   report['DuplicateSignature'] = 'XorgGPUFreeze: instruction %s regs:%s:%s:%s' % (
                     current_instruction, regs[0], regs[1], regs[2])


    report['DuplicateSignature'] = 'PackageFailure: ' + log.splitlines()[-1]

This is integrated into Apport’s already existing CrashDatabase class, which maintains a signature →master bug mapping in a SQLite database. So far these contained the crash signatures (built from executable name, signal number, and the topmost 5 stack trace names). As usual, if an incoming report defines a duplicate signature (from the crash stack trace or from DuplicateSignature), the first one will become the master bug, and all subsequent reports will automatically get closed as a duplicate in Launchpad.

Thanks to Bryce Harrington, who already came up with using this in the latest Intel graphics driver for GPU hangs!

Tags: , , , , , ,

New tool to check support status of dependencies

A common source of unnoticed depwaits or uninstallability are main packages which introduce new build or binary dependencies from universe. These either require fixing, or filing a main inclusion report.

To help with this, I added a new check-mir script into ubuntu-dev-tools version 0.110, which walks through all build and binary dependencies, checks if they are in main/restricted, also considers alternative dependencies, and create a report with a few hints.

For a main package where everything is alright, it looks like this:

$ check-mir 
Checking support status of build dependencies...

Checking support status of binary dependencies...
All dependencies are supported in main or restricted.

Example output for a totally synthetic situation with universe dependencies, non-preferred alternatives in main, and other special cases:

Checking support status of build dependencies...
 * pmount binary and source package is in universe
  ... alternative libexif-dev is already in main; consider preferring it
 * weechat binary and source package is in universe
 * sendmail-bin is in universe, but its source sendmail is already in main; file an ubuntu-archive bug for promoting the current preferred alternative

Checking support status of binary dependencies...
 * wesnot does not exist (pure virtual?)
 * wesnoth binary and source package is in universe

Please check if this source package needs to get into in main/restricted, or reconsider if the package really needs above dependencies.

Tags: , , , ,

Ubuntu Natty: Where did my changelogs go?

Since last Tuesday, packages built in natty don’t come with a Debian changelog included any more. Due to the continuous demand for downsizing both our installation media, as well as the install footprint, we looked for packages which we should eliminate (duplicate libraries, unnecessary runtimes like our current effort to eliminate perl (-modules, not -base), but also for stuff that users generally don’t need and won’t miss. IMO package changelogs very much fall into the latter category, so they were very high on the “first against the wall” list.

Changelogs are of course a valuable developer tool, but for those it is usually less important to have them available locally, as long as there is a convenient method to access them. For that I wrote a helper tool “apt-changelog” which retrieves it from


$ apt-changelog gnome-panel

now replaces

$ zless /usr/share/doc/gnome-panel/changelog.Debian.gz

apt-changelog is now shipped in apt-utils in Natty.

So far this was discussed pre-UDS on and at UDS in a blueprint and various hallway conversations.

However, there were some concerns, so we got a new compromise to just ship the top 10 changelog records, and add a comment about apt-changelog at the bottom. That way, the most interesting entries are still shipped, and developers and users will get used to “apt-changelog”; this will provide a smoother transition, but still get rid of about 90% of the changelog size. This is implemented by pkgbinarymangler version 81 (just uploaded).

We can re-evaluate this after the next LTS, and eventually drop them completely.

I’ll make sure that by the end of the release all packages that got built between last Tuesday and now will be rebuilt at least once and thus get their changelogs back.

Tags: , ,

PostgreSQL 9.0 final released

After 20 days of final polishing and maturing since the release candidate, the PostgreSQL team released the final 9.0 version today.

Hot off the press, I uploaded postgresql-9.0 final into Debian unstable; they will not go into Debian Squeeze, because Squeeze is frozen and it will take a long time to port all the packaged server side extensions to 9.0.

If you are on Ubuntu 10.04 LTS or Ubuntu 10.10, you can add my PostgreSQL backports for stable Ubuntu releases PPA, which will carry 9.0 until it can be moved to the official Ubuntu backports (i. e. when 9.0 goes into Ubuntu Natty).

Enjoy, and kudos to the PostgreSQL team!

Tags: , , , , ,