$ pkcon what-provides  "gstreamer0.10(decoder-audio/ac3)"
[...]
Installed   	gstreamer0.10-plugins-good-0.10.30.2-2ubuntu2.amd64	GStreamer plugins from the "good" set
Available  	gstreamer0.10-plugins-ugly-0.10.18-3ubuntu4.amd64	GStreamer plugins from the "ugly" set

This is the kind of question your video player would ask the system if it encounters a video it cannot play. In reality they of course use the D-BUS or the library API, but it’s easier to demonstrate with the PackageKit command line client.

PackageKit provides a fair number of those concepts; I recently added LANGUAGE_SUPPORT for packages which provide dictionaries, spell checkers, and other language support for a given language or locale code.

However, PackageKit’s apt backend does not actually implement a lot of these (only CODEC and MODALIAS), and aptdaemons’s PackageKit compatibility API does not implement any. That might be because their upstreams do not know enough how to do the mapping for a particular distro/backend, because doing so involves distro specific code which should not go into upstreams, or simply because of the usual chicken-egg problem of app developers rather doing their own thing instead of using generic APIs.

So this got discussed between Sebastian Heinlein and me, and voila, there it is: it is now very easy to provide Python plugins for “what-provides” to implement any of the existing types. For example, language-selector now ships a plugin which implements LANGUAGE_SUPPORT, so you can ask “which packages do I need for Chinese in China” (i. e. simplified Chinese)?

$ pkcon what-provides "locale(zh_CN)"
[...]
Available   	firefox-locale-zh-hans-10.0+build1-0ubuntu1.all	Simplified Chinese language pack for Firefox
Available   	ibus-sunpinyin-2.0.3-2.amd64            	sunpinyin engine for ibus
Available   	language-pack-gnome-zh-hans-1:12.04+20120130.all	GNOME translation updates for language Simplified Chinese
Available   	ttf-arphic-ukai-0.2.20080216.1-1.all    	"AR PL UKai" Chinese Unicode TrueType font collection Kaiti style
[...]

Rodrigo Moya is currently working on implementing the control-center region panel redesign in a branch. This uses exactly this feature.

In Ubuntu we usually do not use PackageKit itself, but aptdaemon and its PackageKit API compatibility shim python-aptdaemon.pkcompat. So I ported that plugin support for aptdaemon-pkcompat as well, so plugins work with either now. Ubuntu Precise got the new aptdaemon (0.43+bzr769-0ubuntu1) and language-selector (0.63) versions today, so you can start playing around with this now.

So how can you write your own plugins? This is a trivial, although rather nonsense example:

from packagekit import enums

def my_what_provides(apt_cache, provides_type, search):
    if provides_type in (enums.PROVIDES_CODEC, enums.PROVIDES_ANY):
        return [apt_cache["gstreamer-moo"]]
    else:
        raise NotImplementedError('cannot handle type ' + str(provides_type))

The function gets an apt.Cache object, one of enums.PROVIDES_* and the actual search type as described in the documentation (above dummy example does not actually use it). It then decides whether it can handle the request and return a list of apt.package.Package objects (i. e. values in an apt.Cache map), or raise a NotImplementedError otherwise.

You register the plugin through Python pkg-resources in your setup.py (this needs setuptools):

   setup(
       [....]

       entry_points="""[packagekit.apt.plugins]
what_provides=my_plugin_module_name:my_what_provides
""",
       [...])

You can register arbitrarily many plugins, they will be all called and their resulting package lists joined.

All this will hopefully help a bit to push distro specifics to the lowest possible levels, and use upstream friendly and distribution agnostic APIs in your applications.

Today’s language-selector upload provides an aptdaemon plugin which automatically marks corresponding language support packages (translated help, dictionaries, spell checker modules, and translations themselves) for installation for any newly installed package, for all languages that are configured on your system.

For example, I have German and English locales on my system, and no KDE packages. Before, installing GIMP got me just that:

$ aptdcon -i gimp
The following NEW package will be installed (1):
gimp


Now it automatically installs the corresponding localized help:


$ aptdcon -i gimp
The following NEW packages will be installed (4):
gimp gimp-help-common gimp-help-de gimp-help-en


I am using aptdcon here as it points out the effect better than software-center doing all this in the background, but both use aptdaemon, so the effect will be the same.

Likewise, installing the first KDE-ish package will automatically install the KDE language packs:


$ aptdcon -i kate
The following NEW packages will be installed (71):
kate kate-data [...] kdelibs5-data [...] language-pack-kde-de language-pack-kde-en [...]


This is now possible because I rewrote the check-language-support logic from scratch; the old code was very slow, hard to read and a nightmare to maintain, and also depended on a lot of data files. The new code is very fast (figuring out all missing language support packages for all installed packages for all available locales takes 8 ms on my system), and has full test coverage.

While the check-language-support program still works (I rewrote it using the new API), it is easier and probably a lot faster to just use the new API now, e. g. in our Ubiquity installer.

Say goodbye to this 2.5 year old bug!

However, it’s time to end that era and provide something better now:

• This probably caused a lot of frustration when a reporter of the crash spent time, bandwidth, and creativity to upload the crash data and create a description for it, only to find that it got closed as a duplicate 20 minutes later.
• Some highly visible crashes sometimes generated up to a hundred duplicates in Launchpad, which was prone to timeouts, and needless catch-up by the retracers.
• We plan to have a real crash database soon, and eventually want to keep Apport enabled in stable releases. This will raise the number of duplicates that we get by several magnitudes.
• For common crashes we had to write manual bug patterns to avoid getting even more duplicates.

So with the just released Apport 1.90 we introduce client-side duplicate checking. So from now, when you report a crash, you are likely to see “We already know about this” right away, without having to upload or type anything, and you will get directed to the bug page. You should mark yourself as affected and/or subscribe to the bug, both to get a notification when it gets fixed, and also to properly raise the “hotness” of the bug to bubble up to developer attention.

For the technically interested, this is how we detect duplicates for the “signal” crashes like SIGSEGV (as opposed to e. g. Python crashes, where we always have a fully symbolic stack trace):
As we cannot rely on symbolic stack traces, and do not want to force every user to download tons of debug symbols, Apport now falls back to generating a “crash address signature” which combines the absolute addresses of the (non-symbolic) stack trace and the /proc/pid/maps mapping to a stack of libraries and the relative offsets within those, which is stable under ASLR for a given set of dependency versions. As the offsets are specific to the architecture, we form the signature as combination of the executable name, the signal number, the architecture, and the offset list. For example, the i386 signature of bug looks like this:

/usr/bin/rhythmbox:11:i686:/usr/lib/libgstpbutils-0.10.so.0.24.0+c284:/usr/lib/i386-linux-gnu/libgobject-2.0.so.0.3000.0+3337a:/usr/lib/i386-linux-gnu/libgobject-2.0.so.0.3000.0+8e0

As library dependencies can change, we have more than one architecture, and the faulty function can be called from different entry points, there can be many address signatures for a bug, so the database maintains an N:1 mapping. In its current form the signatures are taken as-is, which is much more strict than it needs to be. Once this works in principle, we can refine the matching to also detect duplicates from different entry points by reducing the part that needs to match to the common prefix of several signatures which were proven to be a duplicate by the retracer (which gets a fully symbolic stack trace).

The retracer bots now exports the current duplicate/address signature database to http://people.canonical.com/~ubuntu-archive/apport-duplicates in an indexed text format from where Apport clients can quickly check whether a bug is known.

For the Launchpad crash database implementation we actually check if the bug is readable by the reporter, i. e. it is private and the reporter is in a subscribed team, or the bug is public; if not, we let him report the bug anyway and duplicate it later through the existing server-side retracer, so that the reporter has a chance of getting subscribed to the bug. We also let the bug be filed if the currently existing symbolic stack trace is bad (tagged as apport-failed-retrace) or if a developer wants a new symbolic stack trace with the current libraries (tagged as apport-request-retrace).

As this is a major new feature, I decided that it’s time to call this Apport 2.0. This is the first public beta towards it, thus called 1.90. With Apport’s test driven and agile development the version numbers do not mean much anyway (the retracer bots in the data center always just run trunk, for example), so this is as good time as any to reset the rather large “.26″ minor version that we are at right now.

If the apport-retrace package is installed, the crash detail dialog will show a new “Examine locally” button:

After clicking this, you can choose what do do exactly:

I know this dialog is not a beauty, as it’s implemented using the ui_question_choice() API which is used by package hooks. That makes it work for all available UIs (GTK, KDE, CLI), though, and can easily be extended to have more actions. And if you get this far and want to stack traces, you are used to looking at eye-bleeding gibberish anyway..

Presumably the most useful (and default) action is to download all the debug symbols, open a Terminal, and put you into a GDB session with all these, and the core dump loaded, so that you can poke around the crashed program state with all symbols available.

But you can also run gdb without downloading debug symbols, or just update the .crash report file with a fully symbolic stack trace.

This works just as well in apport-cli, but not yet in the KDE version: Someone needs to implement the equivalent of the apport-gtk implementation to apport-kde and kde/bugreport.ui, i. e. show an “Examine locally” button if self.can_examine_locally() is true, and add an appropriate ui_run_terminal() method (which should be fairly similar to the GTK one, just with Qt/KDEish terminal emulators). But as Kubuntu does not currently use Apport (and also because I didn’t have all the dependencies installed on my laptop) I did not yet do this. Please catch me on IRC/mail/merge proposal if you want to work on this. If you look at above commit, the changes to the GtkBuilder file look huge, but that’s only because I haven’t touched it for ages and the current Glade shuffled the elements quite a bit; it just adds the button to the dialog.

For now this is all sitting in trunk, I’ll do a new upstream release and Ubuntu precise upload soon.

Happy debugging!

Enjoy trying out all the cool new features like builtin synchronous replication or per-column collation settings for correctly handling international strings, or an even finer-grained access control for large environments. Please see the detailled explanation of the new features.

As already announced a few days ago, 9.0 is gone from Ubuntu 11.10, as it is still only a development version and not an LTS. 9.1 will be the version which the next 12.04 LTS will support, so this slightly reduces the number of major upgrades Ubuntu users will need to do. However, 9.0 will still be available in Debian unstable and backports, and the Ubuntu backports PPA for a couple of months to give DB administrators some time to migrate.

We always strive to minimize the number of major versions which we have to support. They not only mean more maintenance for developers, but also more upgrade cycles for the users.

9.0 has not been in any stable Debian or Ubuntu release, and 9.1 final will be released soon. So we recently updated the current Ubuntu development release for 11.10 (“oneiric”) to 9.1. In Debian, the migration from 8.4/9.0 to 9.1 is making good progress, and there is not much which is left until postgresql-9.0 can be removed.

Consequently, I also removed 9.0 from my PostgreSQL backports PPA, as there is nothing any more to backport it from. However, that mostly means that people will now set up installations with 9.1 instead of 9.0, and won’t magically make your already installed 9.0 packages go away. They will just be marked as obsolete in the postgresql-common debconf note.

If you want to build future 9.0 packages yourself, you can do this based on the current branch: bzr branch lp:~pitti/postgresql/debian-9.0, get a the new upstream tarball, name it accordingly, add a new changelog with a new upstream version number, and run bzr bd to build the package (you need to install the bzr-builddeb package for this).

Update 2011-09-09: As I got a ton of pleas to continue the 9.0 backports for a couple of months, and to keep it in Debian unstable for a while longer, I put them back now. I also updated the removal request in Debian to point out that I’m mainly interested in getting 9.0 out of testing. I don’t mind much maintaining it for a couple of more months in unstable. My dear, I had no idea that my backports PPA was that popular!

I’m happy to announce that with Apport 1.22, which landed in Oneiric yesterday, this has now become much easier: In the default mode it just calls gdb on the report’s coredump, i. e. expects that all the necessary packages are already installed and will complain about the missing ones. But with the new --sandbox/-Smode, it will just create a temporary directory, download and unpack packages there, and run gdb with some magic options to consider that directory a “virtual root”. These options haven’t been available back when this stuff was written the first time, which is why it used to be so complicated with fakechroots, etc. Now this does not need any root privileges, chroot() calls, etc.

As it only downloads and installs the bare minimum, and does not involve any of the dpkg/apt overhead (maintainer scripts, etc.), it has also become quite a lot faster. That’s how the apport retracers were able to dig through a backlog of about a thousand bugs in just a couple of hours.

So now, if you locally want to retrace or investigate a crash, you can do

   $ apport-retrace -s -S system /var/crash/_usr_bin_gedit.1000.crash

to get the stack traces on stdout, or

   $ apport-retrace -g -S system /var/crash/_usr_bin_gedit.1000.crash

to be put into a gdb session.

If you do this regularly, it’s highly recommended to use a permanent cache dir, where apt can store its indexes and downloaded packages: Use -C ~/.cache/apport-retrace for this (or the long version --cache).

You can also use this to reprocess crashes for a different release than the one you are currently running, by creating a config directory with an appropriate apt sources.list.

The manpage has all the details. (Note that at the time of this writing, manpages.ubuntu.com still has the old version — use the local one instead.)

Enjoy, and let me know how this works for you!

Warning for upgrades from Beta 1: The on-disk database format changed since Beta-1. So if you already have the beta-1 packages installed, you need to pg_dumpall your 9.1 clusters (if you still need them), and pg_dropcluster all 9.1 clusters before the upgrade. I added a check to the pre-install script to make the postgresql-9.1 package fail early to upgrade if you still have existing 9.1 clusters to avoid data loss.

Due to my recent moving and the Ubuntu Developer Summit it took me a bit to package them for Debian and Ubuntu, but here they are at last. I uploaded postgresql-9.1 to Debian experimental; currently they are sitting in the NEW queue, but I’m sure our restless Debian archive admins will get to it in a few days. I also provided builds for Ubuntu 10.04 LTS, 10.10. and 11.04 in my PostgreSQL backports for stable Ubuntu releases PPA.

I provided full postgresql-common integration, i. e. you can use all the usual tools like pg_createcluster, pg_upgradecluster etc. to install 9.1 side by side with your 8.4/9.0 instances, attempt an upgrade of your existing instances to 9.1 without endangering the running clusters, etc. Fortunately this time there were no deprecated configuration options, so pg_upgradecluster does not actually have to touch your postgresql.conf for the 9.0 →9.1 upgrade.

They pass upstream’s and postgresql-common’s integration test suite, so should be reasonably working. But please let me know about everything that doesn’t, so that we can get them in perfect shape in time for the final release.

I anticipate that 9.1 will be the default (and only supported) version in the next Debian release (wheezy), and will most likely be the one shipped in the next Ubuntu LTS (in 12.04). It might be that the next Ubuntu release 11.10 will still ship with 9.0, but that pretty much depends on how many extensions get ported to 9.1 by feature freeze.