$ pkcon what-provides  "gstreamer0.10(decoder-audio/ac3)"
[...]
Installed   	gstreamer0.10-plugins-good-0.10.30.2-2ubuntu2.amd64	GStreamer plugins from the "good" set
Available  	gstreamer0.10-plugins-ugly-0.10.18-3ubuntu4.amd64	GStreamer plugins from the "ugly" set

This is the kind of question your video player would ask the system if it encounters a video it cannot play. In reality they of course use the D-BUS or the library API, but it’s easier to demonstrate with the PackageKit command line client.

PackageKit provides a fair number of those concepts; I recently added LANGUAGE_SUPPORT for packages which provide dictionaries, spell checkers, and other language support for a given language or locale code.

However, PackageKit’s apt backend does not actually implement a lot of these (only CODEC and MODALIAS), and aptdaemons’s PackageKit compatibility API does not implement any. That might be because their upstreams do not know enough how to do the mapping for a particular distro/backend, because doing so involves distro specific code which should not go into upstreams, or simply because of the usual chicken-egg problem of app developers rather doing their own thing instead of using generic APIs.

So this got discussed between Sebastian Heinlein and me, and voila, there it is: it is now very easy to provide Python plugins for “what-provides” to implement any of the existing types. For example, language-selector now ships a plugin which implements LANGUAGE_SUPPORT, so you can ask “which packages do I need for Chinese in China” (i. e. simplified Chinese)?

$ pkcon what-provides "locale(zh_CN)"
[...]
Available   	firefox-locale-zh-hans-10.0+build1-0ubuntu1.all	Simplified Chinese language pack for Firefox
Available   	ibus-sunpinyin-2.0.3-2.amd64            	sunpinyin engine for ibus
Available   	language-pack-gnome-zh-hans-1:12.04+20120130.all	GNOME translation updates for language Simplified Chinese
Available   	ttf-arphic-ukai-0.2.20080216.1-1.all    	"AR PL UKai" Chinese Unicode TrueType font collection Kaiti style
[...]

Rodrigo Moya is currently working on implementing the control-center region panel redesign in a branch. This uses exactly this feature.

In Ubuntu we usually do not use PackageKit itself, but aptdaemon and its PackageKit API compatibility shim python-aptdaemon.pkcompat. So I ported that plugin support for aptdaemon-pkcompat as well, so plugins work with either now. Ubuntu Precise got the new aptdaemon (0.43+bzr769-0ubuntu1) and language-selector (0.63) versions today, so you can start playing around with this now.

So how can you write your own plugins? This is a trivial, although rather nonsense example:

from packagekit import enums

def my_what_provides(apt_cache, provides_type, search):
    if provides_type in (enums.PROVIDES_CODEC, enums.PROVIDES_ANY):
        return [apt_cache["gstreamer-moo"]]
    else:
        raise NotImplementedError('cannot handle type ' + str(provides_type))

The function gets an apt.Cache object, one of enums.PROVIDES_* and the actual search type as described in the documentation (above dummy example does not actually use it). It then decides whether it can handle the request and return a list of apt.package.Package objects (i. e. values in an apt.Cache map), or raise a NotImplementedError otherwise.

You register the plugin through Python pkg-resources in your setup.py (this needs setuptools):

   setup(
       [....]

       entry_points="""[packagekit.apt.plugins]
what_provides=my_plugin_module_name:my_what_provides
""",
       [...])

You can register arbitrarily many plugins, they will be all called and their resulting package lists joined.

All this will hopefully help a bit to push distro specifics to the lowest possible levels, and use upstream friendly and distribution agnostic APIs in your applications.

Nothing blew up in my face, so it seems today is a good day to die^Wupgrade.

If the apport-retrace package is installed, the crash detail dialog will show a new “Examine locally” button:

After clicking this, you can choose what do do exactly:

I know this dialog is not a beauty, as it’s implemented using the ui_question_choice() API which is used by package hooks. That makes it work for all available UIs (GTK, KDE, CLI), though, and can easily be extended to have more actions. And if you get this far and want to stack traces, you are used to looking at eye-bleeding gibberish anyway..

Presumably the most useful (and default) action is to download all the debug symbols, open a Terminal, and put you into a GDB session with all these, and the core dump loaded, so that you can poke around the crashed program state with all symbols available.

But you can also run gdb without downloading debug symbols, or just update the .crash report file with a fully symbolic stack trace.

This works just as well in apport-cli, but not yet in the KDE version: Someone needs to implement the equivalent of the apport-gtk implementation to apport-kde and kde/bugreport.ui, i. e. show an “Examine locally” button if self.can_examine_locally() is true, and add an appropriate ui_run_terminal() method (which should be fairly similar to the GTK one, just with Qt/KDEish terminal emulators). But as Kubuntu does not currently use Apport (and also because I didn’t have all the dependencies installed on my laptop) I did not yet do this. Please catch me on IRC/mail/merge proposal if you want to work on this. If you look at above commit, the changes to the GtkBuilder file look huge, but that’s only because I haven’t touched it for ages and the current Glade shuffled the elements quite a bit; it just adds the button to the dialog.

For now this is all sitting in trunk, I’ll do a new upstream release and Ubuntu precise upload soon.

Happy debugging!

I’m happy to announce that with Apport 1.22, which landed in Oneiric yesterday, this has now become much easier: In the default mode it just calls gdb on the report’s coredump, i. e. expects that all the necessary packages are already installed and will complain about the missing ones. But with the new --sandbox/-Smode, it will just create a temporary directory, download and unpack packages there, and run gdb with some magic options to consider that directory a “virtual root”. These options haven’t been available back when this stuff was written the first time, which is why it used to be so complicated with fakechroots, etc. Now this does not need any root privileges, chroot() calls, etc.

As it only downloads and installs the bare minimum, and does not involve any of the dpkg/apt overhead (maintainer scripts, etc.), it has also become quite a lot faster. That’s how the apport retracers were able to dig through a backlog of about a thousand bugs in just a couple of hours.

So now, if you locally want to retrace or investigate a crash, you can do

   $ apport-retrace -s -S system /var/crash/_usr_bin_gedit.1000.crash

to get the stack traces on stdout, or

   $ apport-retrace -g -S system /var/crash/_usr_bin_gedit.1000.crash

to be put into a gdb session.

If you do this regularly, it’s highly recommended to use a permanent cache dir, where apt can store its indexes and downloaded packages: Use -C ~/.cache/apport-retrace for this (or the long version --cache).

You can also use this to reprocess crashes for a different release than the one you are currently running, by creating a config directory with an appropriate apt sources.list.

The manpage has all the details. (Note that at the time of this writing, manpages.ubuntu.com still has the old version — use the local one instead.)

Enjoy, and let me know how this works for you!

A few months ago, Matt Zimmerman kicked offa new tradition of a quarterly review of the most popular Ubuntu Brainstorm ideas. He did the December review, now it was my turn to coordinate the March review.

7zip desktop support (#26504)

The 7zip compression format becomes increasingly more popular these days; Ubuntu releases up to 10.10 did not support it on the desktop support as well as older formats like zip or bzip2.

Ubuntu developer Sebastien Bacher responds:

The 7z format has in fact been supported by file-roller for quite some time but it does require the installation of the command lines utilities to work. The issue is pretty much addressed in Ubuntu 11.04 (Natty) though since file-roller [...] will ask you if you want to install “p7zip” when you try open an archive using that format.

The other part of the brainstorm request is to also add support for it to gvfs, i. e. that you can browse a 7zip archive as a virtual storage device.This can’t be supported, as the library which is used for this (libarchive) only supports streamable format for efficiency. 7zip is not streamable, and thus would provide a very poor performance.

Empty directories in the Nautilus file manager (#26335)

In tree view mode, nautilus currently displays an expander symbol even if a directory is empty. This looks slightly confusing and makes it harder to see which directories actually have content.

This is indeed a long-standing known problem (the upstream bug is almost ten years old!). Rodrigo Moya, one of the GNOME maintainers in the Ubuntu desktop team, explains why fixing this is actually a lot harder than it might seem initially: Checking each folder to see if it’s got children or not might be time and CPU consuming when displaying lots of subfolders; it gets worse if you are browsing a directory on a remote or slow virtual file system like gphoto cameras or compressed tarballs.

One possible improvement would be to do the test asynchronously and display/hide the expander arrow as the subfolders are checked, and possibly restrict this to local file systems with a maximum number of directory entries. This would create an inconsistency, though.

So unfortunately it is not very realistic to see this being addressed soon.

Login screen (gdm) improvements (#26482)

This item suggests adding features to gdm which make it more useful, such as adding a clock, widgets, or a guest session without requiring an already existing running user session.

Ubuntu and GNOME developer Robert Ancell has a lot of experience with both gdm as well as his own LightDM project.

He points out that in GNOME 3 a clock was added to the login screen and looks similar to the proposed design. So we will get that in Ubuntu 11.10. Other changes to gdm should be discussed and proposed in the upstream bug tracker.

For 11.10 there is an existing proposal to use LightDM by default. LightDM offers a a lot more and easier possibilities for customization and theming, so any contributions for writing widgets or other improvements will be welcome.

Easy side-by-side window arrangement (#26152)

With nowaday’s modern big screens it often is too wasteful or even impractical to run applications fullscreen. A common case is to arrange two applications (such as a web browser and a document editor) side by side. This hasn’t had any particular support up to Ubuntu 10.10, aside from moving and resizing windows manually to fit.

John Lea of the Canonical Design Team explains how the main use case has been implemented in Ubuntu 11.04:

Windows can be opened into semi-maximised state where they occupy 50% of the screen width simply by dragging the window to the left or right border of the screen. A preview shadow informs the user that if they drop the window in this location the window will be resized. This interaction provides a simple, clean solution to the problem without introducing any additional window chrome.

Note that the remaining part of the request, resizing two adjacent windows at the same time, is not currently provided. It is quite a complex interaction which can also trigger false positives, and probably also requires some deeper design studies to get the user experience and definition of “adjacent” right. There are currently no plans to implement this.

man usability (#25975)

First-time users of the man utility often wonder how to quit the program again after they are done reading. Neither the manpage itself nor –help explain that, or other keys for navigation.

Colin Watson is one of the man-db upstream developers. He responds:

I’ve made a change upstream for man-db 2.6.0 which will address this, by adding “(press h for help or q to quit)” to the default prompt string which is displayed on the bottom line of the screen when reading manual pages. I think this is a reasonable balance between providing guidance and taking up too much screen space, and people who get fed up of seeing it can always follow the documentation in man(1) for customising the prompt.

[...] It will definitely be in Ubuntu 11.10.

Naming of Ethernet connections in the UI (#27250)

When connecting to a wired network, it automatically gets assigned a name like “Auto eth0″. Many people will not know what this is, or even if they do, distinguishing between one or another is difficult.

Our NetworkManager maintainer Mathieu Trudel-Lapierre adopted this problem, and wrote a detailled blog entry about how connection naming will be done in Ubuntu 11.10. In particular, network-manager will make the meaning of the default profiles clearer, and notifications will contain “Wired network” in addition to “eth0″. We still need to keep the actual interface name for more experienced users who want to customize their network configuration.

For the case of telling apart multiple ethernet adapters, Ubuntu 11.04 already layed the foundation for integrating biosdevname, which will provide more meaningful names to Ethernet ports than just enumerating them in an arbitrary order, provided that the BIOS provides names for these. It is not enabled by default yet, but might be in 11.10.

Save dialogs should have the three most recently used folders (#26471)

When saving files you often choose the same couple of folders to store your data. Sadly, the drop down menu for the save-as-dialog box only shows the last folder where you have saved a file. Another common use case is to save a document in e. g. Firefox somewhere, and wanting to open it in another application again.

The desktop world is moving towards better tracking of what the user did most recently, so we asked the Zeitgeist developers about the feasibility of this. Manish Sinha discussed the idea within the project and also with the GTK developers, and summarized the possible options in an email to the technical board list.

We don’t currently know about any developer who wants to work on this. GTK developer Federico Mena Quintero said that it is not too difficult to do, and that he would be happy to guide someone who wants to pick this up. So if this interests you, please give him a ping.

Configure auto-mounting of internal drives (#26946)

Ubuntu (and GNOME in general) does not automount internal hard drive partitions in general, as this might cause unwanted data disruption on e. g. Windows system partitions, and also has a performance impact. However, in some use cases it would actually be practical to do so for selected partitions.

David Zeuten and Martin Pitt, the current udisks upstream maintainers, discussed options how this should be integrated and found an agreement (see the response in brainstorm for details). In short, the gnome-disk-utility program will grow some options which allow you to configure individual partitions similar to this:


Automatically mount this drive:
( ) Never
(X) When I log in
( ) On computer startup


(note that this is in no way a finished design or even user fiendly strings).

The current timeline for this is to implement this for GNOME 3.4, which would be in time for Ubuntu 12.04.

It would be great if you could add package name info for Fedora/SUSE/etc., and perhaps add more example links for porting different kinds of software! Please also let me know if you have suggestions how to improve the structure of the page.

The topic that kept me busy most in this cycle was Python gobject-introspection, and porting pygtk2 apps to PyGI (see my initial steps and my report from the PyGI hackfest.)

To spread the love, there will be two talks about this next week: On Monday 17:00 UTC the very Tomeu Vizoso himself will explain what gobject-introspection (“GI”) is, why we need it, and how library developers use it to ship a good and useful GI binding (“typelib”) for application developers. I will then follow up on Tuesday 16:00 UTC about the app developer side, in particular how to use the GI typelibs in Python, and how to port PyGTK2 applications to PyGI.

For the most part these sessions are distribution neutral (we don’t have any special sauce for this in Debian/Ubuntu, it all happened right upstream ); only a very small fraction of it (where I explain package names, etc.) will be specific to Debian/Ubuntu, but shouldn’t be hard to apply to other distributions as well.

So please feel invited to join, and bombard us with questions!

The latest Apport 1.20 (which also just hit current Ubuntu Natty) now allows package hooks to set a special field DuplicateSignature, which abstracts the concept for other kinds of bug reports where Apport doesn’t do automatic duplication. This field should both uniquely identify the problem class (e. g. “XorgGPUFreeze”) as well as the particular problem, i. e. variables which tell this instance apart from different problems. Aside from these requirements, the value can be any free-form string, Apport only treats it as an opaque value. It doesn’t even need to be ASCII only or only be one line, but for better human inspection I recommend this.

So your report could do something like

   report['DuplicateSignature'] = 'XorgGPUFreeze: instruction %s regs:%s:%s:%s' % (
                     current_instruction, regs[0], regs[1], regs[2])

or

    report['DuplicateSignature'] = 'PackageFailure: ' + log.splitlines()[-1]

This is integrated into Apport’s already existing CrashDatabase class, which maintains a signature →master bug mapping in a SQLite database. So far these contained the crash signatures (built from executable name, signal number, and the topmost 5 stack trace names). As usual, if an incoming report defines a duplicate signature (from the crash stack trace or from DuplicateSignature), the first one will become the master bug, and all subsequent reports will automatically get closed as a duplicate in Launchpad.

Thanks to Bryce Harrington, who already came up with using this in the latest Intel X.org graphics driver for GPU hangs!

Last week I was in Prague to attend the GNOME/Python 2011 Hackfest for gobject-introspection, to which Tomeu Vizoso kindly invited me after I started working with PyGI some months ago. It happened at a place called brmlab which was quite the right environment for a bunch of 9 hackers: Some comfy couches and chairs, soldering irons, lots of old TV tubes, chips, and other electronics, a big Pirate flag, really good Wifi, plenty of Club Mate and Coke supplies, and not putting unnecessary effort into mundane things like wallpapers.

It was really nice to get to know the upstream experts John (J5) Palmieri and Tomeu Vizoso (check out Tomeu’s blog post for his summary and some really nice photos). When sitting together in a room, fully focussing on this area for a full week, it’s so much easier to just ask them about something and getting things done and into upstream than on IRC or bugzilla, where you don’t know each other personally. I certainly learned a lot this week (and not only how great Czech beer tastes )!

So what did I do?

Application porting

After already having ported four Ubuntu PyGTK applications to GI before (apport, jockey, aptdaemon, and language-selector),
my main goal and occupation during this week was to start porting a bigger PyGTK application. I picked system-config-printer, as it’s two magnitudes bigger than the previous projects, exercises quite a lot more of the GTK GI bindings, and thus also exposes a lot more GTK annotation and pygobject bugs. This resulted in a new pygi s-c-p branch which has the first 100 rounds of “test, break, fix” iterations. It now at least starts, and you can do a number of things with it, but a lot of functionality is still broken.

As a kind of “finger exercise” and also to check for how well pygi-convert works for small projects now, I also ported computer-janitor. This went really well (I had it working after about 30 minutes), and also led me to finally fixing the unicode vs. str mess for GtkTreeView that you got so far with Python 2.x.

pygobject and GTK fixes

Porting system-config-printer and computer-janitor uncovered a lot of opportunities to improve pygi-convert.sh, a big “perl -e” kind of script to do the mechanical grunt work of the porting process. It doesn’t fix up changed signatures (such as adding missing arguments which were default arguments in PyGTK, or the ubiquitous “user_data” argument for signal handlers), but at least it gets a lot of namespaces, method, and constant names right.

I also fixed three annotation fixes in GTK+. We also collaboratively reviewed and tested Pavel’s annotation branch which helped to fix tons of problems, especially after Steve Frécinaux’s excellent reference leak fix, so if you play around with current pygobject git head, you really also have to use the current GTK+ git head.

Speaking of which, if you want to port applications and always stay on top of the pygobject/GTK development without having to clutter your package system with “make install”s of those, it works very well to have this in your ~/.bashrc:

export GI_TYPELIB_PATH=$HOME/projects/gtk/gtk:$HOME/projects/gtk/gdk
export PYTHONPATH=$HOME/projects/pygobject

Better GVariant/GDBus support

The GNOME world is moving from the old dbus-glib python bindings to GDBus, which is integrated into GLib. However, dbus-python exposed a really nice and convenient way of doing D-Bus calls, while using GDBus from Python was hideously complicated, especially for nontrivial arguments with empty or nested arrays:

from gi.repository import Gio, GLib
from gi._gi import variant_type_from_string

d = Gio.bus_get_sync(Gio.BusType.SESSION, None)
notify = Gio.DBusProxy.new_sync(d, 0, None, 'org.freedesktop.Notifications',
    '/org/freedesktop/Notifications', 'org.freedesktop.Notifications', None)

vb = GLib.VariantBuilder()
vb.init(variant_type_from_string('r'))
vb.add_value(GLib.Variant('s', 'test'))
vb.add_value(GLib.Variant('u', 1))
vb.add_value(GLib.Variant('s', 'gtk-ok'))
vb.add_value(GLib.Variant('s', 'Hello World!'))
vb.add_value(GLib.Variant('s', 'Subtext'))
# add an empty array
eavb = GLib.VariantBuilder()
eavb.init(variant_type_from_string('as'))
vb.add_value(eavb.end())
# add an empty dict
eavb = GLib.VariantBuilder()
eavb.init(variant_type_from_string('a{sv}'))
vb.add_value(eavb.end())
vb.add_value(GLib.Variant('i', 10000))
args = vb.end()

result = notify.call_sync('Notify', args, 0, -1, None)
id = result.get_child_value(0).get_uint32()
print id

So I went to making the GLib.Variant constructor work properly with nested types and boxed variants, adding Pythonic GVariant iterators and indexing (so that you can treat GVariant dictionaries/arrays/tuples just like their Python equivalents), and finally a Variant.unpack() method for converting the return value of a D-Bus call back into a native Python data type. This looks a lot friendlier now:

from gi.repository import Gio, GLib

d = Gio.bus_get_sync(Gio.BusType.SESSION, None)
notify = Gio.DBusProxy.new_sync(d, 0, None, 'org.freedesktop.Notifications',
    '/org/freedesktop/Notifications', 'org.freedesktop.Notifications', None)

args = GLib.Variant('(susssasa{sv}i)', ('test', 1, 'gtk-ok', 'Hello World!',
    'Subtext', [], {}, 10000))
result = notify.call_sync('Notify', args, 0, -1, None)
id = result.unpack()[0]
print id

I also prepared another patch in GNOME#640181 which will provide the icing on the cake, i. e. handle the variant building/unpacking transparently and make the explicit call_sync() unnecessary:

from gi.repository import Gio, GLib

d = Gio.bus_get_sync(Gio.BusType.SESSION, None)
notify = Gio.DBusProxy.new_sync(d, 0, None, 'org.freedesktop.Notifications',
    '/org/freedesktop/Notifications', 'org.freedesktop.Notifications', None)

result = notify.Notify('(susssasa{sv}i)', 'test', 1, 'gtk-ok', 'Hello World!',
            'Subtext', [], {}, 10000)
print result[0]

I hope that I can get this reviewed and land this soon.

Thanks to our sponsors!

Many thanks to the GNOME Foundation and Collabora for sponsoring this event!