Artikel getaggt mit gnome

Automatically generating documentation from GIR files

Many libraries build a GObject introspection repository (*.gir) these days which allows the library to be used from many scripting (Python, JavaScript, Perl, etc.) and other (e. g. Vala) languages without the need for manually writing bindings for each of those.

One issue that I hear surprisingly often is “there is zero documentation for those bindings”. Tools for building documentation out of a .gir have existed for a long time already, just far too many people seem to not know about them.

For example, to build Yelp XML documentation out of the libnotify bindings for Python:

  $ g-ir-doc-tool --language=Python -o /tmp/notify-doc /usr/share/gir-1.0/Notify-0.7.gir

Then you can call yelp /tmp/notify-doc to browse the documentation. You can of course also use the standard Mallard tools to convert them to HTML for sticking them on a website:

  $ cd /tmp/notify-doc
  $ yelp-build html .

Admittedly they are far from pretty, and there are still lots of refinements that should be done for the documentation itself (like adding language specific examples) and also for the generated result (prettification, dynamic search, and what not), but it’s certainly far from “nothign”, and a good start.

If you are interested in working on this, please show up in #introspection or discuss it on bugzilla, desktop-devel-list@, or the library specific lists/bug trackers.

Tags: , , , , ,

PyGObject 3.7.91 released.

I just released a new PyGObject for GNOME 3.7.91. This brings some marshalling fixes, plugs tons of memory leaks, and now raises a Python DeprecationWarning when your code calls a method which is marked as deprecated in the typelib. Please note that Python hides them by default, so if you are interested in those you need to run python with the -Wd option.

Thanks to all contributors!

  • Fix many memory leaks (#675726, #693402, #691501, #510511, #672224, and several more which are detected by our test suite) (Martin Pitt)
  • Dot not clobber original Gdk/Gtk functions with overrides (Martin Pitt) (#686835)
  • Optimize GValue.get/set_value by setting GValue.g_type to a local (Simon Feltman) (#694857)
  • Run tests with G_SLICE=debug_blocks (Martin Pitt) (#691501)
  • Add override helper for stripping boolean returns (Martin Pitt) (#694431)
  • Drop obsolete pygobject_register_sinkfunc() declaration (Martin Pitt) (#639849)
  • Fix marshalling of C arrays with explicit length in signal arguments (Martin Pitt) (#662241)
  • Fix signedness, overflow checking, and 32 bit overflow of GFlags (Martin Pitt) (#693121)
  • gi/pygi-marshal-from-py.c: Fix build on Visual C++ (Chun-wei Fan) (#692856)
  • Raise DeprecationWarning on deprecated callables (Martin Pitt) (#665084)
  • pygtkcompat: Add Widget.window, scroll_to_mark, and window methods (Simon Feltman) (#694067)
  • pygtkcompat: Add Gtk.Window.set_geometry_hints which accepts keyword arguments (Simon Feltman) (#694067)
  • Ship pygobject.doap for autogen.sh (Martin Pitt) (#694591)
  • Fix crashes in various GObject signal handler functions (Simon Feltman) (#633927)
  • pygi-closure: Protect the GSList prepend with the GIL (Olivier Crête) (#684060)
  • generictreemodel: Fix bad default return type for get_column_type (Simon Feltman)

Tags: , , , , , , ,

umockdev 0.2.1 release

Hot on the heels of yesterday’s big 0.2 release, I pushed out umockdev 0.2.1 with a couple of bug fixes:

  • umockdev-wrapper: Use exec to avoid keeping the shell process around and make killing the subprogram from outside work properly.
  • Fix building with automake 1.12, thanks Peter Hutterer.
  • Support opening several netlink sockets (i. e. udev monitors) at the same time.
  • Fix building with older kernels which don’t have the EVIOCGMTSLOTS ioctl yet.

This fixes the “bind: address already in use” errors that were popping up in X.org and upower when running under umockdev, and finally gets us working packages for Ubuntu 12.04 LTS (in the daily-builds PPA).

Tags: , , , , , ,

umockdev 0.2: record/replay input devices

I just released umockdev 0.2.

The big new feature of this release is support for evdev ioctls. I. e. you can now record what e. g. X.org is doing to touchpads, touch screens, etc.:

  $ umockdev-record /dev/input/event15 > /tmp/touchpad.umockdev
  # umockdev-record -i /tmp/touchpad.ioctl /dev/input/event15 -- Xorg -logfile /dev/null

and load that back into a testbed with X.org using the dummy driver:

  cat <<EOF > xorg-dummy.conf
  Section "Device"
        Identifier "test"
        Driver "dummy"
  EndSection
  EOF

  $ umockdev-run -l /tmp/touchpad.umockdev -i /dev/input/event15=/tmp/touchpad.ioctl -- \
       Xorg -config xorg-dummy.conf -logfile /tmp/X.log :5

Then e. g. DISPLAY=:5 xinput will recognize the simulated device. Note that Xvfb won’t work as that does not use udev for device discovery, but only adds the XTest virtual devices and nothing else, so you need to use the real X.org with the dummy driver to run this as a normal user.

This enables easier debugging of new kinds of input devices, as well as writing tests for handling multiple touchscreens/monitors, integration tests of Wacom devices, and so on.

This release now also works with older automakes and Vala 0.16, so that you can use this from Ubuntu 12.04 LTS. The daily PPA now also has packages for that.

Attention: This version does not work any more with recorded ioctl files from version 0.1.

More detailled list of changes:

  • umockdev-run: Fix running of child program to keep stdin.
  • preload: Fix resolution of “/dev” and “/sys”
  • ioctl_tree: Fix endless loop when the first encountered ioctl was unknown
  • preload: Support opening a /dev node multiple times for ioctl emulation (issue #3)
  • Fix parallel build (issue #2)
  • Support xz compressed ioctl files in umockdev_testbed_load_ioctl().
  • Add example umockdev and ioctl files for a gphoto camera and an MTP capable mobile phone.
  • Fix building with automake 1.11.3 and Vala 0.16.
  • Generalize ioctl recording and emulation for ioctls with simple structs, i. e. no pointer fields. This makes it much easier to add more ioctls in the future.
  • Store return values of ioctls in records, as they are not always 0 (like EVIOCGBIT)
  • Add support for ioctl ranges (like EVIOCGABS) and ioctls with variable length (like EVIOCGBIT).
  • Add all reading evdev ioctls, for recording and mocking input devices like touch pads, touch screens, or keyboards. (issue #1)

Tags: , , , , , , , , ,

PyGObject 3.7.90 released

I just released a new PyGObject for GNOME 3.7.90, with a nice set of bug fixes and some internal code cleanup. Thanks to all contributors!

  • overrides: Fix inconsistencies with drag and drop target list API (Simon Feltman) (#680640)
  • pygtkcompat: Add pygtk compatible GenericTreeModel implementation (Simon Feltman) (#682933)
  • overrides: Add support for iterables besides tuples for TreePath creation (Simon Feltman) (#682933)
  • Prefix __module__ attribute of function objects with gi.repository (Niklas Koep) (#693839)
  • configure.ac: only enable code coverage when available (Jonathan Ballet) (#693328)
  • Correctly set properties on object with statically defined properties (Jonathan Ballet) (#693618)
  • autogen.sh: Use gnome-autogen.sh (Martin Pitt) (#693328)
  • Fix reference leaks with transient floating objects (Simon Feltman) (#687522)

Tags: , , , , , ,

umockdev: record and mock hardware for debugging and testing

What is this?

umockdev is a set of tools and a library to mock hardware devices for programs that handle Linux hardware devices. It also provides tools to record the properties and behaviour of particular devices, and to run a program or test suite under a test bed with the previously recorded devices loaded.

This allows developers of software like gphoto or libmtp to receive these records in bug reports and recreate the problem on their system without having access to the affected hardware, as well as writing regression tests for those that do not need any particular privileges and thus are capable of running in standard make check.

After working on it for several weeks and lots of rumbling on G+, it’s now useful and documented enough for the first release 0.1!

Component overview

umockdev consists of the following parts:

  • The umockdev-record program generates text dumps (conventionally called *.umockdev) of some specified, or all of the system’s devices and their sysfs attributes and udev properties. It can also record ioctls that a particular program sends and receives to/from a device, and store them into a text file (conventionally called *.ioctl).
  • The libumockdev library provides the UMockdevTestbed GObject class which builds sysfs and /dev testbeds, provides API to generate devices, attributes, properties, and uevents on the fly, and can load *.umockdev and *.ioctl records into them. It provides VAPI and GI bindings, so you can use it from C, Vala, and any programming language that supports introspection. This is the API that you should use for writing regression tests. You can find the API documentation in docs/reference in the source directory.
  • The libumockdev-preload library intercepts access to /sys, /dev/, the kernel’s netlink socket (for uevents) and ioctl() and re-routes them into the sandbox built by libumockdev. You don’t interface with this library directly, instead you need to run your test suite or other program that uses libumockdev through the umockdev-wrapper program.
  • The umockdev-run program builds a sandbox using libumockdev, can load *.umockdev and *.ioctl files into it, and run a program in that sandbox. I. e. it is a CLI interface to libumockdev, which is useful in the “debug a failure with a particular device” use case if you get the text dumps from a bug report. This automatically takes care of using the preload library, i. e. you don’t need umockdev-wrapper with this. You cannot use this program if you need to simulate uevents or change attributes/properties on the fly; for those you need to use libumockdev directly.

Example: Record and replay PtP/MTP USB devices

So how do you use umockdev? For the “debug a problem” use case you usually don’t want to write a program that uses libumockdev, but just use the command line tools. Let’s capture some runs from libmtp tools, and replay them in a mock environment:

  • Connect your digital camera, mobile phone, or other device which supports PtP or MTP, and locate it in lsusb. For example
      Bus 001 Device 012: ID 0fce:0166 Sony Ericsson Xperia Mini Pro
  • Dump the sysfs device and udev properties:
      $ umockdev-record /dev/bus/usb/001/012 > mobile.umockdev
  • Now record the dynamic behaviour (i. e. usbfs ioctls) of various operations. You can store multiple different operations in the same file, which will share the common communication between them. For example:
      $ umockdev-record --ioctl mobile.ioctl /dev/bus/usb/001/012 mtp-detect
      $ umockdev-record --ioctl mobile.ioctl /dev/bus/usb/001/012 mtp-emptyfolders
  • Now you can disconnect your device, and run the same operations in a mocked testbed. Please note that /dev/bus/usb/001/012 merely echoes what is in mobile.umockdev and it is independent of what is actually in the real /dev directory. You can rename that device in the generated *.umockdev files and on the command line.
      $ umockdev-run --load mobile.umockdev --ioctl /dev/bus/usb/001/012=mobile.ioctl mtp-detect
      $ umockdev-run --load mobile.umockdev --ioctl /dev/bus/usb/001/012=mobile.ioctl mtp-emptyfolders

Example: using the library to fake a battery

If you want to write regression tests, it’s usually more flexible to use the library instead of calling everything through umockdev-run. As a simple example, let’s pretend we want to write tests for upower.

Batteries, and power supplies in general, are simple devices in the sense that userspace programs such as upower only communicate with them through sysfs and uevents. No /dev nor ioctls are necessary. docs/examples/ has two example programs how to use libumockdev to create a fake battery device, change it to low charge, sending an uevent, and running upower on a local test system D-BUS in the testbed, with watching what happens with upower --monitor-detail. battery.c shows how to do that with plain GObject in C, battery.py is the equivalent program in Python that uses the GI binding. You can just run the latter like this:

  umockdev-wrapper python3 docs/examples/battery.py

and you will see that upowerd (which runs on a temporary local system D-BUS in the test bed) will report a single battery with 75% charge, which gets down to 2.5% a second later.

The gist of it is that you create a test bed with

  UMockdevTestbed *testbed = umockdev_testbed_new ();

and add a device with certain sysfs attributes and udev properties with

    gchar *sys_bat = umockdev_testbed_add_device (
            testbed, "power_supply", "fakeBAT0", NULL,
            /* attributes */
            "type", "Battery",
            "present", "1",
            "status", "Discharging",
            "energy_full", "60000000",
            "energy_full_design", "80000000",
            "energy_now", "48000000",
            "voltage_now", "12000000",
            NULL,
            /* properties */
            "POWER_SUPPLY_ONLINE", "1",
            NULL);

You can then e. g. change an attribute and synthesize a “change” uevent with

  umockdev_testbed_set_attribute (testbed, sys_bat, "energy_now", "1500000");
  umockdev_testbed_uevent (testbed, sys_bat, "change");

With Python or other introspected languages, or in Vala it works the same way, except that it looks a bit leaner due to “proper” object semantics.

Packages

I have a packaging branch for Ubuntu and a recipe to do daily builds with the latest upstream code into my daily builds PPA (for 12.10 and raring). I will soon upload it to Raring proper, too.

What’s next?

The current set of features should already get you quite far for a range of devices. I’d love to get feedback from you if you use this for anything useful, in particular how to improve the API, the command line tools, or the text dump format. I’m not really happy with the split between umockdev (sys/dev) and ioctl files and the relatively complicated CLI syntax of umockdev-record, so any suggestion is welcome.

One use case that I have for myself is to extend the coverage of ioctls for input devices such as touch screens and wacom tablets, so that we can write some tests for gnome-settings-daemon plugins.

I also want to find a way to pass ioctls back to the test suite/calling program instead of having to handle them all in the preload library, which would make it a lot more flexible. However, due to the nature of the ioctl ABI this is not easy.

Where to go to

The code is hosted on github in the umockdev project; this started out as a systemd branch to add this functionality to libudev, but after a discussion with Kay we decided to keep it separate. But I kept it in git anyway, given how popular it is today. For the bzr lovers, Launchpad has an import at lp:umockdev.

Release tarballs will be on Launchpad as well. Please file bugs and enhancement requests in the git hub tracker.

Finally, if you have questions or want to discuss something, you can always find me on IRC (pitti on Freenode or GNOME).

Thanks for your attention and happy testing!

Tags: , , , , , , , , ,

PyGObject 3.7.5 released

I just released a new PyGObject for GNOME 3.7.5. Unfortunately master.gnome.org is out of space right now, so I put the new tarball on my Ubuntu people account for the time being.

This again brings a nice set of memory leak and bug fixes, some more reduction of static bindings, and better support for building under Windows.

Thanks to all contributors!

  • Move various signal methods from static bindings to gi and python (Simon Feltman) (#692918)
  • GLib overrides: Support unpacking ‘maybe’ variants (Paolo Borelli) (#693032)
  • Fix ref count leak when creating pygobject wrappers for input args (Mike Gorse) (#675726)
  • Prefix names of typeless enums and flags for GType registration (Simon Feltman) (#692515)
  • Fix compilation with non-C99 compilers such as Visual C++ (Chun-wei Fan) (#692856)
  • gi/overrides/Glib.py: Fix running on Windows/non-Unix (Chun-wei Fan)
  • Do not immediately initialize Gdk and Gtk on import (Martin Pitt) (#692300)
  • Accept ±inf and NaN as float and double values (Martin Pitt) (#692381)
  • Fix repr() of GLib.Variant (Martin Pitt)
  • Fix gtk-demo for Python 3 (Martin Pitt)
  • Define GObject.TYPE_VALUE gtype constant (Martin Pitt)
  • gobject: Go through introspection on property setting (Olivier Crête) (#684062)
  • Clean up caller-allocated GValues and their memory (Mike Gorse) (#691820)
  • Use GNOME_COMPILE_WARNINGS from gnome-common (Martin Pitt)

Tags: , , , , , ,

PyGObject 3.7.4 released

I just released a new PyGObject, for GNOME 3.7.4 which is due on Wednesday.

This release saw a lot of bug and memory leak fixes again, as well as enabling some more data types such as GParamSpec, boxed list properties, or directly setting string members in structs.

Thanks to all contributors!

Summary of changes (see change log for complete details):

  • Allow setting values through GtkTreeModelFilter (Simonas Kazlauskas) (#689624)
  • Support GParamSpec signal arguments from Python (Martin Pitt) (#683099)
  • pygobject_emit(): Fix cleanup on error (Martin Pitt)
  • Add signal emission methods to TreeModel which coerce the path argument (Simon Feltman) (#682933)
  • Add override for GValue (Bastian Winkler) (#677473)
  • Mark caller-allocated boxed structures as having a slice allocated (Mike Gorse) (#699501)
  • pygi-property: Support boxed GSList/GList types (Olivier Crête) (#684059)
  • tests: Add missing backwards compat methods for Python 2.6 (Martin Pitt) (#691646)
  • Allow setting TreeModel values to None (Simon Feltman) (#684094)
  • Set clean-up handler for marshalled arrays (Mike Gorse) (#691509)
  • Support setting string fields in structs (Vadim Rutkovsky) (#678401)
  • Permit plain integers for “gchar” values (Martin Pitt)
  • Allow single byte values for int8 types (Martin Pitt) (#691524)
  • Fix invalid memory access handling errors when registering an enum type (Mike Gorse)
  • Fix (out) arguments in callbacks (Martin Pitt)
  • Fix C to Python marshalling of struct pointer arrays (Martin Pitt)
  • Don’t let Property.setter() method names define property names (Martin Pitt) (#688971)
  • Use g-i stack allocation API (Martin Pitt) (#615982)
  • pyg_value_from_pyobject: support GArray (Ray Strode) (#690514)
  • Fix obsolete automake macros (Marko Lindqvist) (#691101)
  • Change dynamic enum and flag gtype creation to use namespaced naming (Simon Feltman) (#690455)
  • Fix Gtk.UIManager.add_ui_from_string() override for non-ASCII chars (Jonathan Ballet) (#690329)
  • Don’t dup strings before passing them to type registration functions (Mike Gorse) (#690532)
  • Fix marshalling of arrays of boxed struct values (Carlos Garnacho) (#656312)
  • testhelpermodule.c: Do not unref called method (Martin Pitt)

Tags: , , , , , , ,

PyGObject 3.7.3 released

I just released a new PyGObject, for GNOME 3.7.3 which is due on Wednesday.

This is mostly a bug fix release. There is one API addition, it brings back official support for calling GLib.io_add_watch() with a Python file object or fd as first argument, in addition to the official API which expects a GLib.IOChannel object. These modes were marked as deprecated in 3.7.2 (only).

Thanks to all contributors!

Summary of changes (see change log for complete details):

  • Add support for caller-allocated GArray out arguments (Martin Pitt) (#690041)
  • Re-support calling GLib.io_add_watch with an fd or Python file (Martin Pitt)
  • pygtkcompat: Work around IndexError on large flags (Martin Pitt)
  • Fix pyg_value_from_pyobject() range check for uint (Martin Pitt)
  • Fix tests to work with g-i 1.34.2 (Martin Pitt)
  • Fix wrong refcount for GVariant property defaults (Martin Pitt) (#689267)
  • Fix array arguments on 32 bit (Martin Pitt)
  • Add backwards compatible API for GLib.unix_signal_add_full() (Martin Pitt)
  • Drop MININT64/MAXUINT64 workaround, current g-i gets this right now (Martin Pitt)
  • Fix maximum and minimum ranges of TYPE_(U)INT64 properties (Simonas Kazlauskas) (#688949)
  • Ship pygi-convert.sh in tarballs (Martin Pitt) (#688697)
  • Various added and improved tests (Martin Pitt)

Tags: , , , , , , ,

Running a script with unshared mount namespace

When writing system integration tests it often happens that I want to mount some tmpfses over directories like /etc/postgresql/ or /home, and run the whole script with an unshared mount namespace so that (1) it does not interfere with the real system, and (2) is guaranteed to clean up after itself (unmounting etc.) after it ends in any possible way (including SIGKILL, which breaks usual cleanup methods like “trap”, “finally”, “def tearDown()”, “atexit()” and so on).

In gvfs’ and postgresql-common’s tests, which both have been around for a while, I prepare a set of shell commands in a variable and pipe that into unshare -m sh, but that has some major problems: It doesn’t scale well to large programs, looks rather ugly, breaks syntax highlighting in editors, and it destroys the real stdin, so you cannot e. g. call a “bash -i” in your test for interactively debugging a failed test.

I just changed postgresql-common’s test runner to use unshare/tmpfses as well, and needed a better approach. What I eventually figured out preserves stdin, $0, and $@, and still looks like a normal script (i. e. not just a single big string). It still looks a bit hackish, but I can live with that:

#!/bin/sh
set -e
# call ourselves through unshare in a way that keeps normal stdin, $0, and CLI args
unshare -uim sh -- -c "`tail -n +7 $0`" "$0" "$@"
exit $?

# unshared program starts here
set -e
echo "args: $@"
echo "mounting tmpfs"
mount -n -t tmpfs tmpfs /etc
grep /etc /proc/mounts
echo "done"

As Unix/Linux’ shebang parsing is rather limited, I didn’t find a way to do something like

#!/usr/bin/env unshare -m sh

If anyone knows a trick which avoids the “tail -n +7″ hack and having to pay attention to passing around “$@”, I’d appreciate a comment how to simplify this.

Tags: , , , , , ,