Artikel getaggt mit testing

Booting Ubuntu with systemd: Test packages available

On the last UDS we talked about migrating from upstart to systemd to boot Ubuntu, after Mark announced that Ubuntu will follow Debian in that regard. There’s a lot of work to do, but it parallelizes well once developers can run systemd on their workstations or in VMs easily and the system boots up enough to still be able to work with it.

So today I merged our systemd package with Debian again, dropped the systemd-services split (which wasn’t accepted by Debian and will be unnecessary now), and put it into my systemd PPA. Quite surprisingly, this booted a fresh 14.04 VM pretty much right away (of course there’s no Plymouth prettiness). The main two things which were missing were NetworkManager and lightdm, as these don’t have an init.d script at all (NM) or it isn’t enabled (lightdm). Thus the PPA also contains updated packages for these two which provide a proper systemd unit. With that, the desktop is pretty much fully working, except for some details like cron not running. I didn’t go through /etc/init/*.conf with a small comb yet to check which upstart jobs need to be ported, that’s now part of the TODO list.

So, if you want to help with that, or just test and tell us what’s wrong, take the plunge. In a 14.04 VM (or real machine if you feel adventurous), do

  sudo add-apt-repository ppa:pitti/systemd
  sudo apt-get update
  sudo apt-get dist-upgrade

This will replace systemd-services with systemd, update network-manager and lightdm, and a few libraries. Up to now, when you reboot you’ll still get good old upstart. To actually boot with systemd, press Shift during boot to get the grub menu, edit the Ubuntu stanza, and append this to the linux line: init=/lib/systemd/systemd.

For the record, if pressing shift doesn’t work for you (too fast, VM, or similar), enable the grub menu with

  sudo sed -i '/GRUB_HIDDEN_TIMEOUT/ s/^/#/' /etc/default/grub
  sudo update-grub

Once you are satisfied that your system boots well enough, you can make this permanent by adding the init= option to /etc/default/grub (and possibly remove the comment sign from the GRUB_HIDDEN_TIMEOUT lines) and run sudo update-grub again. To go back to upstart, just edit the file again, remove the init=sudo update-grub again.

I’ll be on the Debian systemd/GNOME sprint next weekend, so I feel reasonably well prepared now. :-)

Update: As the comments pointed out, this bricked /etc/resolv.conf. I now uploaded a resolvconf package to the PPA which provides the missing unit (counterpart to the /etc/init/resolvconf.conf upstart job) and this now works fine. If you are in that situation, please boot with upstart, and do the following to clean up:

  sudo rm /etc/resolv.conf
  sudo ln -s ../run/resolvconf/resolv.conf /etc/resolv.conf

Then you can boot back to systemd.

Tags: , , ,

Creating a local swift server on Ubuntu for testing

Our current autopkgtest machinery uses Jenkins (a private and a public one) and lots of “rsync state files between hosts”, both of which have reached a state where they fall over far too often. It’s flakey, hard to maintain, and hard to extend with new test execution slaves (e. g. for new architectures, or using different test runners). So I’m looking into what it would take to replace this with something robust, modern, and more lightweight.

In our new Continuous Integration world the preferred technologies are RabbitMQ for doing the job distribution (which is delightfully simple to install and use from Python), and OpenStack’s swift for distributed data storage. We have a properly configured swift in our data center, but for local development and experimentation I really just want a dead simple throw-away VM or container which gives me the swift API. swift is quite a bit more complex, and it took me several hours of reading and exercising various tutorials, debugging connection problems, and reading stackexchange to set it up. But now it’s working, and I condensed the whole setup into a single setup-swift.sh shell script.

You can run this in a standard ubuntu container or VM as root:

sudo apt-get install lxc
sudo lxc-create -n swift -t ubuntu -- -r trusty
sudo lxc-start -n swift
# log in as ubuntu/ubuntu, and wget or scp setup-swift.sh
sudo ./setup-swift.sh

Then get swift’s IP from sudo lxc-ls --fancy, install the swift client locally, and talk to it:

$ sudo apt-get install python-swiftclient
$ swift -A http://10.0.3.134:8080/auth/v1.0 -U testproj:testuser -K testpwd stat

Caveat: Don’t use this for any production machine! It’s configured to maximum insecurity, with static passwords and everything.

I realize this is just poor man’s juju, but juju-local is currently not working for me (I only just analyzed that). There is a charm for swift as well, but I haven’t tried that yet. In any case, it’s dead simple now, and maybe useful for someone else.

Tags: , , , , ,

What’s the autopilot widget that I want?

Today’s autopilot release provides a new feature for test case writers. Unless the widget you want to test has a direct object name (GtkBuilder ID/Qt objectName), it is often not that easy to find a widget in a deeply nested hierarchy in autopilot vis.

With the new version, if you have some parent widget (like the containing dialog) w in your test, you can now call w.print_tree() to dump the paths and properties of that widget and all its children to stdout. That’s easy enough to grep, so provides a “poor man’s full tree search”. You can also specify a different output sink, like a file object or a file name: w.print_tree('/tmp/dump.txt').

This is a first step towards making it easier to find widgets and properties you are interested in. Arguably this is mostly just a crutch, but I found it to be rather effective. Before this feature I often wrote little snippets like in LP#1241312, now this becomes much easier. A better solution for this would certainly be a “full tree search” in vis itself, but that’s not that easy to implement. It is on the roadmap for this cycle, though.

I am also currently working on a real-time property change monitor for autopilot-gtk, which may also help in some cases. Unfortunately we cannot build such a thing for autopilot-qt, as due to the nature of Qt object properties, changes of them cannot be monitored.

Tags: , , ,

Run autopilot test in autopkgtest

I recently created a test for digicam photo import for Shotwell (using autopilot and umockdev), and made that run as an autopkgtest. It occurred to me that this might be interesting for other desktop applications as well.

The community QA team has written some autopkgtests for desktop applications such as evince, nautilus, or Firefox. We run them regularly in Jenkins on real hardware in a full desktop environment, so that they can use the full desktop integration (3D, indicators, D-BUS services, etc). But of course for those the application already needs to be in Ubuntu.

If you only want to test functionality from the application itself and don’t need 3D, a proper window manager, etc., you can also call your autopilot tests from autopkgtest with a wrapper script like this:

#!/bin/sh
set -e

# start X
(Xvfb :5 >/dev/null 2>&1 &)
XVFB_PID=$!
export DISPLAY=:5

# start local session D-BUS
eval `dbus-launch`
trap "kill $DBUS_SESSION_BUS_PID $XVFB_PID" 0 TERM QUIT INT
export DBUS_SESSION_BUS_ADDRESS
export XAUTHORITY=/dev/null

# change to the directory where your autopilot tests live, and run them
cd `dirname $0`
autopilot run autopilot_tests

This will set up the bare minimum: Xvfb and a session D-BUS, and then run your autopilot tests. Your debian/tests/control should have Depends: yourapp, xvfb, dbus-x11, autopilot-desktop, libautopilot-gtk for this to work. (Note: I didn’t manage to get this running with xvfb-run; any hints to how to simplify this appreciated, but please test that it actually works.)

Please note that this does not replace the “run in full desktop session” tests I mentioned earlier, but it’s a nice addition to check that your package has correct dependencies and to automatically block new libraries/dependencies which break your package from entering Ubuntu.

Tags: , , , ,

umockdev 0.4: Mocking phone calls

umockdev 0.3 introduced the notion of an “umockdev script”, i. e. recording the read()s and write()s that happen on a device node such as ttyUSB0. With that one can successfully run ModemManager in an umockdev testbed to pretend that one has e. g. an USB 3G stick.

However, this didn’t yet apply to the Ubuntu phone stack, where ofonod talks to Android’s “rild” (Radio Interface Layer Daemon) through the Unix socket /dev/socket/rild. Thus over the last days I worked on extending umockdev’s script recording and replaying to Unix sockets as well (which behave quite different and quite a bit more complex than ordinary files and character devices). This is released in 0.4, however you should actually get 0.4.1 if you want to package it.

So you now can make a script from ofonod how it makes a phone call (or other telephony action) through rild, and later replay that in an umockdev testbed without having to have a SIM card, or even a phone. This should help with reproducing and testing bugs like ofonod goes crazy when roaming: It’s enough to record the communication for a person who is in a situation to reproduce the bug, then a developer can study what’s going wrong independent of harware and mobile networks.

How does it work? If you have used umockdev before, the pattern should be clear now: Start ofonod under umockdev-record and tell it to record the communication on /dev/socket/rild:

  sudo pkill ofonod; sudo umockdev-record -s /dev/socket/rild=phonecall.script -- ofonod -n -d

Now launch the phone app and make a call, send a SMS, or anything else you want to replay later. Press Control-C when you are done. After that you can run ofonod in a testbed with the mocked rild:

  sudo pkill ofonod; sudo umockdev-run -u /dev/socket/rild=phonecall.script -- ofonod -n -d

Note the new --unix-stream/-u option which will create /tmp/umockdev.XXXXXX/dev/socket/rild, attach some server threads to accept client connections, and replay the script on each connection.

But wait, that fails with some

   ERROR **: ScriptRunner op_write[/dev/socket/rild]: data mismatch; got block '...', expected block '...'

error! Apparently ofono’s messages are not 100% predictable/reproducible, I guess there are some time stamps or bits of uninitialized memory involved. Normally umockdev requires that the program under test sticks to the previously recorded write() parts of the script, to ensure that the echoed read()s stay in sync and everything works as expected. But for cases like these were some fuzz is expected, umockdev 0.4 introduces setting a “fuzz percentage” in scripts. To allow 5% byte value mismatches, i. e. in a block of n bytes there can be n*0.05 bytes which are different than the script, you’d put a line

  f 5 -

before the ‘w’ block that will get jitter, or just put it at the top of the file to allow it for all messages. Please see the script format documentation for details.

After doing that, ofonod works, and you can do the exact same operations that you recorded, with e. g. the phone app. Doing other operations will fail, of course.

As always, umockdev-run -u is of course just a CLI convenience wrapper around the umockdev API. If you want to do the replay in a C test suite, you can call

   umockdev_testbed_load_socket_script(testbed, "/dev/socket/rild",
                                       SOCK_STREAM, "path/to/phonecall.script", &error);

or the equivalent in Python or Vala, as usual.

If you are an Ubuntu phone developer and want to use this, please don’t hesitate to talk to me. This is all in saucy now, so on the Ubuntu phone it’s a mere “sudo apt-get install umockdev” away.

Tags: , , , , , , ,

umockdev 0.3: record and replay of tty devices

I’m happy to announce a new release 0.3 of umockdev.

The big new feature is the ability to fake character devices and provide recording and replaying of communications on them. This work is driven by our need to create automatic tests for the Ubuntu phone stack, i. e. pretending that we have a 3G or phone driver and ensuring that the higher level stacks behaves as expected without actually having to have a particular modem. I don’t currently have a phone capable of running Ubuntu, so I tested this against the standard ModemManager daemon which we use in the desktop. But the principle is the same, it’s “just” capturing and replaying read() and write() calls from/to a device node.

In principle it ought to work in just the same way for other device nodes than tty, e. g. input devices or DRI control; but that will require some slight tweaks in how the fake device nodes are set up; please let me know if you are intested in a particular use case (preferably as a bug report).

With just using the command line tools, this is how you would capture ModemManager’s talking to an USB 3G stick which creates /dev/ttyUSB{0,1,2}. The communication gets recorded into a text file, which umockdev calls “script” (yay my lack of imagination for names!):

# Dump the sysfs device and udev properties
$ umockdev-record /dev/ttyUSB* > huawei.umockdev

# Record the communication
$ umockdev-record -s /dev/ttyUSB0=0.script -s /dev/ttyUSB1=1.script \
     -s /dev/ttyUSB2=2.script -- modem-manager --debug

The –debug option for ModemManager is not necessary, but it’s nice to see what’s going on. Note that you should shut down the running system instance for that, or run this on a private D-BUS.

Now you can disconnect the stick (not necessary, just to clearly prove that the following does not actually talk to the stick), and replay in a test bed:

$ umockdev-run -d huawei.umockdev -s /dev/ttyUSB0=0.script -s /dev/ttyUSB1=1.script \
    -s /dev/ttyUSB2=2.script -- modem-manager --debug

Please note that the CLI options of umockdev-record and umockdev-run changed to be more consistent and fit the new features.

If you use the API, you can do the same with the new umockdev_testbed_load_script() method, which will spawn a thread that replays the script on the faked device node (which is just a PTY underneath).

If you want full control, you can also do all the communication from your test cases manually: umockdev_testbed_get_fd("/dev/mydevice") will give you a (bidirectional) file descriptor of the “master” end, so that whenever your program under test connects to /dev/mydevice you can directly talk to it and pretend that you are an actual device driver. You can look at the t_tty_data() test case for how this looks like (that’s the test for the Vala binding, but it works in just the same way in C or the GI bindings).

I’m sure that there are lots of open ends here still, but as usual this work is use case driven; so if you want to do something with this, please let me know and we can talk about fine-tuning this.

In other news, with this release you can also cleanly remove mocked devices (umockdev_testbed_remove_device()), a feature requested by the Mir developers. Finally there are a couple of bug fixes; see the release notes for details.

I’ll upload this to Saucy today. If you need it for earlier Ubuntu releases, you can have a look into my daily builds PPA.

Let’s test!

Tags: , , , , , ,

Recent autopilot-gtk improvements for better automatic UI testing

I was asked to pour some love over autopilot-gtk, a GTK module to provide introspection of widget states to Autopilot. For those who don’t know, Autopilot is a QA tool to write automatic testing of GUI applications, without the race conditions and limitations that previous tools had with using only the ATK level. Please see the documentation and tutorial for more information. There are a lot of community members who do great things with it already, such as automating testing for Ubiquity or writing tests for GNOME applications like evince, gedit, nautilus, or Shotwell. This should now hopefully become easier.

Now autopilot-gtk has a proper testsuite, I triaged all bug reports, wrote reproducers for them, and fixed them all in today’s upload to Saucy. In particular, you can now do the following:

  • Access to the GtkBuilder names: Instead of having to find a particular widgets in terms of class, position, label contents, or other (sometimes) non-unique or unstable properties, you can now pick it by its unique and stable GtkBuilder name, which is the ID that most upstream code uses to manipulate widgets: b = self.app.select_single(BuilderName='entry_searchquery')
  • GtkTextBuffer type GObject properties are now translated into plain strings, which allows you to access the textual contents of a GtkTextView widget with my_textview.buffer (both for simple property access as well as for selecting by buffer contents).
  • GEnum and GFlags properties are now accessible. Enums are translated to strings (self.app.select_many('GtkButton', relief='GTK_RELIEF_HALF') or self.assertEqual(btn_greet.resize_mode, 'GTK_RESIZE_PARENT')), and flags are represented as a simple integer (like my_widget.events)); in theory we could represent them as string like FLAG_FOO | FLAG_BAR, but this becomes too unwieldy; for reliable identity matching one would always need to take care to sort them alphabetically, keep a consistent spacing, etc.
  • Please let me know if you need access to other types of properties, it is now quite easy to support more (as long as there is a reasonable way of mapping them to a standard D-BUS data type). So please report bugs.

    Tags: , , , , ,

umockdev 0.2.6: Hello ARM

I released umockdev 0.2.6. Most importantly, this now fully works on ARM platforms, as we want to use it to write tests for/on the Ubuntu phone. I tested it on my Nexus 7, and the tests also succeed on the ARM Ubuntu builder (which are Panda boards). Fixing this revealed some interesting issues in recorded ioctl traces (as they are platform specific in some cases due to different word length) as well as kernel bugs in the Tegra drivers.

This version also fixes compatibility with older automake versions again, so that the daily builds for raring should work again.

I also have a new gvfs test case ready to commit which uses umockdev (if available) to test functionality of the gphoto backend. But that needs the new UMockdevTestbed.clear() API in 0.2.6, so I was holding that back. I will land it soon in upstream git now.

Tags: , , , , , ,

umockdev 0.2.2 released

I did a 0.2.2 maintenance release for umockdev to fix building with Vala 0.16.1, gcc 4.8 (the changed sizeof behaviour caused segfaults), and current udev releases (umockdev-record stumbled over the new “link priority” fields of udevadm). There are also a couple of bug fixes, but no new features.

Tags: , , , , ,

python-dbusmock 0.6 released

I just pushed out a new python-dbusmock release 0.6.

Calling a method on the mock now emits a MethodCalled signal on the org.freedesktop.DBus.Mock interface. In some cases this is easier to track than parsing the mock’s log or using GetMethodCalls. Thanks to Lars Uebernickel for this.

DBusMockObject.AddTemplate() and DBusTestCase.spawn_server_template() can now load local templates from your own project by specifying a path to a *.py file as template name. Thanks to Lucas De Marchi for this feature.

I also wrote a quite comprehensive template for systemd’s logind. It stubs out the power management functionality as well as user/seat/session objects, and is convincing enough for loginctl. Some bits like AttachDevice is missing, as this sounds unlikely to be required for D-BUS mock tests, but please let me know if you need anything else.

The mock processes now terminate automatically if their connected D-BUS goes down, as advertised in the documentation.

You can get the new tarball from Launchpad, and I uploaded it to Debian experimental now.

Enjoy!

Tags: , , , , , , ,