Archiv der Kategorie debian

Running a Samba server as normal user for testing

For writing tests for GVFS (current tests, proposed improvements) I want to run Samba as normal user, so that we can test gvfs’ smb backend without root privileges and thus can run them safely and conveniently in a “make check” environment for developers and in JHBuild for continuous integration testing. Before these tests could only run under gvfs-testbed, which needs root.

Unlike other servers such as ssh or ftp, this turned out surprisingly non-obvious and hard, so I want to document it in this blog post for posterity’s benefit.

Running the server

Running smbd itself is mainly an exercise of figuring out all the options that you need to set; Alex Larsson and I had some fun figuring out all the quirks and hiccups that happen between Ubuntu’s and Fedora’s packaging and 3.6 vs. 4.0, but finally arrived at something working.

First, you need to create an empty directory where smbd can put all its databases and state files in. For tests you would use mkdtemp(), but for easier reading I just assume mkdir /tmp/samba here.

The main knowledge is in the Samba configuration file, let’s call it /tmp/smb.conf:

[global]
workgroup = TESTGROUP
interfaces = lo 127.0.0.0/8
smb ports = 1445
log level = 2
map to guest = Bad User
passdb backend = smbpasswd
smb passwd file = /tmp/smbpasswd
lock directory = /tmp/samba
state directory = /tmp/samba
cache directory = /tmp/samba
pid directory = /tmp/samba
private dir = /tmp/samba
ncalrpc dir = /tmp/samba

[public]
path = /tmp/public
guest ok = yes

[private]
path = /tmp/private
read only = no

For running this as a normal user you need to set a port > 1024, so this uses 1445 to resemble the original (privileged) port 445. The map to guest line makes anonymous logins work on Fedora/Samba 4.0 (I’m not sure whether it’s a distribution or a version issue). Don’t ask about “dir” vs. “directory”, that’s an inconsistency in Samba; with above names it works on both 3.6 and 4.0.

We use the old “smbpasswd” backend as shipping large tdb files is usually too inconvenient and brittle for test suites. I created an smbpasswd file by running smbpasswd on a “real” Samba installation, and then using pdbedit to convert it to a smbpasswd file:

sudo smbpasswd -a martin
sudo pdbedit -i tdbsam:/var/lib/samba/passdb.tdb -e smbpasswd:/tmp/smbpasswd

The result for password “foo” is

myuser:0:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:AC8E657F83DF82BEEA5D43BDAF7800CC:[U          ]:LCT-507C14C7:

which you are welcome to copy&paste (you can replace “myuser” with any valid user name, of course).

This also defines two shares, one public, one authenticated. You need to create the directories and populate them a bit:

mkdir /tmp/public /tmp/private
echo hello > /tmp/public/hello.txt
echo secret > /tmp/private/myfile.txt

Now you can run the server with

smbd -iFS -s /tmp/smb.conf

The main problem with this approach is that smbd exits (“Server exit (failed to receive smb request)”) after a client terminates, so you need to write your tests in a way to only run one connection/request per test, or to start smbd in a loop.

Running the client

If you merely use the smbclient command line tool, this is rather simple: It has a -p option for specifying the port:

$ smbclient -p 1445 //localhost/private
Enter martin's password: [enter "foo" here]
Domain=[TESTGROUP] OS=[Unix] Server=[Samba 3.6.6]
smb: \> dir
  .                                   D        0  Wed Oct 17 08:28:23 2012
  ..                                  D        0  Wed Oct 17 08:31:24 2012
  myfile.txt                                   7  Wed Oct 17 08:28:23 2012

In the case of gvfs it wasn’t so simple, however. Surprisingly, libsmbclient does not have an API to set the port, it always assumes 445. smbclient itself uses some internal “libcli” API which does have a way to change the port, but it’s not exposed through libsmbclient. However, Alex and I found some mailing list posts ([1], [2]) that mention $LIBSMB_PROG, and it’s also mentioned in smbclient’s manpage. It doesn’t quite work as advertised in the second ML post (you can’t set it to smbd, smbd apparently doesn’t speak the socket protocol over stdin/stdout), and it’s not being used anywhere in the current Samba sources, but what does work is to use good old netcat:

export LIBSMB_PROG="nc localhost 1445" 

with that, you can use smbclient or any program using libsmbclient to talk to our test smb server running as user.

Tags: , , , , , ,

Python unittest: Show log on test failure

I found it surprisingly hard to determine in tearDown() whether or not the test that currently ran succeeded or not. I am writing some tests for gnome-settings-daemon and want to show the log output of the daemon if a test failed.

I now cobbled together the following hack, but I wonder if there’s a more elegant way? The interwebs don’t seem to have a good solution for this either.

    def tearDown(self):
        [...]
        # collect log, run() shows it on failures
        with open(self.daemon_log.name) as f:
            self.log_output = f.read()

    def run(self, result=None):
        '''Show log output on failed tests'''

        if result:
            orig_err_fail = result.errors + result.failures
        super().run(result)
        if result and result.errors + result.failures > orig_err_fail:
            print('\n----- daemon log -----\n%s\n------\n' % self.log_output)

Tags: , , ,

Announcing D-Bus mocker library

I was working on writing tests for gnome-settings-daemon a week or so ago, and finally got blocked on being unable to set up upower/ConsoleKit/etc. the way I need them. Also, doing so needs root privileges, I don’t want my test suite to actually suspend my machine, and using the real service is generally not suitable for test suites that are supposed to run during “make check”, in jhbuild, and the like — these do not have the polkit privileges to do all that, and may not even have a system D-Bus running in the first place.

So I wrote a little test_upower.py helper, then realized that I need another one for systemd/ConsoleKit (for the “system idle” property), also looked at the mock polkit in udisks and finally sat down for two days to generalize this and do this properly.

The result is python-dbusmock, I just released the first tarball. With this you can easily create mock objects on D-Bus from any programming language with a D-Bus binding, or even from the shell.

The mock objects look like the real API (or at least the parts that you actually need), but they do not actually do anything (or only some action that you specify yourself). You can configure their state, behaviour and responses as you like in your test, without making any assumptions about the real system status.

When using a local system/session bus, you can do unit or integration testing without needing root privileges or disturbing a running system. The Python API offers some convenience functions like “start_session_bus()“ and “start_system_bus()“ for this, in a “DBusTestCase“ class (subclass of the standard “unittest.TestCase“).

Surprisingly I found very little precedence here. There is a Perl module, but that’s not particuarly helpful for test suites in C/Vala/Python. And there is Phil’s excellent Bendy Bus, but this has a different goal: If you want to thoroughly test a particular D-BUS service, such as ensuring that it does the right thing, doesn’t crash on bad input, etc., then Bendy Bus is for you (and python-dbusmock isn’t). However, it is too much overhead and rather inconvenient if you want to test a client-side program and just need a few system services around it which you want to set up in different states for each test.

You can use python-dbusmock with any programming language, as you can run the mocker as a normal program. The actual setup of the mock (adding objects, methods, properties, etc.) all happen via D-Bus methods on the “org.freedesktop.DBus.Mock“ interface. You just don’t have the convenience D-Bus launch API.

The simplest possible example is to create a mock upower with a single Suspend() method, which you can set up like this from Python:

import dbus
import dbusmock

class TestMyProgram(dbusmock.DBusTestCase):
[...]
    def setUp(self):
        self.p_mock = self.spawn_server('org.freedesktop.UPower',
                                        '/org/freedesktop/UPower',
                                        'org.freedesktop.UPower',
                                        system_bus=True,
                                        stdout=subprocess.PIPE)

        # Get a proxy for the UPower object's Mock interface
        self.dbus_upower_mock = dbus.Interface(self.dbus_con.get_object(
            'org.freedesktop.UPower', '/org/freedesktop/UPower'),
            'org.freedesktop.DBus.Mock')

        self.dbus_upower_mock.AddMethod('', 'Suspend', '', '', '')

[...]

    def test_suspend_on_idle(self):
        # run your program in a way that should trigger one suspend call
        
        # now check the log that we got one Suspend() call
        self.assertRegex(self.p_mock.stdout.readline(), b'^[0-9.]+ Suspend$')

This doesn’t depend on Python, you can just as well run the mocker like this:

python3 -m dbusmock org.freedesktop.UPower /org/freedesktop/UPower org.freedesktop.UPower

and then set up the mocks through D-Bus like

gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddMethod '' Suspend '' '' ''

If you use it with Python, you get access to the dbusmock.DBusTestCase class which provides some convenience functions to set up and tear down local private session and system buses. If you use it from another language, you have to call dbus-launch yourself.

Please see the README for some more details, pointers to documentation and examples.

Update: You can now install this via pip from PyPI or from the daily builds PPA.

Update 2: Adjusted blog entry for version 0.0.3 API, to avoid spreading now false information too far.

Tags: , , , , , ,

PyGObject 3.3.92 released

I just released PyGObject 3.3.92, for GNOME 3.5.92.

There is nothing too exciting in this release; a couple of small bug fixes and a lot of new test cases. See the detailled list of changes below.

Thanks to all contributors!

Changes:

  • release-news: Generate HTML changelog (Martin Pitt)
  • [API add] Add ObjectInfo.get_abstract method (Simon Feltman) (#675581)
  • Add deprecation warning when setting gpointers to anything other than int. (Simon Feltman) (#683599)
  • test_properties: Test accessing a property from a superclass (Martin Pitt) (#684058)
  • test_properties.py: Consistent test names (Martin Pitt)
  • test_everything: Ensure TestSignals callback does get called (Martin Pitt)
  • argument: Fix 64bit integer convertion from GValue (Nicolas Dufresne) (#683596)
  • Add Simon Feltman as a project maintainer (Martin Pitt)
  • test_signals.py: Drop global type variables (Martin Pitt)
  • test_signals.py: Consistent test names (Martin Pitt)
  • Add test cases for GValue signal arguments (Martin Pitt) (#683775)
  • Add test for GValue signal return values (Martin Pitt) (#683596)
  • Improve setting pointer fields/arguments to NULL using None (Simon Feltman) (#683150)
  • Test gint64 C signal arguments and return values (Martin Pitt)
  • Test in/out int64 GValue method arguments. (Martin Pitt) (#683596)
  • Bump g-i dependency to 1.33.10 (Martin Pitt)
  • Fix -uninstalled.pc.in file (Thibault Saunier) (#683379)

Tags: , , , , ,

PostgreSQL 9.2 final available for Debian and Ubuntu

PostgreSQL 9.2 has just been released, after a series of betas and a release candidate. See for yourself what’s new, and try it out!

Packages are available in Debian experimental as well as my PostgreSQL backports PPA for Ubuntu 10.04 to 12.10, as usual.

Please note that 9.2 will not land any more in the feature frozen Debian Wheezy and Ubuntu Quantal (12.10) releases, as none of the server-side extensions are packaged for 9.2 yet.

Tags: , , , ,

PyGObject 3.3.91 released

I just released PyGObject 3.3.91, for GNOME 3.5.91.

The big new feature in this release (thanks to the release team for granting an exception) is Simon Feltman’s new Signal helper class, which makes defining custom signals a whole lot simpler and more obvious. In the past, you had to do

 class C(GObject.GObject):
    __gsignals__ = {
        'my_signal': (GObject.SIGNAL_RUN_FIRST, GObject.TYPE_NONE,
                      (GObject.TYPE_INT,))
    }

    def do_my_signal(self, arg):
        print("my_signal called with %i" % arg)

whereas now this looks like

class C(GObject.GObject):
    @GObject.Signal(arg_types=(int,))
    def my_signal(self, arg):
        print("my_signal called with %i" % arg)

or even more elegantly when using Python 3 and its new type annotations:

class C(GObject.GObject):
    @GObject.Signal
    def my_signal(self, arg:int):
        print("my_signal called with %i" % arg)

Check out the updated example and docstring for other ways how to use it.

Overrides can now be in a directory different from the one that pygobject installs itself into. These overrides need to put this into their __init__.py at the top:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

and put themselves somewhere into the default PYTHONPATH. This should make it a lot easier for library packages to ship their own overrides for Python.

This new version also comes with a couple of new overrides and bug fixes. See the detailled list of changes below.

Thanks to all contributors!

  • Fix exception test case for Python 2 (Martin Pitt)
  • Bump g-i dependency to >= 1.3.9 (Martin Pitt)
  • Show proper exception when trying to allocate a disguised struct (Martin Pitt) (#639972)
  • Support marshalling GParamSpec signal arguments (Mark Nauwelaerts) (#683099)
  • Add test for a signal that returns a GParamSpec (Martin Pitt) (#683265)
  • [API add] Add Signal class for adding and connecting custom signals. (Simon Feltman) (#434924)
  • Fix pygtkcompat’s Gtk.TreeView.insert_column_with_attributes() (Martin Pitt)
  • Add override for Gtk.TreeView.insert_column_with_attributes() (Marta Maria Casetti) (#679415)
  • .gitignore: Add missing built files (Martin Pitt)
  • Ship tests/gi in tarball (Martin Pitt)
  • Split test_overrides.py (Martin Pitt) (#683188)
  • _pygi_argument_to_object(): Clean up array unmarshalling (Martin Pitt)
  • Fix memory leak in _pygi_argument_to_object() (Alban Browaeys) (#682979)
  • Fix setting pointer fields/arguments to NULL using None. (Simon Feltman) (#683150)
  • Fix for python 2.6, officially drop support for < 2.6 (Martin Pitt) (#682422)
  • Allow overrides in other directories than gi itself (Thibault Saunier) (#680913)
  • Clean up sys.path handling in tests (Simon Feltman) (#680913)
  • Fix dynamic creation of enum and flag gi types for Python 3.3 (Simon Feltman) (#682323)
  • [API add] Override g_menu_item_set_attribute (Paolo Borelli) (#682436)

Tags: , , , ,

PostgreSQL 9.2 RC1 available for testing

The unstoppable PostgreSQL team just announced the first release candidate of 9.2, with several bug fixes since the Beta 4. If you haven’t tested 9.2 yet, now is the time! Remember that you can run a copy of your 8.4 or 9.2 cluster in parallel for testing with pg_upgradecluster.

If you use Debian, 9.2rc1 will be available in experimental in a few hours. For Ubuntu, you can get packages for all supported releases from my PostgreSQL backports PPA as usual.

Enjoy!

Tags: , , , ,

PyGObject 3.3.90 released

I just released PyGObject 3.3.90, for GNOME 3.5.90.

This is now working correctly on big-endian 64 bit machines such as powerpc64, and fixes marshalling for GParamSpec attributes and return values, as well as a few small bug fixes.

Thanks to all contributors!

Complete list of changes:

  • Implement marshalling for GParamSpec (Mathieu Duponchelle) (#681565)
  • Fix erronous import statements for Python 3.3 (Simon Feltman) (#682051)
  • Do not fail tests if pyflakes or pep8 are not installed (Martin Pitt)
  • Fix PEP-8 whitespace checking and issues in the code (Martin Pitt)
  • Fix unmarshalling of gssize (David Malcolm) (#680693)
  • Fix various endianess errors (David Malcolm) (#680692)
  • Gtk overrides: Add TreeModelSort.__init__(self, model) (Simon Feltman) (#681477)
  • Convert Gtk.CellRendererState in the pygi-convert script (Manuel Quiñones) (#681596)

Tags: , , , , ,

My impressions from GUADEC

I have had the pleasure of attending GUADEC in full this year. TL;DR: A lot of great presentations + lots of hall conversations about QA stuff + the obligatory be{er,ach} = ♥.

Last year I just went to the hackfest, and I never made it to any previous one, so GUADEC 2012 was a kind of first-time experience for me. It was great to put some faces and personal impressions to a lot of the people I have worked with for many years, as well as catching up with others that I did meet before.

I discussed various hardware/software testing stuff with Colin Walters, Matthias Clasen, Lennart Poettering, Bertrand Lorentz, and others, so I have a better idea how to proceed with automated testing in plumbing and GNOME now. It was also really nice to meet my fellow PyGObject co-maintainer Paolo Borelli, as well as seeing Simon Schampier and Ignacio Casal Quinteiro again.

No need to replicate the whole schedule here (see for yourself on the interwebs), so I just want to point out some personal highlights in the presentations:

  • Jacob Appelbaum’s keynote about Tor brought up some surprising facts about how the project has outgrown its past performance problems and how useful it was during e. g. the Arab revolution
  • .

  • Philip Whitnall’s presented Bendy Bus, a tool to mock D-Bus services for both unit and fuzz testing. He successfully used it to find and replicate bugs in Evolution (by mocking evolution-data-server) as well as libfolks (by mocking the telepathy daemons). It should work just as well to mock system services like upower or NetworkManager to test the UI bits that use it. This is a topic which has been on my wishlist for a long time already, so I’m happy that there is already an existing solution out there. We might have to add some small features to it, but it’s by and large what I had in mind, and in the discussion afterwards Philip said he’d appreciate patches against it.
  • Christophe Fergeau showed how to easily do Windows builds and installers from GNOME tarballs with MinGW-w64, without having to actually touch/use Windows (using cross-building and running tests etc. under wine). I found it surprising how easy that actually is, and it should not be hard to integrate that in a jhbuild-like setup, so that it does not keep breaking every time.
  • Colin Walters gave an introduction to OSTree, a project to build bootable images from kernel/plumbing/desktop upstream git heads on a daily basis. This is mostly to avoid the long delays that we otherwise have with doing upstream releases, packaging them, and getting them into a form that can safely be tested by users. In an afterwards discussion we threw some ideas around how we can integrate existing and future tests into this (something in spirit like our autopkgtest). This will be the area where I’ll put most focus on in the next time.
  • Adam Dingle of yorba fame shared his thoughts about how we can crowdfunding of Free Software Projects work in practice, comparing efforts like codefoundry and kickstarter. Of course he does not have a solution for this yet, but he raised some interesting concerns and it spun off lots of good discussions over lunchtime.
  • Last but not least, the sport event on Saturday evening was awesome! In hindsight I was happy to not have signed up for soccer, as people like Bastian or Jordi played this really seriously. I participated in the Basketball competition instead, which was the right mix of fun and competition without seriously trying to hurt each other. :-)

There were a lot of other good ones, some of them technical and some amusing and enlightening, such as Frederico’s review of the history of GNOME.

On Monday I prepared and participated in a PyGObject hackfest, but I’ll blog about this separately.

I want to thank all presenters for the excellent program, as well as the tireless GUADEC organizer team for making everything so smooth and working flawlessly! Great Job, and see you again in Strasbourg or Brno!

Tags: , , ,

PyGObject hackfest @ GUADEC: Agenda

I started to collect some easy PyGObject bugs which are appropriate for the PyGObject hackfest at GUADEC on July 30th. These are bugs which do not need a lot of previous knowlege and are excellent starters for new contributors, such as adding overrides, fixing build system issues, etc.

I also created an initial idea pool/agenda/coordination page, where participants can add or signup for things to work on.

Feel free to add your own topics! I’m really looking forward to GUADEC and the hackfest, see you there!

GUADEC 2012

Tags: , , , , , , ,