Armin's personal Blog

Gregory Haynes is working on Kobby featuring collaborative text editing in your favourite KDE text editor as a GSoC project. He started to wrap the (still unreleased!) libinfinity (my infinote implementation) API for C++ for this task. I'm really excited to see other collaborative text editors coming up that work together with Gobby, since this is exactly one of the reasons why I took the burden to write the library in GObject-based C.

Speaking of which; I didn't blog on it for a long time. We have moved to git in the meantime. I'm currently working on porting Gobby to use libinfinity instead of obby, heading towards a first usable release, so that the hard work pays off. There are still some things in libinfinity that need to be fixed (performance issues, a few known bugs and way more testing) which are probably easier to fix when a usable Gobby version using it exists. I hope to have these done in a few months, though the most difficult part is always to get the little details right, which is probably why I'm not good at meeting (self-set) deadlines.

I updated the installer. It now includes jpeg62.dll and python25.dll which were missing (thanks to the reporters on my initial post). I also included the glom and gda python modules. If I do understand things right, then installing Python and pygtk makes Python support in Glom automatically work with this. I tried this out by reinstalling Python into a different directory, which worked.

Again, if there are still problems with it, please tell me.

Here it is, Glom's installer for Windows. Thanks to the opinions on my last blog entry. I set up an all-in-one installer, so Glom should run out of the box after installing (please tell me if not). It includes GTK+ from SVN with my patch from bug #506062 (hint, hint) so that recent files filtering works.

Since most people seem to use NSIS for Windows installers (comment on this blog entry, gtk-win.sourceforge.net) I also had a look at it. However, the scripting language reminds me more of assembly than anything else, and I think these days are over. So I used InnoSetup which has Pascal scripting for the rare cases you need it.

The only thing that is not going to work is python scripts for buttons and calculated fields. Ideally, one would just install Python and PyGtk, and having Glom use that installation automatically when present. This shouldn't be too hard, it just needs to tell python where to find the own gda and glom python modules. I am going to tackle this after my physics exam on Thursday.

With added support for self-hosting, Glom for Windows now supports all the major features that the Linux version does (apart from Service Publishing, which is probably way more effort since Avahi is not available on Windows). It's time to think about how to package it, and what dependencies to include, especially whether to ship with GTK+ or not.

All the major GTK+-using projects such as Inkscape, Pidgin and Wireshark include GTK+. Even the GIMP does so since version 2.4. This actually generates a lot of duplication, which is why we (or, rather, I) did decide not to ship it with Gobby. Of course we are getting complaints such as "Hey, this doesn't work because libgtk-win32-2-0-0.dll was not found!!1" from time to time, but if people are not able to read the instructions right above the download link, well, then I'm sorry. Do others think the added duplication is worth that the user needs to install GTK+ manually once? Or, maybe, is this just because there is no "official" GTK+ installer?

Then, there is python, and I have not so much experience here. Glom links statically against libpython, so it should even run without python being installed. However, buttons and calculated fields won't work then, unless we ship with at least pyGTK. Perhaps the best thing is to state that for button scripts and calculated fields to work, python and pyGTK have to be installed. We just have to make sure that Python finds the glom and pygda modules then, even if Python was installed after Glom (so it had no chance to install them into the standard location for such modules), but probably Python has some API for this.

Last but not least, we have gtkmm which also has an installer that is hosted on ftp.gnome.org even. I think we should either ship both GTK+ and gtkmm, or none of them.

If you have any thoughts on this, recommendations and suggestions are welcome.

There are several ways to fix the recent chooser showing no recent files in Glom on Windows, but they are all sort of hacks except one, namely make the recent chooser guess the mime type of an URI on Windows as it already does on Unix. Currently, it always falls back to application/octet-stream and this is why the filter on application/x-glom that glom uses does not show up any files. I looked around a bit in the registry, and it seems that all known file extensions are in HKEY_CLASSES_ROOT/.$ext, and some of them have a "Content Type" key that looks like the mime type we are after. So all we need to do is to register the .glom type there (this can also be used to associate a nice icon with it) and to make the GTK+ recent chooser use that "Content Type" key for the extension of the URI it gets as default mime type.

To implement this, I need to be able to build GTK+ (and glib which GTK+ from SVN trunk depends on) on Windows. The easy method I already used for glom itself was to copy tarballs created by "make dist" to the windows machine so I don't have to create the configure script there, not requiring the autotools stuff. However, this gets horribly annoying when the repositories are updated frequently. It would be much nicer to have a working copy directly on Windows, and to (re)create the configure script there when necessary. In theory, this should work since mingw provides all necessary tools. In practise, I already stumbled upon several ugly problems in the past. I gave it another try, though, and I ended up with a freshly built glib from SVN at the end of the day. Below are the exact steps that got me this far.

I used the environment I set up as described here as a starting point. It is probably equally well possible to start with a freshly set up MSYS+MingW, though.

• First, install termcap, libtool, automake and autoconf from mingw.org into C:\msys (without the usr/ prefix that is in the tarball, such that files end up as /bin/autoconf etc.). Copy /bin/automake to /bin/automake-1.10 and /bin/aclocal to /bin/aclocal-1.10. The -1.10 versions are just 0-byte stubs, but glib's autogen.sh calls them explicitely.
• Then, rename the /bin/perl wrapper script and install the mingw-provided perl package. This is still 5.6, but it is enough for what we need, and the problems with paths in arguments seem to go away with this. Perhaps it would be enough to install ActivePerl directly so that /bin/perl.exe is the ActivePerl one.
• Add gtk-doc stubs to your system. Most GNOME libraries (including glib and GTK) require gtk-doc, but I could not find any success story of getting it to work on Windows. I used the instructions by Hans Breuer:

  cd gtk-doc
  cp gtk-doc.m4 /gtk/share/aclocal
  cp gtkdocize /gtk/bin
  mkdir -p /gtk/share/gtk-doc/data/
  cp gtk-doc.make /gtk/share/gtk-doc/data

  You need a gtk-doc tarball from ftp.gnome.org for this.
• Next, load svn-win32-1.4.5.zip (or a more recent version) from subversion.tigris.org, unpack the zip to C:\svn and add /c/svn/bin to the PATH in your ~/.profile. After a relogin or exec bash -l you should be able to use the SVN client in msys. Try getting glib via svn co http://svn.gnome.org/svn/glib/trunk glib.
• The most hacky thing I had to do was changing the /bin/aclocal file. Add

  $file =~ s/^C:\/msys\//\/usr\//;

  around line 680, right before

  $map_traced_defs{$arg1} = $file
    if ($macro eq 'AC_DEFUN'
        || $macro eq 'AC_DEFUN_ONCE'
        || $macro eq 'AU_DEFUN');
  

  Note that you need to adjust C:\/msys\/ if you installed msys elsewhere. Also remember to copy it to /bin/aclocal-1.10 (By the way, NTFS is capable of symlinks somehow, isn't it)? This is required so that aclocal converts paths starting with C:/msys/share/[...] to /usr/share/[...]. Both refer to the same file on disk, but since aclocal does a simple string comparison, it does not recognize this and always creates an empty aclocal.m4, causing autoconf to fail due to not finding any macros. Better solutions to this are very much appreciated.
• Last, replace AM_CONFIG_HEADER by AC_CONFIG_HEADER in glib's configure.in. autoheader complains about the first. Quick googling seems to suggest that AM_CONFIG_HEADER is deprecated by AC_CONFIG_HEADER, but I am not sure. If that is really the case we should probably change glib.
• Run ./autogen.sh and watch the magic.

So far for today, I am going to try GTK+ and glom tomorrow. Perhaps I even get to what I originally wanted to do, making the GTK+ recent chooser guess mime types on Windows. If you try these steps out and run into further problems, I would be glad to hear about it. I am especially looking forward to John Stowers' work on jhbuild on Windows. Building GNOME libraries and applications directly from SVN via jhbuild on Windows sounds really cool.

After porting Glom to maemo, the next target was Windows. Glom's dependencies are already ported to Windows, so getting it to work wasn't too hard, though some hacks were still necessary to get things going (see the Windows build instructions). Currently, only the client-only mode of Glom runs on Windows, but I hope the full version follows soon.

I installed Windows XP within a virtual machine on my laptop to do the port. The good thing is that this doesn't force me to leave my usual Linux environment, especially when I am not at home where I still have a desktop Linux machine. However, on the other hand compilation is slowed down pretty much, becoming really significant when compiling rather big C++ projects such as gtkmm and glom, probably due to memory constraints.

As always, pictures say more than a thousand words, so enjoy. The third screenshot shows a weird problem when scrolling directly after connecting to the database. The bug disappears when minimizing and re-maximizing the window. The screenshots are truncated to the left because the Glom Window requires more space than the 800x600 assigned to the virtual machine for the Small Business Example (which the screenshots are from). I could also use 1024x768 (the laptop has 1280x800), but I like to have a terminal or other windows next to the VM, and I don't like scrollbars in it either. Other drawbacks the current version suffers from and which I am going to tackle within the next days are listed in the Glom Wiki.

I recently ported glom to the maemo platform for Nokia's internet tablets for Openismus. The major work of this was compiling glom and its dependencies in scratchbox, i.e. make it build and run without using some convencience API in glibmm and friends to save code size. This leads to a lot of #ifdef statements uglifying the code (especially because you cannot use exceptions for the code to compile with the -fno-exceptions flag), but well, that's the price you have to pay if you want to use C++ on maemo (you can at least save the #ifdefs if you are writing a maemo-only application, though). It's still much more convenient than plain C. Note that the maemo version of glom is a client only version, i.e. it does not support self-hosting and developer mode.

The work has already been committed to glom trunk, though I keep making changes in the GLOM_CLIENTONLY branch to be able to make debian packages for sardine extras that are based on glom 1.6. With friendly help from Johannes Schmid and Philipp Kern I managed to build debian packages for glom and its dependencies that are already in sardine extras. Feel free to contact me if there are problems with them, since these are the first ones I ever made, and I am not too familiar with debian based distributions either, because I am not using one.

These screenshots show how the whole thing currently looks like. I made use of the hildonmm APIs where appropriate to get a more native maemo look-and-feel, though there is still some work left in that regard. At some places, glom measures size of text to determine the width and/or height of its widgets, using gtk_widget_create_pango_layout() and pango_layout_get_pixel_size(). However, this seems to result in the size of the text for the normal GTK+ theme, not the one with the bigger font on maemo, and this renders glom's guesses about widget sizes rather unusable. I fixed the most obvious uses (like the height of the tree rows in the glom list view) by hardcoding the size, but of course that is not a permanent solution. I am thankful for any hints how to get a correctly setup PangoLayout to do text measuring on maemo.

The automatic column sizing of GtkTreeView also seems not too useful for glom on maemo (see second screenshot). We probably should rather allow the user to scroll the list view horizontally, and introduce a minimum width for the columns.

Many people using gobby complain that it does not support Undo, and some do not even use it therefore. Of course, they are right. It is not only convenient but may also save you lots of data in case you accidentally delete it. However, it is not easy to implement it in a collaborative editor because between your own do and undo operation pairs others might have altered the document. It is thus not reverted to a previous state as with a single user editor. We got suggestions to add some sort of "primitive" Undo like just undoing the last operation, no matter from whom it was. I didn't like that, though, because I prefer to either do things correctly or not at all.

Fortunately, other people also thought about how to achieve collaborative Undo. I read some papers [1] describing an algorithm (adOPTed) that not only achieves convergence, meaning that two users don't end up with a different document after having applied some operations concurrently (which the obby algorithm (called Jupiter) obviously also achieves), but also allows Undo/Redo (which Jupiter does not when more than two users are involved). This is however not the one-and-only solution. There are other approaches to collaborative Undo (or resolving collisions in general), like the AbiCollab one. adOPTed has some nice properties though:

• It has enough information to not only undo your own but also any other's operation. Actually, when a user wants to Undo something, it just says "I am doing an Undo now" and the others calculate the required operation. This especially allows to easily restore user colors when a deletion is being undone (without allowing arbitrary insertion of text colored with another user's color). It could also be used to undo vandalism issued by another user.
• You can always undo and redo your operations, independent of what others have done in between. However, it might happen that an undo has no effect (for example, if you type something, another one deletes it and you press undo then). I wonder by the way what a user expects in that case: Imagine you are typing two letters, and another user deletes the second one. When you press Undo now, would you expect nothing to happen (because another one already "manually" undid your latest operation) or would you expect that the first letter also disappears (because of exactly the same reason)?
• It doesn't need a server to make some final decisions or to keep things in order. You just have to make the other participants receive your requests somehow. Also, there are no conflict situations or something requiring additional round-trips.

However, as always, everything has its price:

• My implementation requires already a lot of CPU power in my simple test cases to calculate the required transformations, especially when there are lots of operations between a do/undo pair.
• As every user calculates the undo operation by itself, every user needs to keep track of every request made by any user. This also requires to send the whole request log to a joining user. Of course, this can be limited to the latest N operations, but it is still a considerable amount of data (especially when many users are working together) and restricts the number of operations people can undo.

My proof of concept featuring Undo and Redo is here. However, you probably won't understand the interesting part of that code without having read the papers referenced above (which you should do as well if you are interested in more details, by the way). There a still a couple of issues before that can be used in a real-world environment, though. I think the performance problems mentioned above can be fixed by caching transformed requests, so that it does not have to compute the same requests all over again. Another problem that needs to be solved is a dynamic amount of users. My test code only works for a fixed number of users. The difficulty is to find out when it is safe to forget about operations from a user that has left the editing session, to save memory and bandwidth.

I hope to get that done really soon now so that it can be used in Infinote and perhaps even Gobby (I have not yet decided whether I want to implement that in Gobby because it is still a considerable amount of work).

I am off to Ireland at the 23th of July and am visiting some friends near Osnabrück right after that, but when I am back home (presumably at 12th of August) I really hope to get something usable (read: a gedit plugin) before my third semester at university begins in October. Unfortunately, I cannot make it to GUADEC this year because I have to be at university that week, but I really look forward to finally get there next year.

[1] Achieving Convergence with Operational Transformation in Distributed Groupware Systems (by Abdessamad Imine et al.)
An Integrating, Transformation-Oriented Approach to Concurrency Control and Undo in Group Editors (by Matthias Ressel, Rul Gunzenhäuser and Doris Nitsche-Ruhland)
Reducing the problems of group undo (by Matthias Ressel and Rul Gunzenhäuser)

Today, I read the GtkSourceview 1.90 release announcement on gedit-list. After having finished my math excerises much earlier then expected I decided to hack GtkSourceview2 support into Gobby. I committed it a few minutes ago into SVN, after having received some advice from Yevgen Muntyan to get it to work (You have to associate a style theme by hand to get highlighting, which will hopefully be changed in the future so that a default theme is used if none is given).

GtkSourceView2 basically comes with a new syntax highlighting engine that allows embedding languages into others (such as JavaScript into HTML, as shown on the screenshot). It also has theming support and mapping between filename patterns and language definitions, which Gobby had to implement before (and it still does since GtkSourceView1 is still supported).

Gobby development has not stalled. We released 0.4.3 with native Avahi support a few weeks ago. However, it is not planned to add any great new features in favor of the sequel I already announced half a year ago. Unfortunately, it does not make as much progress as one could expect it to do, and especially not as much as I would like it to do. My second semester at university began three weeks ago, so I also don't think I get too much done in the following two to three months, either.

The successor of Gobby is going to be called Infinote. It will offer some (long-requested) features the current Gobby architecture does not support, such as different document types, XMPP-based messaging and showing remote cursors/selections. However, the fundamental change is that I want to provide a kind of framework rather than an editor, and to get people to implement Infinote support into their editors so that ideally you can just continue to use your favorite editor for collaborative work. For this to work, I need to properly document the API and network protocol which is also one of the points Gobby is rather weak in currently.

A propos protocol: More and more collaborative editing efforts pop up lately, like Pisara Editor, AbiCollab (which I should really try out in the near feature...) seems to be mature and a GUADEC session description mentions some gedit/collab (over which I did not find out any further details, though). Then, there are of course ACE and MateEdit which I am already aware of. All of these (including Gobby) use different protocols, and are therefore unable to interoperate with each other. Most seem to agree that XMPP is a good base, but there is still the application-specific stuff on top of it that needs to be compatible. It would be great if there was some project like freedesktop.org to agree on a standard for collaborative (text) editing.

The current Infinote code is available in a public SVN at svn://svn.0x539.de/inifinote for those who want to have a look at it. However, most of the ideas do only exist in my head by now and are not written down somewhere in there.

Do you remember back in school when you learned stuff you certainly would not need anymore in your future life? I thought the same thing when we programmed things in Delphi (those things actually were quite interesting, though). After Phil poked me once again yesterday, I finally decided to fix long-standing bug #145. So, guess what, the setup system we use (InnoSetup) is not only written in Delphi but also allows custom Pascal Code to be executed during setup. So this is what I came up with, and it would certainly have taken me some more hours if I did not know Delphi before.

I wonder however whether I will have to use Java for some real-world application one day. I would be nearly as good prepared as I was for the Delphi task.

I got home from FOSTEL 2007 a few hours ago. I was there to represent Gobby and to spread the word about Openismus a bit.

I met a couple of interesting people there like Philip van Hoof, the author of the Tinymail framework, Robert McQueen from the Telepathy project and Jochen Topf who pointed me to the OpenStreetMap project (and who is also living in Karlsruhe, by the way). There was also a lot of VoIP and Telephony stuff I do not know much about and I did not understand that much, but it was nevertheless pretty interesting and fun. I really enjoyed Sean Egan's talk on XMPP, too.

Things I want to keep in mind for the next conference:

• Do not pronounce Gobby in a way so that people might understand Kopete.
• When doing a five minute talk, only prepare for about four minutes, because it will eventually take longer. Apropos lightning talk, the slides of my Gobby Status talk are here. OpenOffice messed the headers up when exporting to PDF, but I don't want to fiddle around with that at the moment. Also expect a blog entry about that topic in the near future.

Last but not least, I want to thank Dave Neary (and probably OpenWengo) for providing me free accommodation and for the dinner Wednesday night. Perhaps we meet again next year.

If you are stumbling over this while reading Planet Gnome, then I have been added to the latter. Thanks Jeff!

For those who do not know me yet: I am a 20-year old German guy, developer of Gobby, and doing various GNOME-related things as a part-time Openismus employee. Currently, I am working on porting Glom to the new libgda 3.0 API.

According to test test application, gtkmm needs significantly more time when default signal handlers and virtual functions are enabled. What gtkmm basically does is creating a new class inheriting from the corresponding class in GTK and overriding all vfuncs and default signal handlers. In the overridden method, it gets the C++ wrapper object for the C object (if any) and casts it to the actual type (e.g. Gtk::Window). As an optimization it checks already whether the C++ object is a custom one that derives from the gtkmm one. If not, the vfunc could not have been overridden anyway and we just call the parent's (GtkWindow) vfunc implementation. Otherwise, a C++ virtual function is called that classes deriving from Gtk::Window might override. This way, GObject vfuncs can be overridden in the C++ code.

For Openismus, I tried to find out what makes this procedure slow. To do so I used callgrind to collect some profiling data from the test program mentioned above. Then, I installed Qt and kdelibs just to be able to use kcachegrind to visualize that data. Wouldn't someone like to write a kcachegrind-like application for GNOME?

I had a look at Container_Class::forall_vfunc because this was the one called most times (64 025 calls, to be accurate). kcachegrind shows that 3.10% of the total time was spent in forall_vfunc (actually, its not time but CPU cycles or something, but as a physician I assume they are proportional...). Out of these 3.10%, 1.48% are spent in dynamic_cast<>, another 0.99% in ObjectBase::_get_current_wrapper(). The latter call looks up the C++ object from the C object, and the dynamic_cast casts it from ObjectBase to the actual type. Both calls are at the very beginning of the function. Also of notice is that 2.50% of the overall time is in dynamic_cast<>. This means that just the forall_vfunc issues more than the half of all dynamic_casts in the whole program, not counting all the other vfuncs.

What I tried out now is to do the dynamic_cast only if the C++ object is derived from the gtkmm class. In the test application, this is only the case for ExampleWindow but not for all the other widgets that are created. If the object is not derived, the actual C++ type is not needed because we just call the parent vfunc anyway. The result looks like this:

The time spent in forall_vfunc is now nearly only the half as before. It does no more call to dynamic_cast. This is because all containers on which the forall vfunc was called are not overridden by a C++ class. Other vfuncs, as expose_event, still showed a small amount of dynamic_cast calls. These were the expose events the Window received because of being derived by ExampleWindow. There is a bug open about this.

The next thing I had a look at was the ObjectBase::_get_current_wrapper() call. kcachegrind shows a total amount of 1.70% out of which 0.14% are spent in the function itself (and not in functions being called by get_current_wrapper() in turn). However, the function does nothing more than calling g_object_get_qdata on the given GObject to get the C++ wrapper. Therefore, it seems that a significant amount of time is lost due to the overhead of a function call here because the function is often called (106 674 times), so I tried inlining it:

Any calls to get_current_wrapper() disappeared now. Instead, g_object_get_qdata() shows up directly in forall_vfunc. A good way to verify that the compiler actually inlined the function call. Again, there is a performance gain, but way smaller.

So long, but what about actual execution times? When running the test application, there was no noticeable speedup. However, when running under valgrind (meaning about 10-30 times slower), it finished one or two seconds earlier. To conclude, for most users the changes will probably not be visible, but for large applications and/or less powerful machines gtkmm applications might run a bit more smoothly now.