Western Digital Labs and ownCloud

We published a blog post about a collaboration between WDLabs and ownCloud. WDLabs is, in their words, "a division of Western Digital focused on accelerating new solutions". I'd take that as "they look for more ways to sell hard drives" and they have been playing with some interesting stuff. One such thing is their PiDrive Kit. This kit includes a connector cable which powers both the harddrive and the Pi from one power supply, simplifying the setup. We're now working with WD Labs to build something from this starting point.

WD Labs, meet ownCloud

When they approached us a few weeks ago, they told us they had been toying with a more complete kit with bundled operating system based on berry boot. This lets you choose an OS on first boot and they had put together an image with ownCloud as one of the options. They thought it was really cool and asked if we felt the same and had any ideas of what we could do with it. Well, sure, I wrote not too long ago about What's holding ownCloud back and being able to offer potential users a plug-and-play device with ownCloud on it is something we'd love.

So did they. We quickly received a prototype of a kit with Berryboot and ownCloud on it as well as a case and a Pi included. After some more discussions, we developed a plan with which we hope to get some community help to make this happen.

We need your help

You have to know - this is not something ownCloud, Inc. is very much involved with. It simply is not a commercial endeavor - Inc. works for enterprise customers and we haven't found any of them yet running ownCloud on a Pi. So, while WD Labs will take care of the hardware and has committed themselves to doing a serious number of ownCloud branded devices complete with pre installed OS in the first quarter of next year, it will depend on our free time and community help to develop the OS.

And there's serious work to be done. Berryboot can be configured to boot a headless ownCloud but right now, to even boot, you have to connect a keyboard and monitor. And booting isn't even the issue - finding and configuring an IP and then connecting to the Pi is. And once you've done that, the system has to be configured to be accessible from outside so you can use the ownCloud clients on your mobile phone and laptop to always find and connect to your ownCloud and give you the ability to share with friends and family.

There are folks in our community who have experience with this, building our community VM for example (hi Daniel!) and we'd like to solicit their help. We have 10 prototypes to send out to people who want to help build this ready-to-go ownCloud, the blog has more details on how to get one. And if there is a lot of interest, we'll get more prototypes put together, though I think it'll be tight to get them out before Christmas.

Note that it is totally cool, even advantageous, to send in a proposal ad a team. Together you can get more done!!!

Of course - without this kit there is still a LOT you can do to help out and there is no reason to wait for us sending out anything! Help get a discussion started about what this should look like, how to get there and who does what. Subscribe to the developer mailing list and get started!

Future

Now we know the Raspberry Pi 2 isn't the very best device to run ownCloud on, in terms of performance. It won't run with 100 users, indeed, though it isn't too bad for a few users at home. But, aside from the fact that there is a lot of room for tuning (PHP 7, for one, should help a lot, see some stats here), we're not married to the Raspberry with a single hard drive. It is a popular device and thus a great place to start but WD Labs has already shown us prototypes using a Rasberry Pi Compute device and 2 hard drives and we could work with devices like the Banana Pi or more powerful boards, too. That won't happen immediately, it simply depends on how well the first version turns out and the interest from users.

That means it matters a lot now: can we make this work? I'll sure put a portion of my free time in this as I'm quite excited about it. You too?

Meetup and a release party in B'lin?

Today at 7 it is ownCloud meetup time in C-Base again (and also in Munich, by the way!). As ownCloud Server 8.1 is scheduled for the beginning of next month, I hope to grab a daily ownCloud snapshot, put it on a Banana Pi and run the cool Smashbox testing tool against it to see how it fares.

Even more interesting, I hope to see if I can install older ownCloud releases on other Pi's and compare the performance changes between 7, 8 and 8.1 for example. Performance is one of the big improvements that should be coming in 8.1, with 4x speed up of things like file syncing. I'd like to see if I can measure that!

I have several Pi's and (after having given workshops on how to install ownCloud on them) plenty experience in getting them running, but no experience with Smashbox whatsoever. So - I can use some help. If you're interested in joining the fun, be there tonight!

On a different note, ownCloud comes out in the beginning of next month - that means that the NEXT meetup is essentially a release party at C-Base. Sir, yes, sir - release party time! See you there?

There'll be a meetup on the 6th of July in Nuremberg, I guess that's a pre-release party as the release is set for the 7th. Munich also has a meetup on July 15, like Berlin.

ownCloud Meetup to test devices in Berlin

Coming Wednesday, it's time for the monthly Berlin ownCloud meetup again. Last month, we wanted to play with some little development boards, install ownCloud on them and see what they could do. But we had over a dozen new participants join, turning the meetup mostly in a 'how to get an ownCloud development environment up and running' session.

That was great fun and I'd love to see everybody again - but the devices are also in the office, looking at us like they want attention. So, if you're interested in playing with Banana Pi, Raspberry Pi, cubietruck and similar stuff, come and join our meetup. Wednesday, starting at 7 at c-base Berlin in the main room, we'll grab some tables and bring devices, power cords and chargers and get going.

We'll see how far we can get, but goals include getting some decent performance statistics of the differences between the various devices and ownCloud releases as well as finding some ways of optimizing the ownCloud performance on these boards.

Note that ANYBODY, seriously, ANYBODY is welcome. We'd be more than happy to help you get started with any type of ownCloud hacking, so if you don't care about Raspberry Pi's but rather hack on 2048 CPU clusters - fine with me. Feel free to bring one and we'd be happy to turn it into a fun toy ;-)

Oh and if you tend to get hungry in the evening, bring something to eat. We'll probably grab some pizza.

You can RSVP here but you're welcome in any case!

First Berlin meetup of the year!

Next week will be the first Berlin ownCloud meetup of 2015. As the ownCloud Desktop client team is having a code sprint hackathon meeting get together at the office, they'll be at the meetup. So if you wanted to hack on the client, fixing or adding something and live in Berlin - join at the meetup!

The meetup in February I unfortunately can't join but we've already decided that the meetup in March will be All About Devices™ - we will have some Banana Pi, Raspberry Pi etc devices around to play with, do some performance testing and optimization and see where we can take 'em! If you're interested, you can RSVP already.

See you in Berlin?!?

If you're not in Berlin and are sad that you don't meet ownCloud people - check out our events pages to see what can be done about that!

picture viewers/editors

Hi all!

Scanned an A4 using my HP all-in-one. It doesn't print (days of struggeling) but the first time I choose 'scan image' from KWord it worked all fine. Life's like that, I suppose. And I primarily bought it for the printing. Blegh.

Anyway, that's not the topic. The scanned image is, in a way. I scanned with defaults, so that's 600 DPI - resulting in a rather large 7000x5000 something pixels. Enthousiastically I opened it in Gwenview, which took at least 5 seconds to start, then freezed another 15 sec to open the file. And the file looked horrible - while the original was perfectly readable, on screen, resized to fit, it was almost impossible to read!

So I fired up showfoto. Which opened right away, showing a progress bar loading the file. And it looked great, while zooming works much smoother!



But it's not very nice to browse through pictures, like Gwenview, which is very easy to use and has all these nifty features... So I'm wondering... How about putting the thumbnail stuff, the image loading & zooming all in a lib or something and sharing it? Cuz really, showfoto and Gwenview share so much functionality and purpose yet it seems less code than they should ;-)

Yes, I'm not closely involved with either teams and I know it's easy to call for such things from the sideline. But I know both apps are actively developed, both are innovative and not afraid to experiment (I actually use both digikam and gwenview together with dolphin, showing their unique variations on the dolphin file selection mechanisms, during my presentations about KDE and innovation). Come on guys, make me even more proud of you!

GCDS day 3

I'm very surprised that the party we had last night has not been blogged about yet. It was really a great party - my head and stomach remind me of that continuously... Maybe I shouldn't have had that last beer. Or a few of the ones before. But I agree with Troy that our community rocks, and I think we did prove ourselves yesterday. Lots of KDE ppl dancing (I'm most proud of getting Praedeepto to dance). Most of the KDE Board members did their duty, only Cornelius Schumacher disappointed me and refused to dance. I tried talking to his conscience ("you have to set an example!") and his sense of honor ("look, even Sven is dancing and as Amarok dude he knows how sucky the music is") but no dice. Tried bribing him with beer (he's german... How the heck could he refuse?!?), tried to involve some others (thanks to those who tried). In the end I had to give up. So I offer 5 beer to the person who gets Cornelius to dance at one of the next parties here in GC. I do want to see it, however.

There will be another party tonight and I promise I'll be a lot nicer. I won't go around, sneaking up behind ppl and giving them a surprise hug, I won't annoy ppl by 'accidentally' bumping into them, or explain to Gnomes why KDE is a happy community using a very outdated term. As long as you guys & girls don't scream at me today, I'm in a sensitive state. Deal?

Yeah, about that state.It took me until about 15:00 to feel reasonably human again. I just started breakfast. And I wanted to do soo much today... Ah no problem, there will be more days.

Currently listening to Sebas who is trying to get some input from the community on the direction we will be taking in the future. Seems the party has afflicted a lot of damage, everyone is quiet ;-)

Last Performance Blog

According to a reply on my previous blog, I could just as well test if having my hamster run in his wheel would increase drawingspeed as long as I don't use valgrind and cachegrind.

Well, I've never used those tools before, but hey. Let's give it a try.

Valgrind and Dolphin (resizing all the time).
Well, valgrind talked about memleaks. I get that. The details of that, however, don't tell me much (of course, that's an understatement).

==21180== LEAK SUMMARY:
==21180== definitely lost: 8,383 bytes in 342 blocks.


The output for individual parts where something is lost is like this:
==21180== 216 bytes in 1 blocks are definitely lost in loss record 175 of 257
==21180== at 0x4021765: malloc (in /usr/lib/valgrind/x86-linux/vgpreload_memcheck.so)
==21180== by 0x56BB885: _XimOpenIM (in /usr/lib/libX11.so.6.2.0)
==21180== by 0x56B88CF: _XimRegisterIMInstantiateCallback (in /usr/lib/libX11.so.6.2.0)
==21180== by 0x5699517: XRegisterIMInstantiateCallback (in /usr/lib/libX11.so.6.2.0)
==21180== by 0x503F73D: QXIMInputContext::QXIMInputContext() (qximinputcontext_x11.cpp:361)
==21180== by 0x503E4A5: QInputContextFactory::create(QString const&, QObject*) (qinputcontextfactory.cpp:120)
==21180== by 0x4B928F1: QApplication::inputContext() const (qapplication.cpp:4541)
==21180== by 0x4BD5046: QWidget::inputContext() (qwidget.cpp:245)
==21180== by 0x4C051CC: QWidget::destroy(bool, bool) (qwidget_x11.cpp:853)
==21180== by 0x4BD7D8F: QWidget::~QWidget() (qwidget.cpp:1264)
==21180== by 0x4EADB04: QLineEdit::~QLineEdit() (qlineedit.cpp:357)
==21180== by 0x4E76C8A: QComboBox::setLineEdit(QLineEdit*) (qcombobox.cpp:1599)


Above is with QtCurve as style, btw.

Well, I think that would explain it all if I just had more brains. Unfortunately, I don't, so let's go on to kcachegrind. Again with QtCurve, which, as I mentioned before, draws a bit faster than Oxygen.

KCachegrind
Well, what can I say. Lovely graphs, that's for sure. As far as I can tell, this tool shows where the app is spending it's time. I guess the 100% ld-2.7.so doesn't mean that's what the app does all the time.
The < cycle 13 > I see as LibQtCore.so.4.4.0 doesn't explain much either. Digging down (this kcallgrind tool is pretty cool, even though I don't really understand it) I seem to see a lot of calls to QWidgetBackingStore?!? And there are over 2.4 million calls to isPaintOrScrollDoneEvent, but the % spend there (if that's what the % mean) is just 1.5%... Hmmm, again - big ?????

Or is that second line, self, the one which shows how much a function is called? Sounds like it could be true. In that case, at least 12% of the time of Dolphin (when resizing all the time, of course) is spend in varous libc-2.7.so calls.
The funny thing is that the orders for the libc-2.7.so functions seem to come all from the same Qt function: QListView::paintEvent < cycle 13 >. And then some KFileitemDelegate::paint things.

This could mean QListView is the problem (did I say something about QScrollBar? Does that have something to do with QListView?)? Or does this mean my hamster simply didn't run fast enough?

I'm sorry. This does not make anything clear to me. As I said before, I'm going to wait for someone actually knowledgeable in this area to say something.

A commenter on my previous blog did use callgrind, btw, but his findings were not very clear (to him, I mean, they were obviously not clear to me anyway).

Luckily I received also a few private mails, ranging from "Xephyr does indeed weird stuff, Qt4 apps draw much faster in it than in plain X" to "go on blogging about this". Which I won't, btw, as I never intended to go this far. Sure, many ppl seem to find it interesting, but the complaints about "don't whine, do something", while not making me happy, have a point. I can't do anything, so someone who actually CAN should take over. Or not, in that case - well. Shit happens. Maybe just use GTK apps, then. I had to install a bunch of GTK libs for kcachegrind anyway, for some weird reason:
libgnomecanvas, libgnome, libbonoboui, gnome-keyring, libgnomeui, graphviz, gail, gconf, gnome-vfs and libbonobo (including a bunch of errors about missing gtk things and other stuff, even though kcachegrind works fine)
Why KCachegrind needed that stuf: please let me know. I'd love to get rid of it again.

Edit: aah, probably optional (but apparently compiled in) dependencies of Graphviz... Weird, the graphviz page doesn't mention them, so I still think it's faulty.... So far for Arch linux is mean & lean.

Oh, and Happy new year for those who already are in 2008 when they read this ;-)

Edit2: And the Gwenview author, Aurélien Gâteau, just emailed me to ask me to test some improvements to the panning in Gwenview 4 he made in response to my blogs. So I did test it, and I am happy to tell you all the slower panning in Gwenview in KDE 4 is totally gone - the KDE 4 version is now as fast as the KDE 3 version! Thank you, Aurélien!

Edit3: make the blog sound less like I'll be crying all night because someone said bad things - after all, that's not the point, and I don't want to sound like I won't blog anymore if ppl are mean...

performance and Qt 4.4 again

My previous blog resulted in an large amount of reactions, many of which warrant a reply from me.

I blogged about bad drawing performance, and the big question I had was: what was causing it.

Firstly, a few video's were posted which showed the issue:
http://bw.uwcs.co.uk/kde3_resize.ogg
http://bw.uwcs.co.uk/kde4_resize.ogg

I hope the person posting them won't mind the bandwidth sucking....

Now I ended my previous blog with mentioning this issue seemingly was a bug in Qt, and more precisely, QScrollBar, based on the profiling one reader did in the blog before that. But in the comments section another issue came up: first mentioned by Benji, who discovered to his surprise his 3 year old laptop with integrated (Intel) graphics performed better than his NVidia GeForce 8800GTS... Quite a clue as to where the issue might be, I'd say. An article talking about the upcoming (now out) NVidia driver mentioned quite a huge improvement in XRENDER performance, so that might make a difference. I've compiled and installed the driver, and unfortunately, no visible difference. I can show the graph, but just take my word: Konqi still takes between 10% and 20% CPU when scrolling (that's with dualcores, so it's actually between 20 and 40% of a 2.4 ghz AMD core...). Resizing still barely goes over 5 frames a second (mostly a lot less).

So it probably isn't the NVidia driver - somebody with an old ATI Radeon (pretty good FOSS drivers available) also has the issue. But I do wonder about the apparently good Intel performance. More convincing against the NVidia-does-it case is an comment from Rui, who mentions sub-standard Qt 4 performance on Mac OS X:
Hi, FWIW, on Mac OS X, the last.fm client, which bundles some version of Qt4, is also a lot slower redrawing itslef when resizing than, say, a Finder window, even if said Finder window is displaying the new coverflow view of files.

I'll try getting some numbers later. Maybe checking Qt4's performance on windows would be constructive too. By checking on other systems we are eliminating a whole bunch of probable places where the slowdown could be occurring. This kind of performance problem is quite difficult to measure especially because there are so many variables going under (system frameworks) and above (the app itself) Qt.

So, were are we at. A short overview:

- It is not the Oxygen style.
- It is not Qt doublebuffering.
- It is not 3D/compositing stuff.
- It is not the (lack of) alien widgets.
- It is most likely not the XRENDER performance of the NVidia drivers (but maybe Intel does something right?)

So, we're back at Qt again. Why does Qt 4 drawing seem to perform so much worse on some hardware? It could be that Qt 4 is much more dependent on hardware acceleration - and if that's not done properly by the drivers, drawing suffers. I wouldn't know. But if it IS drivers, the new NVidia driver should probably fix it - and it doesn't. I know there is more to drivers than just XRENDER, so I should probably let others talk about that. Actually, that would be a good thing - maybe a graphics guru (our own Graphics Ninja perhaps?) could chime in, say something?


----DISCLAIMER
Now I also have a disclaimer, as my previous blog also resulted in a comment blessing me by the name of 'whistleblower', and someone else wasn't to happy with the "this is Qt's fault".

I think I was clear enough on this, but I want to repeat it: I don't think this is a huge issue. Sure, it looks bad, but that's about it. The faster application startup we have in KDE 4 far outweighs the slower drawing. Performance is much more than drawing, and imho having to wait for an application to start is much more annoying.

Secondly, my measurements are of course entirely unscientific. After all, the difference on can see with the bare eye must be relatively large to really count. And it might be that there isn't one cause but there are several causes. Maybe it is a combination of the double-buffering, bad drivers, Oxygen style and lack of alien widgets. But I have tried resizing dolphin with the QtCurve style and Qt 4.4 (aka alien widgets) with the new NVidia driver - and I still clearly could see it draw. Oh, and turning the double-buffering of leads to 100% cpu usage when resizing even a slight bit so that won't help either ;-)

Third, maybe it isn't Qt's fault at all - it might be that the way Qt4 works (more hardware acceleration, more use of animations, transparency, etc) just exposes bad performance in X.org and/or graphics drivers. Seriously, I find that very likely. Seeing how many Qt developers seem to care about performance, it seems unlikely they would release something which is so much slower than Qt 3. So it is very well possible it works fine on THEIR hardware. Apparently, the FOSS Intel drivers fare pretty well, so maybe they all have these nice MacBooks and such ;-)

Last, I didn't file a bug. First because I'm still trying to figure out what exactly causes this, and secondly I'm simply not knowledgeable enough to start profiling apps. And also because this process seems to work rather well - with the help from the many readers of this blog, I feel we're getting closer to finding the reason of the issue.

So, while I look forward to a comment from someone who can tell me he/she found the issue, this is in no way a blocker for KDE 4.0 or a horrible thing for Qt 4.x. What I would do is recommend to the KWin developers to set KWin to NOT display content in resizing windows by default. That wouldn't look incredibly cool, but it would look far better than horribly slow resizing windows. Lubos? Rivo? Are you guys reading this?

performance and Qt 4.4

And again I blog about performance in KDE 4. In my last blog about that, I did express the hope the Trolls would be able to fix the issue. So, today, I tried to test the cool alien widgets which are supposed to fix flicker-when-resizing! To cut right to the chase, no, it didn't work. Blegh.

I know I'm a sucker for speed. I want my computer to wait for me, not the other way around. I mean, a machine which can easily do millions of computations a second should be fast enough to keep up with my freakingly slow brain, right? It can take up to 1/10th of a second for a signal to travel from one end of the human brain to the other - that's not even the same league as my PC is playing in... Not to mention it has two cores and 3 GB of ram to go with that insane clockspeed ;-)

So I have played with kernel patches (Con Kolivas' work is and has always been amazing in that regard). I've tried Gentoo (aw, yes, I did...). With newer GCC, X.org and linux IO and process scheduler versions, my KDE got much more smooth. Not perfect, no - but my biggest gripe, application startup speed, well - that's covered now. KDE 3.5.x didn't do badly, but if you see the difference with KDE 4 - you're gonna be surprised. Really.

Unfortunately, as I found out some time ago, drawing performance has gone through the drain in KDE 4. Now that issue is less important. Having to wait for an application to start up is annoying - you actually have to wait. If an application redraws slow, that means resizing a window won't look good. That's about it. Not a huge deal, if it wasn't for all the cool eyecandy in KDE 4 (oxygen, animations). Really, it doesn't fit very well if your apps look bloody cool but resizing causes epileptic seizures... Besides, it's not only the resizing, but also the panning of an image in Gwenview and the autoscroll feature in Konqi. The latter manages to use up to 40% of one CPU core, and the former also redraws visibly slow.

As a non-hacker, I'm not sure all issues I find are even related - but hey, one can try to figure out the issue anyway. So, here are suggested reasons for bad performance:
- Oxygen style is slow. Yep, it's a bit slower than for example QtCurve, but not much. Not THE problem.
- Qt doublebuffering is slowing everything down. Nope, disabling makes the application flicker horribly, but it won't be any faster.
- 3D stuff is slow. Sure is, but I ran the apps in KDE 3, no compositing. Not the issue.

So, now I wonder if the slowness could be fixed by the Alien Widgets, which have arrived in Qt 4.4 snapshots.

I downloaded a Qt 4.4 snapshot (today's snapshot, from 29-12-2007), compiled and installed it, and recompiled KDE with it. As I already said - no improvement, at least not as far as I could tell. Sure, with QtCurve the issue did get better, but still - if you have the KDE 3 dolphin resize almost entirely smooth, while the KDE 4 one paints visibly (I could count the number of repaints when resizing if I wanted to) - not good.

The panning in Gwenview isn't faster either, but I guess that doesn't have much to do with the alien widgets thing. I couldn't test konqi scrolling, it doesn't load webpages with Qt 4.4 ;-)

So, it's not the Alien widgets which will come to rescue KDE 4 from bad drawing performance... I see currently one possible reason for the bad speed. An anonymous reader pointed out in my last blog how Qscrollbar isn't that fast:
It seems related to QScrollBar.
Profiling konqueror in auto-scroll mode, I obtain 50% (!!) of the time spent just repainting the scrollbar.
I found no bugreports for this in TT's bugtracker (if the anonymous reader is reading this, maybe he/she should file a bug?). I'm afraid KDE 4.0 and 4.1 will have this issue - IF this gets fixed, it most likely will take until KDE 4.2 is out for this to get into the users hands. Unless the Trolls decide it's important enough for a bugfix release, of course... Personally, I hope so, this really looks bad.


Further info:
I use the following additional cmake-options in my .kdesvn-buildrc:
-DKDE4_BUILD_TESTS:BOOL=OFF -DCMAKE_BUILD_TYPE=release
-> that should lead to no debug symbols, if I understood some recent threads properly.
I've tried several styles, from the with Qt included clearlooks, QtCurve and default Oxygen. Yes, Oxygen is a bit slower with its cool background gradient, but it's not like the other ones are smooth anyway.

One last thing, of course all this might be my fault. I think I do know a little about compiling and stuff, but I'm not a code writer, and profiling is scary for me. So maybe I've made a horrible mistake, and the KDE 4.0 packages my distribution will provide soon after the release (love the rolling release schedule in Arch) will be as fast as a fox. And I don't mean FireFox, as that must be the most horribly slow painting application we have on linux (and windows and Mac OS X).

Let's hope so.

I wrote about the performance of KDE 4 some time ago, and I'd like to revisit it.

After a few threads about debug builds and release builds on kde-core-devel I figured I was wrong in my previous entry, KDE 4 couldn't easily be build without debugging symbols. So my build WAS a debug build. But now it's possible to have it clean and fast, and I did indeed see an increase in performance when trying it.

Apps start up even faster, it's rather weird to see an application which always took a noticeable delay to start up now pop up like its a basic calculator ;-)

Drawing is also a bit faster, but still clearly not as fast as in KDE 3. Resizing dolphin on KDE 3 is almost entirely smooth - very unlike the rather unpleasant experience on KDE 4. Gwenview is a bit faster, it doesn't draw an image visibly when you drag it anymore but clearly isn't as smooth as the KDE 3 version. It IS faster loading large images though, a 6400x3200 image loaded in a snap (!) while the KDE 3 gwenview made me think it got stuck...

And here's a new Konqi-auto-scrolling picture. It starts with XPLANET (nice 20, yes), then 5 seconds pause, and I start scrolling the Konqi from KDE 3. You don't see a visible difference in CPU usage... When I increase the speed of the scrolling, there are some very small spikes but generally the CPU is still barely used.



The picture for KDE 4 is different - approximately 10% of my cpu is needed to scroll a page slowly, which increases to 20% when I increase the speed of scrolling.

Notice that these numbers DO represent a decrease compared to my previous test, which showed twice as high CPU usages in KDE 4. So it might be no more debugging, or some performance work - either way, Konqi got twice as fast already ;-)

As discussed on the other blog post, this is all with a dualcore 2.4 ghz AMD, 3 gb ram, proprietary NVidia drivers (geforce 6600) and NO compositing. Tests are run in a KDE 3 environment. Yes, double-buffering is on in Qt4 - as it should be, it looks horrible without... The whole area flashes when scrolling.

One person mentioned a potential scrolling bug in Qt4, which makes it redraw the whole area when scrolling. This could indeed explain the flickering without double-buffering and the performance problems, I guess. Hopefully a new Qt version fixes this, I found quite a few scroll-speed related bugs in the TT bugtracker.

I think I can conclude the performance issues with drawing are not as bad as I depicted in my previous blog, and it's very well possible they can and will be fixed. Meanwhile, KDE 4 performs amazingly well in many other areas - most noticeably application startup.

EDIT: With some help (see comments) I was able to compile QtCurve for KDE 4, and tried it. Some argued the slow drawing when resizing might be due to Oxygen behing slow, and that showed indeed at least half true. Dolphin resizes faster with Qtcurve - but the KDE 3 version is still much more smooth.

Performance 3

Hi all,

After some raving blogs about how memory-efficient and fast KDE 4 is, I decided to test something myself.

I felt there are regressions in drawing performance in some apps, and I tried to quantify that a little bit. Of course, entirely unscientific, but the patern should be clear... For those who wonder, as far as I can tell, debugging is off in my build and my hardware is pretty nice - a 2.4 ghz dualcore AMD with 3 GB ram. It was all done in a KDE3 environment.

Procedure:
1. Start Ksysguard (the kde 4 version of course, as it looks much cooler), and minimize.
2. Do some Stuff.
3. Use Ksnapshot on Ksysguard ;-)

I tried Konqueror and it's autoscrolling feature (shift-down-arrow) and Gwenview (drag a big picture around). Here the results:



As you can see, Konqi in KDE 3 uses barely any CPU over the baseline when autoscrolling. The KDE 4 version, on the other hand uses quite some CPU.



Here I pan (is that the right word?) the image around. As you see, Gwenview 4 easily uses 100% cpu (one core) and is visually very laggy, too. The KDE 3 version looks entirely smooth, and indeed doesn't get to use a cpu core fully ever. Again, a performance regression.

Well, what can I say. There sure is still a lot to optimize, and maybe these issues can be fixed very easily. After the positive blogs about memory usage I became rather sceptical - I mean, with so much code changed and so many new features - could KDE 4 really be that much faster? Now I didn't test memory usage, which might very well indeed be much better - but at least these two apps got slower in the rendering/painting department, so here's the balance ;-)

Other remarks: yes, ksysguard uses 20% cpu when drawing 4 sensors and put full-screen. Awful indeed, but the KDE 3 version doesn't do much better, and minimized the cpu usage is reduced to almost nothing. Besides, it looks damn sexy, doesn't it?
The big spikes marked 'xplanet' - you guessed it, it's xplanet, which updates my desktop picture every 5 minutes using some pretty pretty big planetary pictures - so it takes a while.



One more thing. I played with the colors in Oxygen again. And in my humble opinion (those who know me know I'm rarely humble, but whatever) Oxygen rocks, it really does. It is one of those rare themes which looks amazingly good in a wide variety of colorschemes. Despite the fact there are a few unfinished things (but that's true for almost all of KDE 4) it is imho the best theme ever made. Original (it looks as much as Vista or Mac OS X as those look like Windows 95) and brilliantly smooth.

The best thing about Oxygen is some quality it shares with Dolphin: it grows on you. At first, Dolphin didn't appeal to me - I love konqi as a filemanager. And I wasn't impressed with Oxygen much either, I normally prefer very glossy themes like liquid or Polyester. Oxygen, I thought, was kind'a boring.

After using it for a while, I've grown very dissatisfied with any theme on KDE 3 - I find Domino, one of the most modern KDE 3 themes, bearable, but that's about it. And I find myself using Dolphin more and more, to the point that I installed the KDE 3 version on my main system...

So, to those bashing Oxygen, Dolphin and Kickoff - wait until you've used it for a while. You'll probably discover you don't wanna go back...

Konsole performance

I had a discussion about the speed of Konsole with some Gnome users at a CodeYard meeting. Some testing on the spot revealed gnome-terminal was much faster. So I tried to test some stuff at home, too - see if anything has changed with KDE 4.

Well, yes, Konsole got almost twice as fast - still slower than gnome-terminal, though.

[root@wietbak log]# time cat messages.log.4
(lots of output)
real 0m1.923s (Konsole KDE 3)
real 0m1.077s (Konsole KDE 4)
real 0m0.412s (gnome-terminal 2.18)

But there are differences - some files show different results.
Take the output of another file, dunno what it's supposed to be but it's in /var/log, too, and contains many weird characters like these: ���E�Ms� �

[root@wietbak log]# time cat btmp
real 0m1.030s (gnome-terminal 2.18)
real 0m0.819 (Konsole KDE 4)
real 0m0.711s (Konsole KDE 3)

So for this file, the result is exactly the opposite, though the differences are much smaller.

Weird, weird.

Let's cat all files in /var/log, then, see what the overall scores are:

[root@wietbak log]# time cat *
real 0m10.797s (Konsole KDE 4)
real 0m18.940s (Konsole KDE 3)
real 0m6.524s (gnome-terminal 2.18)

real 0m16.423s (xterm)

For those who want to know how xterm does... Besides looking horribly (no anti-aliassing, and flashes horribly while scrolling), it sucks only slightly less than Konsole from KDE 3 in terms of speed.

Anyway, that confirms the first results. Konsole got almost twice as fast, but gnome-terminal is still a lot faster ;-)

Now of course Konsole kicks gnome-terminal in terms of features, but speed matters too ;-)
I'm not unhappy with the current speed (hey, twice as fast, that's good!) but there apparently is room for improvement. So let's see what the future (KDE 4.1?) brings...

accelerated rendering

Of course we have some experts in this area in the KDE community (zack is well known, but I wouldn't forget about Mosfet or the people working on KWin.

Now my understanding of this stuff (being not a developer) is always very limited - which does nothing to lower my interest in them... So when I read this blog on the gnome planet, I found it very interesting. I mean, it has pretty pictures, you know... And hardware accelerated rendering, preferably using openGL for everything (or at least more than we do now) is something I'd love to see - it really sounds like a big thing in terms of performance ;-)

Now the person who wrote this blog isn't very positive about this, but I agree with some of the comments there - OpenGL probably won't go away. Or would it be possible to integrate it with toolkits? This blog by Daniel Molkentin talks about using just the normal Qt api, yet the drawing goes through OpenGL. Sounds like neat, but what about all the other drawing in KDE/Qt?

The point of this blog is mostly that I haven't read about lower-level stuff like this for a while in the KDE blogosphere, and I'd love to see that change ;-)

How's the status of Arthur? Animations KDE4 are still kind'a disappointing in some areas - at first I blamed compositing (and it does make matters slightly worse, yes, but just slightly). Take the run dialog - it has an animation when you click options. Lovely, but it never works smooth, even the 100th time (you'd say it has cached stuff, and it has, but it's not enough). It's not just having to resize the window (even though that's bad), but even after that it's not exactly smooth.

And the background gradient in Oxygen has the disadvantage you often see a plain background flashing when resizing or animating things. As I see the CPU usage max out, I wonder if the GPU is used for any of these things - and if not, why not? Where's the bottleneck? X, Qt, other infrastructural stuff, or can't the GPU help at all?

So, who's involved in these things, and willing to write about them - what can we expect for KDE 4.0 and the future? Will these things be solved, is 3D hardware acceleration gonna help us anytime soon? When I see the animations in Vista - they're very smooth, and CPU usage keeps low, so it seems to me they've got the issue solved, haven't they?

Colors again

A bit more work went into the Colors configuration, and imho Matthew Woehlke is doing a great job there. Today I played a little with the 'contrast' slider there, which gave the following screenshots, starting with Oxygen's own colorscheme, and then the 'Midnight Meadow' scheme.

Oxygen Low Contrast

Oxygen High Contrast


Yeah, not a huge difference. Personally, I prefer the 'Beos' color scheme, here in high contrast:


Here we have Midnight Meadow:

low contrast


medium contrast


high contrast


And last but not least, about Plasma performance. Some time ago, KNewsTicker was ported. Now there is a discussions going on about the release status on the KDE development mailinglist, and the KNewsTicker developer argued Plasma performance was bad - his applet used 15% on a 2 ghz Core2Duo, while just scrolling some text. Well - either he fixed the performance issue, he had a configuration problem, or someone else did something cool - but I can't reproduce it at all:



Further complaints about plasma performance (high cpu usage when moving plasmoids around) lead to Aaron testing the difference between KDesktop in KDE 3 and Plasma in KDE 4 - KDesktop used 3 times the cpu for moving an icon than plasma for moving a (transparant, svg, good looking) plasmoid... And to finish the 'performance sucks' argument off really well, Aaron mentioned there still is a lot of lowhanging fruit to fix, and Qt 4.4 is gonna speed things up a lot. Well, if Plasma performs as good as KDesktop with superkaramba, you won't hear me complaining (it does a LOT more). And it really looks it does - not just from Aaron's comments. Try KDesktop and Superkaramba with a few applets and be shocked (and that's not just superkaramba, all applet things in linux suck in terms of cpu usage - looks like Plasma is soon going to be the most sane implementation available).

BTW - see how cool KDE 4 looks ;-)
This is svn from a few hours ago, so the latest & greatest. And after playing with it for some minutes, I must say the main reason I don't switch fully to KDE 4 is because I installed it as a separate user, and I would have to migrate it to my current account or migrate my data to the other account ;-)

KDE 4 - does it work or not?

After a few blogs which where rather positive, we're now also seeing blogs and comments much more negative (I don't feel like linking to them, sorry).

Aside from some speed issues (which I don't see, frankly KDE 4 is clearly faster here in many aspects - see my previous blog), 90% of the complaints is about the state of Plasma

Now I agree Plasma is a big thing, important for KDE - but judging the whole of KDE by the part which is least finished - I think that's unfair. And to be honest, 90% of commments about Plasma are about the panel with the still-very-basic taskbar and the missing menu. The menu still is in playground (could be it was just moved into kdebase, btw) and well, yes, the taskbar - it's not very ready either.

But come on. Think about it. You're complaining about 10% of 10% of KDE 4. Actually, much less, as I've made up these numbers :D

So how much time would it take to write a decent panel, provided the current state? And Plasma will still see a lot of work, it's not frozen yet, unlike the rest of KDE. I'd rather see everyone trying and talking about all the new good stuff instead of complaining about the almost-last portion of KDE still in flux...

When I show people the state of Plasma, they're like "hmm, that's not good". So I then proceed to show the Edu and Games, cheers them right up. And there is Dolphin, Ksysguard, Gwenview, Okular, Kolourpaint, KRDC, Konsole - all those apps which have seen so many improvements. Or these small things, like the file dialog, the character selection dialog, the shortcut dialog... I don't hear anyone raving about those. Is that just because they COULDN'T get past the missing menu and not-so-great taskbar applet? Please...

We should be realistic (KDE4 won't be perfect, and will have regressions) but all this complaining doesn't do much good either. Feel free to report bugs for those things which are supposed to be finished (I just emailed the kwin developers about the not-working window-shortcut feature). And join those working on the things still not ready. And complain to your mum.

Take care.

Who did this?

Here another "KDE4 is really usable" blog. Well, I just want to point that out up front. Oh, and my day was different from Wade's. Yesterday I gave a talk to dutch students about KDE 4, I didn't get machette's to my head. Aaah well, everyone's different.

The real topic is something else, though - I wanna know who did the following:


That person deserves some kudos. I haven't seen any blogs or whatever about it, but I think it's a great job. Solid Usability Work the KDE Way (tm): make it easier to use and more powerful at the same time. Yes, it's a bit harder, but much more rewarding to not take the easy route of 'dumbing down'. Nice work... Of course, there is more to do, maybe the person who did this can spare some time on the 'edit toolbars' dialog as well.... Currently drag'n'drop is broken ;-)

Another thing which I wanted to mention: Cornelius said he was amazed by the speed of KDE 4 apps. Now I took that with a grain of salt, as I hadn't seen any amazing speed yet - Okular was slow, I had seen konqi using 100% cpu and still stutter on scrolling, KWin didn't exactly go smooth... But while making the above screenshots I used kwrite, and indeed - kwrite4 starts faster (feels at least twice as fast) as kwrite3 (both in my kde3 session)! And other apps do the same. Gwenview is like 10 times faster, and konqi4 seems faster without pre-loading than konqi3 with! Konsole seems a tiny bit slower, though.

Sure, you don't want kbuildsycoca4 to run first, but after that's done - applications are wonderfully fast. It's like someone turned 'speed' on in Trunk, somewhere in the last week... Whoever did that, nice work!

Now, the last thing: colors. I've just been trying the colorscheme things, and they work great - see the following screenshots (yes, the second one, called Beos(?), gives the higher contrast many ppl seem to want from Oxygen).





The oxygen style also does a pretty nice job on darker colorschemes (unlike most KDE3 styles):

performance 2

Now I got a few responses on my blog about performance in windows, so I thought let's add a few new irritations.

My university has recently build a really amazing new library. It's architecture is pretty stunning, imho. Of course, it was WAY over budget, but hey, spending a few million on great architecture - I'm not even that opposed to it. Though they cut all college hours by a few % to pay for it... Aaah well.

There are a lot of PC's in there. They use thin clients, win Xp... Big, 19" screens, nice keyboard and mouse. Feels like they choose the wrong resolution, though (non-native). I can't change it, the whole thing is locked down pretty heavy.

So. And how is the performance on those things? Well, it depends. Sometimes, the Start menu reacts in 2-3 seconds, but it can be minutes. It often takes me 3 minutes to start Calculator... It's very irregular, though. SPSS runs reasonably fast, at least when doing it's calculations. Yet, every time I middleclick a link in IE 7, it takes at least a minute before the new tab is opened.

Those of you with a lot of work to do and slow hardware know how frustrating this is.

Aaah, well, that gives some time to play...

Performance - Win & KDE

People often say Linux has a better performance than Windows. Others disagree, and say the opposite is true. Now in my own experience, it's mostly so that both perform differently. windows boots up faster, in the sense of showing the desktop. But you can't actually USE that desktop, clicking the start menu as soon as it's show can take several minutes. In linux/KDE, it'll show slower if your session is still being restored (that functionality doesn't even exist in Windows) but it won't take minutes.

On the other hand, moving and resizing windows goes faster than in Linux. And have you ever seen your mouse anything else but smooth in Windows? I can stop my mouse for a second even on my 2 ghz/3GB dualcore system in linux - just use konsole with 10 tabs, set it to send the commands to each tab and log in as another user (10 times). That's not even hard - but X chokes on it.

Yet I've changed my opinion in the last few months. I've worked with windows a lot, and the deficiencies are becoming clear. Sure, drawing performance is excellent, and the mouse never stops. But start firefox (and let it restore 5 tabs) and your system becomes unusable for a few seconds (not the least because you just can't minimize the starting firefox, it insists on jumping up and taking focus a few times). It takes 10 minutes at least to boot up and start Outlook and firefox. Pretty painful. But the weirdest thing is that the system can become very slow without any apparent reason. The start menu normally takes it's time for showing submenu's, but sometimes I'll have to wait over a minute; the background doesn't redraw, I can't switch windows. My collegues react pretty dull to these kind of things: "you're trying too much, just wait". Well, I got work to do, I don't want to wait.

So I tried to figure out what's wrong. In linux, I use a cpu meter on my panel. That way, you know what's going on - is an app slow, it's because you're compiling, moron!

But the CPU monitoring stuff on windows is entirely useless. Really, wtf is that thing measuring? It seems like some random value... For example. The following screenshot was made a little over 10 minutes after boot. I started firefox, an explorer window (never showed up) and Outlook. So everything stopped to redraw, most things got slow (but not moving mouse, or even a window). So - there must be something going on, right? Well, no. According to the system monitor, those apps which didn't start nor redraw didn't stop working due to a lack of CPU or memory:

(screenshot taken after minutes of waiting for a response - thought the system was busy...)

As I know a little of scheduler things on linux (having followed the -ck mailinglist for years) I suspect a priority inversion or something like that - an higher priority process wants data from a lower prio process, but by being high prio, it makes the low-prio one unable to actually process and send the data. This seems to happen a lot, and most Windows users here at work are so used to it, they don't even notice it anymore. Not being able to work for up to a minute, many times a day - apparently, you can get used to anything. Boy, I get annoyed if the K-menu takes more than half a second to come up.

Guess there's a good reason why Vista takes up so much resources - these hickups of course seem to be smaller on much stronger hardware. Stronger hardware = more room to fuck up. So instead of fixing the scheduler (and other) problems, they up the minimum requirement... Gotta love those guys.


How much money would companies earn on switching to linux? I mean, JUST decent windowmanagement in windows, I'd spend a million bucks on that if I had it. Windows just take focus while you're working, or they seem to have focus but don't (Outlook asks for my pasword everytime it starts. It shows a little 'type pasword here' window, which seems to have focus, but doesn't). And don't get me started on the alt-tab behaviour - even a tooltip over a button seems to count as a full window in alt-tab, so you can't use that to switch to another application if your mouse is over a button!!! blegh. I don't understand why ppl take that pathetic excuse of an OS seriously, and say 'linux isn't ready for the desktop'. No, WINDOWS isn't ready. Linux was ready years ago, but MS still has a long way to go. The only reason it's being used is because ppl gave up on it - they don't try things more efficiently (alt-tab) because it won't work. Just do everything the slow and painful way. Companies should sue MS for RSI...

Strigi performance II

Let's check the performance of some other plugins. The strigi png extractor was clearly faster than the KFileMetaInfo based one, but how about the other fileformats?

So, I went looking for a fileformat which was abundantly available on my pc, and supported by both strigi and kfile. Not much choice, but in the end, xpm turned out to be interesting. Again, xmlindexer was a lot faster. It took xmlindexer 13 seconds to extract the meta-data from all xpm files on my disk, while kfile needed almost 2 minutes (average over 3 runs again).

Now I'm really wondering why the difference is so big. So I went to sysprof to figure out what was happening. It can show you what an app is spending it's time on.

Looking at the results, it seems kfile spends only a very small percentage actually reading the metainfo. Most of it's time, around 50%, it's trying to figure out what mimetype the file is, using kmimetype! So that's the slow part... In the xpm case, a bit more than 3% was spend in reading meta-data, while with the png plugin, it was less than 0.2%!

So the speed difference between kfile and strigi's xmlindexer wasn't really in reading meta-data itself, it was mostly in figuring out the mimetype in KDE. Which strigi does much more efficient.

What does this say about strigi performance? Well, we're back to square one. It's not reliably possible to correct for the slowness of kmimetype, so I can't figure out how much faster (or slower, but I think that's not very likely) the strigi meta-data extractors are compared to kfilemetainfo. At least, we know if someone would write a strigi analyzer for it, figuring out mimetype will be a lot faster in KDE 4...

And of course, strigi could use it's database for meta-data - which might give a big speedup after all. And it will allow things like special directory listings based on some property of the data.

strigi in kdebase

So, Jos van den Oever (yes, the guy working on Strigi) blogged about porting the KFilePlugin's to the new strigi-based infrastructure. Those plugins extract meta-data from files, like in case of a picture, resolution, creation date and such.

This allows Nepomuk integration, less code duplication, and more speed.

Now there have been some insights in Strigi's performance, where it proved to be 40 times faster indexing files compared the the second-best... Of course, the competition has improved since. But how does the new strigi meta-data extraction compare to the old KFileMetaInfo in terms of speed?

I tested this on all png files on my system, using the following two commands:

For KDE 3: time locate png|grep \\.png$|xargs kfile --av
For KDE 4: time locate png|grep \\.png$|xargs xmlindexer

The output of those is comparable, though the xml indexer from strigi extracts stuff in xml form. Both put out the meta information to the commandline, showing things like dimensions, color depth or comments.

Now, on my dualcore 2.2ghz 3gb RAM system, kfile takes 40 minutes to complete (average over 3 runs). Xmlindexer takes an average of 24 seconds. Yes, seconds. That's almost 100x as fast...

xmlindexer (strigi):
real 0m27.262s
real 0m27.056s
real 0m19.328s
Average 24 sec

kfile (kde3):
real 45m5.689s
real 37m14.158s
real 37m17.515s
Average 40m12sec

So, how could this be so much faster? Filecache used by strigi? I first ran xmlindexer (27 seconds), then kfile (45 minutes) and then, after a few hours (during the night), I ran both tests twice again, with a 1 hour interval. Yes, I did use my dualcore in the mean time, mostly watching a movie. So the results aren't exactly scientific, but hey - we're talking a 100 times faster here. A movie won't have such a huge impact... And filecache might have an impact, but if it does - it can't explain the differences. Second and third run of kfile where indeed faster, but still magnitudes behind strigi (which got faster as well).

According to Jos, the speed difference could have two reasons: kfile does something wrong, and KFileMetadnfo isn't that slow, or - well, it is... (I'll have a look at this, btw, expect a second blog about it). It could just be that the png indexer in KFileMetaInfo hasn't seen much work, and most of the speedup could have been gained by rewriting it - so I'll have to check other plugins as well to see how they compare. And currently, the new infrastructure isn't as complete as KFileMetaInfo was - many plugins haven't been ported yet, and there is no KIO integration.

Still, seeing new technology being this closely integrated in KDE, and finding out what it can do makes for some interesting statistics, don't you think? I'll try to test more plugins, and figure out if there is something wrong with kfile. And Konsole, btw, as it crashes every time I try the KDE3 test in it... Maybe the unlimited history gets filled up a bit too fast/much?