Debugging a system without desktop

Sometimes, Linux Desktop bites you in the ass. You just get a command line after booting up. I've ran various bleeding edge distro's over the year, including Cooker, Unstable, Factory, Unmasked stuff; build my own kernels and desktop and more.

Seeing a request for help today made me decide to write down the steps I go through when my favorite desktop doesn't appear in the morning... I bet there's plenty to improve, tips are welcome!

General approach

I try to work quickly from the bottom up: test pieces of the lower stack, go higher until it breaks. While digging through logs can find the issue most likely, I've found that shooting a few quick commands to catch common issues is often faster than immediately going to comb through logs... So, first basic things before diving in logs.

step 1: common issues

I usually of course think back to what I recently changed or updated - that's likely the issue. But to keep it generic, let's assume that doesn't tell us anything. Then it's time to look at a few common causes of problems. Log in on a console with the root user name and password.

disk space

First up for me are disk space and stuff in tmp. A full disk can lead to the weirdest problems - from an end user perspective, it often makes no sense. So check how much space there is on your devices:
df -h
and check the output for 100% full drives:
Filesystem       Size   Used   Avail   Use%   Mounted on
devtmpfs         7.7G   8.0K    7.7G     1%   /dev
tmpfs            7.7G   5.2M    7.7G     1%   /dev/shm
tmpfs            7.7G   319M    7.4G     5%   /run
/dev/sda1          2G   704M    1.2G    63%   /boot
/dev/sda2         20G    14G    5.0G    74%   /
tmpfs            7.7G      0    7.7G     0%   /sys/fs/cgroup
tmpfs            7.7G   139M    7.6G     2%   /tmp
tmpfs            7.7G   319M    7.4G     5%   /var/lock
tmpfs            7.7G   319M    7.4G     5%   /var/run
/dev/sda3        213G    74G    139G    35%   /home
Obviously, if a drive seems full (less than 100 mb free), it's time to clean up. I trust you know enough cd ../cd [directory] and rm -rf [directory]/rm [filename] to get this done...

Temporary directory

Looks all good here, so next step then, just to check: if /tmp is not mounted on a temporary ram drive (like in my case), just clear everything in there to make sure nothing is causing issues. rm -rf /tmp/* will do the trick. Yeah, canon, fly, but /tmp should not be used by applications to store important data (hence the name) so this is safe™. Also, check permissions for /tmp, just to be sure:
ls -la / |grep tmp
drwxrwxrwt 30 root root 760 26.06.2013 15:37 tmp/
Yes, /tmp should be readable and writable by anyone, so the permissions string rwxrwxrwt is important. If it is anything else, execute
chmod +rwxrwxrwt /tmp, remove the contents of /tmp again with rm -rf /tmp/* and try again.

update the system

Third, I always quickly do a zypper refresh && zypper update. If an update broke it, let's see if a newer update can fix it ;-)
Of course, use the commands appropriate to your distribution: apt-get update && apt-get upgrade; pacman -Syu; etcetera.

Step 2: does X work?

Second is seeing if the graphical system (X.org/xorg/X11, or just 'X') still works. Type
startx /usr/bin/xterm
This will start a simple terminal in the graphical environment and nothing else. As failsafe as it gets... If it works go to step 2, if not, see below.

X not working?

Now it's time to check the X.org log file and perhaps the output of the startx command. Any hints there as to what's wrong? If not, look in /var/log/Xorg.0.log (less /var/log/Xorg.0.log will do) and see where you have errors. The Xorg log is long, but has a neat way of showing errors: the codes between parenthesis give hints to what the line represents. Example output:
[199842.753] (WW) Falling back to old probe method for fbdev
[199842.753] (II) Loading sub module "fbdevhw"
[199842.753] (EE) Failed to load module "modesetting" (module does not exist, 0)
First a timestamp, then the code (WW = warning, II is for your info; Look for EE, meaning error!) and then you can try and solve this. In this case, a module doesn't exist - that looks like xorg didn't upgrade properly or got partially uninstalled. Search your package database for xorg and see if possibly-important packages are missing: zypper se x11
Loading repository data...
Reading installed packages...

S | Name                          | Summary       | Type
--+-------------------------------+---------------+-----------
[cut for simplicity]
  | xorg-x11-driver-input         | [cut for simplicity] | package
i | xorg-x11-driver-video         | [cut for simplicity] | package
  | xorg-x11-driver-video         | [cut for simplicity] | srcpackage
  | xorg-x11-driver-video-nouveau | FOSS nVidia driver   | package
[cut for simplicity]

You could imagine the input driver might be missing. Or, if you have nvidia hardware and don't use the proprietary driver, well, nouveau is what you need!

Likely culprit for users of NVidia and AMD hardware is in the proprietary drivers. You're in for a world of pain - I'd just remove them and re-install once your graphical system works again. For NVidia (the only one I have experience with) I usually just remove anything I see with nvidia in it and install the mentioned nouveau driver. Find more Xorg debugging solutions on this excellent Fedora wiki page. Once startx works, I'd reboot.

Step 3: does the desktop work?

Third is finding out if your desktop (plasma desktop, GNOME Shell etc) works. Exit the xterm (note that your mouse has to hover the window before you can type in it) and now start startx without the terminal command:
startx
Because you're root you have a relatively 'clean' user account (you should never daily use the 'root' account for your desktop, we're just testing here). If it doesn't work, you could attempt to remove the desktop user settings in the root user account. For a KDE Plasma Desktop that means:
rm -rf ~/.kde4
Note that I again assume you don't USE this account and the settings are inconsequential. Don't do this on a normal user account, you loose lots of settings, recent files and even data!

If the problem is in user settings, you should by now be on you to your desktop of choice, logged in as root! If that is indeed the case, skip to the next step. If you're not there, let's find out what is wrong.

Desktop doesn't come up?

Ok, step back. xterm did work, Plasma/Shell/XFCE don't? Let's see if we can start them FROM a xterm and catch a glimpse of where we get!
startx xterm
Now back in the terminal? Ok, let's start Plasma Desktop as example. type
startkde
and observe KDE attempting to start. Now you should see some messages and hopefully you can take it from here - going any deeper doesn't really fit this blog post, google is your friend from here onwards...

Step 4: User config broken!

Now we know where the problem is, at least, approximately: the graphical system works, the desktop works, so it's in the user settings. Time to dig there. 90% certain the issue is in the configuration of your desktop, although it could be an application somehow botching stuff. Let's start with startx and xterm again. Log in from the command line as your user and type
startx xterm and then startkde or the equivalent for the desktop you're looking for. Observe the failure, see if it tells you anything. If that doesn't help, next up the Real Hard Work: finding out what setting in your ~/.kde4 configuration folder is the culprit.

I suggest to back up your config folder:
cp -R ~/.kde4 kde4-config-backup
Then go in and randomly nuke files and folders until you get the problem. Ok, random is a bit too bad, I'd start removing ~/.kde4/share/config and if that doesn't solve it put it back and remove ~/.kde/share/config/apps instead. One of these two contains the problem, most likely...

More?

In the end, you'll have to find and solve the problem with googling and simply some hard work... The above is a guide which I personally follow, essentially a heuristic based on lots of fixing. I bet lots of readers have tips and tricks that are far more helpful and I'll happily update this post to include them...

Hugs and good luck!

oSC2013 next!

After Akademy 2013 in Bilbao, we will fly (via Berlin...) to Thessaloniki, Greece, where the openSUSE Conference will take place. Like I argued for Akademy, oSC is a relevant and useful event for the openSUSE folk.

History - oSC 10, 11 and 12

This year oSC takes place in Greece, a fact far more relevant than it might seem. The first three openSUSE Conferences I attended took place in Nürnberg. First in a conference center, the third was the legendary oSC2011 in the Zentrifuge, an old factory building creating an absolutely amazing atmosphere. Both events were largely organized by SUSE employees from the Nürnberg office but oSC11 already had a fair involvement of volunteers and a strong focus on BoF sessions and 'getting stuff done'. We had a lively marketing and ambassador team by then. I vividly remember the day before the event, when I rode a van full of crazy geekos by a few stores, buying everything from carpets and plants to lights to dress up the location. Every day we figured out who would staff the bar, organize stuff in the rooms - and moving the chairs was a matter of asking outside if a few folk would be willing to help out. At this event, there was already talk about doing the openSUSE conference in Greece.

First Prague

But going from doing the event in Nue and by SUSE people (with help) to doing it in Greece by volunteers (with help) seemed a big step and the team also had limitations as to the date and time: the event would have to be in May or June. As it was already September when oSC'11 took place, that was very close.

There came a different proposal from Michal and others in Prague, and with an office there, we decided it made much more sense to do it there and told Kostas that he'd have his oSC in Thessaloniki, but one year later. I joined Michal in scouting for a location in Prague and discovered a local community was about to organize another Linux event a mere two weeks after the openSUSE Conference! I proposed to merge the two, and the format of oSC'12 was born. Like oSC'10 two years earlier, this event tried to focus on collaboration and bringing communities together. I unfortunately had to scale back my involvement in the conference significantly after LinuxTag Berlin in May 2012 due to health issues and barely could make it to the event itself. Meanwhile, the Greeks had already started preparing the organization of oSC'13 and were present with a large team at oSC'12 to help run the show.

Greece in 2013

Feedback from the event did show that the openSUSE community wanted more 'Geeko time', so the Greek team has made sure that there will be a good openSUSE focus at the event. Of course, without compromising our open nature: there will be plenty colors besides green. Naturally, SUSE input will be lower than usual - the Nürnberg can't easily take a afternoon off to visit, even a bus trip is not enough. But we have made sure that many central developers will be there and there is a host of great sessions coming - a post about that on news.opensuse.org is coming soon!

Due to the internal changes at SUSE, I haven't been as much involved in the event as I would have liked but just seeing the conference team work has been amazing. It isn't just a Greek affair, mind you - for example, the logo and much artwork comes from the other side of the world, by the hands of US citizen Anditosan and Brazilian Carlos Ribeiro! And as with the Prague conference, Izabel Valverde again played a major role in handling travel support and sponsors. And we have Robert Schweikert and Henne Vogelsang, working on the program and website, Matt Barringer building OSEM (!!!) and some others.

Awesome

I might be overusing that word - but I really, really think oSC'13 will be awesome. As far as it is up to the Greekos organizing it, there's no doubt! And like with Akademy, if you haven't booked yet - you should...

Have a lot of fun and see you in Greece ;-)





PS: note that the above is mostly my memory of How Stuff Happened™. There were plenty more people involved, pushing, making it happen. Thanks to all of them, as we're gonna have a great conference in Greece ;-)

Akademy for everybody

About six weeks from now the yearly KDE conference 'Akademy' will kick off in Bilbao, Spain! Looking forward to that. I had to ramble a bit on why I think it's worth going there. And a tip for if you still have to book!

Have not booked yet?

Aw, I was late, and I'm paying the price, in terms of money and convenience. There's work and family and in my case a dose of always-lateness with this stuff that got in the way. Probably not different for others. But keep in mind - I never met anybody who who was sorry they went through the trouble of going to Akademy! It's worth it...

Relevant

Akademy is relevant, useful and fun above and beyond just another Free Software event. Seriously. Even in a world of Android, KDE still builds major Free Desktop stuff. Think about Krita going places lately. Think about how KDE PIM still is arguably the only serious Free enterprise-ready mail-calendar-etc solution on the market. There is plenty more at the event: KParts might get a successor, we move forwards with new task-centered UI paradigms, we move to new devices and more.

Directions

Akademy is important for setting directions. We don't all agree on where KDE is going and that's why we meet. To SET goals and CHECK expectations. That is hard to do over mailing lists, blogs, social media and all that - so we need to talk in person. We're doing new things and it takes a while, often a long while, to get stuff together. We have to address misconceptions, improve common understanding, find out why we disagree. And this works, we're making progress! Nepomuk is now getting the love it needed and same for KDE PIM, the Plasma team can finally implement their ambitious plans without the technical limitations of old thanks to QMLv2 and openGL etc etc.

And I'm betting that those who think KDE is doing the wrong thing actually have arguments and reasons for that. Are these reasons so vague and unconvincing that nobody will listen? Or would the KDE folk be so stubborn they wouldn't listen to obvious facts? That is either under-estimating who-ever-disagrees or the 300-odd KDE contributors. And both, I think, are undeserving of that.

Family

Akademy is about more than listening to talks or giving them. Even if you haven't had time or motivation to contribute, if you didn't agree with one thing or another or got yelled at by (or did yell at) somebody, does that matter? KDE is family: family has plenty of disagreements and fights, yet in the end, you all hug and make it up to each other, yes? Being at Akademy is not just about learning new things and deciding on the future of what we do (and, to a large degree, where Free Software for end users is going) but it is also about meeting, having energizing and inspiring conversations, learning from each other, sharing great ideas. Something we all enjoy, don't we?

Booking

I'm sure there'll be a lot of people at Akademy but I think that should include you... Now, about the booking. It will be hard to get a hotel room, esp if you need space for two people, I'm afraid... I booked the first 3 days (Thursday-Friday-Saturday) for a painfully high price in the Holiday Inn Express Bilbao and the rest in RÍA DE BILBAO which is far cheaper.

And last but not least, if you're going to Akademy and want to make sure all relevant people are there, be sure to talk to those you'd like to have a chat with, convince them to go! And perhaps add the awesome going to Akademy badges to your blog or website!