Triggered

Somebody pointed me to a research article about how many app developers fail to comply with the GDPR and data requests in general.

The sender suggested that I could use it in marketing for Nextcloud.

I appreciate such help, obviously, and often such articles are interesting. This one - I read it for a while but honestly, while I think it is good this is researched and attention is paid for it, I neither find the results very surprising NOR that horrible.

What, a privacy advocate NOT deeply upset at bad privacy practices?

Sir, yes, sir. You see, while the letter of the law is important, I think that intentions also are extremely important. Let me explain.

Not all GDPR violations are made equal

If you or your small business develops an app or runs a website to sell a product and you simply and honestly try to do a decent job while being a decent person, the GDPR is a big burden. Yes, the GDPR is good, giving people important rights. But if you run a mailing list on your local pottery sales website, with no intention other than to inform your prospective customers and followers of what you're up to, it can be a burden to have people send you GDPR takedown and 'delete me' requests instead of just having them, you know - unsubscribe via the link under your newsletter!

The goal of the GDPR, and of my personal privacy concerns, isn't at all related to such a business. If anything, their additional hardship (and we at Nextcloud have this issue too) is at best a by product of the goal. That byproduct isn't all bad - we all make mistakes, and being more aware of privacy is good, even for small businesses. The GDPR has forced many small businesses to re-think how they deal with private data, and that isn't a bad thing at all. But it isn't the main benefit or goal of the GDPR in my eyes. There are big businesses who totally COULD do better but never bothered, and now the GDPR forces them to get their act together. While that's a real good thing, even THAT is not, in my opinion, what the GDPR is about.

Privacy violation as a business

You see, there are businesses who don't violate privacy of people by accident. Or even because it is convenient. There are businesses who do it as their core business model. You know who I'm talking about - Facebook, Google. To a lesser but still serious degree - Microsoft and yes, even Apple, though you can argue they are perhaps in the "side hustle" rather than "it's their primary revenue stream" category.

For these organizations, gathering your private data is their life blood. They exploit it in many ways - some sell it, which is in my opinion definitely among the most egregious 'options'. Others, like Google and Facebook, hoard but also aggressively protect your data - they don't want to leak it too much, they want to monetize it themselves! Of course, in the process of that, they often leak it anyway - think Cambridge Analytica - that was in no way an incident, hundreds of apps get your private data via Google, Facebook, Microsoft and others. But by and large, they want to keep that data to themselves so they can use it to offer services - targeted ads. Which in turn, of course, get abused sometimes too.

My issue with this business model, even without the outright sale of data, is two-fold.

Ads work better than you think

First, in principle - while people might feel ads don't effect them, there is a reason companies pay for them. They DO effect your behavior. Maybe not as much or in the way marketing departments think or hope, but the effect exists.

How bad is that? Depends, I guess. To some degree, it is of course entirely legitimate that companies have a way to present their product to people. But modern targeting does more, including allowing companies to charge specific people different prices, and of course a wide arrange of sometimes nasty psychological tricks is used. The example Facebook once gave to potential advertisers, of targeting an insecure youth "at their most vulnerable" with an ad is... rather disgusting.

This gets worse when we're not just talking about product ads but political ads, either from political countries or, of course, from foreign non-democratic adversaries trying to influence our freedoms in a rather direct and dangerous way. And again - this is more effective than most people realize or are willing to admit and has swayed elections already, making is all less free.

Centralization is bad

Second, there is simply a HUGE issue with all-our-eggs in one basket. Especial when that basket is in a foreign country and not protected by privacy and security laws compatible with those in your own country. Having a single point of failure, how well protected - is just not smart. Things WILL fail, always. Better have slightly more breaches that each are just a single provider, than one breach of all private data of everyone in a country...

And that's not even talking about the fact that this data helps these companies get incredibly huge and then allows them to suppress or kill competition (or just buy it) - think Amazon, Microsoft. These tech molochs are just plain bad because of many reasons. They are anti-competitive, which raises prices, decreases choice, and the much lower innovation-per-dollar they produce is of course a worse deal for society too. They are too easy to control by law enforcement and censorship, impacting our freedoms - even when they're not 'foreign' to you. Yes, it is harder to censor 50000 private servers than one Google server farm!

Triggered

 So, as you notice, this question triggered me. Not all privacy violations are equal. Intentions matter. As does market power. And the GDPR is not a golden bullet. It has downsides - compliance is often easier for big companies than small ones, a serious issue.

Luckily, our judicial system tends to look at the intentions behind law, and I would expect a judge to fine an organization heavier for truly bad business models than for honest mistakes. I hope I'm not too optimistic here.

From my side, I don't want to bang on people's head for mistakes. I want to attack and challenge bad business models and bad intentions. A local, small app maker who fails to respond quickly enough to GDPR requests - not my target. Facebook - yes.

And by the way. Maybe it doesn't need to be said to most of you, dear readers, but of course - our open source world is, I still believe, a huge part of solving this problem. KDE, openSUSE and other Linuxes and desktops - and of course Nextcloud, Mastodon, Matrix and other decentralized and distributed and self-hosted platforms. We have ways to go, but we're making progress!

As I concluded to the person who triggered me - I know, this is far too long a reply to what they said

But it triggered me ;-)

Best reply over twitter, (twitter.com/jospoortvliet) or so, this awful Google platform makes commenting nearly impossible. And I know, the irony, replying on twitter, and I still have not moved away from blogger.com... Some day, some day. When I find time.

Rant of the day: well, at least Microsoft is making loads of money...

Sadly, many if not most of our schools today are suddenly pumping lots of extra money into Microsoft, Zoom and other proprietary software companies, because they need online collaboration. We all know there are many alternatives to giving their students' data away to foreign companies but most don't bother. It is annoying, there is always budget for Microsoft, but not for proper, local, privacy-protecting open source solutions, even if those are better. Why is that?

Reputation, I'm convinced, is the main reason for that.

We teach them the wrong thing

Unfortunately, a lot of people try to convince schools, governments, charitable organizations and even companies to not pay anything at all. They are promoting open source solutions as an alternative that is cheaper or free, which just makes it look inferior to management. They are not telling organizations to pay local and open source product companies instead of Microsoft.

Open source/Free Software advocates hammer on "but it is free"! And when they do, THEY probably think of Freedom. But the person they talk to just thinks "cheap and bad", no matter how you try to explain freedom. Nobody gets that, really, even if they nod friendly while thinking what a silly, idealistic nerd you are. Been there, done that.

I love the enthusiasm, yes, but in the end it is not helpful: it presents open source as a crappy but cheaper alternative without any real support. Well, there are a few overloaded volunteer enthusiasts who might do a great job for a volunteer but can't compete with a bunch of full time paid people at Microsoft. So the schools and governments and companies will simply use those 'free' (as in cheap and crappy) services as a stop-gap and then beg their bosses for budget to be able to pay a "proper" Microsoft service. There goes more public money in NOT public code.

We need to stop teaching companies that open source is a crappy, cheaper alternative to proper, paid alternatives from big American companies and instead tell them that they can pay for an open source solution that has real good support, no vendor lock-in, doesn't leak your data, protects your privacy and is actually better in many other ways. That way open source companies can actually hire people to make products better instead of just doing consulting one customer at a time.

And yes, some companies and some business areas have figured this out - Red Hat and SUSE are obvious examples, and projects like OpenStack have lots of paid people involved. But lots of other companies, from Bareos (backup) to Kolab (groupware) have struggled for years if not decades to build a product, instead getting sucked into consulting.

It doesn't work that way

I have seen loads of open source product companies go bankrupt or just give up and become consulting firms because their customers simply expected everything for free and to only pay a bit for consulting. Lots of open source people work at or set up their own consulting firms, occasionally even contributing a patch to upstream - but not building a product. Not that they don't want to, but they quickly find out that working your ass off for a maybe decent hourly rate does not leave you time to actually work on the thing you wanted to improve in the first place.

Indeed, you can't build a good end user product that way. Frank and myself put together a talk about this recently:



I have also recently written an article about this entire thing, explaining why of all the business models around open source, only subscriptions can lead to a sustainable business that actually builds a great product. Will hopefully soon be on opensource.com.

Yeah but volunteers...

Are fundamental to open source, yes, no doubt. At Nextcloud we could not have build what we did without lots of volunteers, heck, nearly everybody at Nextcloud was a volunteer at some point. And yes, all code we write is AGPL, and that, too is important. I am NOT arguing against that, not in the least.

What I say is:

• You can't build a great product without paid developers*
• You can't build a great product on consulting and only getting paid for setting it up/hosting

But let me then also add:

• You can build a better product collaboratively
• And the (A)GPL are the best licenses to do that

I'm sure there are exceptions to those rules, yes. But compare a great product like Krita, see how its developers struggle every day to be able to pay the bills of just a few full-time volunteers. Do you know how they are currently paying most of them? Last time I spoke to Boudewijn, the reality was sad: the Microsoft App store. Yup. How many does Adobe manage to pay to work on its products? Why should our ambition not be to have as many people working on Krita? Of course it should be. And yes, keep it open source. Is that doable?

Of course it is. Well, maybe not Adobe levels, but we can absolutely do better.

Missed opportunities

I said this was a rant, so I do have to complain a bit. My biggest regret is that KDE failed to catch up during the netbook period (around 2005). I believe that it is in no small part because we failed to work with businesses. Idealism can be super helpful and can also totally keep you irrelevant.

KDE is, lately, working more with companies, trying to build up more business around its product. GNOME has been far better at that for a far longer time, by the way. It is hard, and companies like Kolab, struggling for the last ~20 years to make things work, have shown that. Just being a for-profit obviously doesn't solve all problems. Idealism and hard work are not enough to make a business work. But we can do better, and Nextcloud is an example that shows we can. Now not all things are freaking awesome at Nextcloud, really - we work our a**** off and it is hard. We put on our best face in public but sometimes I just want to bang my head on and in the wall...

Still, see the video, read the blog hopefully soon on opensource.com - there are ways.

Thoughts welcome.

Edit:
* let me qualify that statement. You can do it without paid developers in a small project, I dunno, grep or ls or the awesome simplescreenrecorder and tools like that. With those there is a risk of the apps going unmaintained and new ones popping up all the time - look at music players in the KDE community. I'd rather see one well maintained than new ones pop up with all their different flaws, but I totally get that for a volunteer it is often easier and more fun to start fresh. In either case, once you start building something huge, it gets pretty hard without long term dedicated resources. Note that it can be donations-run (like Krita and many others), with a charitable organization. I do think it is about more than 'just' the resources. If somebody 'just' sponsored 25 people to work full-time on Nextcloud, the end result would be different than the situation today. The need to deliver something that makes customers happy (which means focus on details, scalability etc!) and pressure to do things you wouldn't want to do in your free time (developer documentation...) make a big difference.

In any case, I really don't think projects like LibreOffice, Firefox, Nextcloud, KDE or GNOME and the Linux kernel itself would be where they are today without people paid to work on them.

Principles

We recently did a post about the Nextcloud Mission and Principles we discussed at the previous Contributor Week. I guess it is mostly the easy-to-agree on stuff, so let me ruin the conversation a bit with the harder stuff. Warning: black and white don't exist beyond this point.

Open Source

In an internal conversation about some community pushback on something we did, I linked to islinuxaboutchoice.com - people often think that 'just' because a product is open source, it can't advertise to them, it has to be chock full of options, it has to be made by volunteers, it can't cost money and so on...

But if you want to build a successful product and change the world, you have to be different. You have to keep an eye on usability. You have to promote what you do - nobody sees the great work that isn't talked about. You have to try and build a business so you can pay people for their work and speed up development. Or at least make sure that people can build businesses around your project to push it forward.

I personally think this is a major difference between KDE and GNOME, with the former being far less friendly to 'business' and thus most entrepreneurial folks and the resources they bring go into GNOME. And I've had beers with people discussing SUSE's business and its relationship with openSUSE - just like Fedora folks must think about how they work with Red Hat, all the time. I think the openSUSE foundation is a good idea (I've pushed for it when I was community manager), but going forward I think the board should have a keen eye on how they can enable and support commercial efforts around openSUSE. In my humble opinion the KDE board has been far to little focused on that (I've ran for the board on this platform) and you also see the LibreOffice's Document Foundation having trouble in this area. To help the projects be successful, the boards on these organizations need to have people on them who understand business and its needs, just like they need to have community members who understand the needs of open source contributors.

But companies bring lots of complications to open source. When they compete (as in the LibreOffice ecosystem), when they advertise, when they push for changes in release cycles... Remember Mark Shuttleworth arguing KDE should adopt a 6-month release cycle? In hindsight, I think we should have!

Principles

So, going back to the list of Nextcloud's Mission and Principles, I say they are the easy stuff, because they are. They show we want to do the right thing, they show what our core motivation was behind starting this company: building a project that helps people regain control over their privacy. But, in day to day, I see myself focus almost exclusively on the needs of business. And you know what, businesses don't need privacy... That isn't why we do this.

Oh, I'm very proud we put in significant effort in home users when we can - our Simple Signup program has cost us a lot of effort and won't ever make us a dime. The Nextcloud Box was, similarly, purely associated with our goals, not a commercial project. Though you can argue both had marketing benefits - in the end, a bigger Nextcloud ecosystem helps us find customers.

I guess that's what keeps me motivated - customers help us improve Nextcloud, more Nextcloud users help us find more customers and so both benefit.

Pragmatism and the real hard questions

Personally, I'd add an item about 'pragmatism' to the list, though you can say it is inferred from our rather large ambitions. We want to make a difference, a real difference. That means you have to keep focused on the goal, put in the work and be pragmatic.

An example is the conversation about github. Would we prefer a more decentralized solution? Absolutely. Are we going to compromise our goals by moving away from the largest open source collaboration network to a platform which will result in less contributions? No.... As long as github isn't making our work actively harder, does not act unethically and its network provides the biggest benefits to our community by helping us reach our goals, we will stay...

More questions and the rabbit hole

Would you buy a list of email addresses to send them information about Nextcloud? No, because it harms those users' privacy and probably isn't even really legal. Would you work with a large network to reach its members, even if you don't like that network and its practices? Yes - that is why we're on Facebook and Twitter, even though we're not fans of either.

Let's make it even harder. How about the choice of who you sell to. Should we not sell to Company X even if that deal would allow us to hire 10 great developers on making Nextcloud better for the whole world and further our goals? Would you work with a company that builds rockets and bombs to earn money for Nextcloud development? We've decided 'nope' a few times already, we don't want that money. But what about their suppliers? And suppliers of suppliers? A company that makes screws might occasionally sell to Boeing which also makes money from army fighters... Hard choices, right?

And do you work with countries that are less than entirely awesome? Some would argue that would include Russia and China, others would say the USA should be on a black list, too... What about Brazil under its current president? The UK? You can't stop anyone from using an open source product anyway, of course... It gets political quick, we've decided to stick to EU export regulations but it's a tough set of questions. Mother Teresa took money from dictators. Should she have? No?

It might seem easy to say, in a very principled way, no to all the above questions, but then your project won't be successful. And your project wants to make the world better, does it not?

Conclusion?

We discuss these things internally and try to be both principled and pragmatic. That is difficult and I would absolutely appreciate thoughts, feedback, maybe links to how other organizations make these choices. Please, post them here, or in the comments section of the original blog. I can totally imagine you'd rather not comment here as this blog is hosted by blogger.com - yes, a Google company. For pragmatic reasons... I haven't had time to set up something else!

There's lots of grey areas in this, it isn't always easy, and sometimes you do something that makes a few people upset. As the Dutch say - **Waar gehakt wordt vallen spaanders**.



PS and if you, despite all the hard questions, still would want to work at a company that tries to make the world better, we're hiring! Personally, I need somebody in marketing to help me organize events like the Nextcloud Conference, design flyers and slide decks for sales and so on... Want to work with me? Shoot me an email!

Nasty fall-out from Spectre and Meltdown

I guess it's hard to miss Spectre and Meltdown so you probably read about it. And there's more bad news than what's been widely reported, it seems.

You trust the cloud? HAHAHAHA

What surprised me a little was how few journalists paid attention to the fact that Meltdown in particular breaks the isolation between containers and Virtual Machines - making it quite dangerous to run your code in places like Amazon S3. Meltdown means: anything you have ran on Amazon S3 or competing clouds from Google and Microsoft has been exposed to other code running on the same systems.

And storage isn't per-se safe, as the systems handling the storage just might also be used for running apps from other customers - who then thus could have gotten at that data. I wrote a bit more about this in an opinion post for Nextcloud.

We don't know if any breaches happened, of course. We also don't know that they didn't.

That's one of my main issues with the big public cloud providers: we KNOW they hide breaches from us. All the time. For YEARS. Yahoo did particularly nasty, but was it really such an outlier? Uber hid data stolen from 57 million users for a year, which came out just November last year.

Particularly annoying if you're legally obliged to report security breaches to the users it has affected, or to your government. Which is, by the way, the case in more and more countries. You effectively can't do that if you put any data in a public cloud...

Considering the sales of the maximum allowed amount of stock just last November by the Intel CEO, forgive me if I have little trust in the ethical standards at that company, or any other for that matter. (oh, and if you thought the selling of the stock by the Intel CEO is just typical stuff, nah, it was noticed as interesting BEFORE Meltdown & Spectre became public)

So no, there's no reason to trust these guys (and girls) on their blue, brown, green or black eyes. None whatsoever.

Vendors screwed up a fair bit. More to come?

But there's more. GregKH, the inofficial number two in Linux kernel development, blogged about what-to-do wrt Meltdown/Spectre and he shared an interesting nugget of information:

We had no real information on exactly what the Spectre problem was at all

Wait. What? So the guys who had to fix the infrastructure for EVERY public and private cloud and home computer and everything else out there had... no... idea?

Yeap. Golem.de notes (in German) that the coordination around Meltdown didn't take place over the usual closed kernel security mailing list, but instead distributions created their own patches. The cleanup of the resulting mess is ongoing and might take a few more weeks. Oh, and some issues regarding Meltdown & Spectre might not be fix-able at all.

But I'm mostly curious to find out what went wrong in the communication that resulted in the folks who were supposed to write the code to protect us didn't know what the problem was. Because that just seems a little crazy to me. just a little.

Client-side, server-side and zero-knowledge end-to-end encryption in Nextcloud

Android encryption in action

Yesterday, Nextcloud published plans, designs and code for end-to-end encryption in the clients. A recent report from Forrester named data encryption as one of the top global Cybersecurity trends for 2017, so this is a big deal! I thought it'd be good to go over the differences between client-side, server-side and end-to-end encryption and position what we developed.

End-to-End Encryption

End-to-end encryption is usually associated with communication channels, think chat or video calls. Signal, Telegram and lately WhatsApp employ end-to-end encryption. So what does it mean?

The main property of end-to-end encryption is that the data is encrypted on the one end and decrypted on the other end, so only the sender and receiver can read it.

As an example, take the Nextcloud Video Calls app. When you share a link for a call and the other person joins, the Nextcloud Server essentially gives the other user the address of your computer and helps you connect to each other. It will also send signals when others join the call, when you mute and so on. But the actual call takes place between your system and that of your conversational partner, directly. And before it is sent out, your browser will encrypt the data for the other side to decrypt. This way, nobody in between, like your internet provider, can listen in!

The main downside of the term is that the definition of that 'end' can vary: you can say that normal https encryption is 'end-to-end', defining one end as the browser and the other end as the server. So this brings us to the difference between server-side and client-side encryption.

Server-side encryption

Server-side encryption serves to protect data on or going through a server: as soon as the data arrives, the server encrypts it. When you use a cloud storage like Amazon S3 or a Dropbox account or a FTP at another office with Nextcloud, our Server-side Encryption encrypts the data before it gets sent to the other storage and decrypt it only after it has been retrieved. Server-side encryption thus protects your data from access by a third party storage solution.

Note that encryption in the browser is essentially server-side encryption: the code that does the work comes from the server and thus the server controls what is going on. If a server is compromised, the attacker (or evil system administrator or government agency) can simply make a minor modification to that code so it gives them access to your private key!

When the data is on the server itself, however, the server admin has access to it as the keys are managed by the server. In case of Nextcloud, we encrypt that key with your login, making it impossible for the server to decrypt the files unless you are logged. That is called "protection of data at rest", but, of course, you log in all the time if you have the client running.

So the limitation of server-side encryption lies in what it does not protect you from: the server being compromised.

Client-side encryption

This is why server-side encryption is often contrasted with client-side encryption, which is what is employed by the end-to-end encryption Nextcloud introduced today. Where server-side encryption happens after transmission to the server, we encrypt the data on the Android, iOS or desktop client already. Then, only at the receiving end, it is decrypted again. That receiving end can be another device owned by the same user or a device owned by another user who has been given access to the data. But not anyone in between or otherwise not authorized! This is sometimes also called a zero-knowledge privacy: at no point in time can the server have any knowledge of the data.

The process of creating keys

A little bit about keys

Usually, public and private keys are used for the encryption and decryption. How does that work?

Everybody has a set of closely related keys, a 'public' and a 'private' key. They work a bit like a mail box works: anyone can throw a letter in the box, but only the mailman (or woman) can open and empty it. The public key is, as the name implies, public. Anyone can use it to encrypt something. But to decrypt the result, the private key is needed!

Nextcloud generates a public and private key pair the first time a user enables end-to-end encryption in their client. The public key gets signed by the server with 'certificate' to verify the user identity (our Cryptographic Identity Protection feature) and stored there for other users to encrypt files to that they wish to share. The private key gets encrypted with a locally, app-generated very secure and very long (12 word!) passcode which is displayed to the user and then the key gets stored on the server as well.

Another device owned by the user can download the private key, the user can enter the 12 word passcode to decrypt it, and this device will then also be able to encrypt and decrypt files. That is how you add your phone and desktop after you enabled end-to-end encryption on your laptop.

You can learn much more about exactly how the encryption and decryption work on the end-to-end encryption web page we made as well as the whitepaper you can download there.

Enterprise challenges

While consumers use end-to-end encryption for chat for years, larger organizations like companies and governments have struggled to find solutions that protect their sensitive data without disrupting productivity and legal requirements for audit logs.

Indeed, many solutions create a new layer over existing solutions like Box and Dropbox or have poor key management, making sharing cumbersome and less secure. A real enterprise solution needs to take the burden of complexity away from users and, if it needs to be anywhere, put it on the shoulders of the system administrators. Nextcloud has done exactly that, not even allowing users to create their own, potentially insecure password and making adding devices easy. The 12-word passcode can be recovered from any of the devices an user owns and it is possible to enable a system wide recovery key. If enabled, users will get warned of this and the system administrator gets to see and note down the key once and only once. After that, the code is destroyed and no attacker would be able to steal it, provided the server administrator puts it in a safe location: we'd recommend an actual safe.

Another important feature for enterprises is support for a Hardware Security Module which could be used to generate user certificates. Without it, our design does not allow users to switch identities (as this could be abused by an attacker with control over the server) but this restriction can be relaxed if a secure HSM is in the mix.

You can learn more on our webpage about end-to-end encryption!

Happy Birthday ownCloud

Seven years ago at Camp KDE in San Diego, Frank announced a project to help people protect their privacy, building an alternative to Dropbox: ownCloud.

I was there, sharing a room with Frank at the infamous Banana Bungalow. Epic times, I can tell you that - there was lots of rum, lots of rain and loads of good conversations and making new friends.





Since then, a lot has changed. But the people who started building a self-hosted, privacy protecting alternative in 2010 and 2011 are still on it! In 2011, a first meetup was held, and the 5 participants at that meetup recently got on stage at the Nextcloud conference to recall some good memories:



Of course, today we continue the work at Nextcloud, that just yesterday published its latest bugfix- and security update. It is great to see so many people have stuck with us for all these years - just this month, the KDE sysadmins migrated their ownCloud instance to Nextcloud!

We'll keep up the good work and you're welcome to join, either if you're looking for a job or just want to code. In both cases I can promise you: working with such a motivated, dedicated, professional team is just plain amazing.

I also published a blog on our Nextcloud blog about this milestone.

EDIT: By the way - there's a meetup tonight in C-Base, B'lin, 19:00 - would be fun to drink a beer on ownCloud's birthday and talk about the future! Join! It will be at least until 10 or so, so if you can't be there before then - still come! ;-)

Brittain’s Snoopers charter threatens your privacy

pic from the ZDNet article

The United Kingdom this week passed the so called Snoopers Charter, a law which forces UK internet providers to store the browsing history of UK citizens for a full year. You, your family, visitors or any devices in your household which have been hacked (the government is now allowed to do that, by the way) better not visit anything bad as the government can get their hands on this data quite easily. What does this mean and what can you do?

An attack on privacy

There is a global siege on privacy. Governments all over the world have introduced legislation (sometimes secret) which forces email, internet or data storage providers to track what you do and make that data available to their governments. This, of course, also means third parties who gain access to the storage systems can see and abuse it. And because so many of us have put so much of our data at just a few providers, we're at great risk as events like last week's shutdown of hundreds of Google accounts did show.

While Google, Dropbox and others lure customers in with 'free' data storage and great online services, governments benefit from centralized data storages as it makes it easy for them to hack in or demand data from these companies.

Why this surveillance?

While governments usually claim they need access to this data to find terrorists or child pornography, experts point out that it will not be helpful at all. As multiple experts (even internally) put it, growing the haystack makes it harder to find the needle. Intelligence agencies are swamped with data and nearly every terrorist attack in western states over the last decade took place despite the agencies having all information they would have needed to prevent it. The Paris attackers, for example, coordinated their attack using plain SMS messages. The Guardian thus rightly points out that:

"Paris is being used to justify agendas that had nothing to do with the attack"

which has become a familiar refrain after nearly every terrorist attack.

Indeed, we all know the argument But you have nothing to hide, do you? and indeed, we probably don't. But some people do, so they'll try to avoid being seen. That being illegal won't change their behavior...

And as Phill Zimmermann, the inventor of the PGP encryption pointed out:

"When privacy is outlawed, only outlaws will have privacy"

So not terrorists. Then what?

Experts agree that the vast majority of these surveillance and anti-privacy laws have little or no effect on real criminals. The crime syndicates, corrupt politicians and large corporations evading taxes and anti-trust/health/environmental laws, they DO have something to hide, and thus they would use encryption or avoid surveilled communication methods even if it were outlawed.

However, ordinary citizens, including grass-roots local activists, charitable organizations, journalists and others, who DO have nothing to hide, would be surveilled closely. And with that information, the real criminals mentioned earlier - crime syndicates, corporations or corrupt politicians - would have weapons in hand to keep these citizens from bothering them. Whistle blowers can be found out and killed (like in Mexico), journalists can be harassed and charged for trivial transgressions (like was recently done at the US pipeline protest) and charities can be extorted.

What can we do?

Luckily, there are initiatives like the Stanford Law Schools' Crypto Policy Project which aim to train, for example, journalists in the use of encryption. Tools and initiatives like Signal, PGP email encryption, Let's Encrypt and Nextcloud provide the ability for users to protect themselves and their loved ones from surveillance. More importantly, these at the same time making it harder and more costly to conduct mass surveillance.

There is nothing wrong with governments targeting criminals with surveillance but just vacuuming up all data of all citizens that might, some day, be used is a massive risk for our democracy. We all have a responsibility to decentralize and use tools to protect our privacy so those who need it (press, activists and others) have a place to hide.

Get started with Nextcloud App development in 6 easy steps!

The brand new app scaffolding tool in our app store

Last night, Bernhard Posselt finished the app scaffold tool in the app store, making it easy to get up and running with app development. I was asked on twitter to blog about setting up a development environment, so... here goes.

What's simpler than downloading a zip file, extracting it and running a command in the resulting folder to get an Nextcloud server up on localhost for hacking?

Yes, it can be that simple, though it might require a few minor tweaks and you have to make sure to have all Nextcloud dependencies installed.

Note that this is useful if you want to develop an Nextcloud app. If you want to develop on the Nextcloud core, a git checkout is the way to go and you'll need some extra steps to get the dependencies in place, get started here. Feedback on this process is highly appreciated, especially if it comes with a pull request for our documentation of course ;-)

Step 1 and Two: Dependencies

• Install PHP and the modules mentioned here
  Your distro should make the installation easy. Try these:
  
  • openSUSE: zypper in php5 php5-ctype php5-curl php5-dom php5-fileinfo php5-gd php5-iconv php5-json php5-ldap php5-mbstring php5-openssl php5-pdo php5-pear php5-posix php5-sqlite php5-tokenizer php5-xmlreader php5-xmlwriter php5-zip php5-zlib
  • Debian: apt-get install php5 php5-json php5-gd php5-sqlite curl libcurl3 libcurl3-dev php5-curl php5-common php-xml-parser php5-ldap bzip2
• Make Nextcloud session management work under your own user account.
  Either change the path of php session files or chmod 777 the folder they are in, usually something like /var/lib/php (debian/SUSE) or /var/lib/php/session (Red Hat).

The Final Four Steps

• Download the latest Nextcloud zip file.
• Extract it in a folder where you want to get going.
• Enter the folder and execute: php -S localhost:5000
• Visit http://localhost:5000 in your browser

Nextcloud should present you with its installation steps! Give your username and password and you're up and running with SQLite.

Start with the app

Now you create a subfolder in the nextcloud/apps with the name of your app and put in a skeleton. You can generate an app skeleton really easy: use the scaffolding tool, part of our new app store for Nextcloud 11!

It's probably wise to now get going with the app development tutorial here. This isn't updated for the scaffolding tool yet, so you'll have a head start here. Be sure to check out the changelog, we try to make sure the latest changes are noted there so even if we didn't manage to fully update the tutorial, you can find out what will and won't work in the changelog. Also, be sure to update the links to get the latest dev doc - this all links to 11, once that is out it is probably better to directly target 12 and so on.

Help and feedback

Your input is very much welcome! If you run through these steps and get stuck somewhere, let me know and I'll update the documentation. Or, of course better still, do a pull request on the documentation right in github. You don't even have to do a full checkout, smaller fixes can easily be done in the web interface on github.

Last but not least, ask questions on our forums in the app dev channel or on IRC. Here is the Nextloud development IRC chat channel on freenode.net, also accessible via webchat.

Thanks, good luck, and have fun building Nextcloud apps!

Western Digital Labs and ownCloud

We published a blog post about a collaboration between WDLabs and ownCloud. WDLabs is, in their words, "a division of Western Digital focused on accelerating new solutions". I'd take that as "they look for more ways to sell hard drives" and they have been playing with some interesting stuff. One such thing is their PiDrive Kit. This kit includes a connector cable which powers both the harddrive and the Pi from one power supply, simplifying the setup. We're now working with WD Labs to build something from this starting point.

WD Labs, meet ownCloud

When they approached us a few weeks ago, they told us they had been toying with a more complete kit with bundled operating system based on berry boot. This lets you choose an OS on first boot and they had put together an image with ownCloud as one of the options. They thought it was really cool and asked if we felt the same and had any ideas of what we could do with it. Well, sure, I wrote not too long ago about What's holding ownCloud back and being able to offer potential users a plug-and-play device with ownCloud on it is something we'd love.

So did they. We quickly received a prototype of a kit with Berryboot and ownCloud on it as well as a case and a Pi included. After some more discussions, we developed a plan with which we hope to get some community help to make this happen.

We need your help

You have to know - this is not something ownCloud, Inc. is very much involved with. It simply is not a commercial endeavor - Inc. works for enterprise customers and we haven't found any of them yet running ownCloud on a Pi. So, while WD Labs will take care of the hardware and has committed themselves to doing a serious number of ownCloud branded devices complete with pre installed OS in the first quarter of next year, it will depend on our free time and community help to develop the OS.

And there's serious work to be done. Berryboot can be configured to boot a headless ownCloud but right now, to even boot, you have to connect a keyboard and monitor. And booting isn't even the issue - finding and configuring an IP and then connecting to the Pi is. And once you've done that, the system has to be configured to be accessible from outside so you can use the ownCloud clients on your mobile phone and laptop to always find and connect to your ownCloud and give you the ability to share with friends and family.

There are folks in our community who have experience with this, building our community VM for example (hi Daniel!) and we'd like to solicit their help. We have 10 prototypes to send out to people who want to help build this ready-to-go ownCloud, the blog has more details on how to get one. And if there is a lot of interest, we'll get more prototypes put together, though I think it'll be tight to get them out before Christmas.

Note that it is totally cool, even advantageous, to send in a proposal ad a team. Together you can get more done!!!

Of course - without this kit there is still a LOT you can do to help out and there is no reason to wait for us sending out anything! Help get a discussion started about what this should look like, how to get there and who does what. Subscribe to the developer mailing list and get started!

Future

Now we know the Raspberry Pi 2 isn't the very best device to run ownCloud on, in terms of performance. It won't run with 100 users, indeed, though it isn't too bad for a few users at home. But, aside from the fact that there is a lot of room for tuning (PHP 7, for one, should help a lot, see some stats here), we're not married to the Raspberry with a single hard drive. It is a popular device and thus a great place to start but WD Labs has already shown us prototypes using a Rasberry Pi Compute device and 2 hard drives and we could work with devices like the Banana Pi or more powerful boards, too. That won't happen immediately, it simply depends on how well the first version turns out and the interest from users.

That means it matters a lot now: can we make this work? I'll sure put a portion of my free time in this as I'm quite excited about it. You too?

Home Automation, AI and the Crownstone

Waiting for that future where you're dressed automatically? I'd rather do that myself and don't have a Roomba yet but as tech marches on, we'll get more automation in our homes.

With more and more 1984, Brave New World and Farenheit 451 being brought to this century by our esteemed leadership and our corporate overlords, I'm hoping that this tech will be ours, rather than theirs.

Enter a nice player in the "automate stuff" arena: Dobots. In their own words:

Our goal: Really smart buildings. Really smart robots. We're a startup in AiTech with applications in CleanTech and GreenTech.

What makes them nice is that they are real Open Source folks, putting what they code on github under a free license and working with and building on open tech.

They previously did industrial cleaning robots and other stuff. Currently, they are running a kickstarter for a smart power outlet as a first foray into home automation. EDIT: they've instead decided to, for the first run, only create EU plugs as there was little interest in the US. You can order them on the crownstone.rocks website.

Enter the Crownstone

How is AI relevant for power outlets? More than you'd think. One example is SLAM. SLAM (Simultaneous Localization And Mapping) tries to answer two key questions in Robotics:

• Robot Localization: “Where is the Robot?”
• Robot Mapping: “What does the world around the Robot look like?”

(read more about SLAM here)

Dobots has turned this around in the crownstone, using these algorithms to map your house and localize people in it. Yes, the Crownstones know where THEY are and where YOU are. While not exactly a localization tech, the Crownstone uses Bluetooth LE for this so it works with pretty much any device. And the best part: their localization code is, in Javascript, to be found on github!

This means you need no motion detectors or cameras to know where people are in the house, keeping things simpler and cheaper. But Bluetooth LE is used for managing the Crownstones, too. Rather than turning to WiFi or (expensive) central control hubs, Dobots lets the Crownstones automatically create a mesh network. Your smartphone acts as control hub (apps on github, of course) though you can also take a Raspberry Pi, plug in a bluetooth usb module and have it act as hub. That way, even when you're not at home, you can control devices or get notificationsthat, say, your television, computer or other devices are getting unplugged. It uses XMPP and WebRTC to get through your firewall and guess where you can find the Cordova based Raspberry Pi app...



But there's another innovation in there which helps avoid extra devices and complicated configuration: extremely fine grained power usage monitoring. Yeah, it can tell you how much power your fridge and TV use but more importantly: it can automatically detect which device you connect to it! So when you leave the house, it can kill power to your tv but not your fridge... I've been told they think they eventually will be able to figure out different makes and models of laptops based on their power signature. Now is that cool, or what?

The algorithms to do this are currently Matlab code but once finalized, they'll be on github, too.

Dobots regularly works with students who get to play with their tech and come up with new, innovative use cases. And this works wonders. Just some results from a recent hackathon:

• Team 1 developed ChildLock. Devices are only turned on when adults are in proximity.
• Team 3 developed Start VR. Virtual reality that allows you to picture your own furniture in an Ikea store.
• Team 4 introduced Never Lose. Lights indicate for elderly people where they have lost items with iBeacons.
• Team 5 used Crownstones to indicate the way to store employees.
• Team 6 worked on Tommy. An AI that analyzes patterns of daily life to combat loneliness.
• Team 7 developed Any Morning. Your phone using the Crownstones guides you to your morning routines to make you leave your home on time.
• Team 10 implemented Tipspromenad. Kids have to find objects in a place combined with solving puzzles for fun!
• Team 11 developed SpotOn. In emergency situations lights indicate how to flee a building

All together

The Crownstone has some nice advantages over the competition:

• Functional. Dimmer, iBeacon, a current measuring device, a standby killer and more in one.
• Cheap: you get 2 ready-made or 3 do-it-yourself for Eur 75 but there's no need for a hub, motion sensors or other stuff.
• Open. Using open protocols and code on github and I expect interesting applications to come from the community.

I did an interview with their COO - for more background, read it on LinuxVeda.

So I say - if you are looking to start with some basic home automation, go and get some Crownstones and get started!

The Washington Post again demanded that tech companies create special 'golden keys' for authorities to keep and use for access to private communication. Protected by a warrant, of course. For the benefit of this discussion (which is really getting old), I just put together the reasons why it is a dumb idea.

First of all. It is a pure fantasy, an entirely unrealistic wish of the Pink Unicorn variety that it is possible to create a key which only the US goverment (and other sanctioned agencies) would have access to. It is technically not possible. Ever. I explained that before so let me now just quote Bruce Schneier:

"We have one infrastructure. We can't choose a world where the US gets to spy and the Chinese don't. We get to choose a world where everyone can spy, or a world where no one can spy. We can be secure from everyone, or vulnerable to anyone. And I'm tired of us choosing surveillance over security."

And let's be clear - we've been over this, the Clinton government wanted a similar thing with the Clipper chip and as security researcher Matthew Green pointed out:

Clipper is only one of several examples of 'government access' mechanisms that failed and blew back on us catastrophically.

A second issue with the proposal is that it doesn't do anything. Just like all the spying programs that came before in this and previous decade. Here's Bruce talking about that, here the Guardian, the Newyorker, Wired and Washingtonsblog. Whatever these spy programs do - from spying on German Chancellor Merkel to US congress (that's the Washington Post itself!) to the United Nations and Unicef - the government spying programs certainly don't target or are helpful against terrorism or pedophilia or any of the other stuff they are claimed to be for. And neither will these 'golden keys' be used to catch terrorists.

Last, and this should already be blindingly clear if you see the list above of some of the targets of surveillance, you should doubt if the government agencies will abide by the rules - they haven't in the past.

I also want to point out that the very reason we're having this conversation in the first place is because we're idiots.

Dealing with our flaws in thinking

This is a follow-up on a post about the limited human rationality. In that post I described some facts - just a few - that perhaps left you a little more in doubt about your cognitive abilities. Or at least more aware of the limitations our human condition comes with!

Consequences

Unfortunately, the mentioned and the many other flaws in our thinking have consequences for decision making in our society, especially when there's money to be made. The lobby of weapon manufacturing is rather stronger than that of companies creating anti-slip mats in showers and car manufacturers, well, security is merely a factor increasing the costs of cars so there's little incentive for them to hammer on that issue either. The combination of our innate inability to judge the likelihood of these and other things to harm us and the financial pressure on politicians results in massive over-spending on what is in essence irrelevant or even dangerous and harming our society. The NSA, for one, stupidity around net neutrality is another and the war on drugs is third rather prominent example. And now Ebola, of course - a disease so unlikely to kill you, you're probably more likely to be killed by a falling piano.

I think it is pretty clear, as I mentioned above, that politics and business happily abuse our lack of rationality. But probably more often, 'the system' causes issues by itself, as the insanely huge political divide in the US shows. It pays of for the media to put extreme people in front of their audience - and today, we have a country where you can't discuss politics at the office because people see the world so vastly different that only conflict can come out of a conversation. Think of the biases I discussed earlier: these world views aren't likely to get fixed easily, either.

Never attribute to malice that which is adequately explained by stupidity.

I don't think anybody set out to create this divide - but it is with us now.

Now indeed, the media are part of a system working against us. They get rewarded for creating an impression of problems; and they are often uninformed and biased themselves. As John Oliver pointed out, we don't put a believer in regular human abductions by aliens in front of a camera to debate a scientist, attempting to give both sides of the debate an equal chance! We pity the few who still don't get that this, and many other issues, are settled.


Yet this is what often happens in areas where science has long come to a conclusion. Not just the moon landing but also vaccinations, global warming, evolution and a great many more things. Take the recent "Snowden wants to come home" media frenzy!

I don't think any of that is intentional. It's the system rewarding the wrong things. We are part of that 'system': we prefer news that supports our view point; and we prefer new and exciting things - a balanced point of view is boring.

Quality decision making gets harder and harder.

Dealing

One way of dealing with disagreement has been to essentially declare all facts 'up for discussion'. It all depends on your point of view, proponents of this idea say. But reality isn't as malleable as relativists make it out to be. You can choose to leave your house through the front window on the 3rd floor, but gravity's a bitch. It's nice that some want to value everybody's opinion, but the universe imposes limits to that.

We have to realize that the world is real. People can be right or wrong about it and the choices we make matter!

As a society, we need to find new ways to make decisions in a healthy way. We've done good things - we eliminated polio and smallpox, diseases that were around for many thousands of years and nobody has had them in a long, long time. River blindness is hopefully next, and others will follow. We also drove half the worlds' animals near extinction and are abusing this planet to the point where it just will become a much more hostile place in a century or two unless we change something. You can guess I'm not much into libertarianism - it is clear that we can and do impact the world and going it all alone does not solve the tragedy of the commons style issues we have. There's a problem - and the inherent complexity of the world is certainly part of it, as is our lack of rationality.

How do we deal? I used to be an optimist - when I discovered the Internet, I thought it would democratize knowledge (it has) and news (not so much). Social media, sites like Digg where people vote on what the 'best news' is, it seemed a new and improved reality. No more single points of failure. No journalists who can be bought or oppressed. And then there were open source communities, with their flat structure of decision making, ideals of equality and meritocracy. Democracy would thrive!

Reality was harsh. The Internet has allowed us to hide in our corners with like minded people. It has lots of good stuff (if you're not into economy or net neutrality, this is a good read on both) but the Internet didn't kill conspiracy theories, it fuelled them. And open source works for some, but has its own issues of inequality (and that is just one problem).

Perhaps technology can help - Google has apparently found ways to find out what's true and what isn't. I'm not so sure. I wonder what it would do to sites like this proving that even with mere facts you can create conspiracy nonsense.

Methods to the rescue

I think the solution has to be found in a system or a process in the way the scientific method works. Wikipedia describes it as follows:

The scientific method is a body of techniques for investigating phenomena, acquiring new knowledge, or correcting and integrating previous knowledge.

I prefer to call it a process, rather than a 'body of techniques'. The key is that if left to common sense, humanity decides that life emerges from lifeless matter until more than 2000 years later Louis Pasteur shows it really, really doesn't (except for this). The scientific method thus aims to take human decision making out of the equation, or at least, rigorously deal with the biases that cloud our judgment. Lots of books have been written on the subject of philosophy of science - I got my portion by way of Chalmers, worth reading.

However it works, key is that while science relies on people and thus makes mistakes, it has a process for dealing with these mistakes, correcting them over time. Confidence is gained over long periods and the result is that we have been largely refining our knowledge gained since the scientific method became widely used, rather than rewriting the world as we know it over and over again. Yes, Newton's theories on physics still stand - quantum mechanics and Einsteinian physics merely refines it, providing better results in areas Newton can't reach. Uncertainty exists in science, but only at the 'edges', where new knowledge is created. While many facts of evolution are debated, since evolutionary synthesis, we've settled on a core which is as solid as Newton's ideas about gravity; climate models might be imperfect today but much of what we discovered does not have to be debated over again and again.

Method for decision making

We have methods, systems, processes for decision making, too. Democracy is one, the trias politica part of it. But it has flaws and needs some refinement, ideally in the opposite direction of Citizens United. I don't think we can make a Philosopher King system work, so whatever we come up with has to be a bureaucracy, evolved from today's system. I think decentralization is part of the solution (majors should rule the world?), but we live together on this planet so there have to be over-arching structures, too.

I know there is research being done on the topic. And we've already come up with some strategies like the advocate for the devil.

What exactly the solution should look like - don't ask me. I'm a psychologist, I can merely tell you not to trust people and their gut instincts. If this feels like an anti-climax, well, it should. We will have to come up with solutions together - not one blogger alone!

Soon?

But we should hurry.

I believe, with self-described plutocrat Nick Hanauer, that the pitchforks are coming. Perhaps the militarization of police and governments (the NSA in particular) disrupting online security are attempts of governments to prepare for social unrest.

Maybe.

What I'm certain about is that humanity can't continue the way it is functioning now. If social inequality doesn't put a stop it, the depletion of natural resources or religious fundamentalists will. The Dutch would say: the wall will turn the ship.

Let's see how hard we'll hit it.

Why can the NSA do what it does?

This is part one of two blogs about how we make decisions and how our lack of rationality results in much of the mess we have today. I'll start with addressing the title of the blog - spying.

Much has been said about NSA by very qualified people like Bruce Schneier, comparing the NSA to

the Maginot Line (..) in the years before World War II: ineffective and wasteful.

The costs in terms of civil liberties and money resulted in one confirmed case where somebody was caught thanks to NSA spying (probably unfairly, though).

Like most technical people, I'm not impressed but very worried about the erosion of our civil rights, through the NSA spying and in other ways. And I am sure I share with others the impression that if only politicians and the general public knew more about the problem, we wouldn't make such bad decisions.

At the same time, I know I'm probably wrong about that. Like most people, I also care about global warming; health care; poverty; war; and the countless other things arguably Wrong With The World. And collectively, we know all there is to know about them.

Somebody, or a small cabal, must be causing this, then - an argument you often hear about many things gone wrong.

Is there a cabal? Let me invoke Hanlon's razor:

Never attribute to malice that which is adequately explained by stupidity.

Because I think it's the human condition that got us here, not malice of anybody in particular. We just, collectively AND individually, fail at making the right decisions.

So the question should be: what makes us so unreliable? So easy to lie to, especially in groups? Why do we believe conspiracy theories that often require us to believe far more fantastic things than the reality they try to disprove? How can fans kill people only in South Korea?

I'd like to dig into that a bit in this post, more of an essay than a blog, I suppose. The immediate reason is the mess surrounding the NSA (probably not news to most readers of my blog), where one-liners and the inherent complexity of the issue have ensured most people I talk to don't see the problem.

Reality is that the more you know, the harder it is to have a firm opinion. In reality, often conflicts are like Israel vs Palestine - if you pick a side, you're wrong. The complexity of real life issues makes it easy for governments and companies to play people - I feel an urge to point to Russia, but how do we know they are not right claiming Israel-backed Neo-Nazis used US supplied weapons to shoot down Flight 17? You can point out that historically, (neo)Nazis and Jews haven't gotten along very well. That's a fact. But so is the support of the EU for 'political reform' (overthrowing a legitimate, democratically elected government) in Ukraine. How valuable are facts and reason in a complicated situation?

I don't want to talk about the facts of any specific situation here. I'm not an expert in pretty much every relevant domain and neither are all of you! But we do have decision power - that is how democracy works. Luckily, with the Internet today, it is possible to have the most important facts around. And unfortunately, it does not lead to better decisions. So I want to look at it from my background in psychology and talk about how we deal with knowledge and how we make decisions.

Most of the time, we make decisions using 'common sense'. Our gut feeling, which was useful when we were still in Africa. But what works when you're hunting antelope might just not do the trick when you have to decide for an insurance company.

Why we don't get it

The problem lies in cognitive biases, described by Wikipedia as a "pattern of deviation in judgment". That's a nice understatement if you ask me.

A cognitive bias is a remainder of our Africa times, setting us up for certain errors. You'd be surprised how much of our thinking is in our genes. It is why most people are afraid of snakes and spiders: somehow, our genes have programmed our brain to quickly learn to fear snake and spider like objects, but be totally fine with bunnies and flowers.

To understand how limited our ability for logical thinking is, let's remind ourselves of Optical Illusions. I'm sure you've seen many by now and you should realize how flawed our vision is. Yet we have evolved to 'see stuff' for millions of years. We have a big part of our brain dedicated to it. And we use our eyes all day long, every day of our life. Yet, these images keep confounding us. To top it all off, there is change blindness, showing it isn't just bad - we are terrible.

Logic, on the other hand, is quite new. While some brilliant people were inventing math and logic thousands of years ago, most humans kept busy hunting, later growing food. And most of the time, in day to day life, we work on the automatic pilot. There's a reason we learn to recognize objects with little effort as babies already but have to be taught the mere basics of math during excruciatingly long sessions at school! Humans are naturally absolutely horrible at math and logic, to the point where somebody with some reasonable knowledge of probability could have won half the riches of the ancient world by gambling - probability theory was only developed in the 17th century.

And if you think you did well in those 16 years of school, answer this:

A bat and ball together cost $1.10. 
The bat costs $1.00 more than the ball.
How much does the ball cost?

See the answer two chapters further, just before the end.

It shouldn't be a surprise to learn that the list of cognitive biases is as long as it is. Most of the time, instead of 'proper' logic thinking, we use heuristics, shortcuts to 'good enough' solutions. They work great - if you're hunting antelopes.

To state the obvious: we're not. To go back to the NSA - we're letting them go wild. And: global warming, poverty, war... We're not doing that well, as Douglas Adams pointed out:

“This planet has - or rather had - a problem, which was this: most of the people living on it were unhappy for pretty much of the time. Many solutions were suggested for this problem, but most of these were largely concerned with the movement of small green pieces of paper, which was odd because on the whole it wasn't the small green pieces of paper that were unhappy.”

Some biases explained

You might have clicked some of the links above and seen some examples already, but lets bring this to life with a familiar example: why are so many people afraid of flying?

Let's turn the question around. SHOULD you be afraid of flying? There's a risk, certainly. But it is a well known fact (I won't even link to anything) that the chances of an accident when traveling by plane are larger on the way to and from the airport than while in the air! So why do many people still feel flying is dangerous? It is due to the way our brain does statistics. It's nothing like a calculator...

Our brains have ways of estimating how likely something is. They do that by digging in our memory: the easier it is to recall multiple instances of something, the more likely it is to happen. That makes sense - if you found food three times when walking a certain path, there is a high likelyhood of you finding more there rather than on a path where you never found anything to eat. It's clear where you should go if you are real hungry.

But our memory isn't terribly precise. First of all, it stores information in relation to emotion. This means when you're sad you can remember sad things better and when you're happy, happy events come to mind easier. It also stores things better when they are associated with strong emotions. This makes sense: events which upset, anger or scare you are probably more important than those which don't elicit any particular emotional response. You'll remember less details from your home-work travel from Tuesday a week ago then that crazy ride in a roller coaster, even though the latter didn't even get you to anywhere special.

This effect causes us to make massive errors in estimating things like the likelihood of getting a car accident, slipping in the shower or getting killed by a terrorist. And flying, of course - plane accidents feature prominently in the press and cause a lot of anxiety. But let me remind you:

Never attribute to malice that which is adequately explained by stupidity.

These accidents are worth talking about - don't blame the media for reporting the news. It's just that we fail at judging how likely it is that it happens to us.

And this is just one flaw related to just one aspect of how we think - statistics. We're not too good when it comes to statistics and causality either, as XKCD points out below.



Our failure to handle statistics can be abused quite easily, and not just in gambling or other situations where statistics obviously play a role. In marriage counseling, a therapist might ask you to try and come up with five good things about your partner. This wouldn't be too hard and put you in a more positive mood. If the therapist was evil, however, he/she could ask for ten good things instead. While you might get to ten, a number that should give more confidence than a mere five, paradoxically the difficulty in coming up with such properties is not going to be good for the relationship.

And there's (again) no malice needed. Stereotypes, for example, are known to cause behavioral changes without awareness of the subject ("you do something but don't know why you are doing it"). In an interesting experiment, it was shown that if you let a group of white people and people of color complete a standardized test, they can either do equally well, or not. It depends on what you tell them it is a test of "diagnostic of intellectual ability" or not!


That's right: when they know it is an IQ test, African-Americans score significantly worse than white people, while they actually do a little better when the same test is done without any mention of its diagnostic abilities. They let the stereotype come true, unintentional and unconscious.

An awesome example of how prejudices influence us on a fundamental level can be found in this great video of a talk by Mahzarin Banaji, who shows to an audience of liberal, well educated scientists the gender bias in their brains. Watch the section with the test below and while she does the test, participate! Speak out the 'left' and 'right' so you can experience the effect for yourself.


If you participate, you will notice that you easily do the first three tasks - but the fourth is inexplicably harder. This shows the fundamental bias in your brain: you take twice as long to answer a question that isn't congruent with your unconscious gender prejudice! Professor Mahzarin notes that she, herself, exhibits this same bias - about 75% of men and women do. Racism exists even in people with the best of intentions.

Nobody can be blamed for this - it is how we, humans, function, just another one of our many flaws. Stereotypes are central to how we function, influencing our behavior at every corner in our lives. They ARE the biases, the short cuts of thought, themselves!

What does this all add up to? For one, you can't judge how dangerous something is. That is why we spend lots of time and money to reduce risks that are tiny, see also this and this.

Confirmation bias

Let's discuss one more bias, confirmation bias. This is one particularly nasty bugger:

Confirmation bias, also called myside bias, is the tendency to search for, interpret, or recall information in a way that confirms one's beliefs or hypotheses. It is a type of cognitive bias and a systematic error of inductive reasoning.

Remember, the inductive reasoning Wikipedia mentions here is the one based on probability: after you've seen over 50 expensive cars in a new place you're visiting, you might be inclined to think it is a rich city. Maybe. But remember what I said earlier: how does your brain assess probability? Memory recall.

Wikipedia mentions explicitly that your brain has a tendency to remember things that fit the theory you are trying to verify! That means that it will fail to bring to mind the slum around the airport you saw from the airplane. Yes, your brain will explicitly take away your ability to properly judge a situation.

It sure makes you feel more confident about your decision. But not any less wrong.

And that is just the memory side of this bias. Countless studies have shown we explicitly look for evidence that fits our expectation. A truly impressive example is in the video below:

A little background on this experiment is here and an opinion piece by the LA Times connects it to how white people demonize people of color in the aftermath of Fergusson. Of course, when it comes to minorities there are many biases in play, this is just one of them.

About the Bat and the Ball. The bat was a dollar more expensive than the ball, and no, the ball isn't 10 cents with the bat being 1 dollar. That would make the bat only 90 cent more expensive. 5 ct and 1 dollar and 5 cent. Of course, if you did the math, you wouldn't get it wrong. But if you think quickly - you are. Because you're then using a shortcut. Sadly, you do that most of the time, and not just when math is involved.

The human intellect is particularly good at over-estimating itself. We even consider unrealistic positivism about oneself a healthy attitude - people who are more realistic are clinically depressed. There's a sad list of self deceptions here. And if you think that high intelligence means less cognitive biases - think again. It might even be that the opposite is true!

In a way, this video gives a philosophical view on the subject, inspired by Plato's cave:


I hope that with some of the facts and thoughts above, I've made you a little bit more humble, dear reader. Because if we're to solve the problems we have, we need exactly that...

Read part two for some thoughts on the consequences and dealing with our limitations. Feel free to comment below!