23 September, 2020

Triggered

Somebody pointed me to a research article about how many app developers fail to comply with the GDPR and data requests in general.

The sender suggested that I could use it in marketing for Nextcloud.

I appreciate such help, obviously, and often such articles are interesting. This one - I read it for a while but honestly, while I think it is good this is researched and attention is paid for it, I neither find the results very surprising NOR that horrible.

What, a privacy advocate NOT deeply upset at bad privacy practices?

Sir, yes, sir. You see, while the letter of the law is important, I think that intentions also are extremely important. Let me explain.

Not all GDPR violations are made equal

If you or your small business develops an app or runs a website to sell a product and you simply and honestly try to do a decent job while being a decent person, the GDPR is a big burden. Yes, the GDPR is good, giving people important rights. But if you run a mailing list on your local pottery sales website, with no intention other than to inform your prospective customers and followers of what you're up to, it can be a burden to have people send you GDPR takedown and 'delete me' requests instead of just having them, you know - unsubscribe via the link under your newsletter!

The goal of the GDPR, and of my personal privacy concerns, isn't at all related to such a business. If anything, their additional hardship (and we at Nextcloud have this issue too) is at best a by product of the goal. That byproduct isn't all bad - we all make mistakes, and being more aware of privacy is good, even for small businesses. The GDPR has forced many small businesses to re-think how they deal with private data, and that isn't a bad thing at all. But it isn't the main benefit or goal of the GDPR in my eyes. There are big businesses who totally COULD do better but never bothered, and now the GDPR forces them to get their act together. While that's a real good thing, even THAT is not, in my opinion, what the GDPR is about.

Privacy violation as a business

You see, there are businesses who don't violate privacy of people by accident. Or even because it is convenient. There are businesses who do it as their core business model. You know who I'm talking about - Facebook, Google. To a lesser but still serious degree - Microsoft and yes, even Apple, though you can argue they are perhaps in the "side hustle" rather than "it's their primary revenue stream" category.

For these organizations, gathering your private data is their life blood. They exploit it in many ways - some sell it, which is in my opinion definitely among the most egregious 'options'. Others, like Google and Facebook, hoard but also aggressively protect your data - they don't want to leak it too much, they want to monetize it themselves! Of course, in the process of that, they often leak it anyway - think Cambridge Analytica - that was in no way an incident, hundreds of apps get your private data via Google, Facebook, Microsoft and others. But by and large, they want to keep that data to themselves so they can use it to offer services - targeted ads. Which in turn, of course, get abused sometimes too.

My issue with this business model, even without the outright sale of data, is two-fold.

Ads work better than you think

First, in principle - while people might feel ads don't effect them, there is a reason companies pay for them. They DO effect your behavior. Maybe not as much or in the way marketing departments think or hope, but the effect exists.

How bad is that? Depends, I guess. To some degree, it is of course entirely legitimate that companies have a way to present their product to people. But modern targeting does more, including allowing companies to charge specific people different prices, and of course a wide arrange of sometimes nasty psychological tricks is used. The example Facebook once gave to potential advertisers, of targeting an insecure youth "at their most vulnerable" with an ad is... rather disgusting.

This gets worse when we're not just talking about product ads but political ads, either from political countries or, of course, from foreign non-democratic adversaries trying to influence our freedoms in a rather direct and dangerous way. And again - this is more effective than most people realize or are willing to admit and has swayed elections already, making is all less free.

Centralization is bad

Second, there is simply a HUGE issue with all-our-eggs in one basket. Especial when that basket is in a foreign country and not protected by privacy and security laws compatible with those in your own country. Having a single point of failure, how well protected - is just not smart. Things WILL fail, always. Better have slightly more breaches that each are just a single provider, than one breach of all private data of everyone in a country...

And that's not even talking about the fact that this data helps these companies get incredibly huge and then allows them to suppress or kill competition (or just buy it) - think Amazon, Microsoft. These tech molochs are just plain bad because of many reasons. They are anti-competitive, which raises prices, decreases choice, and the much lower innovation-per-dollar they produce is of course a worse deal for society too. They are too easy to control by law enforcement and censorship, impacting our freedoms - even when they're not 'foreign' to you. Yes, it is harder to censor 50000 private servers than one Google server farm!

Triggered

 So, as you notice, this question triggered me. Not all privacy violations are equal. Intentions matter. As does market power. And the GDPR is not a golden bullet. It has downsides - compliance is often easier for big companies than small ones, a serious issue.

Luckily, our judicial system tends to look at the intentions behind law, and I would expect a judge to fine an organization heavier for truly bad business models than for honest mistakes. I hope I'm not too optimistic here.

From my side, I don't want to bang on people's head for mistakes. I want to attack and challenge bad business models and bad intentions. A local, small app maker who fails to respond quickly enough to GDPR requests - not my target. Facebook - yes.

And by the way. Maybe it doesn't need to be said to most of you, dear readers, but of course - our open source world is, I still believe, a huge part of solving this problem. KDE, openSUSE and other Linuxes and desktops - and of course Nextcloud, Mastodon, Matrix and other decentralized and distributed and self-hosted platforms. We have ways to go, but we're making progress!


As I concluded to the person who triggered me - I know, this is far too long a reply to what they said


But it triggered me ;-)


Best reply over twitter, (twitter.com/jospoortvliet) or so, this awful Google platform makes commenting nearly impossible. And I know, the irony, replying on twitter, and I still have not moved away from blogger.com... Some day, some day. When I find time.

08 June, 2020

Collabora vs ONLYOFFICE

Since the Nextcloud Hub release switched from ONLYOFFICE to Collabora Online as default, lots of people have asked why. Is one better than the other? Let's talk about this.

History

Let me first say - the decision wasn't pure technical. As always, relations and other reasons play a role. I'll try to cover both aspects, but there is always more. With that out of the way, let's first look at how ONLYOFFICE got into Nextcloud.

Frank, myself and others in the Nextcloud community have wanted to integrate office in our collaboration platform for most of the past decade. Previously, we* had invested quite a bit in getting a collaborative document editor into our private cloud. The Documents app was a from-the-ground-up developed ODF editor with a unique and very clever design, built by KO GmbH (now sadly defunct). We together put resources in integration and further development and we hoped other (open source) businesses would invest and contribute too, so the solution would grow in time. Also, we had hoped some customers would be willing to pay for it. Both of these did not really come true, and KO sadly didn't survive.

* Note that I use 'we' here loosely as I wasn't really involved back then, so think 'the core team', as a slowly-changing team of people, including Frank, Jan, Arthur and others.

Fast forward to our launch on June 2 2016 (happy birthday!), and a few months later we announced Collabora Online integration. We had worked with Collabora to make this available not just to enterprise customers, as before, but to all users thanks to the 'CODE' docker image. As you know, we care deeply about community/private home users and this was of course a great step forward.

But running docker, setting up a reverse proxy on a second domain with proper certificates - it isn't easy and does not work for everyone. So we had to keep maintaining the Documents app a little, as some users still could only use that.

ONLYOFFICE vs Collabora

Meanwhile, a new open source online office solution came around, ONLYOFFICE. Let's talk for a sec how they compare Collabora, as the two could not be more technically and non-technically different!

Technical: how they work

The way Collabora Online works is:
An embedded version of Libreoffice runs on the server. It reads the document, then 'streams' the rendered document as image tiles to the browser client, which shows it to the user. The browser client does some of the menu's and lots of smart things like showing the cursor, other users, text selection etc, but many other components like pop-up menu's and sidebars are also streamed from the back-end, giving relatively good feature parity with LibreOffice. This strategy is responsible for giving LibreOffice, for example, desktop-level table style editing, better than any other online office solution.

The way ONLYOFFICE works is:
The document is converted on the server to a JSON file which is streamed to the browser client. The browser client is the full office suite, editing the document. Once done, it sends back the JSON and the server merges and exports it back to a file. A fully html5 canvas based front-end means a relatively pretty user interface and any javascript dev can go hacking.

So what does this mean?

  • LibreOffice is much heavier on the server and network connection, but uses a bit less client resources which tends to help mobile devices with battery life during editing
  • You get the full Libreoffice file type support. Decades worth of obscure file formats, it is all there.
  • ONLYOFFICE has a more modern UI, writing it all in Javascript so it is far easier to be mobile-friendly. You can imagine how useless those old LibreOffice paragraph settings dialogs are on a mobile phone screen!
  • In theory ONLYOFFICE would be much easier to integrate in web apps in general. Most app frameworks can consume a javascript or json component, a simply streamed, tiled image is far less flexible...

Compatibility

On document support, three things.

First, with regard to the Microsoft file compatibility - this is ALWAYS hit and miss. I can't objectively claim either is better or worse, you will always find a file that works well in one but not the other. But you will also find lots of MS Office files that won't work in Office 365, or break the desktop version between Mac and Windows or even just from older versions, because Microsoft screwed up their own compatibility.

Second, one thing I can say: if you migrate from Collabora Online to ONLYOFFICE and most of your files are ODF files because that's what Collabora uses by default, you're in for a bad experience. The ODF support in ONLYOFFICE is quite basic. But with MS Office files they feel on-par to me and that's what probably matters for most people. (sadly, yes)

Third, if you need any other file types - Collabora can handle a LOT, due to its long legacy. Word Perfect anyone?

For other technical capabilities - I probably be best off simply pointing to the comparisons both made themselves:

Social/historical differences

Let's talk about the second big difference between Collabora and ONLYOFFICE: their roots. Collabora builds on and is part of the LibreOffice community, a decades-old project, and consists of long time open source believers. Development is open and accessible and there are lots of individuals and companies that work on and can provide services for its code base. ONLYOFFICE on the other hand, is quite new to open source and only a bit over a dozen people have contributed to the code base. Their open core model if of course less than favorite in the open source world, though it is still miles better than proprietary - some people seem to lose sight of that sometimes, if you ask me. For an end user, the development model makes little difference, in either case.

let me emphasize two things.
First, it is awesome that we have TWO open source office suits. Building one is an amazing accomplishment - we have had others in the past but most are no longer really viable due to the massive amount of resources required to keep up.
Second, I think it is great that ONLYOFFICE decided to open source their product. I believe most people really under estimate what it takes to turn around your business model so radically. And if you're unhappy with decisions made, in either case - contribute, get involved. That is how you change things in open source.

Getting Office in Nextcloud

So, as I said in the History section, by 2017 we had three office solutions integrated in Nextcloud. One was easy to install but unmaintained and quickly deteriorating. The other two were harder to install but much more complete.

You know we're ambitious people, so indeed we have thought about and discussed this situation forever. And at some point, Robin started to really investigate what would be possible. After looking deeply at both, he finally managed to create a proof of concept with ONLYOFFICE. What he did was:

1. Separate the 'converter' part from ONLYOFFICE, the javascript front-end and the 'rest'
2. Made a separate binary of the converter, package the javascript and rewrite all the glue that lets them interact in PHP
3. Make this thing install-able as one big blob, acting as alternative 'server' with a proxy component that ties it all together

This was a LOT of work, but after polishing it, we had something we could show to the ONLYOFFICE people. They were initially not huge fans of what we did - no surprise, as it was an ugly solution. We discussed this for a fair bit and in the end, we agreed on an approach.

The result was what we made available last January with the first release of Nextcloud hub. We saw it as a first step towards deeper integration. Watch the video below to get an idea of what it looked like!

📺 view video on YouTube

And then...

After release, two things happened.
First, ONLYOFFICE has sadly been unable to focus much on the integration with Nextcloud. There was a long wish list we had - there is a lot you can do to make the experience nicer, from removing/disabling/hiding duplicated features like the build in chat and file handling to making file collaboration work in other apps like Talk, or adding certain features that connect even deeper like @mentioning users for example. Unfortunately, this didn't happen. No blame, there is a lot happening in the world right now!
Second, Collabora was inspired by the work and while we didn't think we could make it install-able with such ease, they obviously know their own technology better. And indeed, they did make it happen! Besides that, we worked with them to improve the already pretty good integration further, allowing you to edit documents while in a video call or chat in Talk.

As our focus continues to be on providing the best experience possible, we simply looked at that: what gives, right now, the best experience. And thus our latest video shows Collabora instead...

📺 view video on YouTube

Note that this doesn't mean we don't like ONLYOFFICE. 😍 This just changed the default you get on installation. Both solutions are very good and continue to be available for users! And perhaps things will change for the next release. Given the large differences at every level between the two, I consider it a benefit to have both approaches available for Nextcloud users!

So is Collabora better?

I will let Captain Marvel answer that.

03 April, 2020

Rant of the day: well, at least Microsoft is making loads of money...

Sadly, many if not most of our schools today are suddenly pumping lots of extra money into Microsoft, Zoom and other proprietary software companies, because they need online collaboration. We all know there are many alternatives to giving their students' data away to foreign companies but most don't bother. It is annoying, there is always budget for Microsoft, but not for proper, local, privacy-protecting open source solutions, even if those are better. Why is that?

Reputation, I'm convinced, is the main reason for that.

We teach them the wrong thing

Unfortunately, a lot of people try to convince schools, governments, charitable organizations and even companies to not pay anything at all. They are promoting open source solutions as an alternative that is cheaper or free, which just makes it look inferior to management. They are not telling organizations to pay local and open source product companies instead of Microsoft.

Open source/Free Software advocates hammer on "but it is free"! And when they do, THEY probably think of Freedom. But the person they talk to just thinks "cheap and bad", no matter how you try to explain freedom. Nobody gets that, really, even if they nod friendly while thinking what a silly, idealistic nerd you are. Been there, done that.

I love the enthusiasm, yes, but in the end it is not helpful: it presents open source as a crappy but cheaper alternative without any real support. Well, there are a few overloaded volunteer enthusiasts who might do a great job for a volunteer but can't compete with a bunch of full time paid people at Microsoft. So the schools and governments and companies will simply use those 'free' (as in cheap and crappy) services as a stop-gap and then beg their bosses for budget to be able to pay a "proper" Microsoft service. There goes more public money in NOT public code.

We need to stop teaching companies that open source is a crappy, cheaper alternative to proper, paid alternatives from big American companies and instead tell them that they can pay for an open source solution that has real good support, no vendor lock-in, doesn't leak your data, protects your privacy and is actually better in many other ways. That way open source companies can actually hire people to make products better instead of just doing consulting one customer at a time.

And yes, some companies and some business areas have figured this out - Red Hat and SUSE are obvious examples, and projects like OpenStack have lots of paid people involved. But lots of other companies, from Bareos (backup) to Kolab (groupware) have struggled for years if not decades to build a product, instead getting sucked into consulting.

It doesn't work that way

I have seen loads of open source product companies go bankrupt or just give up and become consulting firms because their customers simply expected everything for free and to only pay a bit for consulting. Lots of open source people work at or set up their own consulting firms, occasionally even contributing a patch to upstream - but not building a product. Not that they don't want to, but they quickly find out that working your ass off for a maybe decent hourly rate does not leave you time to actually work on the thing you wanted to improve in the first place.

Indeed, you can't build a good end user product that way. Frank and myself put together a talk about this recently:



I have also recently written an article about this entire thing, explaining why of all the business models around open source, only subscriptions can lead to a sustainable business that actually builds a great product. Will hopefully soon be on opensource.com.

Yeah but volunteers...

Are fundamental to open source, yes, no doubt. At Nextcloud we could not have build what we did without lots of volunteers, heck, nearly everybody at Nextcloud was a volunteer at some point. And yes, all code we write is AGPL, and that, too is important. I am NOT arguing against that, not in the least.

What I say is:
  • You can't build a great product without paid developers*
  • You can't build a great product on consulting and only getting paid for setting it up/hosting
But let me then also add:
  • You can build a better product collaboratively
  • And the (A)GPL are the best licenses to do that

I'm sure there are exceptions to those rules, yes. But compare a great product like Krita, see how its developers struggle every day to be able to pay the bills of just a few full-time volunteers. Do you know how they are currently paying most of them? Last time I spoke to Boudewijn, the reality was sad: the Microsoft App store. Yup. How many does Adobe manage to pay to work on its products? Why should our ambition not be to have as many people working on Krita? Of course it should be. And yes, keep it open source. Is that doable?

Of course it is. Well, maybe not Adobe levels, but we can absolutely do better.

Missed opportunities

I said this was a rant, so I do have to complain a bit. My biggest regret is that KDE failed to catch up during the netbook period (around 2005). I believe that it is in no small part because we failed to work with businesses. Idealism can be super helpful and can also totally keep you irrelevant.

KDE is, lately, working more with companies, trying to build up more business around its product. GNOME has been far better at that for a far longer time, by the way. It is hard, and companies like Kolab, struggling for the last ~20 years to make things work, have shown that. Just being a for-profit obviously doesn't solve all problems. Idealism and hard work are not enough to make a business work. But we can do better, and Nextcloud is an example that shows we can. Now not all things are freaking awesome at Nextcloud, really - we work our a**** off and it is hard. We put on our best face in public but sometimes I just want to bang my head on and in the wall...

Still, see the video, read the blog hopefully soon on opensource.com - there are ways.

Thoughts welcome.

Edit:
* let me qualify that statement. You can do it without paid developers in a small project, I dunno, grep or ls or the awesome simplescreenrecorder and tools like that. With those there is a risk of the apps going unmaintained and new ones popping up all the time - look at music players in the KDE community. I'd rather see one well maintained than new ones pop up with all their different flaws, but I totally get that for a volunteer it is often easier and more fun to start fresh. In either case, once you start building something huge, it gets pretty hard without long term dedicated resources. Note that it can be donations-run (like Krita and many others), with a charitable organization. I do think it is about more than 'just' the resources. If somebody 'just' sponsored 25 people to work full-time on Nextcloud, the end result would be different than the situation today. The need to deliver something that makes customers happy (which means focus on details, scalability etc!) and pressure to do things you wouldn't want to do in your free time (developer documentation...) make a big difference.

In any case, I really don't think projects like LibreOffice, Firefox, Nextcloud, KDE or GNOME and the Linux kernel itself would be where they are today without people paid to work on them.

07 September, 2019

04 September, 2019

Principles

dafuq?
We recently did a post about the Nextcloud Mission and Principles we discussed at the previous Contributor Week. I guess it is mostly the easy-to-agree on stuff, so let me ruin the conversation a bit with the harder stuff. Warning: black and white don't exist beyond this point.

Open Source

In an internal conversation about some community pushback on something we did, I linked to islinuxaboutchoice.com - people often think that 'just' because a product is open source, it can't advertise to them, it has to be chock full of options, it has to be made by volunteers, it can't cost money and so on...

But if you want to build a successful product and change the world, you have to be different. You have to keep an eye on usability. You have to promote what you do - nobody sees the great work that isn't talked about. You have to try and build a business so you can pay people for their work and speed up development. Or at least make sure that people can build businesses around your project to push it forward.

I personally think this is a major difference between KDE and GNOME, with the former being far less friendly to 'business' and thus most entrepreneurial folks and the resources they bring go into GNOME. And I've had beers with people discussing SUSE's business and its relationship with openSUSE - just like Fedora folks must think about how they work with Red Hat, all the time. I think the openSUSE foundation is a good idea (I've pushed for it when I was community manager), but going forward I think the board should have a keen eye on how they can enable and support commercial efforts around openSUSE. In my humble opinion the KDE board has been far to little focused on that (I've ran for the board on this platform) and you also see the LibreOffice's Document Foundation having trouble in this area. To help the projects be successful, the boards on these organizations need to have people on them who understand business and its needs, just like they need to have community members who understand the needs of open source contributors.

But companies bring lots of complications to open source. When they compete (as in the LibreOffice ecosystem), when they advertise, when they push for changes in release cycles... Remember Mark Shuttleworth arguing KDE should adopt a 6-month release cycle? In hindsight, I think we should have!

Principles

So, going back to the list of Nextcloud's Mission and Principles, I say they are the easy stuff, because they are. They show we want to do the right thing, they show what our core motivation was behind starting this company: building a project that helps people regain control over their privacy. But, in day to day, I see myself focus almost exclusively on the needs of business. And you know what, businesses don't need privacy... That isn't why we do this.

Oh, I'm very proud we put in significant effort in home users when we can - our Simple Signup program has cost us a lot of effort and won't ever make us a dime. The Nextcloud Box was, similarly, purely associated with our goals, not a commercial project. Though you can argue both had marketing benefits - in the end, a bigger Nextcloud ecosystem helps us find customers.

I guess that's what keeps me motivated - customers help us improve Nextcloud, more Nextcloud users help us find more customers and so both benefit.

Pragmatism and the real hard questions

Personally, I'd add an item about 'pragmatism' to the list, though you can say it is inferred from our rather large ambitions. We want to make a difference, a real difference. That means you have to keep focused on the goal, put in the work and be pragmatic.

An example is the conversation about github. Would we prefer a more decentralized solution? Absolutely. Are we going to compromise our goals by moving away from the largest open source collaboration network to a platform which will result in less contributions? No.... As long as github isn't making our work actively harder, does not act unethically and its network provides the biggest benefits to our community by helping us reach our goals, we will stay...

More questions and the rabbit hole

Would you buy a list of email addresses to send them information about Nextcloud? No, because it harms those users' privacy and probably isn't even really legal. Would you work with a large network to reach its members, even if you don't like that network and its practices? Yes - that is why we're on Facebook and Twitter, even though we're not fans of either.

Let's make it even harder. How about the choice of who you sell to. Should we not sell to Company X even if that deal would allow us to hire 10 great developers on making Nextcloud better for the whole world and further our goals? Would you work with a company that builds rockets and bombs to earn money for Nextcloud development? We've decided 'nope' a few times already, we don't want that money. But what about their suppliers? And suppliers of suppliers? A company that makes screws might occasionally sell to Boeing which also makes money from army fighters... Hard choices, right?

And do you work with countries that are less than entirely awesome? Some would argue that would include Russia and China, others would say the USA should be on a black list, too... What about Brazil under its current president? The UK? You can't stop anyone from using an open source product anyway, of course... It gets political quick, we've decided to stick to EU export regulations but it's a tough set of questions. Mother Teresa took money from dictators. Should she have? No?

It might seem easy to say, in a very principled way, no to all the above questions, but then your project won't be successful. And your project wants to make the world better, does it not?

Conclusion?

We discuss these things internally and try to be both principled and pragmatic. That is difficult and I would absolutely appreciate thoughts, feedback, maybe links to how other organizations make these choices. Please, post them here, or in the comments section of the original blog. I can totally imagine you'd rather not comment here as this blog is hosted by blogger.com - yes, a Google company. For pragmatic reasons... I haven't had time to set up something else!

There's lots of grey areas in this, it isn't always easy, and sometimes you do something that makes a few people upset. As the Dutch say - **Waar gehakt wordt vallen spaanders**.



PS and if you, despite all the hard questions, still would want to work at a company that tries to make the world better, we're hiring! Personally, I need somebody in marketing to help me organize events like the Nextcloud Conference, design flyers and slide decks for sales and so on... Want to work with me? Shoot me an email!

25 June, 2018

Working at Nextcloud

I've been around in communities like KDE, openSUSE, Mandrake/Mandriva and others... and various open source and closed companies. Seen some do a good job. Seen others be mismanaged. This one: the most fun. Serious.

Working at Nextcloud is special. For one, we're a distributed company. Is it hard? Well, yes and no. Working from home is great with such a motivated team with very little management overhead and good communication. Our company is entirely built on it, that is why it works.

As an example, while our head of sales lives in Hannover, the rest of the sales people is spread over Berlin, Switzerland, Stuttgart... Engineers can be found in Germany, Netherlands, Spain, even Croatia and as far as Cape Verde. I'm sure I forgot some countries. Our biggest office in Stuttgart has less people than we have in Berlin!

But we connect in person: roughly every second month, at a company-wide meeting in a single place, usually Stuttgart, for a full week of coding and having a great time. And once a year we go to Berlin for our conference, happening the last week of August! All those meetings are open, with often lots of community members participating in the whole process of designing and deciding around our software.

And yes, the sales people join there, too. I have NEVER worked in a company where the sales people, the marketing team and the engineers were so good with each other. Respect between these three departments is extremely rare, as I'm sure every one of my readers knows from experience.


Me handing the mic to the guys that started it all back in 2010


What else is crazy about Nextcloud? Here's another one: where lots of companies struggle to find good engineers, that is literally the LEAST of our problems. We drown in amazingly good CV's and have a big pool of enthusiastic, qualified engineers who contribute to Nextcloud and already know the code. I wish we could hire them all but growing more than 50-80% per year isn't really health for a company culture...

Also special: other companies struggle to get sales leads and pay lots of (advertising) money for them. We, we drown in leads... Even without marketing automation. Our biggest challenge, instead, is answering all the requests from companies that want to buy our product - we need more sales people!

Yes, we're a pretty unique company in how we approach open source business and we're successfully taking on much bigger companies. Yes, it works! Just check how we're doing on Google Trends. Love that!

If you want to work for us, especially in sales, or know somebody who should, tell me ;-)

Or first learn about us by meeting us - you're welcome at our conference! Or at one of our meetups, there's a monthly one in Berlin for example.



12 January, 2018

Nasty fall-out from Spectre and Meltdown

I guess it's hard to miss Spectre and Meltdown so you probably read about it. And there's more bad news than what's been widely reported, it seems.

You trust the cloud? HAHAHAHA

What surprised me a little was how few journalists paid attention to the fact that Meltdown in particular breaks the isolation between containers and Virtual Machines - making it quite dangerous to run your code in places like Amazon S3. Meltdown means: anything you have ran on Amazon S3 or competing clouds from Google and Microsoft has been exposed to other code running on the same systems.

And storage isn't per-se safe, as the systems handling the storage just might also be used for running apps from other customers - who then thus could have gotten at that data. I wrote a bit more about this in an opinion post for Nextcloud.

We don't know if any breaches happened, of course. We also don't know that they didn't.

That's one of my main issues with the big public cloud providers: we KNOW they hide breaches from us. All the time. For YEARS. Yahoo did particularly nasty, but was it really such an outlier? Uber hid data stolen from 57 million users for a year, which came out just November last year.

Particularly annoying if you're legally obliged to report security breaches to the users it has affected, or to your government. Which is, by the way, the case in more and more countries. You effectively can't do that if you put any data in a public cloud...

Considering the sales of the maximum allowed amount of stock just last November by the Intel CEO, forgive me if I have little trust in the ethical standards at that company, or any other for that matter. (oh, and if you thought the selling of the stock by the Intel CEO is just typical stuff, nah, it was noticed as interesting BEFORE Meltdown & Spectre became public)

So no, there's no reason to trust these guys (and girls) on their blue, brown, green or black eyes. None whatsoever.

Vendors screwed up a fair bit. More to come?

But there's more. GregKH, the inofficial number two in Linux kernel development, blogged about what-to-do wrt Meltdown/Spectre and he shared an interesting nugget of information:
We had no real information on exactly what the Spectre problem was at all
Wait. What? So the guys who had to fix the infrastructure for EVERY public and private cloud and home computer and everything else out there had... no... idea?

Yeap. Golem.de notes (in German) that the coordination around Meltdown didn't take place over the usual closed kernel security mailing list, but instead distributions created their own patches. The cleanup of the resulting mess is ongoing and might take a few more weeks. Oh, and some issues regarding Meltdown & Spectre might not be fix-able at all.

But I'm mostly curious to find out what went wrong in the communication that resulted in the folks who were supposed to write the code to protect us didn't know what the problem was. Because that just seems a little crazy to me. just a little.