I’m going to stray away from usual topics and write about a recent experience with a discussion I had on an issue tracker for the OSMC open-source project (it’s a media center / linux distribution, based around Kodi). This wasn’t a pleasant experience, essentially because of one toxic developer, and I find myself still needing to write down some things about it that still trouble me, but there is no other suitable medium for doing so. I don’t feel that I was in the wrong, however, I’ll try to focus on what I could have handled better (especially in terms of winning the technical argument) rather than what others might have said or done that I find offensive. This is a somewhat personal blog entry, but I suppose these situations happen to most of us at some point or other.
The original discussion in full is still available here, though further comments have been blocked, but I’ll try to re-cap in what I hope is an even-handed manner.
1. The Issue
A user had noticed that OSMC was enabling the SSH service in its default configuration, with a default username and password, and without prompting or informing the user in any way. I would like to think at this point that any readers are already horrified, and I certainly was, but this is certainly not a universal reaction: the issue was summarily closed without being addressed. One of the developers gave some reasons for this: essentially, SSH is useful as a recovery option in case of failure or misconfiguration that prevents other access, but the user could not be expected to remember a password; furthermore, OSMC is designed to be used behind a NAT (by which he meant firewall, but that’s nit-picking). Finally, OSMC can be configured without a keyboard attached (using just a remote control), which makes choosing a password difficult.
These reasons weren’t good enough for the original reporter nor for several others who chimed in (before I did), and there was some ongoing discussion (with the issue held in the CLOSED state). Points of contention included:
- Whether being on a private network is a good enough security measure
- To what level user convenience (such as not having to remember a password and being able to use SSH when necessary without having to enable it first) trumps security, and vice versa. (Bear in mind that forgetting the SSH password potentially requires the user to re-image their device, though I personally don’t consider this necessarily adds much weight, relatively speaking).
- Whether restricting incoming SSH connections to end points on the local network would suffice as a mitigation.
- Whether disabling SSH by default would be a reasonable change.
Some arguments were made for making no changes, that I felt were markedly dubious, including:
If we disable SSH, then where should the line be drawn? There is a libmicrohttpd service on Port 80 running with a JSON-RPC interface. Unfortunately, there is a Kodi 0-day vulnerability which allows a system to be compromised with this.
(I mean, your excuse for not securing one part of your system is that there’s a known vulnerability in another part? What?) I eventually stepped in, and I wanted to hit hard – really drive the point home:
You leave a service like SSH running with a default password and I pretty much guarantee you just created a botnet. Well done. News flash: your users won’t do what you want or expect them to do. A proportion of the devices will get exposed to the wider world, through accident or negligence – yes, even a proportion of that alleged 99% that sit behind a router; you can’t prevent this, but you can mitigate the damage, and it would be irresponsible not to do so.
Now, the first sentence was (I thought) obvious hyperbole designed to stress a particular point. This brings me to my first mistake:
#1 Avoid Hyperbole and Exaggeration
Although in this case I though it was obvious that I wasn’t saying there would be an actual botnet at any particular point in time if this issue is not addressed, the problem was that people later argued against this point as if I had meant it literally (I was essentially challenged to produce evidence of the existence of a botnet). And though this seems ridiculous, it gave people a way to argue in which it looked (superficially) like they were making a valid point which reduced my own.
I continued with some brief conjecture about how the issue could be resolved in terms of implementation details. I think this was also a mistake, but it’s more of a technical nature. I shouldn’t have gotten caught up in implementation details, and instead just stuck to the main point: ssh with default password enabled by default is very bad. (You can always argue the less important details later, once the important battles are won).
#2 Don’t bow out early
After rebutting another user’s comment (which was apparently later deleted), I concluded with:
(I’m done with the discussion. I hope you guys see the light eventually. Good luck).
I should not have said this. I wasn’t done. I got a reply, and felt compelled to respond.
#3 Don’t make throwaway comments
I continued arguing against another user (who later I discovered was actually an OSMC developer), one “KODeKarnage”. KODeKarnage had said:
There is no such thing as 100% secure, at some point you have to be ok with good enough, and blithely ignoring the costs of “mitigation” doesn’t make them disappear.
I responded with:
What, the “cost” of users needing to actually remember their password? In my view cost/benefit analysis weighs heavily in favor of not having a default password. I’m surprised that anyone would disagree with this, but there it is, I guess.
There’s probably a couple of things to be said about this. Firstly, I was too flippant about the costs KODeKarnage was referring to. At this stage as then, I feel those costs are insignificant compared to the problem of having a trivially compromisable system on your network, but being so flippant allowed KODeKarnage to claim that I hadn’t really considered their point, which indeed became an ongoing theme (along with, shortly afterwards, insults to my intelligence and insinuations of arrogance).
Secondly, that last sentence. I was surprised that people disagreed, but saying so didn’t help matters; indeed, apparently, it was construed as an insult, which the same KODeKarnage revealed much further down the line. The problem with this sentence, even if one doesn’t consider it inflammatory (and I don’t), is that it is unnecessary to the discussion at hand. Any time you say something unnecessary, you’re giving those who disagree potential fuel to use to detract (or at least distract) from your valid arguments.
A more immediate response from KODeKarnage was:
I rebutted the first two points and ended by quoting the final sentence, responding with:
I don’t see the point in arguing with you at this stage; you’re being arrogant and rude.
Which brings us to point #4.
#4 If you’re not going to argue with someone, don’t argue with them.
I shouldn’t have bothered rebutting the first two points. They were insignificant compared to the personal insult which had been issued. Replying to them detracted from the real issue here, which was that this person was being a prat. I could have either ignored them, or dismissed them without trying to counter them – normally not a valid tactic in a technical debate, but at this point, the debate had gone beyond the technical.
I left the conversation, for some time, after being insulted. For a while, nothing happened. Some time later, I noticed that KODeKarnage had made a commit that looked as if he was going to implement the ability to change the SSH password, as well as disable the service, during the initial setup. A little later, the password change functionality (which had only reached the stage of being a stub) was removed, though the ability to disable SSH remained in the code base.
More time passed, during which nothing much else happened.
Then, the discussion awoke as it was joined by another user, who agreed that having a default username/password was bad and didn’t think that the option to disable SSH, as had been implemented, really was a satisfactory solution. There was a little more to-and-fro, and at this point I got used as an excuse to forgo further discussion:
However, outlandish statements such as
pretty much guarantee you just created a botnet
make it increasingly challenging for us to take further comments on this issue seriously.
Yep, that’s my “botnet” quote – refer to mistake #1. Now the price for that mistake was being paid. I feel the statement quoted above (originally by one “samnazarko”) is ridiculous, and indeed samnazarko was called out for taking my statement far too literally by the other user (not by me) – but arguably the damage was done. Essentially, the opposition had been given a gift in the form of a “clearly ridiculous statement”. It didn’t matter to them that it was fundamentally incorrect to take this statement literally, because by doing so they were essentially able to hold up a hand and say, “no, stop, you’re obviously wrong, conversation is over”.
However, the argument continued. There was a lot of fluff about IPv6 and “NAT vs firewall” that doesn’t really matter too much. Eventually the original reporter of the issue (vexto1) waded back in and revealed that a private discussion between himself and samnazarko had resulted in the promise that it would become possible to disable SSH. Indeed, this had been delivered, but as vext01 noted (and as had been noted by the other user who was relatively new to the discussion, joepie91), the option was enabled by default and the help text was insufficient to dissuade users from enabling it on no more than a whim. Although this had been pointed out earlier, samnazarko now relented:
I am sure we can make that change
Battle won, right? I took a moment to stick one, ever so gently, to this KODeKarnage who had been obnoxious earlier:
I know that some people will consider this mistake #5, though I honestly don’t feel it was. The technical battle had, fortunately, been won by this point. Was it cheeky? yes. Provocative? I suppose, though in all honesty I never expected KODeKarnage to bite on this.
They did so, however:
What ensued was a to-and-fro which got heated, though I did avoid throwing any insults of my own until towards the end, at which stage I felt that it was clear that no amount of reasonable discussion was going to get anywhere.
Which takes me the final valuable lesson:
#5 Don’t keep arguing when you’ve already won.
It just frustrates you and takes the pleasure of the victory away.
I had the right, I believe, to feel insulted at being called arrogant. I was also outraged at the fact that I was being accused of “insulting the intelligence of people who didn’t value the same” – when that was clearly what had been done to me, not the other way around. I wanted to respond, and I believe that defending yourself against an allegation like the above is reasonable. But when I responded, I tried to rebuff each (or at least several) of the points in turn, which just led to a shit-storm of messages to and fro which were almost certainly of no interest to anyone but the two of us. I even broke rule #2 (don’t bow out early) again.
What I should have done was dismissed KODeKarnage’s statements above – all of them – with one or two simple sentences – along the lines of “No, you’re misrepresenting my position, but I refuse to argue with you further” – and then denied the allegation categorically – in a nice, straightforward message that couldn’t easily lead to further argument. I regret not doing so, now. It would have saved me some time and some stress.
I also fired off a twitter message to express my frustration, not thinking that it might get beyond my immediate circle of followers – unfortunately it was seen by samnazarko who used it as a reason to lock the discussion (I would have thought that various of KODeKarnage’s insulting statements would have been enough to warrant that on their own, but I suspect that developers within a project have a certain tolerance for each other, and it’s possible that they know each other in person).
In any case, what’s done is done. I said all that I really needed to say in the discussion, and we got the right result, but I should have reined myself in to a much greater extent. I want to try harder to do that the next time I’m drawn into such a discussion.
I’ll just leave these two great quotes from KODeKarnage here:
Both of those facts are obvious to anyone who has actually considered the issue. Maybe come back when you have given this more than a few seconds thought.
Either it was obvious or it took more than a few seconds thought. You can’t have it both ways.
Indeed, you can’t.