Jan Harasym

On The Shame We Share

2026-02-19T02:52:00+07:00

Peter Lübeck, a prominent figure in Malmö's gamedev scene, recently wrote that he's ashamed to be a man.</a> which is a sad thing to say for someone who is a role model to many.

The thing is, that he's not wrong about any of the things he's angry about. The Pelicot case</a>. Epstein. The call centre CEO who spent years running what was, essentially, an abuse operation against his own staff</a>. Women being harassed on a fucking secondhand clothing app</a> of all things. These are real, and they're really awful.

I wrote a reply suggesting demographic shame was the wrong response. The replies taking me apart got significantly more support than my point did. A few suggested I was strawmanning.

Fine. Worth talking about then.

His Strongest Point</h2>
Men commit approximately 97% of reported sexual offences, consistently, across every Western jurisdiction that tracks it. Not contested. Not arguing that this isn't the case.
The Pelicot case is the sharpest version of this argument. Within a small geographical area, dozens of ordinary men (truck drivers, firefighters, IT guys, fathers) raped an unconscious woman who had been habitually drugged by her husband. There was no shared pathology, nor obvious common thread. Just men who lived nearby and said yes when asked. One walked away when he realised what was happening, but he didn't call the police.
For Peter; this kills his more "comfortable" idea that male violence belongs exclusively to disturbed individuals. He's right that this idea was wrong. Ordinary people, without obvious pathology, do terrible things when the conditions allow it.
I'm not going to argue with that. The question is what follows from it.

What The Language Is Doing</h2>
Peter writes that all men know men who have "stepped out of line, spoken derogatorily of women, or wouldn't take no for an answer."
"Stepped out of line" is doing enormous work in that sentence.
At one end: someone who told a bad joke fifteen years ago that went unchallenged. At the other: the exceptional depravity of the Pelicot case. His evidence supports the first, but he reaches for the second. Most readers won't catch the slide, which means they end up somewhere the argument hasn't actually taken them.
Asking people to comment about being uncomfortable when someone tells an off-colour joke: Entirely reasonable. Carrying collective guilt for crimes committed by strangers: a smidge less reasonable. He requests the second but only lays a passable argument for the first.

What Speaking Up Costs</h2>
Look, I've stepped in on domestic violence twice. The phrase "men should speak up" hits your ears very different when you know what it means.
The first time was on a street between Charing Cross and St Martin's Lane in London, on my way home from work. The attacker rose to meet my condemnation with further aggression- directed at me, I hesitated (in the UK you think twice; wrong move and you're the assailant in the eyes of the law) and he put me on the ground, destroyed my laptop, a bag I couldn't afford to replace (or afford in the first place, realistically) & my nose - then went right back to what he was doing, angrier. The second time, the woman being dragged around by the back of her neck became furious at me for interfering. I was around 22 years old. I'll tell you concretely: ieither felt like I was doing the right thing in any meaningful way, I was attacked and scolded and it changed nothing. Both were frightening, and bare in mind: I grew up in violent places and had martial arts training.
There are other options, of course. Calling the police, filming it, trying to gather people nearby. None of them are free either. The police often don't arrive (and if they do: it's post-fact, obviously), filming can escalate in the same way confrontation can, persuading bystanders to act together under pressure is harder than it sounds from a safe distance. The point isn't that physical intervention is the only option. It's that the cost of intervening is real regardless of the form it takes, and not everyone is built to absorb it. I think those people aren't "lesser men" for it.

The Part He's Right About</h2>
Men are responsible for the culture immediately around them. I think Peter is right about that, even if I'd put it differently. (IE; that men are not unique in that aspect).
Most of it isn't dramatic. If someone makes a joke you found disagreeable and you changed the subject, to me that is a signal. A real one. Most people calibrate their behaviour on reinforced feedback; jokes that don't land stop getting made. People corrected without being destroyed tend to actually change.. but these aren't the great sacrifices being called for: but they're low-cost and they compound over time. The ask should be that: the ordinary, undramatic correction of the culture within reach. Not martyrdom. Not career destruction.
I've been in Malmö's gamedev scene for twelve years. If you want to know what that culture looked like on the ground: men performing contrition for existing, internal mailing lists weaponised for social-justice pile-ons until they had to be shut down, and at least one occasion where I was told, without irony, that my next hire had better be a woman (a move strongly defended by the other men in the studio). Serge^{[more on him later], meanwhile, was running his operation completely untouched. The people with no institutional power were being punished for their demographic. The person with all of it was unreachable until a criminal court got involved. People subsequently pointed to Serge as proof the punishment of the former was justified. I've never quite worked out the logic of that, but I can say that it was stated with great confidence.}

The pinch</h2>
Peter says there's a lack of solid male role models. I agree.
Look, though, at what his post models as the correct masculine response: shame. He doesn't mention that if you're ashamed, you're confirming maleness is the problem, and implicitly, if you're not ashamed, that proves you're part of the bad stuff men do. The framework doesn't seem to distinguish between men who feel solidarity with victims and men who are complicit; absence of visible shame reads as evidence that you are comfortable with the suffering women face at the hands of some men. It's a closed loop. Congratulations: you're guilty.
Some people reading this will say that shame about male behaviour doesn't have to mean shame about being male: that you can feel ashamed of what some men do the same way you feel ashamed, as a human, about what humans are doing to the planet. That's a coherent position. That's not what is happening.
And, what I'm arguing against is that it is not the position the post builds. A teenage boy reading Lübecks emotional appeal doesn't take away "here's how to be part of the solution." He takes away a debt he incurred before he was old enough to have done anything.
Peter ends with terror for his daughter, which I understand completely, but he forgets that son's exist too; and what is he putting into the world for them to grow from? Collective guilt that can never be put down without being labelled as "the problem".
</a>

My friend Charlene.</h2>
The structures that enable abuse don't much care about the demographics of who's exploiting them. They care about who has access and who won't be believed.
I know this more concretely than I'd like to.
I had a friend at school- Charlene, the year above me, six months older. We used to make up rather inventive stories together. One of those stories, as it turned out later, wasn't made up: men coming into her room at night to lie in her bed with her. I was eight. I thought she was joking. I didn't tell anyone. (See, I was part of the problem!)
My mum had caught wind that social services suspected a paedophile in the family home, and she assumed it was the father. Reasonable assumption; he was violent and angry. Though not, as it happens, in the way anyone assumed.
It was the grandmother. Selling access to her granddaughters to people from the pub.
The assumption my mother made (reasonable, statistically defensible, the kind anyone would make) meant the father was kept away at exactly the moment it mattered most. Which goes some way toward explaining the violence and the anger, if you think about it. Social services eventually cut funding for the family home on the basis that the mother had maintained a living situation around a suspected abuser. They came to stay with us while they figured out what came next. The father wasn't permitted to come.
Granny slept in my room. With me.
They moved to Blackpool shortly after, directly from our house to avoid social services. Charlene disappeared some time later. Nobody has ever been convicted.
I don't think all grandmothers are doing this. But then, that's rather the point. She happens to be a hundred percent of the paedophiles I've personally known about.
It wasn't institutions that protected her. It was assumptions. It was her own daughter; who hid it, covered for her, kept it going even while it destroyed her own children's lives. Nobody had named the father. Nobody needed to. There was a man in the house and a suspected paedophile, then that has to be the same person. He couldn't be there for his daughters because we drew the "obvious" conclusion. While granny slept in my room, with me.
Meanwhile, In Malmö</h2>
Serge Hascoët</a> was Ubisoft's chief creative officer and a generally rather unlikable cunt.. It's relevant to Lübeck because it's: Ubisoft Massive (as-in: roughly 30% of Malmö's entire gamedev scene) that is in his collective view. A French court finally convicted Serge of psychological harassment and "complicity" in sexual harassment, if you know the rumours then you know that this is a fucking joke. Eight years of complaints buried, staff silenced, total systematic suppression, all while the company rolled out DEI initiatives to bring more women in presumably for the aforementioned harassment. He got one suspended sentence. No jail time.
The person whose job was specifically to stop this was Cécile Cornet, Ubisoft's global head of human resources</a>. Named in the lawsuit</a> as directly responsible for maintaining the conditions that let it continue. Also a woman.
The point isn't that her presence exonerates anyone. It's that she held the one institutional role designed to catch exactly this, and she failed the victims just as completely as the men above her. I worked at Massive, which means I share Malmö's general collective shame for the whole thing. But the idea that I bore more institutional responsibility than Cornet, because I happen to be a man, doesn't hold. She had the power and the mandate. What would my speaking up, had I even known, have cost me? I know exactly what. I also know it wouldn't have been enough.
The variable that predicted abuse at Ubisoft wasn't demographics. It was impunity, and the cost of ending it landing on individuals rather than institutions.

The structure underneath</h2>
Strip away the specifics and you have: a demographic commits crimes at elevated rates, therefore members of that demographic carry collective guilt.
Apply that to any other group. You probably already have one in mind. Notice the discomfort.
That discomfort is the argument.
The standard objection is that the analogy breaks down because gender involves structural power in ways that race/religion/whatever doesn't. But the objection to collective demographic guilt was never only about power: it's that guilt doesn't transfer between individuals who share an immutable characteristic. That principle either holds or it doesn't. Deciding it holds for some demographics and not others isn't a principled position, it becomes bigotry looking for a justification.

What Would Fix It</h2>
Think about what speaking up about Hascoët would have cost anyone, not just someone at the bottom like me. Your career at Ubisoft? Definitely. Probably your reputation in an industry where everyone knows everyone too. Some of the people who said nothing weren't moral cowards; they'd watched what happened to the ones who spoke, and they knew they couldn't afford to be next. Rose McGowan spent years being surveilled, smeared, and quietly blacklisted before anyone would print a word. I still feel bad for the ones who did the maths and stayed quiet, not as much as the victims of course, but I can't see it as black-and-white.
"Men should hold men accountable" doesn't grapple with any of that. It asks individuals to take career-ending, sometimes physically dangerous personal risk to compensate for institutions that won't do their job. You can't build a functional system on the expectation that enough people will keep volunteering to take the hit. At some point the pool shrinks to nothing, either by attrition or because the only people left in the room have already decided the culture suits them fine.
The calculation changes when speaking up stops being a personal sacrifice and starts being a protected act. Right now, a complaint at Ubisoft went to Cornet. Cornet buried it. Eight years of that. The person who reported got nothing, maybe even a reprimand and a private blacklisting; the person who buried it kept her salary and her title. That's the sum the system is currently running.
Obviously: fix the fucking sum. Reporting needs somewhere to go that isn't inside the organisation being reported on. Consequences need to land on the institution when complaints are suppressed, not solely on the individual brave or foolish enough to file them. When the career risk runs in the other direction (when burying a complaint is the thing that ends careers), the culture shifts without requiring anyone to be destroyed in the process.
That's not a "men's problem" to solve, it's collective: it's legislation with actual teeth, aimed at the organisations that currently let this run for years on the very real assumption that nobody will pay for it.

anyway</h2>
Peter asked, in good faith, what male role models look like. Here's one answer: a man who thinks sexism is wrong, that abusers should face consequences, that when someone comes forward the response shouldn't be to bury it, and who also thinks his gender isn't something to apologise for.
Most men in this industry won't say that. I've watched, over twelve years, that the cost of dissent become progressively less negotiable, and the quiet normalisation of contempt for men treated as a reasonable position. The pool of people willing to say anything shrinks accordingly. The silence makes a bad argument look unanimous when it isn't.
For the juniors, the students, the boys reading this: you don't have to accept demographic guilt to be a decent person. Do what you can, when you can. Support victims. Don't be an arsehole. That's it.
Sexism doesn't stop being sexism because it's well-intentioned.
And, don't feel bad for being a boy.

The only good cloud is a google cloud

2024-09-24T19:21:00+07:00
Your browser does not support the audio element. </audio>
First, this is not marketing. I'm not paid nor do I have any incentive to say what comes next. My personal opinion overall is that we're all ~dumb as fuck~ for giving so much of our foundational infrastructure to three US companies. That said:
Yours truly once dared to suggest on the hallowed grounds of HackerNews that Amazon's cloud computing juggernaut, AWS, had cleverly exploited the very shortsightedness that plagued operations teams</a> and caused people to believe that we could not do better internally. A stroke of genius, for sure.
But, let's be frank, outside of S3 have any of their so-called "innovations" truly served anyone beyond lining the pockets of some fat cats in Seattle?
I've traversed our industry as a counter-culture goblin for a long time (though I prefer the term "independent thinker"), and I've heard every snake oil pitch imaginable about the cloud being the true technological messiah. Meanwhile, I, for daring to suggest that perhaps virtual machines shouldn't cost a king's ransom</a> ^[HN comments</a> ], am painted as vaudvillian villain, cartoonishly outdated or- in one case: that I lacked testicles to learn. (though I am certainly unsure how your brain is wired if it is dependent on your testicles.)
"Why would you give people the Sisyphean task of operating servers, that is for peasants! Economies of scale make it cheaper to operate in a cloud!"; as if wrangling vendor-locked terraform and stodgy YAML manifests is somehow intellectually superior, or that we couldn't possibly measure our outcomes related to cost.
I won't waste column inches lambasting AWS for failing to deliver on their promise of leaner tech teams (though statistics prove me right). Instead, allow me to air a few grievances, then make a bold claim: Google Cloud Platform, is the superior choice in every way that counts for a cloud.
Why? Because I refuse to believe this *gestures broadly at AWS console* is as good as it gets, but I've come to believe that we are a bunch of technological copycats, too afraid to deviate from the herd. But today, I, your friendly neighborhood goblin, will hopefully piss you off enough to respond to me angrily, and in doing so, force you to confront the true reality of why you choose inferiority and mediocrity.
Of course, Google isn't without its faults. They've been known to pull the rug out from under their users. But desperate times call for desperate measures, and sometimes, you have to risk a nibble from the alligator to escape the clutches of the wolf; and I contend: that this is actually a good thing, as it prevents you from being too comfortable sticking around.
The lure of cloud: MIRAGE OR MIRACLE?</h2>
What is it that draws folks to this nebulous "cloud," anyway? If you're a pain in the ass like me you hear it constantly, and if you boil it down, and you'll find the same trio of promises: "it's reliable" (supposedly), "it's easy" (they say), and "it's economical" (or so we're told).
Well, I'm calling their bluff. If that's all the cloud's got going for it, then Google's offering leaves Amazon's in the dust. And if you think I'm wrong, I'm all ears. Name one more genuine advantage AWS has over GCP when compared to colocation of rented machines, and I'll eat my hat.
This post is an indirect response to: "it's hard to recommend Google Cloud"</a> ^[lobste.rs comments</a> ]
1. Reliability</h2>
VPCs</h3>
AWS is zonal by default, it's clear. VPC's require manual networking intervention (outside the happy path) to set up a "peered" network. This, by itself and with the context of "cloud helps you do the right thing for HA", is fucking moronic.
You could try to argue that it's hard to do networking right, or that they have a legacy to take care of. -- But why the fuck is that my problem if I'm paying them hand over fist for them to commoditise my compute? As soon as Google released global based routing Bezos should have been commanding improvement from the telescreens</a> that I am certain are placed on every engineers desk (and cars, and homes).
To add insult to injury, doing it properly, for years, costed more and is so complex there's an actual certification for it!</a>.
(I guess google has one too</a> but I'm not aware of people actually taking it or needing it, AWS claims 1.31M certificate holders though</a>).
Newcomers often assume that basic networking within a cloud environment would be region-wide or at least span multiple Availability Zones (AZs) for redundancy.
EC2 instances are isolated within a single AZ by default - not obvious to newcomers. It violates principle of least surprise when talking about a "cloud" that's supposed to be helping you become more reliable, and again, doing it properly costed more, meaning people naturally deviated their dev environments from prod.
ELBs</h3>
The classic ELB (now considered a legacy service) has a default configuration that doesn't distribute traffic across multiple AZs.
This might now be a legacy service, but let me repeat that.
THE FUCKING LOAD BALANCER -FOR 15 YEARS- DID NOT LOAD BALANCE ACROSS ZONES BY DEFAULT.
You need to choose the "Application Load Balancer" or "Network Load Balancer" and configure them appropriately for high availability.
Obviously this is not the case in GCP, networking is global by default and load balancers actually balance across zones; disks can also be regional- it's just one click away (and it costs double, but, it works, and this is what I'd expect).
I guess they consider it a legacy service, so they finally fixed it after a decade and a half of service, so I should give them some credit...
But even the new "ALB" has weird quirks</a> even though it does the right thing by default:

After you disable an Availability Zone, the targets in that Availability Zone remain registered with the load balancer. However, even though they remain registered, the load balancer does not route traffic to them. </blockquote>
WHAT?! WHY?!
EC2</h3>
Reliability is hard when the underlying systems become unavailable, Google was the first to support live migration between hosts- I don't even notice it happening and I ran some performance sensitive applications.
AWS though? notice of termination, you have 4-24 hours to comply</a>. Or, none. Sometimes.

In some cases, Amazon will notice their hardware is in a degraded state and tell you to get off of it (stop and start your instance) by a certain date or it will be stopped automatically. </blockquote>

In some cases, there will be no warning and it will just stop. Or not enter STOP state, and simply become unreachable. It may or may not reboot after they take care of it. Sometimes, there will be an apology mail after the fact. </blockquote>
You probably think that this shit is "good enough", or maybe that we've recieved enough collective brain damage over time that it has become "good enough" to get by... even if it's not perfect.. well...
2. "Good Enough" != Ease of Use</h2>
This section used to be titled: "Good enough" helps you limp along or: you can't 60% your way to good UX.
The insidious nature of the "good enough" approach is that it works 20-40% of the time, that's why it's pervasive; however, people like to take the "good enough" approach every time: and on a long enough timeline you end up with AWS.
I mean, it works, right, but why can't I see what project I'm in? (sorry: "Account")...

Look, there was a really big reason why Windows kicked the absolute crap out of everything else back in the day, and part of that reason was a slavish devotion to user interface design, with an eye for accessibility and consistency</a>; they forgot this, but remains a large part of what put them on top. ^[HN Comments</a> ]
Clearly AWS doesn't believe this.
I mean, what is wrong with my SSH key?

There's no "help" dialog, or information about supported key types, or examples, literally sweet fuck-all.
Additionally why would I care about "terminated" machines? I can't do anything with the info here? If I need to know what's terminated shouldn't I just look at logs? I can't recover anything from these; it's not like I can "un"-terminate them or recover the drive...

And.. while I'm here... why can't I see instances hosted in other regions?
Clearly UI weirdness isn't a deal-breaker, but it's a constant reminder that AWS settles for 'good enough' instead of striving for excellence, which they should be able to afford</a>.
Let's see how Google handles those things:
First, logging in allows you to have a standard google account or federated identity</a>
and.. project's/workspaces are human readable...

Better yet, my VM list doesn't have dead resources and is actually global by default, meaning no random resources in Frankfurt that are chewing our bill silently for years on end that we never noticed.. (Yes, that happened because we took our eye off the ball).

(ignore the fact they're in the same zone, they have regional disks :') )
And they even have human readable names, which are used by the API and CLI too...
... Oh wait, what the fuck is that!? In three places in that image I see the word "Save"!
3. Cost optimisations</h2> AWS Cost management is basically a meme at this point, they even invented a term for it FinOps</a> (hey, weren't we supposed to be reducing operations?! - we just invented a whole fucking role, and a complicated boring one!).

Meanwhile Google bakes cost saving into its offering with a comprehensive pricing calculator, sustained usage based discounts (in addition to comitted use, like in AWS), and a "FinOps hub"^{🤮 to help you organise and further reduce your spend...}
The actual cost reporting system is pretty nice too, and you can export it to BigQuery and get extremely detailed reporting if you really wanted.
Yeah, it's still a goddamn rip-off! Cloud's gonna bleed you dry compared to a colo, no question. You'll be paying five to even eleven times the price sometimes.
At least they're not sneaky bastards about it, though. They'll show you how to trim some fat off that bill.
And let's face it, one engineer can actually move mountains on this platform. No bullshit, no hidden gotchas - it's compute the way it should be: comodditised and without brainrot.
Imagine getting paged at 3am. You're bleary-eyed, trying to figure out if i-0256162531f6a2ed</code> or i-0256162531f6a2ec</code> is the problem VM. They both have the same 'Name' label/tag, and you accidentally opened the dev environment instead of production. You need some obscure browser extension just to tell them apart!
And don't even get me started on naming conventions. Instances can share the same 'Name' label so you can't trust, and if you want to auto-generate unique names in an autoscaling group, you need to write a Lambda function. Seriously, I laughed out loud when I saw that was the main recommended solution</a>. It's ridiculous!
Google is scary though</h2> Google is a scary proposition for two reasons; Google itself tends to deprecate fucking everything</a> it seems. </li> It's not popular enough, so "enhancements" are AWS first, like cloudcraft</a>. </li> </ol> Point #2 is self-fulfilling, so let's not even bother talking about it. Point #1 however, is important. Windows is well-known for its commitment to backward compatibility, sometimes to a fault. This can lead to frustrating situations, like having 32 different USER_INFO</code> struct "levels" for interacting with win32 functions. The problem is, many of these structures don't play nicely with group functions because groups don't let you work with usernames directly. Groups in Windows fundamentally, only understand security identifiers (sids), not usernames</a>, which can make things tricky for developers as there's no USER_INFO</code> struct level that understands SID</code>-s.. A fucking nightmare. (ask how I know</a>, go on..) A slavish devotion to backwards compatibility stifles innovation, this is largely the reason Moxie Marlinspike did not enjoy the idea of federating Signal</a>. Thus, we should allow our clouds to shed the dead weight. Crucially, it's better for us too: since more work can go into working on better developer experience instead of complicated cruft that makes things slower, cruftier, complex and more idiosyncratic over time (like Windows). Maybe I'm biased, because as a game developer we're usually quick to move fast and abandon things</a> if they're not working. Cloud in general</h2> Look, the cloud is a rip-off. All that talk about its benefits is just short-term thinking in disguise. Even GCP, which I think is the best of the bunch, is still highway robbery. Sure, it's great for handling sudden traffic spikes or figuring out your needs, but in the long run, you'll save a fortune by ditching the cloud and running your own hardware. My build servers paid for themselves in two damn months! And don't give me that "humans cost money" shit: AWS experts cost way more than old-school sysadmins - and I haven't touched my build servers in almost two years. It's all a scam to funnel your startup's cash straight to Bezos. The whole cloud industry is a racket designed to suck your startup dry because you're too stupid to think beyond 6 months and you have heard stories regarding heavily underfunded ops teams. This is made worse by slick consultants, professional services salesmen and developer evangelists whispering sweet nothings in your ear about "scale" and "abstraction" while you rack up a bill you can't afford, and learn things you can never apply outside. Hopefully they at least lubed you up with $100,000 in startup credits, if not, what were you fucking thinking? You traded freedom for a little comfort in being told you were smart. Dumbass. GCP might have its quirks, but at least it doesn't try to rewire your brain like AWS. You can actually get shit done without memorizing their entire goddamn encyclopedia, and the skills you learn are useful elsewhere, unlike AWS's proprietary nonsense. GCP is a tool, not a religion – except maybe for their IAM, which is still less painful than AWS's convoluted mess. Azure?</h3> "Jan, you're talking a lot about AWS vs GCP (and, no cloud at all) but what about Azure?" </blockquote> Well... If you're using Azure, it's because your CFO decided that since you already have a deal with Microsoft for Office, why not also put it on the same bill instead of having another relationship with another vendor. Nobody chooses Azure for Azure, even the Azure representatives didn't recommend Azure to me, so they're even aware of this. (which, honestly endears me to other service offerings from Microsoft...). Conclusion</h2> Your browser does not support the audio element. </audio> So, GCP: Makes it easier to do the right thing by default, has significantly improved DX/UX (no missing instances because they're in another castle or garbage errors, or missing help, and you can reference instances by human readable names) and does it's best to help you understand costs and save money... the three reasons you would even use a cloud in the first place (supposedly). Issues raised with AWS could be answered with "you're holding it wrong", but isn't it the entire fucking point that it should be easy to hold it right? This is supposed to be replacing Ops specialisation, not changing ops specialisation to be vendor specific instead! Isn't the point that "the cloud" is meant to be a clean abstraction over a commodity system, so we spend less time and energy on this cruft? That it requires less staff, less time and more focus on the product itself? If that's the case, then.. Google cloud is the only game in town. But, they might deprecate your shit</a>. (In fairness, Google Domains was not a Google Cloud product (which was actually extremely annoying when using it before); it's a notable distinction because Google Cloud products seem to always have worthwhile replacements waiting for them. That said, yes, it pissed me off. I recommend porkbun</a> as they suck less.) Personally, I see that as a good thing, the absolute best use of cloud is to scale a bit for peaks, or to save you time and energy up front when you are figuring things out. Staying in any cloud long-term is a money losing strategy on all fronts.
Enforcement can have the inverse effect 2023-01-27T00:00:00+00:00 I live in a small city which is extraordinarily easy to cycle through. Everything is a short distance, the cycle infrastructure is kept tidy and there is a distinct absence of anything that could even remotely be considered a hill. One issue that keeps cycling down (if there are any) is that unscrupulous ball-bags1</a> tend to steal bicycles rather often, there are even remarks that "bikes are socially owned, you never own one personally". Which, if you enjoy having a bike with good gearing and a high degree of comfort (especially one that cost a lot) is unsettling to hear. The advent of e-Scooters all but eliminated these worries, they are a communal asset, they are sufficiently fast, low effort (so you're not sweaty when you arrive) and ubiquitious enough that you can be fairly certain you will find one. These nippy little things are also fantastic on the cycling infrastructure, going just marginally faster than the average cyclist; allowing you to overtake excessively slow cyclists easily, but keep (or match pace) with ordinary ones. Unfortunately, people are ball-bags. </a> Even the bikes aren't safe. </a> People park these things wherever they feel like without regard to foot, bike or car traffic, ride as fast as possible through pedestrian areas and often ride with two or even three people on a scooter at a time! Truly, something must be done. Fix 1: Speed limit pedestrian areas and tight areas</h2> I imagine the discussion happening at HQ: Our scooters have in-built GPS and we can control the speed of the motor, they're also connected via LTE. </blockquote> This is perfect. </blockquote> We will draw a geofence and limit the vehicle speed in those areas! </blockquote> Genius! </blockquote> Sounds good? Right? The problem is for purely software approaches like this that rely on hardware is: GPS is not actually perfectly precise, cars tend to fake it by assuming you'll be on the road (which you can usually tell is happening when you start pulling off of a road that the map knows about to a road that the map doesn't know about). Additionally, GPS on low energy devices also tends to... poll. Leaving the GPS streaming is computationally expensive and antennas area major power suck. So, if you're running an eScooter company, you would probably set that polling rate to be as low as you can get away with, maybe once every 90 seconds. You can see this battery drain in real life by leaving (Google|Apple) Maps open on your iPhone. What this means is, you can enter a designated slow zone, but the scooter does not immediately know it's in a slow zone. Thus you can go full speed through slow zones! At least for a little while. This works the inverse too, if you're in a slow zone and leave: well, be prepared to wait for the scooter to realise you left! Commonly the issue becomes, though, that small areas of the map that are designated slow zones but that you pass by can add a minute or two on your 5 minute journey. If you spend enough time in the slow zone for it to notice that you're in a slow zone. Thus, the incentive is to absolutely gun it through those little sections, in the hope the GPS doesn't catch you. </a> Doing the right thing here and slowing down is going to make your journey longer, which makes you pay more. You're basically punishing the correct behaviour. Worse: it can be more dangerous either way as the slowdown is completely unpredictable and the GPS can put you in a slow area even if you're not actually riding in one. Fix 2: Parking in designated areas</h2> I can almost hear another conversation at HQ: People keep parking like dumbasses across pavements, </blockquote> but we have a wonderfully precise GPS system that we're already using and we have a collection of painted areas on a city map that can fence parking! </blockquote> What we can do, is prevent ending the ride and locking the scooter in any un-designated parking area! </blockquote> So, Another GPS solution, but one that isn't affected by a low polling rate, but given how imprecise GPS can be, I don't think I'd ever be convinced that this was going to work. Hrm, yeah, this is me standing in a blue square area. </a> If I was trying to park a scooter the ride sharing app would complain about this. I would need to move my expected coordinates (orange dot) to the middle of the blue painted area by walking east to end my ride, if I walk east though I will be leaving the scooter awkwardly placed across a pedestrian footpath. (those light grey areas are pedestrian walk ways) This is, again, incentivising the wrong behaviour. Other example: Piracy</h2> Movie piracy is the most blatant example of what I’m talking about. Those buying DVDs to watch movies legitimately in the 00’s had to sit through minutes of unskippable ads and egregious anti-piracy warnings. Those that actually pirated had a better experience in their movie watching. Almost assuredly leading to (a non-zero) number of people into the loving embrace of torrent sites. Conclusion, if there is one.</h2> I consider myself a conscientious rider, I park away from pedestrian traffic, usually next to a bicycle stand -- I ride with care around areas where people cross -- I ride alone, and never on the pavement. However, even I have been tempted to say "fuck it" and speed through "slow" areas. When I am late for something and the parking does not allow me to lock my scooter, I am tempted to just walk to where my GPS allows parking and abandoning the scooter wherever that might be. Note: I used VOI for these examples, and to their credit, they are allowing you to lock your scooters outside of those areas and they have improved the polling rate of the scooters a lot, though it does remain a problem. The main point I'm driving here is that sometimes enforcement encourages ordinary users into doing the wrong thing, or needlessly punishes them. I don't have any solutions for the problems I presented, perhaps clear license plates and harsh penalties if reported for dangerous driving (along with looking at the telemetry of the ride) and actually validating the parking is free from obstructing the path which could be done with the picture you send at the end of ride. However, I'm mostly interested in the realisation I had while riding that if I didn't care about slowing down or where I lock my scooter; I would have been better off in terms of time and money. Derogatory British slang word for a thoughtless and ignorant individual. ↩</a> </li> </ol> </section> That time my manager spent $1M on a backup server that I never used 2022-10-21T00:00:00+00:00 The games industry is weird: It simultaneously lags behind the rest of the tech industry by half-a-decade in some areas and yet it can be years ahead in others. What attracted me to the industry was not the glossy veneer working on entertainment products, or making products that I enjoyed using (I wouldn't describe myself as a gamer): I love solving problems, especially problems that are not easily solved. When I joined Ubisoft in 2014 I was put in the Online Programming Team as a person who would run Ops; this was awful because everything was Windows-based. Kubernetes wasn't on the horizon, and even if it was, Docker itself was extremely immature and could not run native Windows binaries yet. What we had instead was our own implementation of distributed systems. The Environment</h2> A highly-optimised and extremely robust service discovery system, reverse proxies which were intelligent enough to force exponential backoff of clients without taking in any load on themselves, a supervisor that could be instrumented via web-sockets, internal service-to-service encryption with a centralised rotating key system, in-memory log viewers that could be reached with a browser over the network, and even stats collectors that ran in-browser. -- all of this, written by hand in C++, nothing off the shelf, very minimal dependencies (OpenSSL being the only one of note), everything running on Windows and completely bespoke. As a predominantly Unix Adminsys1</a> you can do one of two things in this situation: Double down on what you know and try to bend the problem into a solvable one. (think: Wine, I guess) </li> Lean in to the surrounding ecosystem and re-learn the best way of doing things. (Do things the Microsoft Way with SCCM + GPO). </li> </ol> I chose option 3: treat Windows like a appliance or black box, lean on an execution framework that has a solid Windows agent, write all of our tooling in a general scripting language that the remote execution framework can call. (We chose SaltStack+Python). What was great about this approach is that we ultimately understood exactly what was happening in our environment. Nothing was unknown, there was no “magic” program or service doing anything, but simultaneously there was nothing to lean back on: no shell, no Unix tools like sed/awk, no SSH. If you need to modify a file, you have to write a program to do that. If you need to make Windows do something that GPO</a> normally does, you’re writing registry entries by hand, otherwise you’re doing a weird dance of daisy-chaining RDP sessions over a double VPN (yay corporate policies!). An astute reader might be wondering at this point: "Doesn't Ubisoft have a way of doing this properly? They're a large games publisher and games probably had Windows servers before! Right?" You are very right. WWUD :: What Would Ubisoft Do?</h2> Ubisoft's pedigree in online games was exclusively tiny, barely reliable systems. Think: Infrequently accessed NAT punching servers, minor single-use matchmaking to facilitate peer-to-peer client connections, leaderboards and the very occasional real game-server (but that was an oddity). The online subsystems needed to create a game like The Division</a> was a step well beyond what the organisation had ever done, the closest online system of note would be the despised always-online DRM system</a> internally named "Orbit" and Uplay which was also reviled. Ubisoft had built an organisation optimised for treating developers like fools and thus it had built itself into a corner; all processes were designed around the idea that hardware is a single size, that all needs are similar, that it's an afterthought and especially: that developers do not understand what is required, so don't let them set requirements. So believe me when I say we had to fight tooth and nail to get even an extra disk installed into our bare-metal machines. Data Consistency as a Requirement</h2> Games as a Service (puke) have an additional burden that may seem absurdly obvious but I think often goes overlooked: We are the arbiters and stewards of your player profile. We do not store your player profile on your console or PC, you never get to see it in its binary form, instead what we do is we pass your player profile from game-server to central-point to game-server. This process is actually fairly fast and involves some minor locking. I was given the task of ensuring that the data-storage of this thing is extremely performant and extremely durable. "Downtime is preferable to losing committed data." You can see why; imagine that you just completed an herculean task and had been rewarded with a highly coveted extremely rare prize, a dizzyingly low chance of being replicated. Well, if we lost that, you would be rightfully angry. My responsibility would be that we do not lose committed data. This might seem very easy to do actually, lots of people think that disks are relatively reliable, but when you're making the statement that "I do not lose data" and begin real investigation you will quickly find that many databases that are popular are totally fine losing data. MongoDB being the most famous example that I can think off of the top of my head. Others: like hbase, only ensure persistence to VFS</a>, as-in, they're not flushing their writes all the way to disk, they just assume that it's your operating systems responsibility now. Not comforting when you know that VFS is caching in memory.. Given our previous track record of building everything ourselves, I felt that this was probably the one place where you definitely do not want to do that, database maturity takes roughly 10 years, it's very risky - and my Linux administration skills have actual use when it comes to managing the most popular database systems as they run on Linux! At the time I joined we were using MySQL as the only backing store of the game; I spent 3 solid months dissecting MySQL, performance testing on "unrealistic" hardware to find internal locking bottlenecks, finding where it would lose data and under what conditions. The conclusion was mostly that MySQL can be convinced not to lose data but internal locking caused it to perform worse on many-core systems, PostgresSQL performed much better and had the additional benefit of being able to cleanly split write-ahead logs (which are largely sequential) and data to separate RAID devices. Something that MySQL doesn't really support and would have to be hacked in using Symlinks on every table create. PostgreSQL is robust in this regard, you can increase the commitment to guaranteeing data is persisted on disk further than most database engines, paired with disabling the performance giving "write-back" mode in the RAID controller and ultimately you will almost certainly never lose data, except for that one thing</a>. Any persistence you deem important should of course be backed up, so I began investigating off-the-shelf solutions for database backups for Postgres. Backups :: Enter PgBackRest</h2> After evaluating a few options (including some manual ones) I settled on a tool called PgBackRest</a> which had a bunch of interesting features, but the best part of it was the fact that it ensured consistency of your backups! I tested this and ordered the storage I would need to have a rolling 90 day window of backups (with older backups being taken off-site).. The hardware request was rejected. When I inquired as to why, I was told that Ubisoft has a standard backup solution which is replicated globally and sent to cold storage in a bank vault somewhere in Paris. I was told this is because we had lost some source code once upon a time and we could no longer build certain games because of that. -- Of course "Source code" was not even available in that network as we had a clear segmentation there, but I heeded the message. "That's fine", I said, "less for me to order!" I tested my solution with a couple of 400GiB SAS HDD's and it seemed well and good, so on I continued. When I eventually leveraged the right people to get access to this system (basically I was handed an IP and an instruction that it was NFS) it seemed very snappy, it was very quick to send data to it: I had even been given direct fibre lines attached to the database servers themselves, and in my testing I could completely saturate the drives I had been using for local backups. I was happy. Until the second day of using it. You see: PgBackRest is "smart"; it will read the data that you previously wrote to create "incremental" backups (looks like a full backup, so you only need to restore it and continue replaying WAL from that point, which means faster restores). This means that you can have your big backup once per day which locks the database and causes a little bit of a backlog and then you can have hourly incremental backups which take less disk space and are much cheaper to take. In order to generate and additionally verify these incremental backups PgBackRest must read data. The Storage Appliance</h2> Our backup appliance didn't like anyone reading data back from it, performance was abysmal, the only direct parable is AWS Glacier, but this was NFS and anyone who knows what NFS does when the remote is slow or unresponsive will tell you: this can kill your server. Linux will basically keep putting I/O operations onto the pending task queue and eventually everything will just fall over as the kernel spends all it's CPU time trying to evaluate what it needs to do next since the pending IO queue is full of things that are essentially just waiting, and the list just keeps growing. Think: load average 900. After talking to the storage admins, the architects, the managers, my managers, producers in increasing levels of agitation; one thing was clear: we will not buy dedicated hardware for storing backups, even if we cannot reliably make backups using the current system. I investigated alternatives, dumping the data directly to this system, but reading it back was impossible, our recovery times would be measured in weeks, not minutes or hours as was my objective. Eventually I found that this appliance was called a "DataDomain" and after reading the spec sheet, it was working as intended. "Rehydration" is an expensive operation for the device and it's meant for more long-term archival storage. If only I had known what my target was... When I pressed for why this was the case, why would you put the project at so much risk after spending hundreds of millions on a new game engine and a new IP (and it's marketing) and an entirely new online subsystem... The answer was simply: "We have spent $1M of Yves</a>' money, and it will look bad if you do not use it". Lost Data</h2> Ironically, and as if there was some deity wishing to vindicate me, a rogue and out of date game server node rose from the dead and began corrupting player profiles</a> shortly after. The backups I had been creating during my tests was the only reason we had the ability to restore those corrupted profiles (albeit they were older than I would have liked). Not long after: We got the hardware. What can I learn from this?</h2> There are times where buying what you need makes sense, it's reasonable to question though what features are a priority for the service or product you buy. Our EMC DataDomain system was optimised primarily for ingesting huge volumes of traffic, but if we want incremental backups then perhaps we needed a something a little less intelligent. Ubisoft had positioned itself into a strong position for a single type of workload and the organisation was unable to see any other way of working, I see some amount of echo's of this in our cloud providers and the way we all bend our workflow to fit the limitations presented (or, sometimes, not presented). Just because you spent $1M on a product because it fits the generic case does not mean it will fit every case. Which brings me to a comment that someone said recently, which inspired this little tirade; when someone says that Amazon has invested a lot of money into security</a> I think about the fact that Ubisoft spent $1M on a backup solution that didn't work for the game that would have had the best use of it. I wonder about what our providers really do with our money, since support is usually out of the question with cloud providers2</a>, I think about the fact that being opaque was a larger part of the problem than the thing itself being not fit for purpose -- it took months for me to even know the endpoint of my NFS IP endpoint was even called "DataDomain" -- and the fact that changing it was near impossible. The solution had to fail catastrophically first. It reminds me a little about that time that Amazon refused to tell me why my instances were unavailable because they were hiding a huge outage. I get the same vibe from these incidents. I wonder further about "build vs buy"- because the things we built, always worked.3</a> The only problem was unruly providers and the power they held over us. I don’t know what else to take away from this. An affectionate corruption of "Sysadmin", usually uttered by those who remember the times when Sysadmin was doing what "devops as a job title" folks do now, and before it was resigned to history as a helpdesk role. ↩</a> </li> I'm well aware you can pay for some level of support but then the cost of cloud goes from a mere 10x higher to a eye-watering 12-13x higher. ↩</a> </li> With the very notable and public exception of our rogue instance springing to life and murdering everyones profile. ↩</a> </li> </ol> </section> Microsoft Teams; using one monopoly to aid another 2022-02-02T00:00:00+00:00 The title is not going to be a surprise to anyone reading this, but I'm getting frustrated and I have to vent. At Sharkmob1</a> we use Microsoft products... not really surprising; Microsoft Office is extremely common and- we make games. What is nice about Sharkmob is that we get a lot more freedom to pick tools that work well for us, at Ubisoft2</a> it was often an uphill struggle to get anything paid for (so: things like SSO never happened, and you had a seperate account for everything! BAH). The reason I say this is because Teams being free for Office users is an immediate sell. It doesn't matter if anything is better or if teams is awful. (and I'm not saying that, even if I personally dislike it), but this is enough for Ubisoft to just use it without further reason. However, at Sharkmob we can have Slack enterprise, Github Premium, Zoom, 1Password, Jetbrains, whatever works best, and we use Okta SSO for literally everything. I love this, but apparently Microsoft do not, as they have not been chosen as the blessed instant messaging system, and they will do anything in their power short of pointing a gun at our heads to ensure Teams is used. Let me explain what I mean: I only really started noticing Teams being pushed quite heavily when we migrated to Office365 and AzureAD, suddenly Outlook would make every meeting a teams meeting automatically and you had to manually opt-out each time. They do warn you of this, but only if you hover over the little icon: And sure enough if you add an attendee: But I explicitly did not enable that, I unchecked the automatically checked box and pasted in my tasty little zoom link. However when I open my phone I'm greeted with: I discussed this with my IT department jmh></code> Can't we disable Teams with GPO? IT_01></code> We have our install of Office set to not install teams and we still find it getting pushed to machines... IT_02></code> Don't forget windows 11 forces teams, cant be removed. And its personal teams only, you cant login with a "professional account" on it . that is a a separate app </blockquote> </blockquote> On searching for "How to remove teams via GPO Windows 11" I was greeted with a lovely well instructed page which basically says: "You need to make a scheduled task that removes teams every day"</a> Hell, there's even an ugly "Teams Call" text button in the web version of outlook; you can't escape it. I might be in the minority: but I don't like being told what to do, so I'm going to find some way of sabotaging the Teams installation mechanism for all machines at Sharkmob. Sharkmob is a studio and publisher of video games, and we're hiring: https://career.sharkmob.com/ ↩</a> </li> Ubisoft is the 3rd largest publisher of AAA games globally, responsible for such franchises as Far Cry</a>, Assassins Creed</a> and Tom Clancy</a> games. I worked at Ubisoft Massive supporting The Division 1 and it's sequel. ↩</a> </li> </ol> </section> DevOps; a decade of confusion and frustration 2021-06-29T00:00:00+00:00 What is "DevOps"? is a question I've heard a lot, often I've asked it implicitly to myself when reading job ads for "DevOps Engineers". According to Patrick Debois, a Belgian "agile" consultant and former sysadmin who coined the term in January of 2009: the term "Devops" (not "DevOps") was supposed to be "Agile System Administrator".1</a> it is a compound of "Developers"+"Operations"+"Days"2</a> and was not intended to be a methodology by that name or even a job title.1</a> There are some who refer to this talk, titled "10+ Deploys Per day"</a> as the true origin of the DevOps methodology, and primary launch pad for the name. The talk regales the challenges in contemporary companies, especially young ones: Operations staff were oft instructed: "Do not break the site or let it get broke.. We consider slowness is broke-ness", while, contrarily Development staff were told: "Move fast, deliver value", and both were measured on success of these objectives. Now, of course moving fast means breaking things, a tradeoff Facebook fully embraced in its old mantra; "move fast and break things"</a> (though we did not presume they were talking about the very fabric of democracy); but even they came back to the notion that reliability is important, and changed their mantra to "Move Fast With Stable Infra."</a>. </a> "But, what if..." said many people, who did not enjoy this fighting between reliability and feature development: ".. we put developers and operations staff together to make DevOps!?", and thus the silo-slash-wall was broken down and everyone rejoiced. Except of course, that's not what happened.</h2> Operations staff were renamed to be devops. </li> Developers were renamed to be devops. </li> What we might call "release engineering" was also renamed devops. </li> Whole teams of people with a singular discipline: devops </li> All engineering: Devops </li> </ul> Now you could be a junior sysadmin or a senior backend programmer only fluent in the most arcane elements of .NET and somehow you're both: "DevOps" at least to someone. There are also other methodolgies based on this: DevSecOps</a>, ArchOps</a>, TestOps</a>, CloudOps</a> DevOops</center> Lack of Cohesive Vision</h2> DevOps as a movement has no truly coherent definition or mantra other than removing the wall and a few images of a horizontal lemniscate3</a>. People make their own definitions and it differs from person to person. Additionally; the keen eyed (or ancient) among you may have noticed though that "the wall" was not always inherently bad, as long as teams had shared ownership of the product and a clear contract of concerns then velocity could be achieved. This brings me to SRE4</a>, which, ironically had existed long before DevOps. SRE was founded as "Production" in 2003 at Google. Described by its founder, Benjamin Treynor Sloss as "What happens when you send Developers to do Operations"</a>. The whole video is worth a watch but tl;dr: he codifies the contract between Feature Development and SRE ^{(which, are other developers doing Operations work as stated). This does not break "the wall", it is promoting shared ownership and shared responsibility, but SRE is not embedded in development and is not putting perpetual Ops support onto staff whom are focused on developing features. SRE only codifies and formalises contracts of what it means to hand something over, or to control releases. In my mind: this is a true evolution of Operations. But it should be noted that this was business as usual for a lot of established companies; tenured sysadmins generally grew into being quite close in ability to feature developers, many even becoming developers themselves, the inverse was also true with feature developers joining what was often referred to as "Platform Ops"} Benjamins talk is fantastic for other reasons too; he also critically points to the fact that you need people that can be free to focus on reliability. He indicates that developers/coders are the best people served to do this and I tend to agree with him, we should all be comfortable scripting/programming. I was headed in a direction and I'm just going to go there now: SRE is a nice concept and adds some good ideas to Operations, but it's still operations. You're still a Sysadmin. Yes, Sysadmins could code *gasps from the audience*. In fact sysadmins generally code some of the more gnarly stuff that keeps a business running, database migration systems, failover systems. Chatops as a concept, for example, is founded not from Development but from Operations who used to use IRC to instrument changes. It is Operations who pushed the idea of message queues for reliability reasons at my previous jobs. Developers at that company would have used MySQL for everything. and.. devops... as a job title.. Well, my opinion is: If you're a devops who can't sort a binary tree, you're probably a sysadmin.5</a> </li> if you're a devops who can't forcefully unload a stuck file descriptor from a running process, you're probably a developer.6</a> </li> if your job is to maintain CI pipelines: you're a release engineer. </li> </ul> And there is no shame in these job descriptions or titles.. it's just saying what the focus is. As I mentioned to my directors when they were keen on me hiring "devops engineers": DevOps, are not Developers who can install apache and read infra docs. </li> DevOps, are not Sysadmins who learned Python. </li> DevOps is not the people who run your Jenkins pipelines. </li> </ol> Personally; I'm sick to absolute death of hearing "DevOps Engineer", it tells me nothing about what you do, how you do it, how you approach problems or what your real responsibilities are, it doesn't even tell me what tools you use although there are definitely some "DEVOPS TOOLS!!1!" which are trying to make you believe you need to be using them if your title is devops engineer.. It's akin to a doctor titled: "BodyFixing Engineer"; doctors specialise because the problem scope of a human body is wide, and when they generalise they misdiagnose.. unfortunately quite often</a>.. and even amongst doctors generalists have a specific title indicating their focus (General Practitioners). There also seem to be misconceptions floating around, especially on hackernews and lobsters: Sysadmins did know how to code back in the day, they were coders, it's weird to assume they didn't code. If your sysadmin didn't know your program it's because they weren't allowed (compliance) or there was a cultural problem in your company, it's unlikely that your new "DevOps Engineers" know the codebase if your company was one of these. </li> Developers knew a fair bit about infrastructure back in the day, if only to get their staging/dev environments to work. It is weird to assume they didn't. </li> </ol> Today, life is a little bit easier on either side of these camps: but please stop assuming you can do everything, you will only do both poorly. And if you're a hiring manager and you only want to hire "DevOps" from only from infrastructure backgrounds or only from software engineering backgrounds: you are actually hiring sysadmins or developers; using the term "DevOps Engineer" only serves to remove information about the focus of the role. It is likely increasing the amount of noise in your hiring pipeline. Source: https://newrelic.com/blog/nerd-life/devops-name ↩</a> ↩2</a> </li> Devopsdays is a worldwide series of technical conferences covering topics of software development, IT infrastructure operations, and the intersection between them. Each event is run by volunteers from the local area. ↩</a> </li> The geometric symbol of infinite. ∞ Example: devopedia</a> ↩</a> </li> Site reliability engineering is a set of principles and practices that incorporates aspects of software engineering and applies them to infrastructure and operations problems. The main goals are to create scalable and highly reliable software systems. ↩</a> </li> Not specifically this, "why would anybody want to do this!?" but distributed debugging, consistency of data, continuity of service and scalability is a difficult problem to do right, if you know more about this than about the intricacies of memory padding, then this applies to you. ↩</a> </li> Overly simplistic and arbitrary, but the point is you probably know more algorithms and coding standards than infra standards. ↩</a> </li> </ol> </section> I don't trust Signal 2021-06-21T00:00:00+00:00 I'm sure you have already formulated an opinion about how I'm wrong. That's fine, but I invite you to at least open your mind a little before you hit back and inform me of how stupid I am. After the hackernews reaction I should also preface this post by saying that the title should really be "I don't inherently trust Signal". This is an important point because nothing of what I talk about here is a reason to not use Signal by itself; it just lends a skeptical person to the conclusion that there's no concrete reason to trust them, and that ultimately Signal makes it hard to function the way they do without half-blindly trusting them. </blockquote> This is decidedly not a rehashing of Drew Devaults</a> essay of the same name, he mostly talks about Google Play and Federation. I am here to talk purely about trust, about how it's something you verify- something that is hard earned, something you try to avoid giving, that's easily lost and worries you when people forcefully ask for it. Let me start with a perhaps controversial statement: I do not believe that end-to-end encryption means anything at all when the network and the client are the same entity. </blockquote> What do I mean? Well, back in the old days, by pure virtue of not having large companies that could do everything, we used to bolt on security mechanisms to insecure transports. PGP</a>, OMEMO</a> and the foundation of Signals encryption</a> OTR are all known primarily for being developed as third-party client side implementations. -- for everyone else TLS was good enough since if you trust MSN or AIM (the client) then you trust Microsoft or AOL, right!? What running an encryption overlay means in practice is that your transport could never collude against you with your client; better: it usually means multiple client implementations of the same standard (though not always; in the case of PGP for example) -- and often the clients are fully open source. That leads directly into two of my next grievances: Signal is not open source</h2> Why would I say something so provably untrue? "Of course signal is open source, it's on f-droid! (it's not, actually1</a>); there are even sources</a> on github!" ... I can already hear it coming. How is it then dear reader, that they developed MobileCoin integrations for over a year without anyone knowing? That would be because, they stopped updating sources</a>. We can be reasonably sure that private & unpublished code was in production, otherwise they left some security vulnerabilities unpatched</a> for a long time2</a>. This throws into question the entire nature of what they consider "open source" to mean, they are clearly comfortable deploying non-public software. It's also vanishingly small amounts of people who will use the from-FOSS versions of the client, nearly everyone will be downloading it from Google Play or Apple's App Store; and they have a long way to go when it comes to verified builds</a> which seems to work when you google it and there's a page; but in reality if you read the page you'd realise is not possible. Which gives a false appearance in my opinion, and that is a large part of my issue honestly; that there is a surface level of "everything is by the book" but underlying it all is: nothing, really. Signal doesn't give you any option to verify their claims</h2> If I were in a situation to be signal, if there was a competing implementation that I could point my clients to (similar to how headscale</a> is an implementation of tailscale</a>'s control server); I'd certainly be a lot more comfortable, since then I could be in a situation where I can see all traffic to my server and jail/inspect all traffic coming from the binary distributed Signal client; thus it would allow for independent verification of the binary distributions delivered via Play or the iOS App Store. As it stands the whole thing is built on trust and people believe that someone else will do the hard part of reverse engineering every version. Which I don't have to tell you is significantly more effort, requires much more advanced skills and might not even yield results even if there were concerning items yet to be discovered. "Moxie says you can run your own server though!"3</a>; I'd like to see where I can change the endpoint in the signal app that's distributed via Play or App Store; my claim is purely that I can't verify those and that few enough people run the custom compiled versions to be meaningful. If I was to be smart and want to hide a back door I'd only need one side of every conversation. -- please note though, I'm not saying they do this, I'm just saying that they could do this and the only thing that says they don't is "trust me". Signal is fairly hostile to any other clients</h2> OK, so, it's unlikely you run a from-source client, it's less likely everyone you know runs a from-source client. It's less likely that everyone you know audited it-- but that's easier than reverse engineering of course. However something that could increase trust is to decouple that client/network collusion possibility, perhaps by having independent clients based on a spec. Moxie has explicitly said several times that third-party clients connecting to the main Signal servers are actively not supported and has threatened to start blocking them or enforcing the Signal trademark if they get big enough4</a>. Signal took money from the US Government</h2> I've heard the argument about this, NSA</del> OTF</a>5</a> funds loads of projects, "You're being a conspiracy theorist Jan!" Sure. NSA gave us SELinux, NRL gave us Tor (which the CIA loves</a>), sometimes the stars align and the security services actually release something that makes us more secure. However I still find Signal an odd choice, it's not inherently better than any other client that supports OMEMO, including Jabber clients. The only things it's better in is that it's a foundation that is under US jurisdiction- it was founded around the same time as Telegram which was likely seen as a competitor- and... it has good marketing? I don't honestly see any reason to fund Signal over anything else. Additionally: Tor and SELinux genuinely are used by their respective agencies, yet Signal is not being used by NSA. I know this for fact.</del>6</a> Signal seems to have a lot of strong and emotional advocates</h2> This is also conspiratorial, but if you take my first point as fact: that E2EE is meaningless if the client and the network are the same; then Signal seems to have a lot of people foaming at the mouth on popular sites like Reddit and HackerNews doing everything possible to convince you that it's the one true secure messenger. If you do anything more than what signal provides: you're paranoid and probably doing it wrong anyway; if you do anything less or god forbid you use something like Telegram; you might as well telegraph all your messages to every person on the planet! Ok, I'm being hyperbolic; but there is a really strong sentiment that cannot be argued or reasoned with (especially on hackernews), and legitimate complaints are brushed aside with snide remarks about paranoia or trust or that you're not doing enough for privacy. Which, if you really do buy my first argument: feels massively disingenuous. Signal requires a phone number</h2> I know, this ugly thing. People say that it's to combat spam. Unfortunately you know what else it combats: basically anybody being able to register with signal without disclosing their ID to someone. Even more annoying is that locating someone via phone number is pretty trivial if you have the right equipment or you have the ability to ask a carrier. Heck, that's how they got Mitnick</a>. I am really not a major privacy nut, and when you get to the end of this blog post you'll see just how true that is, but my point here is simple: You cannot claim to be running a secure messenger and have your only method of connecting with other person be a globally unique number that is easily tied to a real world person. Physical security is a pretty major part of security. They say they're working on this, and someone mentioned something recently about a very complicated command-line, I haven't looked into it any further honestly -- However people are definitely advocating on hackernews and reddit to keep the phone numbers because (and I quote: "Keeping the numbers makes it easy, if I wanted usernames I would use Riot/Element"). The hagiographies of Moxie</h2> OK, I actually have a soft spot for Moxie, he gave a talk on not trusting CA's and instead developed (a now defunct) system that used multiple third party brokers to act as notaries. It was called Convergence</a>7</a>. One of the things I really liked about that approach was that it inherently didn't trust the "authority". Now it seems Moxie really likes the idea of authority, so long as it's his foundation. I'll be honest, despite me having a soft spot for Moxie, I am inherently distrustful of being told what to think, I am even more distrustful of anything that uses emotive language (such as Fox News or the Daily Mail) in order to illicit a particular feeling on the state of the world. When I read articles like his profile in The New Yorker</a> I am left thinking: Who paid for this? Why? From everything I personally know about the media, articles like that are usually paid for, though almost never directly8</a>. And it goes back again to "authority" for me; I'm being told to trust this guy, this foundation, that they've got the right moxie^haha^, that they're in it for good reasons. But, only if they're the authority. You could argue that Convergence, the anti-authority system, is defunct and thus his new approach is more poised for success as he has learned that authorities are good; and honestly I wouldn't have a good argument against that. It's possible. Coincidentally though the best form of government is absolute dictatorship; so long as the dictator is benevolent. it says nothing about future corruptibility... which brings me to my final point: Signal wants to move fast9</a></h2> This, is the common argument used against federation, and when I first read it I thought that basically they want the ability to forcefully change the software and protocol actively used for users without any consent (much less informed consent), which renders it functionally immune to any criticism or review because any aspect of the protocol could be changed ('improved') at a moments notice. Final Word</h2> OK, I talked about trust, I don't think any individual issue I've mentioned here is a dealbreaker, and most in isolation can be argued away. For me, though, in the larger context with all these pieces I can't really say that I have full faith in Signal. It's fine for me as an insecure messenger, but the UX is just worse than other insecure messengers. I don't personally have any reason to trust it more than telegram; other than that people get mad when you say that. Which, is incredibly unconvincing. I mean, we have an ecosystem that: Can change at a moments notice10</a>; and works hard to keep it that way;</li> Attempts to avoid you extending their messengers;</li> Is centrally controlled;</li> Handles all traffic (via the USA, no less);</li> Took money from US intelligence agencies;</li> Is not used by at least one US intelligence agency that I know of;</li> Has engaged hiding updates before;</li> Can be easily tied to your person;</li> Asks for your contact list and "encrypts" them in a way that is trivially broken11</a>;</li> </ul> Those things combined, with the strong push that it is truly the "secure" messenger gives me enormous pause. Telegram might be cryptographically flawed12</a> and does not have E2EE enabled by default; but you know what it has? An open protocol, third-party clients, accounts without phone numbers, it's eas(y|ier) to use- and if I get paranoid: fuck it, I'll customise one of their open source clients to use OMEMO. Ironically the messenger which is widely thought to be less secure has a similar enough trust stance but is open enough to actually be more secure... Or maybe we should all chip in with what Matrix/Element are up to, instead of allowing these walled garden authorities to exist with "trust me bro" marketing and a cool looking hacker dude as the frontman being the only major selling points. (yeah, you too Telegram) Thanks to Signal's centralized model, implementations of backdoors are one (perhaps even targeted) software update away. By the time the "nerds" find out, it'd probably be far too late and lives could be at stake. It's unfortunately such the nature of the beast that being half-hearted about security does not yield a half-secure product, or a product that's fully secure against half the hostile actors, it yields a product that only gives the presumption of safety, which is far more dangerous. I use many messaging services in my life as security absolutism leads to a miserable, paranoid life, but my expectations are accordingly tempered when I use them, and I let my contacts know my expectations too. Everyday chat? Sure. Sensitive, personal info? Maybe, depends on the exact topic. Trade/state secrets (if I were to handle them)? Hell no. If Signal's security boils down to reputation and community trust, why not just use WhatsApp or Facebook Messenger or really any chat product where the makers claim it's secure and private? Common Responses</h3> "Who do you trust? they literally do all they can." </blockquote> Commercially, truthfully, I only really trust Mullvad; they do everything possible to keep their customers anonymous even to themselves and are pushing open source systems transparency that we can all benefit from even as non-customers of the service. Otherwise; I will always have more trust toward the people who create FOSS that is easy to use by yourself or allows you to be independent from central control which aids in making you a small enough target that the government isn't going to knock your door. Federated or independent alternatives that live outside centralised control, that do not need to consider nation state threats because they are practically anonymous if used correctly. "it's easy to criticise, what have you built!" </blockquote> IDK, games and stuff</a>. I'm aware of how people can be overly critical; but Signal invited these problems and has been ignorant or dismissive of peoples concerns on these points. My point is mainly that we should probably be funnelling money, resources and time to things like Matrix/Element -- rather than the authoritarians. Or otherwise that Signal's tradeoffs don't amount to more than trusting them. Which for me defeats the whole point. "They make trade-offs to make it easier" </blockquote> Yep, but if you're willing to take those tradeoffs there's no difference between a fully E2EE messenger program where the client and network are the same entity; and a TLS connection to the network with any client. It's the same level of trust. "x problem is so small/x problem is not an issue!" </blockquote> Unfortunately all of these points are dots on a line, none big enough to cause any real distress, most easy to hand-wave away. However when put together they paint a picture: a picture of a company that is being heavily promoted but who's principles boil down to ensuring they are in total control. There is no straw that breaks the camels back here, none of these are strong objections, just minor and quite numerous. Handwave as many of these issues away as you'd like, I'm certain there are some core factual errors here. At the core of it though: I contend that E2EE means nothing if the clients and the network collude are one entity- and I see no true external reason why I should trust Signal more than anyone else if you consider that statement to be true. "You are wrong about y". </blockquote> Yeah, probably, this is my outside observation, I wrote this at 2am on a night when I couldn't sleep and someone argued passionately in favour of Signal over everything else (to them: more paranoid solutions are completely unusable, less paranoid solutions are tantamount to a billboard in times square!) and I got frustrated. Not everything is post is fully fact checked, mostly it's based on gut feeling. I can be wrong about any and all of this article but the problem is that the core of the argument doesn't lie in the semantics. The issue is one of centralised control and the notion that E2EE is functionally useless if your client can be updated randomly. This post is a collection of minor grievances or feelings that make me not trust them more than other providers. It's not on f-droid and Signal has "preferred"</a> to avoid it's inclusion there; https://forum.f-droid.org/t/signal-on-f-droid/13742 ↩</a> </li> HN Thread discussing this: https://news.ycombinator.com/item?id=26717134 ↩</a> </li> Moxie says to run your own network (regarding federation): https://news.ycombinator.com/item?id=12883410 ↩</a> </li> https://github.com/LibreSignal/LibreSignal/issues/37#issuecomment-217339450 ↩</a> </li> OTF has a long and storied history as "Radio Free Asia" which is a propaganda outlet</a>. It was devised under Secretary of State Hilary Clinton who coincidentally endorses Signal</a> and is featured heavily in the book Propaganda, Power and Persuasion: From World War I to Wikileaks</a>. ↩</a> </li> NSA not using Signal is completely irrelevant, I had erroneously "remembered" that NSA funded Signal, however in reality it was OTF</a> who donated to the Signal Foundation. ↩</a> </li> Convergence was presented at BlackHat 2011: https://www.youtube.com/watch?v=Z7Wl2FW2TcA ↩</a> </li> To be clear about payment for articles, it's often not the case that established journals will directly take payment for an article. It is much more common to take lunches with PR firms whom you've hired, for your marketing spend to drive up your profile in other ways and it always helps to be close to an advertiser in some way. This is not conspiracy, that's just how the media machine works. Drive up your image, someone will write about you. Be it via PR lunches or marketing spend. ↩</a> </li> https://signal.org/blog/the-ecosystem-is-moving/ & https://news.ycombinator.com/item?id=17172203 ↩</a> </li> Signal updates without consent: https://github.com/signalapp/Signal-Desktop/issues/4578 ↩</a> </li> they use SGX</a> which is broken</a> ↩</a> </li> there is no evidence or reason to believe that is the case, there was a great outcry when Telegram launched that they had home-rolled their own crypto; after getting a security review they appear to have fixed four flaws that were discovered: https://ethz.ch/en/news-and-events/eth-news/news/2021/07/four-cryptographic-vulnerabilities-in-telegram.html ↩</a> </li> </ol> </section> GPG::SSH; notes for current best practices 2020-08-16T00:00:00+00:00 When I start at a new company, I always do a refresher on my key security. One thing I always hate about SSH is that the encryption scheme is pretty basic actually, and once your ssh-agent is loaded- anything can just request a sign/authorize. So, in tried and true "over engineering" fashion, I've taken to using my GPG key as my ssh key instead, and using gpg-agent instead of ssh-agent. Another thing is to use elliptic curves instead of RSA, RSA is still secure, but ECC (ECDSA) is faster and theoretically more resistant, and everything from 2016 onwards supports it, so it's fair to assume it is supported in my SSH programs of choice. :) First, to create a ECDSA key we have to use expert mode with the --full-gen-key</code>: jan.harasym@sm-mbp-jmh ~ % gpg2 --full-gen-key --expert gpg (GnuPG/MacGPG2) 2.2.20; Copyright (C) 2020 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Please select what kind of key you want: (1) RSA and RSA (default) (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) (7) DSA (set your own capabilities) (8) RSA (set your own capabilities) (9) ECC and ECC (10) ECC (sign only) (11) ECC (set your own capabilities) (13) Existing key (14) Existing key from card</code></pre> We want to use ECC and ECC here, or "ECC (set your own capabilities)" and skip to the addkey</code> section. Your selection? 9</code></pre> Next we choose our ciphers: Please select which elliptic curve you want: (1) Curve 25519 (3) NIST P-256 (4) NIST P-384 (5) NIST P-521 (6) Brainpool P-256 (7) Brainpool P-384 (8) Brainpool P-512 (9) secp256k1</code></pre> NIST P-521 is the strongest, but NIST P-256 will be more compatible. Your selection? 5</code></pre> And finally expiry and user-info. Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) 0 Key does not expire at all Is this correct? (y/N) y GnuPG needs to construct a user ID to identify your key. Real name: Jan Martin Harasym Email address: jmh@xxx.com Comment: Expert Online Infrastructure Engineer :: Live Operations You selected this USER-ID: "Jan Martin Harasym (Expert Online Infrastructure Engineer :: Live Operations) <jmh@xxx.com>" Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O <...> pub nistp521 2020-08-16 [SC] 6F1AA563C75BA41387FDDAD7DE3F72240989604A uid Jan Martin Harasym (Expert Online Infrastructure Engineer :: Live Operations) <jmh@xxx.com> sub nistp521 2020-08-16 [E] </code></pre> Now we need to add a key which can be used for authorization, for this we can create a subkey, remove the encryption capability and enable the authorization one. (Failing this, we can ask for setting our own capabilities during the initial key creation above) Make sure it's ECC (set your own capabilities)</code> when selecting a key type. jan.harasym@sm-mbp-jmh ~ % gpg2 --expert --edit-key 6F1AA563C75BA41387FDDAD7DE3F72240989604A Secret key is available. gpg: checking the trustdb gpg: marginals needed: 3 completes needed: 1 trust model: pgp gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u sec nistp521/DE3F72240989604A created: 2020-08-16 expires: never usage: SC trust: ultimate validity: ultimate ssb nistp521/697423BAAFF6B653 created: 2020-08-16 expires: never usage: E [ultimate] (1). Jan Martin Harasym (Expert Online Infrastructure Engineer :: Live Operations) <jmh@xxx.com> gpg> addkey Please select what kind of key you want: (3) DSA (sign only) (4) RSA (sign only) (5) Elgamal (encrypt only) (6) RSA (encrypt only) (7) DSA (set your own capabilities) (8) RSA (set your own capabilities) (10) ECC (sign only) (11) ECC (set your own capabilities) (12) ECC (encrypt only) (13) Existing key (14) Existing key from card Your selection? 11 Possible actions for a ECDSA/EdDSA key: Sign Authenticate Current allowed actions: Sign (S) Toggle the sign capability (A) Toggle the authenticate capability (Q) Finished Your selection? S Possible actions for a ECDSA/EdDSA key: Sign Authenticate Current allowed actions: (S) Toggle the sign capability (A) Toggle the authenticate capability (Q) Finished Your selection? A Possible actions for a ECDSA/EdDSA key: Sign Authenticate Current allowed actions: Authenticate (S) Toggle the sign capability (A) Toggle the authenticate capability (Q) Finished Your selection? q Please select which elliptic curve you want: (1) Curve 25519 (3) NIST P-256 (4) NIST P-384 (5) NIST P-521 (6) Brainpool P-256 (7) Brainpool P-384 (8) Brainpool P-512 (9) secp256k1 Your selection? 5 Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) Key does not expire at all Is this correct? (y/N) y Really create? (y/N) y We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. sec nistp521/DE3F72240989604A created: 2020-08-16 expires: never usage: SC trust: ultimate validity: ultimate ssb nistp521/697423BAAFF6B653 created: 2020-08-16 expires: never usage: E ssb nistp521/5A4C21BF1FC787DB created: 2020-08-16 expires: never usage: A [ultimate] (1). Jan Martin Harasym (Expert Online Infrastructure Engineer :: Live Operations) <jmh@xxx.com> gpg> Save changes? (y/N) y</code></pre> Tell gpg-agent to enable ssh support: echo > ~/.gnupg/gpg-agent <<EOF default-cache-ttl 600 max-cache-ttl 7200 enable-ssh-support write-env-file ~/.gpg-agent-info EOF</code></pre> And finally we need to tell the gpg-agent which key we want to load, however this is done by using the 'keygrip', not the 'keyID', you can get the keygrip with the following: jan.harasym@sm-mbp-jmh ~ % gpg -K --with-keygrip /Users/jan.harasym/.gnupg/pubring.kbx ------------------------------------- sec nistp521 2020-08-16 [SC] 6F1AA563C75BA41387FDDAD7DE3F72240989604A Keygrip = 1114A87C954A99D1BE2BBBCEFD2FEF4A8F81A17B uid [ultimate] Jan Martin Harasym (Expert Online Infrastructure Engineer :: Live Operations) <jmh@xxx.com> uid [ultimate] [jpeg image of size 8037] ssb nistp521 2020-08-16 [E] Keygrip = 8E504AC5A41C7C71905604E83B2F150B562ADFAE ssb nistp521 2020-08-16 [A] Keygrip = 1DB1E97B20FD54DF2BAB906EA64C30081DEA8C32</code></pre> Look for the keygrip which is associated with the [A]</code> capability in my case: ssb nistp521 2020-08-16 [A] Keygrip = 1DB1E97B20FD54DF2BAB906EA64C30081DEA8C32</code></pre> And enable the "A" key by adding it to the bottom of the sshcontrol file: echo '1DB1E97B20FD54DF2BAB906EA64C30081DEA8C32' > ~/.gnupg/sshcontrol</code></pre> EDIT: AWS does not like strong keys, so for AWS I did the same as above but when I added a key for authorization I chose RSA:4096 instead. jan.harasym@sm-mbp-jmh / % gpg2 --expert --edit-key 6F1AA563C75BA41387FDDAD7DE3F72240989604A gpg (GnuPG/MacGPG2) 2.2.20; Copyright (C) 2020 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Secret key is available. sec nistp521/DE3F72240989604A created: 2020-08-16 expires: never usage: SC trust: ultimate validity: ultimate ssb nistp521/697423BAAFF6B653 created: 2020-08-16 expires: never usage: E ssb nistp521/5A4C21BF1FC787DB created: 2020-08-16 expires: never usage: A [ultimate] (1). Jan Martin Harasym (Expert Online Infrastructure Engineer :: Live Operations) <jmh@xxx.com> [ultimate] (2) [jpeg image of size 8037] gpg> addkey Please select what kind of key you want: (3) DSA (sign only) (4) RSA (sign only) (5) Elgamal (encrypt only) (6) RSA (encrypt only) (7) DSA (set your own capabilities) (8) RSA (set your own capabilities) (10) ECC (sign only) (11) ECC (set your own capabilities) (12) ECC (encrypt only) (13) Existing key (14) Existing key from card Your selection? 8 Possible actions for a RSA key: Sign Encrypt Authenticate Current allowed actions: Sign Encrypt (S) Toggle the sign capability (E) Toggle the encrypt capability (A) Toggle the authenticate capability (Q) Finished Your selection? A Possible actions for a RSA key: Sign Encrypt Authenticate Current allowed actions: Sign Encrypt Authenticate (S) Toggle the sign capability (E) Toggle the encrypt capability (A) Toggle the authenticate capability (Q) Finished Your selection? S Possible actions for a RSA key: Sign Encrypt Authenticate Current allowed actions: Encrypt Authenticate (S) Toggle the sign capability (E) Toggle the encrypt capability (A) Toggle the authenticate capability (Q) Finished Your selection? E Possible actions for a RSA key: Sign Encrypt Authenticate Current allowed actions: Authenticate (S) Toggle the sign capability (E) Toggle the encrypt capability (A) Toggle the authenticate capability (Q) Finished Your selection? Q RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) 4096 Requested keysize is 4096 bits Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) Key does not expire at all Is this correct? (y/N) y Really create? (y/N) y We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. sec nistp521/DE3F72240989604A created: 2020-08-16 expires: never usage: SC trust: ultimate validity: ultimate ssb nistp521/697423BAAFF6B653 created: 2020-08-16 expires: never usage: E ssb nistp521/5A4C21BF1FC787DB created: 2020-08-16 expires: never usage: A ssb rsa4096/7085DF63EBE406FD created: 2020-08-20 expires: never usage: A [ultimate] (1). Jan Martin Harasym (Expert Online Infrastructure Engineer :: Live Operations) <jmh@xxx.com> [ultimate] (2) [jpeg image of size 8037] gpg> save jan.harasym@sm-mbp-jmh / % gpg -K --with-keygrip /Users/jan.harasym/.gnupg/pubring.kbx ------------------------------------- sec nistp521 2020-08-16 [SC] 6F1AA563C75BA41387FDDAD7DE3F72240989604A Keygrip = 1114A87C954A99D1BE2BBBCEFD2FEF4A8F81A17B uid [ultimate] Jan Martin Harasym (Expert Online Infrastructure Engineer :: Live Operations) <jmh@xxx.com> uid [ultimate] [jpeg image of size 8037] ssb nistp521 2020-08-16 [E] Keygrip = 8E504AC5A41C7C71905604E83B2F150B562ADFAE ssb nistp521 2020-08-16 [A] Keygrip = 1DB1E97B20FD54DF2BAB906EA64C30081DEA8C32 ssb rsa4096 2020-08-20 [A] Keygrip = 933F367496118EF7CB6C457C48011DD859AA215A jan.harasym@sm-mbp-jmh / % echo "933F367496118EF7CB6C457C48011DD859AA215A" >> ~/.gnupg/sshcontrol</code></pre> sources: https://www.gnupg.org/faq/whats-new-in-2.1.html#ecc https://gregrs-uk.github.io/2018-08-06/gpg-key-ssh-mac-debian/ https://www.linode.com/docs/security/authentication/gpg-key-for-ssh-authentication/ Cloudflare is turning off the internet for me 2020-01-21T10:01:00+00:00 Ok, I'll admit, I'm not the largest fan of centralisation, but rarely do I so swiftly and effectively feel the crushing weight of it. I happen to use a very nice Chromium-based web-browser</a> which, when it opens has javascript disabled. Often I find that nothing works so I re-enable javascript and continue about my day. This morning I went to work, as normal, turned on my laptop and as my laptop dutifully reloaded all my tabs from the day before I saw a few sites error-ing out. This is relatively common when I haven't connected to the network yet, or some sites which don't even attempt to load without javascript, so I check my connection, enable javascript and went about reloading the offending pages. But I noticed quite a few of the pages were the following: Let me copy that for those who don't like to read images: Sorry, you have been blocked </blockquote> Why have I been blocked? This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data. </blockquote> What can I do to resolve this? You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page. </blockquote> Ok, so I've been outright blocked, no captcha, and a suggestion to email the owners of the sites... which could be difficult given that I can't actually reach the sites to find the site-owner for each one. I figured this might be because I had javascript disabled and tried to load a few pages, so I refreshed the page, noted that no captcha appeared and continued my day and figured the ban would eventually be lifted; Noting that I should keep my browser open so that it doesn't happen again when I get home. home</h2> Ok, so I got home 30 minutes ago and I'm following some reddit links from /r/irc, when I get a "You have been blocked message". "Oh," I thought. "I forgot about that." So now I'm curious as to why, and what I can do about it. Any ideas? Please help</a> EDIT: if my blog goes down; let it be known that I tried to read the docs on nginx.org but was not able: EDIT2: plot thickens, it seems to work fine in chrome. EDIT3: Conclusion!</h1> I had set a custom user agent string (D'OH!)</a> some time ago, and forgot about it. Which is now biting me on the butt. Ironically I set that user agent string before because Google was not allowing me to log in to my account</a> anymore, I got that working but now half the internet doesn't work for me. Kudos to RyanK24 on HackerNews for following up. Much, much appreciated. ¯\_(ツ)_/¯</code></pre> How to survive an open office. 2019-07-18T04:30:00+07:00 I've been struggling for some time to find a decent enough guide to actually accomplish anything meaningful (other than ad-hoc break-fix work) in my office. One of the things I know is that this problem seems to affect me more than others, so for many people this advice (or lamentation) might seem like it comes from a weird place. Especially since this is the first-worldiest of first-world problems. However, for me, if I have some work that really must be done I end up doing it at home. When I'm in the office I just work on things as I get interrupted and cannot possibly focus on anything for more than 15 minutes. When I go home I feel exhausted, if I have enough energy I do the work I really needed to do in the day. -- it's not even that I don't have the time to do it during the day, it's just that I end up procrastinating because I can't get focused. Ironically I get significantly more accomplished for my job on vacation. Anyway, in my searches for a way to cope I came across this video</a> from "The Globe and Mail" where a young lady by the name of Leanne Devi (VP of Knightsbridge leadership solutions) gives some advice. https://youtu.be/DDlJ5zrnc5I?t=271 Her opening statement and a brief of her comments: One of the big issues we see all the time is productivity in modern offices, so we've got more stress than ever, more to accomplish but increasingly we're using open office spaces to create collaboration. The problem is, as soon as you try and get work done in an open space like that, distractions are everywhere. </blockquote> Her advice is: So one of the things that I really recommend to people is that you Create an imaginary office even when theres' no door to actually close. There are several ways to do that, in our office we've done that with those really great noise cancelling headphones. </blockquote> Her other points fall under: Have a unique ringtone </blockquote> So that when a phone rings you know if it's yours or not. (idk anyone who leaves there ringer on in 2019 though) Learn hand signals </blockquote> Not quite sign language, but more polite gestures. Avoid conversations </blockquote> She says "in the main area" but, let's be real here, it's all the main area. -- Now, I know this advice is from 2014 (4.5 years old at this point), but her advice is basically all I ever really hear, aside from the ringtone thing. There's a bunch of reasons the above advice is just, really bad, and personally I find the suggestions of "avoiding conversations" to be completely contradicting to the supposed objective of open office space. I'm not going to go into "why they suck" or "what is a better office" because; if you're reading this, then you've almost certainly not got the backing of management. And ultimately this is their call. Although if you're in tech I would highly advise mentioning that you need more money if there's an open office in your next interview (somewhere around 20% as that's what the estimates say they're saving in the short term). Personally I don't see any upside to open office aside short-term cost savings</a>, since studies have consistently shown that over the long term the cost is higher</a>, taking into account lost productivity from sickness</a>, distraction</a> and worker attrition</a>. Headphones</h2> Headphone use has obvious benefits for enhancing your sound privacy, however the drawbacks are that: your ears get warm</li> they don't fully isolate sound</li> they compress your head</li> make you feel tired with >1hr use</li> you might feel that there must be something playing audio</li> </ul> They're not a panacea, it's not practical to use them all day, but I'll come back to this. Avoid Conversations</h2> Obviously this is not possible and the antithesis of the purported reason for doing open office space. What can you do instead? Well, avoid interrupting people. This is super tempting when they're "right there" but honestly just do your absolute best to use the tech tools that are available, hit them up on slack or IRC or MS teams or whatever. Why would we do this? because the purported reason we have open offices is actually astonishingly false. If everyone feels like they won't be interrupted, perhaps we can sit shoulder to shoulder and make a "virtual" office space, as the lady in the video said in her opening statement. So, yeah, avoid starting new conversations, and if everyone is able to do so, it might just help. No management buy-in needed. Doesn't do much for those people on conference calls all the time though. Ideally you should push for "people closets" or "phone booths", they also work as the inverse like "quiet rooms". Try to get head high (when seated) dividers</h2> Now, this needs more management buy-in and you might stick out. BUT! there is significant evidence that removing visual distractions while seated goes a really long way to helping preserve the feeling of visual privacy and enhances focus. Productivity lowers linearly as the size of the divider falls to eye height. Avoid any company doing "Hot desking"</h2> Just don't work anywhere that does this, unless they also have private rooms and much more desks than employees. This is just a nice name for competitive desk hunting. Aim for flexibility</h2> Ideally when you start out, talk to your boss, talk to your manager, explain that distractions come easy and that you'd really prefer to do more remote working or work odd hours. Make sure you frame it correctly, because companies/managers might see it as you trying to take advantage, you need it just to get the work done, in the most productive way you know how. Perception is reality</h2> People who are seen, are seen as hard working. So you need to remain visible, which might be hard because you're balancing perception with productivity. Bosses love people who come in before them, but hate people that stay late. They might not say this directly, and it certainly doesn't apply everywhere. But the perception is that "the early bird gets the worm" vs "oh, I didn't manage to get all my work done". As a night owl this is especially displeasing, but it's the way it is. Take care of yourself</h2> Make sure you're finding a good time to recharge, the open offices space can be draining and being in that very stimulating environment for an unbroken 9 hours is especially draining. Finding ways to break up the day and find quiet space, or getting a good music list setup that takes you out of the environment. (noise cancelling headphones would probably help here) Get practical</h2> You can make visual cues to signal if you're interruptible. People use noise cancelling headphones for this too (see, they're useful, just not a panacea), but the issue is it conflicts with perception is reality, you need to be available sometimes, Ask to have quiet times</h2> This is different from asking for flexible work, this is more about asking for a time to put on "do not disturb" mode, or setting up some kind of sign that you're not to be interrupted. This can be invaluable if you really need to be in the office. Shift your hours</h2> Not much different from having more flexibility in general, but if you can't do that to work from a coffee shop or something then try to shift your hours, this is easier as most companies have some kind of flexi-time. Ideally this would be to work earlier hours, because as I mentioned it's perceived as being more hardworking, but this can be a huge saving grace, even if your company doesn't allow you to get too flexible with the hours you work in the day or where you work from, you can usually come in early and leave late. Compartmentalise the type of work</h2> This can be frustrating, but if you're coming in early in the morning and leaving earlier in the day to really ensure the minimum amount of "interrupt" hours, it is well worth using the interrupt time to do small improvements to your next days agenda. Knowing what your next task in the morning will be when you have that quiet focus time and just pounding it out can be very fulfilling and might even give you the energy needed to get through the rest of the day. Save email responses to your interrupt time, don't check emails in the morning when you come in. Don't eat at your desk</h2> This is an awful habit, you need the break from the open office environment. It's also quite distracting for your colleagues. Switch jobs</h2> If you really can't work, there's really no point sticking around. You don't owe anything to the company really, and if they can't respect your productivity then maybe it's not worth burning yourself out trying to please them. We're not cattle, we have significant leverage as a group (technology), and it's not worth your health. Even if you take a pay cut, it can be worth it, remote work is another alternative since remote workers are roughly 13.5% more productive</a> on average than even the most highly performing office workers, and all that saved commute time! On the importance of self-hosted backups. 2017-01-16T22:21:00+07:00 A long</a> time ago I built a pretty big storage computer (16TB) which I built because SSDs at the time were pretty small and most laptops came with only a single possible SATA drive bay for storage. I'm also quite a large proponent of self-hosting/federation and things of this nature as evidenced by our gitlab</a>, mail</a> service and IRC</a>. However, most people assume this huge storage unit is for piracy. In fact Sweden has a tax on storage devices</a> because they may be used to store pirated media, that is also true in other EU countries such as the Netherlands</a> [PDF]. But my unit is not for piracy- despite there being digital creative works on there I prefer to buy all my music through iTunes. </a> Why? Because then it's synced to all of my devices, I don't worry about ownership or getting it synced, or worrying about the bitrate/collecting albumart/removing weird watermarks people add... it's simple, convenient and unobtrusive to me. That said, I do take a copy of everything because you never know, and I like Linux/BSD and I use my music on those platforms too. However, we were talking about backups and I've gone on to piracy for some reason... One of the other reasons I like iCloud/iTunes is that I change phones with relative frequency (I replace my phone yearly pretty much); and having my music, photos' and apps automatically downloaded with no extra work on my behalf is certainly a convenience worth paying for. (And I do, with iCloud). However, there are issues with the iTunes store as anybody will tell you, one of those issues is that I cannot download the music I purchased when I lived in the UK, unless I have a UK credit/debit card on hand so I can change my store. But, sure, as long as I don't move from Sweden this wont become an unmanageable mess. File it under: things I wish I knew before I moved</a> Oh yes, backups.. Last week I went looking for an album that had a distinct cover and a small nostalgia attached to it. Since my phone is 128G I tend to download everything I purchase onto it without leaving it "in the cloud" for later download but I couldn't find it. I scrolled through my albums maybe 6 times, thinking perhaps I misremembered the album name. I couldn't find it, so I checked my purchases in the itunes store, where it also was not... so I checked the store itself, searching for an unpurchased album of the same name... which also yielded no results. Thinking that it must only be available to people in the UK, or that because I purchased it in the UK which could explain why it's not displaying; I reached for my credit card and changed stores. Nope. Maybe I imagined the album? maybe I can't remember the real name.. and maybe I never bought the album from iTunes- I mean, six years ago I had just moved to Helsinki to work for Nokia, I was not using Apple devices.. So I check google.. It doesn't help that the album name "Nightclub Life" was so non-unique, but thinking google images would have the album cover somewhere I checked.. I even went to page 2 of the results. </a> Nothing. I must have imagined the whole thing.. but it was so distinct! Last resort.. maybe I can grep the NAS. </a> Success! I questioned my own sanity there; of course I can just import the thing back into iTunes; but I can't put it on my phone since that requires wiping**!** the damn thing first... Apples ecosystem leaves much to be desired. </a> When I listened to the album it was garbage, I suppose I remembered one or two songs fondly and the album cover.. But that doesn't mean Apple gets to decide they can remove stuff from the store that people paid for (and I did buy this from the store since all files are in the apple format, correctly tagged and have ownership information in them). Thank god they don't have DRM any longer</a> </a> Let that be a lesson to me. Don't assume Apple wont rm</code> your shit when they feel like it, don't depend on iCloud or iTunes entitlements when you purchase media. Keep it backed up.

Jan Harasym

On The Shame We Share

The only good cloud is a google cloud

VPCs</h3> AWS is zonal by default, it's clear. VPC's require manual networking intervention (outside the happy path) to set up a "peered" network. This, by itself and with the context of "cloud helps you do the right thing for HA", is fucking moronic.</p>

EC2</h3> Reliability is hard when the underlying systems become unavailable, Google was the first to support live migration between hosts- I don't even notice it happening and I ran some performance sensitive applications.</p>

Enforcement can have the inverse effect

Fix 1: Speed limit pedestrian areas and tight areas</h2> I imagine the discussion happening at HQ:</p> Our scooters have in-built GPS and we can control the speed of the motor, they're also connected via LTE.</p> </blockquote>

Fix 2: Parking in designated areas</h2> I can almost hear another conversation at HQ:</p> People keep parking like dumbasses across pavements,</p> </blockquote>

That time my manager spent $1M on a backup server that I never used

Microsoft Teams; using one monopoly to aid another

DevOps; a decade of confusion and frustration

Except of course, that's not what happened.</h2> Operations staff were renamed to be devops.</p> </li> Developers were renamed to be devops.</p> </li> What we might call "release engineering" was also renamed devops.</p> </li> Whole teams of people with a singular discipline: devops</p> </li>

I don't trust Signal

GPG::SSH; notes for current best practices

Cloudflare is turning off the internet for me

home</h2> Ok, so I got home 30 minutes ago and I'm following some reddit links from /r/irc, when I get a "You have been blocked message".</p> "Oh,"</em> I thought.</p> "I forgot about that."</em></p> </p> So now I'm curious as to why, and what I can do about it.</p>

How to survive an open office.

Headphones</h2> Headphone use has obvious benefits for enhancing your sound privacy, however the drawbacks are that:</p> your ears get warm</li> they don't fully isolate sound</li> they compress your head</li> make you feel tired with >1hr use</li>

Don't eat at your desk</h2> This is an awful habit, you need the break from the open office environment.</p> It's also quite distracting for your colleagues.</p>

On the importance of self-hosted backups.

Don't eat at your desk</h2>
This is an awful habit, you need the break from the open office environment.</p>
It's also quite distracting for your colleagues.</p>