Jan Harasym

Designing highly scalable/resilient infrastructure by day; running hacker communities by night.

Page 3

SaltStack notes



Minions: salt “clients”, aka hosts / provision targets. (not to be confused with the salt command-line client salt)


master: the salt server, drives the provisioning of minions. the salt cli client runs on the master. The master is an ensemble of several services and worker processes.

  • Publisher (port 4505): which minions must be able to access for pull-mode
  • EventPublisher (IPC only):
  • MWorker: one or more “master workers”, which handle salt operations concurrently
  • ReqServer (port 4506): pop work and push to MWorker, plus receiving replies so MWorker doesn’t have to block
  • File Server (?): transfers files to minions on demand from the state tree


Grains are basically facts in the ansible/puppet world.


Pillar is a global value/config storage, spelled out on the master. This is basically YAML which is laid out in folder hierarchies...

Continue reading →

Friends don’t let friends use BTRFS for OLTP

I usually write rant-style posts, and today is no exception. A few months ago I was working on a benchmark comparing how PostgreSQL performs on a variety of Linux/BSD filesystems, both traditional ones (EXT3, EXT4, XFS) and new ones (BTRFS, ZFS, F2FS, HAMMER). Sometimes the results came out a bit worse than I hoped for, but most of the time the filesystems behaved quite reasonably and predictably. The one exception is BTRFS …

Now, don’t get me wrong - I’m well aware that filesystem engineering is complex task and takes non-trivial amount of time, especially when the filesystem aims to integrate so much functionality as BTRFS (some would say way too much). Dave Chinner stated that it takes 8-10 years for a filesystem to mature, and I have no reason not to trust his words. I’m not a XFS/EXT4 zealot, I’m actually a huge fan of filesystem improvements (and I don’t really like EXT4 so much)...

Continue reading →

Theatre: Lolita

(@ London Theatre)

I recently (as of 20 minutes ago actually) attended a production of Lolita, a representation of Stanley Kubricks work (they say on posters).

I had gone in with no expectations, well, when you purchase tickets for “The London Theatre” online you expect something grandiose in the heart of theatreland.

However, this was not one of those. This was a “Fringe Theatre”, which I’ve never heard of- but I’m open minded enough, although it’s situated in New Cross (not exactly known for it’s cultural prowess).

When we arrived at New Cross Gate station we were invited to walk over a rather sketchy looking scaffold bridge between platforms if we wanted to leave; once we got outside we navigated through the even sketchier neighbourhood

I’ve walked through New Cross before (back when I lived in Lewisham) and back then I had been hardened from my time in Coventry however, I’m a...

Continue reading →

The Sad State of British Broadband

I suppose I should change the title; it’s unfair to blame the broadband provider[0] and exclude the misdeeds of 3G/4G providers.

I have been at odds with the only true ADSL provider in the UK for some time. In fact, for as long as I’ve lived in the Capital.
I’ve lived in Lewisham (SE13), Aldgate (E1), and I’ve been living in Bow (E3) since July of last year and during this time I have achieved the average speed of 0.21Mb/s (yes, bits).

4G Rollout

During this time 4G was rolled out across London, and despite not having signal in my home, I can in fact, use this new technology.

However this rollout has been delayed by almost 2 years, there was an auction for the 4g spectrum from ofcom[1], however, “EE” (formerly T-mobile and Orange) seem to have deployed nearly a year before anyone else. Whether that was ability or willingness I’ll never know. (my initial guess was that the company...

Continue reading →

Windows 7 Clients on Samba Domain

Today, I had to face the undocumented mess that is: adding a windows machine to our UNIX infrastructure.


Where I work, we’re mostly UNIX and Linux, with UNIX on the backend for everything (solaris) and Linux for the e-commerce platform, along with the Customer Service computers. This is a stark contrast with people who are only accustomed to using Windows. Combine these factors (undocumented unix/windows + requirement to run windows) and the approaching April end of support deadline, and you have my heart racing and cold sweats.

I noticed that I can get some HP Prodesks (with windows 7 Pro) for less than the price of a Windows 7 license, so I bought one.

it was a modest machine with an AMD processor running 1.5Ghz and quad-core, but the improvements in CPU and harddisk design put it far above the other machines in the office for performance, which, shocked me somewhat.

As for...

Continue reading →

Failing to monitor, dying without dignity.

Today, I’m going to tell you about the story of an obscure kernel bug, how we missed it, and how we’re still recovering from the effect

I should preface this by saying that, generally, I like virtual machines.
I have 5 actual servers doing actual things- everything else is a VM in a racked bunch of servers hosted at Telecity in east London.

Generally, these servers are catered for with two uncontested fibre-to-the-rack lines which are layer-4 DDoS scrubbed and redundant power from two seperate generators and dirty feeds. – believe me when I say, no expense is spared on that rack, it’s where 90% of my budget goes and as well it should, given it’s the core business of the company.
I should also preface this by saying ubuntu has held hatred of mine for some time- given we had a development server here in the office and it failed due to a name change of lvm2 to lvm in initramfs causing our...

Continue reading →