Personal Blog

View the blog index.

RSS feed Click the icon for the blog RSS feed.


5 most recent items

14 Feb 2022 : Backups all the way down #
My home server has spent the weekend completing an offsite backup. Duplicity provides a rather nice stats dump at the end of the process, so I now know that what felt like a full two days of poor internet connectivity was actually only 37 hours 10 minutes and 41.54 seconds. During this time it uploaded 122 GB of new and changed file data resulting from 113 GB of actual changes.

That's quite a lot of delta, considering the server stores 381 GB of data in total, and that's more than I actually thought was on there. Most of this is probably backups from my other devices which go to my home server, then into this backup here. Backups of backups. I've had need to rely on these backups in the past, so it's reasuring to know I have something recent now to fall back on.

2 Jan 2022 : Reckoning and Renewal, Part II #
New year, new resolutions. At the start of last year I made some new year's resolutions, and the time has come to judge my actions against them.

Unlike 2020, my 2021 achievements aren't best measured in television series and computer games. Which isn't to say I didn't get through a lot of television (How to Get Away with Murder, Borgen, Capitani, Lupin, The Expanse, Witcher, Another Life, Cursed, all 172 episodes of Voyager) or computer games (Hades, the Banner Saga, Creaks, Dishonored Death to the Outsider, Titanfall 2, Trine 4). But unlike in 2020 these weren't the completion of multi-decade odysseys.

Instead, I spent much of my year focussing on work. A surprising amount of my free time was taken up writing the Sailfish OS community newsletter. I also spent a lot of time trying to learn Finnish, wracking up a 339 day streak on Duolingo and, in the second half of the year, attending intensive Finnish classes. Unfortunately my actual progress in being able to communicate doesn't match what my Duolingo stats imply.


I managed to eradicate some long-standing sores in my life. In September I finally released my website code as open source, thereby completing the first of my 2020 resolutions.

"1. Put this website code into a public git repository."

This had been on my to-do list for fifteen years, so I was pretty happy to get it finally resolved. Also by September I'd managed to transition my password approach from the Stanford to the Cambridge algorithm, releasing an updated PwdHash app in the process. This has been on my to-do list for five years. In December I managed to upgrade Constantia, my home server, to the latest versions of the software it runs (Linux, NextCloud, jitsi, etc.). Upgrades are always daunting in prospect and painful in practice, so I'm really happy to have this back under control. Finally Joanna and I have arranged to have our gas central heating converted to a heat pump. It's taken a lot of organising from us both, and it won't actually happen until later this year, but this is another long-standing issue I'm really happy we're going to get fixed.

The other positive from 2021 was that I managed to reduce my carbon footprint, and my waste output.

Those are the big positives from 2021 for me. It wasn't all positives of course. Joanna and I still haven't addressed our living-in-two-different-countries challenge. We really need to find a better solution. This was made worse by the travel restrictions this year, which really meant we spent much less time together than is right.

While I succeeded with my first resolution, I did much worse with my other two.

"2. Each week by the end of the weekend, spend at least an hour doing something calming that doesn't require a computer."

This was always going to be a challenge, and even though I spent ages trying to formulate this resolution into something measurable, it still didn't work out. My year was busy, sometimes stressful, and I didn't make progress in the non-computer calming things, such as reading, that I'd hoped to. I did at least do a lot of walking in the forest and by the lake, and even managed to do some work in the forest and by the lake. But I need to keep track of it better in the future.

"3. Complete the bisection analysis that Frajo and I started working on a year ago."

Again, this was a fail. I made some progress, but the work isn't even close to being complete. The fact I've not pushed my write-up to git is a good indication that it needs a lot more work.

So that was my 2021. What about 2022? Based on the above, what are my resolutions for the next year going to be?
  1. Maintain a digital record of all of my grocery purchases during the year. Since I already keep track of my waste output, I think it'd be interesting to compare this against my goods input
  2. Learn quantum programming, measured by completing the book I'm reading on the topic, and writing at least one quantum computer programme. The programme should do something.
  3. Complete my Curious Correspondence course with Joanna. We've already completed four out of the twelve tasks, so we have another eight to complete this year at least.
  4. Write up the bisection analysis work.
Plus keep my average waste down to below 200g per day on average, which is a reduction on my 2021 goal, but above my 2021 level of 114 g/day, in order to keep it achievable.

I'm dropping my aim to spend a measurable hour away from my computer each week in favour of these specific tasks instead. If I can manage the above, then I think I'll have achieved something similar.

Finally, despite my usual cynicism I was won over by this article about improving without trying. I think it's because the sentences are so short. Here are five I really buy into and will be test-driving this year.
  1. Sharpen your knives.
  2. Start a Saturday morning with some classical music – it sets the tone for a calm weekend.
  3. Be polite to rude strangers — it’s oddly thrilling.
  4. Learn the names of 10 trees.
  5. Politely decline invitations if you don’t want to go.
One thing I'm pleased to note is that none of my 2022 resolutions are intended to fix long-term sores, which I think means I must have tackled the most serious of them. In which case, I can now focus on preventing any new ones from forming.

Plenty of things to aim for in 2022. Let's see how this one goes. I predict 3% chance of success.
31 Dec 2021 : How lightly did I tread in 2021? #
I'm trying to make doing an environmental check-up an annual habit. I have so many bad habits, it feels like getting a good habit would make a nice change. So this is my attempt. Looking back is also a lot safer than making future commitments.

It turns out that 2021 was a good year for me environmentally, or that it at least looks that way on paper. Hemmed in by the pandemic and forced to reduce flying, it wasn't hard to do less this year. On top of that 2021 made my third year of collecting waste data, which — even unconsciously — has trained me into throwing less stuff away.

So let's start with my 2021 household carbon footprint. According to the Carbon Footprint Calculator, in 2021 Joanna and I contributed a combined total of 7.73 tonnes of CO2 to the atmosphere. That's a lot of CO2, but our output is at least following a downward trend. In 2019 we contributed 14.47 tonnes and in 2020 it was 8.50 tonnes. The following table summarises where all that gas came from.
Source Details for 2021 CO2 output 2019 (t) CO2 output 2020 (t) CO2 output 2021 (t)
Electricity 3 009 kWh 0.50 0.40 0.59
Natural gas 9 089 kWh 1.18 1.26 1.66
Flights 3 HEL-LHR, 4 TMP-STA 5.76 2.26 1.90
Car 3 219 km 1.45 0.39 0.39
Bus 168 km 0.00 0.01 0.02
National rail 676 km 0.08 0.01 0.02
International rail 513 km 0.02 0.01 0.00
Taxi 100 km 0.01 0.01 0.01
Food and drink   1.69 1.11 1.05
Pharmaceuticals   0.26 0.32 0.31
Clothing   0.03 0.06 0.06
Paper-based products   0.34 0.15 0.14
Computer usage   1.30 1.48 0.75
Electrical   0.12 0.29 0.19
Non-fuel car   0.00 0.10 0.00
Manufactured goods   0.50 0.03 0.03
Hotels, restaurants   0.51 0.16 0.15
Telecoms   0.15 0.05 0.04
Finance   0.24 0.24 0.22
Insurance   0.19 0.11 0.10
Education   0.05 0.00 0.04
Recreation   0.09 0.06 0.05
Total   14.47 8.50 7.73

The main reasons for the reduction compared to 2020 were fewer flights, and fewer computer purchases (I purchased precisely one less laptop than the one I purchased in 2020). Laptops, it turns out, are surprisingly carbon-intensive to make.

So those reductions are benefits, but I'm not sure they're benefits we'll be able to maintain over time. In early 2022 we've arranged to have a heat pump installed to replace our gas central heating. This is a big change, with the main aim to reduce that 9 089 kWh of natural gas usage in the table above. Gas is clean to burn, but as a non-renewable fossil fuel it's especially problematic, with no easily switchable environmentally-friendly alternative. Hopefully a heat pump will reduce our overall power usage, not just our non-renewable usage.

Our numbers equate to an average of 3.87 tonnes of CO2 per person in 2021. That compares favourably to the UK average of 5.4 tonnes, an EU average of 6.4 tonnes and a world average of 4.8 tonnes according to the World Bank.

How about waste output? My average waste output for 2021 was 114.69 g/day. You can see how this came about, and how it was split across different types of waste, in the graph below.
Daily waste data histocurve

This average is equivalent to a total waste output of 42 kg for the year. In theory everything except the General waste shown in the graph was recycled. The total is also a reduction on previous years, comparing to 57 kg of output in 2020 and 118 kg in 2019. These number are slightly lower than the actual amount. For example this year I've spent around six weeks in the UK, during which I'm not able to collect waste output data.

This all looks quite positive, but I'm becoming increasingly aware that waste output is a volatile metric. For example, if at some point I have to replace a piece of furniture, my waste output will go through the roof for the year. This does honestly motivate me to try to fix things rather than throw them away, but it's also a source of angst, knowing that it'll happen eventually.

According to eurostate, average per capita municipal waste output across the EU was 505 kg per person, with the average in Finland being slightly higher at 596 kg. Compared to this, my 42 kg of output looks pretty good. Still, I'm supposing that at least some of that 505 kg was made up of chairs.

So in summary I'm happy that Joanna's and my CO2 output was down on 2020, as was my waste output. We both trod a bit more lightly, even if it's not yet light enough. We've not quite reached that fully circular economy. The main driving factor for the reduction seems to have been the pandemic, so it will at least be interesting to see what happens next year.

12 Sep 2021 : Changing password approaches is really hard #
Last month I spent a couple of weeks with Joanna in a cottage next to a lake in the Finnish countryside. It was a time for reflection and an opportunity to re-evaluate my life choices. A chance happening raised the topics of passwords and phishing. It's a subject I used to be well-acquainted with, but which my work has drifted away from more recently.

Everyone needs a way to manage their passwords and ideally everyone should have a good way to manage their passwords. About a decade ago I started using PwdHash as my method. It has several advantages that are similar to those offered by the more-familiar database-backed password managers. For example it ensures you use a different password for every website; it guards against phishing attacks; it avoids the need to remember anything other than a master password; and it works across all devices (desktop, phone, browser, etc.). Because it generates passwords on-the-fly, it also has the benefit of not needing to store a database of passwords, neither in the cloud nor locally. Pretty neat. This last point makes it particularly attractive to me, since I'm generally uncomfortable relying on cloud services I don't host myself.

It has downsides too though. The main practical downside is that — because they're deterministically generated — the passwords can't be amended. This causes issues for systems that require regular password changes (thankfully less of an issue now than it was five years ago), or if you have to change your password on a specific site for some other reason (e.g. the site's passwords are compromised by an attacker). There are ways to work around this, but they're pretty awkward and user-unfriendly.

A further downside is that the password generation can be reversed, meaning that compromising a password for a single account could lead to compromise of the master password used to generate all of the individual site passwords, and hence to all of your accounts. This was such a major concern that my Cambridge colleague Graham Rymer and I investigated it back in 2016 and showed it to be a very real threat. We even managed to extract 79 PwdHash master passwords from three well-known compromised and publicly-leaked password databases (Stratfor, Rootkit and LinkedIn). In the paper we published on it we discussed ways to mitigate this threat, presenting our own improved alternative scheme. Other than technical improvements, the most crucial countermeasure we suggested was to use a really strong master password. This may seem obvious, but apparently it wasn't for many users of PwdHash up to that point.

Another significant problem with PwdHash is that changing to a different scheme at a later date is a tremendously painful experience. And that's exactly what I'm experiencing now.

The work Graham and I did convinced me I needed to change my approach. And what better approach than the one we recommended?

Yet it's been five years and I'm only just now starting to make the switch, which maybe tells you something about the effort involved. In theory I should have been able to switch over gradually, one site at a time, just updating my passwords on my next visit to each site. In practice the infrastructure wasn't there to allow me to do this easily enough. What I needed was a website and an app that would allow me to generate my old password, then seamlessly generate the new password without having to type in a new URL, open a new app or whatever. Basically, for something I'm going to have to perform hundreds of times, the process has to be as effortless as possible.

So this is what I've spent the last few weeks arranging. First off was the website. There are already sites for the original Stanford PwdHash and for our updated variant. But the thought of having to switch from one to the other across hundreds of sites was just too much. So I combined them into a single site that allows switching between the two with a single click. It's a really small, simple thing. but it's enough to make the difference between inertia holding me back and momentum pushing me forwards. I also made it easier to get the generated password onto the clipboard, allowing my web workflow to be fully optimised.

That deals with the web but what about apps on my phone? I've been using a PwdHash app on my phone for many years, written by Robert Gerlach. It's a really simple app, but all the more effective for it. The app only supported the original Stanford variant, so I've just spent my weekend updating it to support the new algorithm as well. Not only have I added support for the new algorithm, but following the advice from our paper of five years ago, I also added a password strength meter using the zxcvbn algorithm. Plus I also made a few other improvements to better suit my usual workflow.

So now the fun of changing all my passwords can finally begin. So far it's been more preparation than progress, but I have managed to convert the passwords for three sites, so there's no going back now. The whole experience has reaffirmed my empathy for everyone struggling with password management. There are plenty of good solutions out there for managing passwords of course, but frankly the fact there are so many options just adds to the complexity of making a good choice. Especially when it's the sort of decision you really want to get right first time. Or in my case, hopefully, second time.
29 Aug 2021 : New and improved waste data graphs #
I've just hit two full years of waste output data, which has given me a nice idea about how much waste I generate on a daily basis. Since I started back in August 2019 I've been updating a graph showing the results on my waste page. It's provided quite a fascinating picture. Not only has my waste output gone down over time, but it's also become more consistent.

I attribute this improvement squarely to the act of measuring my data each fortnight. The process has made me far more aware, not just about how much waste I produce, but also the sorts of products that generate more or less waste.

For example, glass is really heavy and it became clear quite early on that it was contributing significantly to the weight of waste I was producing. This motivated me to look into it more deeply, which ultimately resulted in me almost completely eradicating glass from my daily usage.

As a result of this and other changes, my daily usage has gone down from 322.80 g/day in 2019 to 154.98 g/day in 2020, and now in 2021 I'm currently averaging 123.34 g/day. Admittedly my average this year is likely to increase during the winter (and Christmas especially) but my aim is to keep it at least as low as my 2020 average.

One of the downsides to accumulating all this data is that the graphs I've been posting here have become increasingly hard to read. Placing all of the data onto a single graph has become unsustainable, so over the last week I've been updating my graph-generating scripts to make them more flexible. As a result, I'm now going to only show data for the current year on the main waste page. The data for previous years can still be viewed on the pages for 2019 and 2020, and I'll add new pages as the years tick forwards.

I've also created a new page showing the complete data set. These "all-data" graphs are plotted wider now, and while this makes it easier to read the individual entries, it also makes them impractically long and thin. The "fixed in time" preview below already gives an idea of the problem, but the graphs will only get wider, and the issue more accuate, over time. So they're really only going to be of interest for the masochistic.
Daily waste data histocurve snapshot 29/08/2021

While the full-data graph is interesting by virtue of its absurdity, splitting the graph up into annual chunks turns out to be the more interesting case. In particular, because I take readings when I take out the rubbish, these rarely actually fall on the first or last day of the year. So, how to split the readings across the year boundaries?

The solution I've came up with is to scale the readings at each end of the year in proportion to how much of the period falls into the year in question. For example, here are the actual readings I took over the 2020-2021 year boundary.

Date Paper Card Glass Metal Returnables Composts Plastic General
12/12/2020 57 515 0 0 0 449 107 322
14/3/2021 641 225 0 0 93 443 88 473

This covers an unusually long period of time because I was stuck in the UK for January, February and most of March due to Covid travel restrictions. But this is also convenient for making a more exaggerated example. So the period between 12th December and 14th March contains a total of 92 days. That splits into the two periods "12th December - 31st December" and "1st January - 14th March", which contain 20 and 72 days respectively. The proportion of time for each of these periods is therefore 20 / 92 = 21.74% that falls into 2020 and 72 / 92 = 78.26% that falls into 2021.

To manage the data split across the year, we therefore have to scale it appropriately. Each entry represents the end of a period, so the 12th December data falls entirely within 2020. The 14th March data represents the period that's split across both years. We can therefore scale this entry and turn it into two separate entries like this, scaling each of the data points based on the proportions calculated above.
Date Paper Card Glass Metal Returnables Composts Plastic General
31/12/2020 139.35 48.91 0 0 20.22 96.30 19.13 102.83
14/3/2021 501.65 176.09 0 0 72.78 346.70 68.87 370.17

To get the correct picture this has to be done at both ends of the year being plotted.

Managing the data this way makes some obvious assumptions which may not necessarily be true (it assumes I generate waste uniformly across the time period, which is obviously not the case). However it has several nice properties. The annual histograms get drawn in a way that broadly speaking matches up across the year boundary; and the annual averages also match up correctly. At least, it seems to me to be the most honest way to tackle the issue when apportioning the data across year boundaries.

Check back to my waste page over time to see how I'm getting on with keping my waste output down (or not), and whether I'm able to hit my 2021 target.