planet.uknot.org

February 22, 2017

Neil McGovern

A new journey – GNOME Foundation Executive Director

IMG_0726For those who haven’t heard, I’ve been appointed as the new Executive Director of the GNOME Foundation, and I started last week on the 15th February.

It’s been an interesting week so far, mainly meeting lots of people and trying to get up to speed with what looks like an enormous job! However, I’m thoroughly excited by the opportunity and am very grateful for everyone’s warm words of welcome so far.

One of the main things I’m here to do is to try and help. GNOME is strong because of its community. It’s because of all of you that GNOME can produce world leading technologies and a desktop that is intuitive, clean and functional. So, if you’re stuck with something, or if there’s a way that either myself or the Foundation can help, then please speak up!

Additionally, I intend on making this blog a much more frequently updated one – letting people know what I’m doing, and highlighting cool things that are happening around the project. In that vein, this week I’ve also started contacting all our fantastic Advisory Board members. I’m also looking at finding sponsors for GUADEC and GNOME.Asia, so if you know of anyone, let me know! I also booked my travel to the GTK+ hackfest and to LibrePlanet – if you’re going to either of those, make sure you come and introduce yourself :)

Finally, a small advertisement for Friends of GNOME. Your generosity really does help the Foundation support development of GNOME. Join up today!

by Neil McGovern at February 22, 2017 04:50 PM

Roger Bell-West

Large spool mount

I've been printing a fair bit for 3dhubs lately: the user uploads a model and chooses a material and colour, I print and post it, I get paid. Since most people opt for black, silver-grey or white, I've started buying larger spools of those, since they're about 20% cheaper per length.

February 22, 2017 09:00 AM

February 21, 2017

Roger Bell-West

Stitchers season 2

2016 science fiction, 10 episodes. Kirsten Clark continues to have her consciousness inserted into the minds of the recently-dead, while hunting for more information about her father.

February 21, 2017 09:04 AM

February 20, 2017

Roger Bell-West

Died in the Wool, Ngaio Marsh

1945 classic English detective fiction; thirteenth of Marsh's novels of Inspector Roderick Alleyn. Florence Rubrick, sheep station owner and local MP, vanished one night from her home; her body was found some weeks later, packed into a bale of wool. Eighteen months later, Alleyn is hunting for spies in New Zealand, and informally takes on the case.

February 20, 2017 09:04 AM

February 19, 2017

Roger Bell-West

Teaching Grifters

Today I'll introduce Grifters. Anything in square brackets is to be thought about rather than read aloud.

February 19, 2017 09:01 AM

February 18, 2017

Roger Bell-West

Pyramid 98: Welcome to Dungeon Fantasy

Pyramid, edited by Steven Marsh, is the monthly GURPS supplement containing short articles with a loose linking theme. This time, to support the forthcoming Dungeon Fantasy RPG, it's all about dungeon-bashing.

February 18, 2017 09:01 AM

February 17, 2017

Roger Bell-West

The Thirty-Nine Steps, John Buchan

1915 thriller; first of Buchan's books about Richard Hannay. Bored in London, Hannay invites his worried neighbour into his flat, and soon finds both the authorities and a cunning group of terrorists against him.

February 17, 2017 09:02 AM

February 16, 2017

Roger Bell-West

Boardgaming At Home, January 2017

A seven-, then five-player games session with some larger games that don't come out often enough. Images follow: cc-by-sa on everything.

February 16, 2017 09:04 AM

February 15, 2017

Roger Bell-West

Joker Game

2016 historical espionage, adaptation of a novel series, 12 episodes: AniDB. In the years just before the Second World War, a maverick Japanese spymaster establishes a new intelligence agency.

February 15, 2017 09:01 AM

February 14, 2017

Roger Bell-West

Teaching Coup Rebellion G54

Today I'll introduce Coup: Rebellion G54. Anything in square brackets is to be thought about rather than read aloud.

February 14, 2017 09:04 AM

February 13, 2017

Liam Proven

USB C. Everyone's complaining. I can't wait. I still hope for cable nirvana.

Things have been getting better for a while now. For smaller gadgets, micro-USB is now the standard charging connector. Cables are becoming
a consumable for me, but they're cheap and easy to find.

But it only goes in one way and it's hard to see and to tell. And not all my gadgets want it the same way round, meaning I have to either remember or peer at a tiny socket and try to guess.

So conditions were right for an either-way-round USB connector.


Apple led the way with its symmetrical Lightning connector:

https://en.wikipedia.org/wiki/Lightning_(connector)

This introduced millions of customers to a USB-sized single plug for data, audio & power that could go into its socket either way. That
"primed the pump".

Then a design student published a, well, um, a design for a USB plug that could go in either way:

http://dornob.com/double-sided-usb-solves-painfully-universal-design-problem/

This was widely admired and discussed, or as they say now, "went viral". I think some companies implemented it but it violates the USB formal spec.

(Not that that bothers the cheapo vendors -- e.g. I have both external hard disks and a laptop cooling stand that both take a USB A to A cable. I.e. the computer-end connector on both ends. This is highly illegal -- you could connect 2 computers directly and blow at least one of them up thereby -- but nobody enforces the rules.)

The bi-directional plug and Lightning both demonstrated that this was desirable, possible, and that there was demand.

Also, there were the bodges of USB 3 extensions to the micro-USB spec such as this:

https://en.wikipedia.org/wiki/USB_3.0#/media/File:Connector_USB_3_IMGP6033_wp.jpg

... and this...

https://en.wikipedia.org/wiki/USB_3.0#/media/File:USB-3.0-Stecker_(Typ_B).jpg

And commentary such as this:

http://www.theverge.com/2013/9/5/4696874/galaxy-note-3-usb-3-0-power-connector-explained

Sort of Siamese-twin double connectors -- big and ugly and a very visible kludge.

Something Had To Be Done.

USB C is the result.

And although owners and prospective owners of the new MacBook Pro laptops are complaining widely that they *only* have USB-C ports, it's actually a good thing, IMHO.

As this eloquently explains:

https://medium.com/@ageitgey/the-new-macbook-pro-is-kind-of-great-for-hackers-64c1c577a4d2#.bfca5smcu

The higher-end MacBook Pro has 4 of them -- *and nothing else*.

You can plug the power cable into any of them. Doesn't matter. Plug a display in to any of them. Doesn't matter. Phone, memory stick, wired network cable, docking station. Any port. Doesn't matter.

Plug the power supply into the MacBook, the MacBook charges. Plug the same cable into your phone, the phone charges. Plug the phone into the MacBook -- same cable -- the phone charges and syncs.

I really like the idea of a small silent computer that's got a bunch of ports and anything plugs into anything. 2 or 3 screens? Just works. Sync several phones? Just works. Where's the power socket? There isn't one. Plug it in anywhere. If it fits, it works, either way round.

February 13, 2017 06:51 PM

Roger Bell-West

Elementary season 4

2015-2016, 24 episodes. In modern New York, (a completely different) Sherlock Holmes and Dr Watson continue to consult for the police.

February 13, 2017 09:02 AM

February 12, 2017

Roger Bell-West

Handycon 2017

This new convention happened just on the other side of High Wycombe from me, at a hotel that has some conference space. (Why would you choose to have a conference there? I suppose if you applied some sort of travel-time-minimising algorithm and worked out that you got the least total driving that way…)

With images; cc-by-sa on everything.

February 12, 2017 09:03 AM

February 11, 2017

Roger Bell-West

The Three Body Problem, Catherine Shaw

2004 historical epistolary mystery. In Cambridge in 1888, a young schoolmistress tries to solve the murders of three mathematicians before her beau is convicted of them.

February 11, 2017 09:01 AM

February 10, 2017

Roger Bell-West

Teaching One Night Revolution

I've had a certain amount of practice teaching board games, so I'm going to post some of my introductions to them here. These are all put together from multiple sessions, generally at Essen. As a general principle, I have components lying on the table, and point to them and/or pick them up as they are mentioned.

Today I'll introduce One Night Revolution. Anything in square brackets is to be thought about rather than necessarily read aloud.

February 10, 2017 09:03 AM

February 09, 2017

Roger Bell-West

Kirkaldy Testing Museum

Just off Southwark Street in London is an unexpected piece of industrial history. Images follow: cc-by-sa on everything.

February 09, 2017 09:01 AM

February 08, 2017

Roger Bell-West

Flash Point solo: Let's Try That Again

Another run of Flash Point, on the two-door map from the base game. This time I decided to go with six firefighters, to give me some leeway for using the non-firefighting roles, in Veteran mode as usual. Photos are taken at the end of each round, six turns.

February 08, 2017 09:01 AM

February 07, 2017

Jonathan McDowell

GnuK on the Maple Mini

Last weekend, as a result of my addiction to buying random microcontrollers to play with, I received some Maple Minis. I bought the Baite clone direct from AliExpress - so just under £3 each including delivery. Not bad for something that’s USB capable, is based on an ARM and has plenty of IO pins.

I’m not entirely sure what my plan is for the devices, but as a first step I thought I’d look at getting GnuK up and running on it. Only to discover that chopstx already has support for the Maple Mini and it was just a matter of doing a ./configure --vidpid=234b:0000 --target=MAPLE_MINI --enable-factory-reset ; make. I’d hoped to install via the DFU bootloader already on the Mini but ended up making it unhappy so used SWD by following the same steps with OpenOCD as for the FST-01/BusPirate. (SWCLK is D21 and SWDIO is D22 on the Mini). Reset after flashing and the device is detected just fine:

usb 1-1.1: new full-speed USB device number 73 using xhci_hcd
usb 1-1.1: New USB device found, idVendor=234b, idProduct=0000
usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-1.1: Product: Gnuk Token
usb 1-1.1: Manufacturer: Free Software Initiative of Japan
usb 1-1.1: SerialNumber: FSIJ-1.2.3-87155426

And GPG is happy:

$ gpg --card-status
Reader ...........: 234B:0000:FSIJ-1.2.3-87155426:0
Application ID ...: D276000124010200FFFE871554260000
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 87155426
Name of cardholder: [not set]
Language prefs ...: [not set]
Sex ..............: unspecified
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]

While GnuK isn’t the fastest OpenPGP smart card implementation this certainly seems to be one of the cheapest ways to get it up and running. (Plus the fact that chopstx already runs on the Mini provides me with a useful basis for other experimentation.)

February 07, 2017 06:34 PM

Roger Bell-West

Chapel Noir, Carole Nelson Douglas

2001 historical mystery, fifth of Douglas's novels about Irene Adler. In Paris in 1889, the Exposition Universelle is in full swing… but a notorious killer seems to have come over from London.

February 07, 2017 09:00 AM

February 06, 2017

Liam Proven

On being boggled by technological advances [tech blog b

I had Sinclair Microdrives on my ZX Spectrum. They were better than tape cassette but nothing else -- ~90 kB of slowish, rather unreliable storage.

So I bought an MGT DISCiPLE and an old cheap 5¼" 80-track, DS/DD drive.

780 kB of storage! On demand! Programs loaded in seconds! Even when I upgraded to an ex-demo 128K Spectrum from Curry's, even 128 kB programs loaded in a (literal) flash!

(MGT's firmware strobed the Spectrum's screen border, in homage to loading from tape, so you could see the data streaming into memory.)

That was the first time I remember being excited by the size and speed of my computer's storage.

After an interval on an Amstrad PCW9512 -- in hindsight a bit of an odd choice, really -- I got an Acorn Archimides A310. 1 megabyte of RAM and 20 meg of hard disk!

I was a bit less dazzled because what paid for it was my first job, so I was working on PCs. My work PC was an actual IBM PC-AT, 6MHz I think, with 512 kB of RAM and a 20 MB 5¼" hard disk the size and weight of half a breeze-block. I added a second 15MB drive for SCO Xenix.

But the Archie was about 4x faster than the fastest PC the company sold, an IBM PS/2 Model 70-H21, with a 25MHz 80386DX with cache. So what dazzled me about my Archie wasn't it vast epic tracts of storage -- I had that at work -- it was the blinding speed of the thing. My own 8MHz RISC workstation. BASIC apps that ran as quick as Z80 machine code on a Speccy or PCW.

That would be about 1989.

The next few weren't my own.

There was the first 3½ inch hard drive I saw. So tiny. 120MB in a fag packet. Amazing.

Then I remember the first 330MB hard disk at work -- another full-height 5¼" thing, but SO MUCH SPACE. It also cost a few months' pay, mind. And the custard's server OS ran on PC DOS 3.3, so I had to partition it into 32MB slices -- drives C:, D:, E:, F:, G:, H:, I:, J:, K: and L: with L being a tiddler of about 11MB.

I think everything since has been a bit of a let-down.

Until last year when I bought a used 1TB notebook HD from a chap on FesseBouc here in Brno. (He bought a new lappy and got them to bung an SSD in it), so he sold me a new, unused 1TB drive...

... for thirty quid.

I bought one for my big desktop-replacement laptop which was my main machine for my first year here. That and a 128GB SSD. Later they went in my Mac mini. But they cost real money in 2014.

But thirty quid? I've spent more than that on a curry. For this tiny thing, the size of a dozen business cards stacked up. Not worth keeping for him.

The 20-somethings in the pub that night were unable to understand why I was so boggled by this. A terabyte is a meaningless word to them -- it's a lot, it's 2-3x what their laptops hold, but not vast. No sense of size, I think.

It's equivalent to FIFTY THOUSAND of the first hard disks I ever used, probably a Seagate  ST-4026:



Here's a nice retrospective.

Only the top-end PC-AT came with a hard disk, they were that expensive. (OK, mine was about 4 years old when I inherited it.)

3 and a quarter kilos!

That little £30 1TB thing holds as much as 162,500 kilograms of the first hard disk I personally used.

More than a blue whale, more than 2 space shuttles.

And it's the price of a central London curry and smaller than the battery of my first mobile phone.
That was hard to handle.

February 06, 2017 02:35 PM

Roger Bell-West

Thirsty Meeples January 2017

Back to the boardgame café on a mildly snowy night. With images; cc-by-sa on everything.

February 06, 2017 09:04 AM

February 05, 2017

Roger Bell-West

Borderline, Nevada Barr

2009 mystery, fifteenth in Barr's Anna Pigeon series, murder mysteries in US National Parks. After the events of the last book, Anna's on administrative leave; she travels with her husband to the Rio Grande for a rafting trip. They come across a nearly-dead, very pregnant woman; and apparently someone wants to finish the job.

February 05, 2017 09:01 AM

February 04, 2017

Roger Bell-West

DanMachi

2015 fantasy, adaptation of a light novel series, 12 episodes: AniDB. In a world where gods dwell among humans, Bell Cranel is a novice adventurer. Full title: Dungeon ni Deai o Motomeru no wa Machigatte Iru Darouka: Familia Myth, or Is It Wrong to Try to Pick Up Girls in a Dungeon?.

February 04, 2017 09:04 AM

February 03, 2017

Roger Bell-West

GURPS house rule: completion time

This is a small extension of the GURPS rules to generate the time taken for tasks of variable length.

February 03, 2017 09:00 AM

February 02, 2017

Steve Kennedy

Blink (and the thieves are gone)

There's a new home security security system, it was originally a Kickstarter project and was delayed a long time. But now it's here and it's called Blink.

There's a sync module and then camera units (the system can cope with 10 camera units in total). Currently it all works over WiFi, but as there's an Ethernet port on the sync unit, it's expected that it will be enabled with some future version of the firmware.

Set-up is relatively simple, install the iOS (or Android) app, create an account and then it will look for the sync unit (it initially uses its own WiFi), connect to it and select the WiFi network you want it to work with.

It then asks to set-up the cameras, which is done by adding the number printed inside them (you have to open them to put the batteries in anyway), they then get added (and you can name them) and they also connect to the WiFi network.

The camera install is relatively straight forward too, open the back, put in the batteries, push out the bit of plastic in the hole where the mount goes, put it all back together (snaps), put the mount on, then affix the two sided pad and stick it on a wall (or wherever suits).

Though they're battery powered, they should operate for a year under normal operating conditions (a new software update now takes this to 2 years). The cameras have a motion detection on them (infrared) and also an LED flash (for night time usage) which is VERY bright. If a burglar were to break in, they'd probably be more put off by being blinded rather than a camera being in the room. The cameras can take a photo or video (720p with audio) and the system can be alarmed and when triggered, your phone will notify you and get a video or photo.

People have used the cameras outside, but they're not waterproof so if that's required, mount them somewhere where they'll be protected from direct rain (like under the eaves or similar).

A single sync module and camera costs $99 (available from Amazon.com or directly from Blink), a 2 camera kit is $159, 3 camera kit $219 and 5 camera kit $399. Additional cameras are $70.

Update, all the Blink modules are available from Amazon UK, The sync module with 2 cameras is £189.99 and sync module with 3 cameras is £259.99 and an add-on camera is £89.99.

Watch this space for a new external camera which is coming out in 2017.

by Steve Karmeinsky (noreply@blogger.com) at February 02, 2017 01:18 PM

Roger Bell-West

Justice Calling, Annie Bellet

2014 modern fantasy/romance. Jade Crow runs a comics-and-games shop in Wylde, Idaho, where lots of magical creatures congregate. She's hiding from someone. But she isn't going to be able to stay hidden.

February 02, 2017 09:02 AM

February 01, 2017

Roger Bell-West

January 2017 Trailers

Some trailers I've seen recently, and my thoughts on them. (Links are to youtube. Opinions are thoroughly personal.)

February 01, 2017 09:04 AM

January 31, 2017

Roger Bell-West

Stabcon 2017

This long-running games convention had another instance at the start of January. With images; cc-by-sa on everything.

January 31, 2017 09:00 AM

January 30, 2017

Roger Bell-West

Coroner's Pidgin, Margery Allingham

1945 classic English detective fiction; twelfth of Allingham's novels of Albert Campion. Back in London after three years overseas on mysterious war work, Campion just wants to catch a train; but Lugg and an unknown lady turn up in his flat with a corpse. US vt Pearls Before Swine.

January 30, 2017 09:00 AM

January 29, 2017

Jonathan McDowell

BelFOSS 2017

On Friday I attended the second BelFOSS conference. I’d spoken about my involvement with Debian at the conference last year, which seemed to be well received. This year I’d planned to just be a normal attendee, but ended up roped in at a late stage to be part of a panel discussing various licensing issues. I had a thoroughly enjoyable day - there were many great speakers, and plenty of opportunity for interesting chats with other attendees.

The conference largely happens through the tireless efforts of Jonny McCullagh, though of course many people are involved in bringing it together. It’s a low budget single day conference which has still managed to fill its single track attendee capacity both years, and attract more than enough speakers. Last year Red Hat and LPI turned up, this year Matt Curry from Allstate’s Arizona office appeared, but in general it’s local speakers talking to a local audience. This is really good to see - I don’t think Jonny would object at all if he managed to score a `big name’ speaker, but one of his aims is to get students interested and aware of Free Software, and I think it helps a lot that the conference allows them to see that it’s actively in use in lots of aspects of the industry here in Northern Ireland.

Here’s hoping that BelFOSS becomes an annual fixture in the NI tech calendar!

January 29, 2017 10:18 PM

Roger Bell-West

Limitless season 1

2015-2016, 22 episodes. Brian Finch is an unemployed slacker, until he's introduced to a brain-boosting drug. Naturally he goes to work for the FBI to catch crooks.

January 29, 2017 09:00 AM

January 28, 2017

Roger Bell-West

Dune, Frank Herbert

1965 Hugo- and Nebula-award-winning science fiction. In the distant future, plots whirl within plots, and the control of the most valuable planet in the universe is a poisoned chalice.

January 28, 2017 09:04 AM

January 27, 2017

Roger Bell-West

Limitless

2011 science fiction, dir. Neil Burger, Bradley Cooper, Robert de Niro: IMDb / allmovie

Eddie Morra is a struggling writer, until his ex-brother-in-law offers him a new drug that makes him smarter.

January 27, 2017 09:02 AM

January 26, 2017

Roger Bell-West

Be My Enemy, Christopher Brookmyre

2004 tartan noir. Jack Parlabane is invited to the junket weekend of a new firm that plans to run team-building retreats with a twist. But that twist is going to turn out to be rather more twisted than anyone expects.

January 26, 2017 09:03 AM

January 25, 2017

Roger Bell-West

On-line postage with Royal Mail and MyHermes

With 3D printing sales and occasional bulk imports of board games I send 4-5 packages a month, and sometimes larger batches. There seem to be two main ways of achieving this easily.

January 25, 2017 09:02 AM

January 24, 2017

Jonathan McDowell

Experiments with 1-Wire

As previously mentioned, at the end of last year I got involved with a project involving the use of 1-Wire. In particular a DS28E15 device, intended to be used as a royalty tracker for a licensed piece of hardware IP. I’d no previous experience with 1-Wire (other than knowing it’s commonly used for driving temperature sensors), so I took it as an opportunity to learn a bit more about it.

The primary goal was to program a suitable shared key into the DS28E15 device that would also be present in the corresponding hardware device. A Maxim programmer had been ordered, but wasn’t available in stock so had to be back ordered. Of course I turned to my trusty Bus Pirate, which claimed 1-Wire support. However it failed to recognise the presence of the device at all. After much head scratching I finally listened to a co-worker who had suggested it was a clock speed issue - the absence of any option to select the 1-Wire speed in the Bus Pirate or any mention of different speeds in the documentation I had read had made me doubt it was an issue. Turns out that the Bus Pirate was talking “standard” 1-Wire and the DS28E15 only talks “overdrive” 1-Wire, to the extent that it won’t even announce its presence if the reset pulse conforms to the standard, rather than overdrive, reset time period. Lesson learned: listen to your co-workers sooner.

A brief period of yak shaving led to adding support to the Bus Pirate for the overdrive mode (since landed in upstream), and resulted in a search request via the BP interface correctly finding the device and displaying its ROM ID. This allowed exploration of the various commands the authenticator supports, to verify that the programming sequence operated as expected. These allow for setting the shared secret, performing a SHA256 MAC against this secret and a suitable nonce, and retrieving the result.

Next problem: the retrieved SHA256 MAC did not match the locally computed value. Initially endianness issues were suspected, but trying the relevant permutations did not help. Some searching found an implementation of SHA256 for the DS28E15 that showed differences between a standard SHA256 computation and what the authenticator performs. In particular SHA256 normally adds the current working state (a-g) to the current hash value (h0-h7) at the end of every block. The authenticator does this for all but the final block, where instead the hash value is set to the working state. I haven’t been able to find any documentation from Maxim that this is how things are calculated, nor have I seen any generic implementation of SHA256 which supports this mode. However rolling my own C implementation based on the code I found and using it to compare the results retrieved from the device confirms that this is what’s happening.

So at this point we’re done, right? Wait for the proper programming hardware to turn up, write the key to the devices, profit? Well, no. There was a bit of a saga involving the programmer (actually programmers, one with at least some documentation that allowed the creation of a Python tool to allow setting the key and reading + recording the ROM ID for tracking, and one with no programming documentation that came with a fancy GUI for manually doing the programming), but more importantly it was necessary to confirm that the programmed device interacted with the hardware correctly.

Initial testing with the hardware was unsuccessful. Again endianness issues were considered and permutations tried, but without success. A simple key constructed to avoid such issues was tried, and things worked fine. There was a hardware simulation of both components available, so it was decided to run that and obtain a capture of the traffic between them. As the secret key was known this would then allow the random nonce to be captured, and the corresponding (correct) hash value. Tests could then be performed in software to determine what the issue was & how to generate the same hash for verification.

Two sets of analyzer software were tried, OpenBench LogicSniffer (OLS) and sigrok. As it happened both failed to correctly decode the bitstream detected as 1-Wire, but were able to show the captured data graphically, allowing for decoding by eye. A slight patch to OLS to relax the timing constraints allowed it to successfully decode the full capture and provided the appropriate data for software reproduction. The end issue? A 256 bit number (as defined in VHDL) is not the same as 32 element byte array… Obvious when you know what the issue is!

So? What did I learn, other than a lot about 1-Wire? Firstly, don’t offhandedly discount suggestions that you don’t think make sense. Secondly, having a tool (in this case the Bus Pirate) that lets you easily play with a protocol via a simple interface is invaluable in understanding it. Thirdly, don’t trust manufacturers to be doing something in a normal fashion when they claim to be using a well defined technology. Fourthly, be conscious about all of the different ways bitstreams can be actually processed in memory. It’s not just endianness. Finally, spending the time to actually understand what’s going on up front can really help when things don’t work as you’d expect later on - without the yak shaving to support Overdrive on the BP I wouldn’t have been able to so quickly use the simulation capture to help diagnose the issue.

January 24, 2017 08:49 PM

Roger Bell-West

Mike, P. G. Wodehouse

1909 school stories, compilation from magazine publication in 1907-1908: Mike Jackson, youngest of five cricketing brothers, goes first to Wrykin and then to Sedleigh. Also republished in separate parts later, as Mike at Wrykin and Mike and Psmith; the latter also as Enter Psmith.

January 24, 2017 09:03 AM

January 23, 2017

Roger Bell-West

Bucks Boardgames pre-new-year games

This group usually meets on Tuesday evenings, so although they're fairly nearby I hadn't had a chance to join them before. But over the Christmas holiday they met on a Friday afternoon as well.

January 23, 2017 09:01 AM

January 22, 2017

Roger Bell-West

Night Without End, Alistair MacLean

1959 thriller. An airliner makes a forced rough landing near a weather observation post in the highlands of Greenland. What was it doing that far north, what foul play was involved, and who is responsible?

January 22, 2017 09:02 AM

January 21, 2017

Roger Bell-West

Schwarzesmarken

2016 alternate history mecha science fiction, adaptation of a manga spinoff of a visual novel, 12 episodes: AniDB. In 1983, the 666th Tactical Surface Fighter Squadron of the East German Army is the only force between the invading alien hordes and the rest of Germany.

January 21, 2017 09:00 AM

January 20, 2017

Roger Bell-West

Cinderella Ate My Daughter, Peggy Orenstein

2011 non-fiction. Orenstein breaks down various elements of the pink-princess culture as marketed to young girls.

January 20, 2017 09:00 AM

January 19, 2017

Roger Bell-West

Boxing Day Boardgames 2016

On Boxing Day because it's a tradition now (i.e. we did it last year), a bunch of us without family commitments got together for boardgames. Like the last session here, we stayed together as one big group.

January 19, 2017 09:02 AM

January 18, 2017

Roger Bell-West

The Last Frontier, Alistair MacLean

1959 thriller. British agent Michael Reynolds travels to Budapest to retrieve a British scientist kidnapped by the Russians and due to be shown off at a conference. US vt The Secret Ways.

January 18, 2017 10:26 AM

January 17, 2017

Roger Bell-West

Colour Scheme, Ngaio Marsh

1943 classic English detective fiction; twelfth of Marsh's novels of Inspector Roderick Alleyn. At a run-down hot-springs resort in the back country of New Zealand, one guest seems to be going out of his way to offend everyone; then he vanishes.

January 17, 2017 09:03 AM

January 16, 2017

Roger Bell-West

Castle season 8

2015-2016, 22 episodes. Famous writer Richard Castle continues to work with NYPD homicide detective (now Captain) Kate Beckett, in the final season of this police procedural.

January 16, 2017 09:03 AM

January 15, 2017

Liam Proven

6music

So... when the lack of apps for my beloved Blackberry Passport, and the issues with running sideloaded Android apps, became problematic, I decided to check out a cheap Chinese Android Phablet.

(P.S. The Passport is for sale! Let me know if you're interested.)

The Passport superseded a Samsung Galaxy Note 2, which subsequently got stolen, unfortunately. It was decent, occasionally sluggish, ran an elderly version of Android with no updates in ages, and had a totally useless stylus I never used. It replaced an iPhone 4 which replaced an HTC Desire HD, which replaced a Nokia Communicator E90 -- the best form-factor for a smartphone I've ever had, but nothing like it exists any more.

I wanted a dual-core or quad-core phablet, bigger than 5.5", with dual SIM and a memory card. That was my starting point.  I don't have or use a tablet and never have -- I'm a keyboard junkie. I spend a lot of time surfing the web, on social networks, reading books and things on my phone. I wanted one as big as I could get, but still pocketable. My nicked Samsing was 5.5" and I wanted a little larger. I tried a 6" phablet in a shop and wanted still bigger if possible. I also tried a 6.8" Lenovo Phab Pro in a shop and that was a bit too big (but I might be persuaded -- with a tiny bezel, such a device might be usable).

I looked at several sites -- the main 2 are:
* Coolicool -- http://coolicool.com/
* GearBest -- http://www.gearbest.com/
I've also been told DX.com is good ( http://www.dx.com/ ) but not tried it myself.

... and Googled in general. I found tons of them. I even started to built a spreadsheet to compare them, there were so many.

So I started looking for things to eliminate certain offerings.

Old models are easy to eliminate. 512MB RAM or something? Too little, strike that line out. I wanted at least 2GB. Tiny amount of Flash? 2, 4, 8GB Flash -- strike that line out.

Then it got a bit harder and needed to pay closer attention to the specs.

  • Dual SIM but one of them is the card slot -- strike that line out.

  • Not an ARM chip (e.g. Intel) -- strike that line out.

  • No removable battery -- strike that line out. (I didn't think to check that spares are available for mine. They aren't. :-( )

  • Low-res screen. At least 720P is normal now. 1280*600 is grainy and low-res these days. Anything less than 720P across (i.e. the narrow axis) got struck out.

I still had more than half a dozen and much confusion.

So then I started looking at Android versions.

Android 4.2 was the oldest I'd consider. It's the last version to play Flash, which is handy, but it's really old now. That had mostly already gone with the models with very low memory or storage.

Android 4.4 is OK, but still old.

So I looked for devices with Android 5.1 (as 5.0 and 5.0.x were buggy and leaked memory).

Bing, everything went away. The iRulu Victory v3 was the only one left, so I bought one.

Basically, now, there are 3 layers in the market.

There's high-end, low-end, and a poorly-defined layer in between.

There are premium big-name devices: Samsung, LG, Sony, HTC, Google Nexus. These are typically poor at what I consider useful extras like dual-SIM, memory card slots, removable batteries. But on the other hand, they abound in very hi-res screens (even with pointless gimmickry like curved screens or bevelled edges), extras like fingerprint readers and NFC and so on that I don't want. Even external expansion connectors like headphone sockets are now disappearing. They are 3x -- 4x the price of low-end devices.

There's the low-end: mostly Chinese, you've never heard of them. Basic but workable. Beware of ancient versions of Android, tiny amounts of memory and Flash, very slow processors or very low-res screens. As of last year, if it's less than £100, it's junk.

But they extend upwards slightly into decent, usable, cheap phones (and tablets, I guess -- I've not looked.)

The high end of the Chinese no-brand generics blurs into the mid-range of smaller brands.

Lenovo sits here, perhaps unexpectedly. Internationally it's a PC company but in its home market it makes hundreds of phones and tablets that non-Chinese markets rarely see. On international sale are about 1% of its models. It has some tempting devices and next time I might go for one, but as ever, they're always compromised -- the memory card is in SIM slot 2, the batteries aren't removable, etc.

Some of the semi-premium brands sit here too -- Huawei, Asus, Xiaomi.

A bit more expensive, but better build quality and support. Software-wise, you might get updates, which is good, but then again, you'll get some horrible customised skin on Android, and preloaded crapware, which is bad.

There are also smaller European companies, making deals with unheard-of Chinese vendors, rebranding them and offering European-customised Android and a bit of after-sales support. If you want to buy in the UK and are scared of dealing directly with China or whatever, this is a good choice.

Main examples:

* WileyFox -- https://www.wileyfox.com/
* OnePlus -- https://oneplus.net/uk/

So, the iRulu Victory v3 I'm currently using...

https://www.irulu.com/phone/p-irulu-victory-v3-smartphone-6.5-inch-4g-lte-unlock-cell-phone-android-5.1-gsm-wcdma-ips-touch-screen-qualcomm-quad-core-gms-certified_1123492.html

I bought it off eBay, new, from Singapore.

I found the best site for checking specs is GSMArena but there are others, e.g.

http://www.devicespecifications.com/en/model/502a3723

It's cheap and plasticky, but it works and it cost £125 new. I spent an extra fiver on a flip case, from a different vendor on eBay. It's one of those where you replace the back cover and the case attaches to that. Another fiver bought a pack of screen protectors. I'll only ever need one, but it was cheaper to buy 5 than 3. I made a balls up of fitting the first one, so I got a shop to do the next, and they did a lovely job.

It's a huge, fairly high-spec phablet for the money -- 2GB RAM, 16GB Flash, both Dual SIM and a µSD slot. (Often makers/vendors claim this but a memory card goes in the 2nd socket. I needed dual SIM plus a memory card.)

The only downside is that there's no compass. Not no compass app -- that's trivial -- no compass chip. Makes navigation on foot trickier. Not a problem in a car but I very rarely need that .

It's huge but it's light as the case and frame aren't metal. As it's not Gorilla Glass or other super-tough screen covering, I regard a screen protector and a hard case as essential. I've dropped it once or twice and it's not cracked or anything yet.

It fits in most trouser pockets fine, even jeans, and it's a talking point and conversation starter because it's so big. But then I have huge hands and can just, barely, use this one-handed.

The battery life is poor, despite an alleged 3000 mAh battery. I carry a power bank. I need to reboot daily, and carefully move all apps that can be moved from internal storage to my memory card.

The iRulu isn't lasting well. I have occasionally errors with the SIM not registering, the backlight sometimes flickers, and the volume-down key no longer works. I have to press volume up and then use a fingertip to move the onscreen slider. However, it remains usable. If it lasts a year, I'll be happy. If it only lasts 9 months, for the money, I won't be gutted. In my experience, premium-brand Android phones are on their last legs after 2 years. If my ChiPhone lasts half as long but costs a quarter of the money, I'm quids in. If it lasts a third as long and costs a quarter, I'm still good. If it lasts a third and costs a third, it's still not bad -- I'm better off if it gets lost or stolen.

A tip of the hat to my mate Louis Dobson ( http://thegentlemansnapper.blogspot.cz/ ) who told me that this was an option right after I bought my Samsung Galaxy Note 2, about 3 years ago. Go adopt one of his dogs.

January 15, 2017 02:36 PM

Roger Bell-West

Psycho, Robert Bloch

1959 horror. Mary Crane has stolen $40,000 from her employer and is running away to get married. Unfortunately she's chosen to stay at the wrong motel.

January 15, 2017 09:03 AM

January 14, 2017

Jonathan McDowell

Cloning a USB LED device

A month or so ago I got involved in a discussion on IRC about notification methods for a headless NAS. One of the options considered was some sort of USB attached LED. DealExtreme had a cheap “Webmail notifier”, which was already supported by mainline kernels as a “Riso Kagaku” device but it had been sold out for some time.

This seemed like a fun problem to solve with a tinyAVR and V-USB. I had my USB relay board so I figured I could use that to at least get some code to the point that the kernel detected it as the right device, and the relay output could be configured as one of the colours to ensure it was being driven in roughly the right manner. The lack of a full lsusb dump (at least when I started out) made things a bit harder, plus the fact that the Riso uses an output report unlike the relay code, which uses a control message. However I had the kernel source for the driver and with a little bit of experimentation had something which would cause the driver to be loaded and the appropriate files in /sys/class/leds/ to be created. The relay was then successfully activated when the red LED was supposed to be on.

hid-led 0003:1294:1320.0001: hidraw0: USB HID v1.01 Device [MAIL  MAIL ] on usb-0000:00:14.0-6.2/input0
hid-led 0003:1294:1320.0001: Riso Kagaku Webmail Notifier initialized

I subsequently ordered some Digispark clones and modified the code to reflect the pins there (my relay board used pins 1+2 for USB, the Digispark uses pins 3+4). I then soldered a tricolour LED to the board, plugged it in and had a clone of the Riso Kaguku device for about £1.50 in parts (no doubt much cheaper in bulk). Very chuffed.

In case it’s useful to someone, the code is released under GPLv3+ and is available at https://the.earth.li/gitweb/?p=riso-kagaku-clone.git;a=summary or on GitHub at https://github.com/u1f35c/riso-kagaku-clone. I’m seeing occasional issues on an older Dell machine that only does USB2 with enumeration, but it generally is fine once it gets over that.

(FWIW, Jon, who started the original discussion, ended up with a BlinkStick Nano which is a neater device with 2 LEDs but still based on an Tiny85.)

January 14, 2017 11:53 AM

Roger Bell-West

Marlow Tabletop and Board Games 5 December 2016

Third time at the Marlow Donkey and the sixth monthly meeting of this Meetup-based boardgames group.

January 14, 2017 09:04 AM

January 13, 2017

Roger Bell-West

Twilight Robbery, Frances Hardinge

2011 non-historical fiction, sequel to Fly By Night. Mosca Mye and Eponymous Clent are still in trouble, and attempting to flee through Toll, the town that controls the only bridge across the big river. But both entering and leaving have their price. US vt Fly Trap.

January 13, 2017 09:03 AM

January 12, 2017

Roger Bell-West

Pyramid 97: Strange Powers

Pyramid, edited by Steven Marsh, is the monthly GURPS supplement containing short articles with a loose linking theme. This time it's the loose idea of bizarre phenomena and, well, strange powers.

January 12, 2017 09:00 AM

January 11, 2017

Roger Bell-West

One Fine Day in the Middle of the Night, Christopher Brookmyre

1999 tartan noir. Gavin Hutchison has a brilliant idea: convert an oil rig to an offshore resort hotel for xenophobic tourists, with all the comforts of home but warmer weather since it'll be moored off the African coast. What better way to show it off than by hosting a school reunion there?

January 11, 2017 09:01 AM

January 10, 2017

Andy Smith (strugglers.net)

XFS, Reflinks and Deduplication

btrfs Past

This post is about XFS but it’s about features that first hit Linux in btrfs, so we need to talk about btrfs for a bit first.

For a long time now, btrfs has had a useful feature called reflinks. Basically this is exposed as cp --reflink=always and takes advantage of extents and copy-on-write in order to do a quick copy of data by merely adding another reference to the extents that the data is currently using, rather than having to read all the data and write it out again, as would be the case in other filesystems.

Here’s an excerpt from the man page for cp:

When –reflink[=always] is specified, perform a lightweight copy, where the data blocks are copied only when modified. If this is not possible the copy fails, or if –reflink=auto is specified, fall back to a standard copy.

Without reflinks a common technique for making a quick copy of a file is the hardlink. Hardlinks have a number of disadvantages though, mainly due to the fact that since there is only one inode all hardlinked copies must have the same metadata (owner, group, permissions, etc.). Software that might modify the files also needs to be aware of hardlinks: naive modification of a hardlinked file modifies all copies of the file.

With reflinks, life becomes much easier:

  • Each copy has its own inode so can have different metadata. Only the data extents are shared.
  • The filesystem ensures that any write causes a copy-on-write, so applications don’t need to do anything special.
  • Space is saved on a per-extent basis so changing one extent still allows all the other extents to remain shared. A change to a hardlinked file requires a new copy of the whole file.

Another feature that extents and copy-on-write allow is block-level out-of-band deduplication.

  • Deduplication – the technique of finding and removing duplicate copies of data.
  • Block-level – operating on the blocks of data on storage, not just whole files.
  • Out-of-band – something that happens only when triggered or scheduled, not automatically as part of the normal operation of the filesystem.

btrfs has an ioctl that a userspace program can use—presumably after finding a sequence of blocks that are identical—to tell the kernel to turn one into a reference to the other, thus saving some space.

It’s necessary that the kernel does it so that any IO that may be going on at the same time that may modify the data can be dealt with. Modifications after the data is reflinked will just case a copy-on-write. If you tried to do it all in a userspace app then you’d risk something else modifying the files at the same time, but by having the kernel do it then in theory it becomes completely safe to do it at any time. The kernel also checks that the sequence of extents really are identical.

In-band deduplication is a feature that’s being worked on in btrfs. It already exists in ZFS though, and there is it rarely recommended for use as it requires a huge amount of memory for keeping hashes of data that has been written. It’s going to be the same story with btrfs, so out-of-band deduplication is still something that will remain useful. And it exists as a feature right now, which is always a bonus.

XFS Future

So what has all this got to do with XFS?

Well, in recognition that there might be more than one Linux filesystem with extents and so that reflinks might be more generally useful, the extent-same ioctl got lifted up to be in the VFS layer of the kernel instead of just in btrfs. And the good news is that XFS recently became able to make use of it.

When I say “recently” I do mean really recently. I mean like kernel release 4.9.1 which came out on 2017-01-04. At the moment it comes with massive EXPERIMENTAL warnings, requires a new filesystem to be created with a special format option, and will need an xfsprogs compiled from recent git in order to have a mkfs.xfs that can create such a filesystem.

So before going further, I’m going to assume you’ve compiled a new enough kernel and booted into it, then compiled up a new enough xfsprogs. Both of these are quite simple things to do, for example the Debian documentation for building kernel packages from upstream code works fine.

XFS Reflink Demo

Make yourself a new filesystem, with the reflink=1 format option.

# mkfs.xfs -L reflinkdemo -m reflink=1 /dev/xvdc
meta-data=/dev/xvdc              isize=512    agcount=4, agsize=3276800 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0, rmapbt=0, reflink=1
data     =                       bsize=4096   blocks=13107200, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=6400, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Put it in /etc/fstab for convenience, and mount it somewhere.

# echo "LABEL=reflinkdemo /mnt/xfs xfs relatime 0 2" >> /etc/fstab
# mkdir -vp /mnt/xfs
mkdir: created directory ‘/mnt/xfs’
# mount /mnt/xfs
# df -h /mnt/xfs
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdc        50G  339M   50G   1% /mnt/xfs

Create a few files with random data.

# mkdir -vp /mnt/xfs/reflink
mkdir: created directory ‘/mnt/xfs/reflink’
# chown -c andy: /mnt/xfs/reflink
changed ownership of ‘/mnt/xfs/reflink’ from root:root to andy:andy
# exit
$ for i in {1..5}; do
> echo "Writing $i…"; dd if=/dev/urandom of=/mnt/xfs/reflink/$i bs=1M count=1024;
> done
Writing 1…
1024+0 records in 
1024+0 records out
1073741824 bytes (1.1 GB) copied, 4.34193 s, 247 MB/s
Writing 2…
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 4.33207 s, 248 MB/s
Writing 3…
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 4.33527 s, 248 MB/s
Writing 4…
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 4.33362 s, 248 MB/s
Writing 5…
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 4.32859 s, 248 MB/s
$ df -h /mnt/xfs
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdc        50G  5.4G   45G  11% /mnt/xfs
$ du -csh /mnt/xfs
5.0G    /mnt/xfs
5.0G    total

Copy a file and as expected usage will go up by 1GiB. And it will take a little while, even on my nice fast SSDs.

$ time cp -v /mnt/xfs/reflink/{,copy_}1
‘/mnt/xfs/reflink/1’ -> ‘/mnt/xfs/reflink/copy_1’
 
real    0m3.420s
user    0m0.008s
sys     0m0.676s
$ df -h /mnt/xfs; du -csh /mnt/xfs/reflink
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdc        50G  6.4G   44G  13% /mnt/xfs
6.0G    /mnt/xfs/reflink
6.0G    total

So what about a reflink copy?

$ time cp -v --reflink=always /mnt/xfs/reflink/{,reflink_}1
‘/mnt/xfs/reflink/1’ -> ‘/mnt/xfs/reflink/reflink_1’
 
real    0m0.003s
user    0m0.000s
sys     0m0.004s
$ df -h /mnt/xfs; du -csh /mnt/xfs/reflink
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdc        50G  6.4G   44G  13% /mnt/xfs
7.0G    /mnt/xfs/reflink
7.0G    total

The apparent usage went up by 1GiB but the amount of free space as shown by df stayed the same. No more actual storage was used because the new copy is a reflink. And the copy got done in 4ms as opposed to 3,420ms.

Can we tell more about how these files are laid out? Yes, we can use the filefrag -v command to tell us more.

$ filefrag -v /mnt/xfs/reflink/{,copy_,reflink_}1
Filesystem type is: 58465342
File size of /mnt/xfs/reflink/1 is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  262143:    1572884..   1835027: 262144:             last,shared,eof
/mnt/xfs/reflink/1: 1 extent found
File size of /mnt/xfs/reflink/copy_1 is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  262143:     917508..   1179651: 262144:             last,eof
/mnt/xfs/reflink/copy_1: 1 extent found
File size of /mnt/xfs/reflink/reflink_1 is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  262143:    1572884..   1835027: 262144:             last,shared,eof
/mnt/xfs/reflink/reflink_1: 1 extent found

What we can see here is that all three files are composed of a single extent which is 262,144 4KiB blocks in size, but it also tells us that /mnt/xfs/reflink/1 and /mnt/xfs/reflink/reflink_1 are using the same range of physical blocks: 1572884..1835027.

XFS Deduplication Demo

We’ve demonstrated that you can use cp --reflink=always to take a cheap copy of your data, but what about data that may already be duplicates without your knowledge? Is there any way to take advantage of the extent-same ioctl for deduplication?

There’s a couple of software solutions for out-of-band deduplication in btrfs, but one I know that works also in XFS is duperemove. You will need to use a git checkout of duperemove for this to work.

A quick reminder of the storage use before we start.

$ df -h /mnt/xfs; du -csh /mnt/xfs/reflink
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdc        50G  6.4G   44G  13% /mnt/xfs
7.0G    /mnt/xfs/reflink
7.0G    total
$ filefrag -v /mnt/xfs/reflink/{,copy_,reflink_}1
Filesystem type is: 58465342
File size of /mnt/xfs/reflink/1 is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  262143:    1572884..   1835027: 262144:             last,shared,eof
/mnt/xfs/reflink/1: 1 extent found
File size of /mnt/xfs/reflink/copy_1 is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  262143:     917508..   1179651: 262144:             last,eof
/mnt/xfs/reflink/copy_1: 1 extent found
File size of /mnt/xfs/reflink/reflink_1 is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  262143:    1572884..   1835027: 262144:             last,shared,eof
/mnt/xfs/reflink/reflink_1: 1 extent found

Run duperemove.

# duperemove -hdr --hashfile=/var/tmp/dr.hash /mnt/xfs/reflink
Using 128K blocks
Using hash: murmur3
Gathering file list...
Adding files from database for hashing.
Loading only duplicated hashes from hashfile.
Using 2 threads for dedupe phase
Kernel processed data (excludes target files): 4.0G
Comparison of extent info shows a net change in shared extents of: 1.0G
$ df -h /mnt/xfs; du -csh /mnt/xfs/reflink
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdc        50G  5.4G   45G  11% /mnt/xfs
7.0G    /mnt/xfs/reflink
7.0G    total
$ filefrag -v /mnt/xfs/reflink/{,copy_,reflink_}1
Filesystem type is: 58465342
File size of /mnt/xfs/reflink/1 is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  262143:    1572884..   1835027: 262144:             last,shared,eof
/mnt/xfs/reflink/1: 1 extent found
File size of /mnt/xfs/reflink/copy_1 is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  262143:    1572884..   1835027: 262144:             last,shared,eof
/mnt/xfs/reflink/copy_1: 1 extent found
File size of /mnt/xfs/reflink/reflink_1 is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  262143:    1572884..   1835027: 262144:             last,shared,eof
/mnt/xfs/reflink/reflink_1: 1 extent found

The output of du remained the same, but df says that there’s now 1GiB more free space, and filefrag confirms that what’s changed is that copy_1 now uses the same extents as 1 and reflink_1. The duplicate data in copy_1 that in theory we did not know was there, has been discovered and safely reference-linked to the extent from 1, saving us 1GiB of storage.

By the way, I told duperemove to use a hash file because otherwise it will keep that in RAM. For the sake of 7 files that won’t matter but it will if I have millions of files so it’s a habit I get into. It uses that hash file to avoid having to repeatedly re-hash files that haven’t changed.

All that has been demonstrated so far though is whole-file deduplication, as copy_1 was just a regular copy of 1. What about when a file is only partially composed of duplicate data? Well okay.

$ cat /mnt/xfs/reflink/{1,2} > /mnt/xfs/reflink/1_2
$ ls -lah /mnt/xfs/reflink/{1,2,1_2}
-rw-r--r-- 1 andy andy 1.0G Jan 10 15:41 /mnt/xfs/reflink/1
-rw-r--r-- 1 andy andy 2.0G Jan 10 16:55 /mnt/xfs/reflink/1_2
-rw-r--r-- 1 andy andy 1.0G Jan 10 15:41 /mnt/xfs/reflink/2
$ df -h /mnt/xfs; du -csh /mnt/xfs/reflink
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdc        50G  7.4G   43G  15% /mnt/xfs
9.0G    /mnt/xfs/reflink
9.0G    total
$ filefrag -v /mnt/xfs/reflink/{1,2,1_2}
Filesystem type is: 58465342
File size of /mnt/xfs/reflink/1 is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  262143:    1572884..   1835027: 262144:             last,shared,eof
/mnt/xfs/reflink/1: 1 extent found
File size of /mnt/xfs/reflink/2 is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  262127:         20..    262147: 262128:            
   1:   262128..  262143:    2129908..   2129923:     16:     262148: last,eof
/mnt/xfs/reflink/2: 2 extents found
File size of /mnt/xfs/reflink/1_2 is 2147483648 (524288 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  262127:     262164..    524291: 262128:            
   1:   262128..  524287:     655380..    917539: 262160:     524292: last,eof
/mnt/xfs/reflink/1_2: 2 extents found

I’ve concatenated 1 and 2 together into a file called 1_2 and as expected, usage goes up by 2GiB. filefrag confirms that the physical extents in 1_2 are new. We should be able to do better because this 1_2 file does not contain any new unique data.

$ duperemove -hdr --hashfile=/var/tmp/dr.hash /mnt/xfs/reflink
Using 128K blocks
Using hash: murmur3
Gathering file list...
Adding files from database for hashing.
Using 2 threads for file hashing phase
Kernel processed data (excludes target files): 4.0G
Comparison of extent info shows a net change in shared extents of: 3.0G
$ df -h /mnt/xfs; du -csh /mnt/xfs/reflink
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdc        50G  5.4G   45G  11% /mnt/xfs
9.0G    /mnt/xfs/reflink
9.0G    total

We can. Apparent usage stays at 9GiB but real usage went back to 5.4GiB which is where we were before we created 1_2.

And the physical layout of the files?

$ filefrag -v /mnt/xfs/reflink/{1,2,1_2}
Filesystem type is: 58465342
File size of /mnt/xfs/reflink/1 is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  262143:    1572884..   1835027: 262144:             last,shared,eof
/mnt/xfs/reflink/1: 1 extent found
File size of /mnt/xfs/reflink/2 is 1073741824 (262144 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  262127:         20..    262147: 262128:             shared
   1:   262128..  262143:    2129908..   2129923:     16:     262148: last,shared,eof
/mnt/xfs/reflink/2: 2 extents found
File size of /mnt/xfs/reflink/1_2 is 2147483648 (524288 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  262143:    1572884..   1835027: 262144:             shared
   1:   262144..  524271:         20..    262147: 262128:    1835028: shared
   2:   524272..  524287:    2129908..   2129923:     16:     262148: last,shared,eof
/mnt/xfs/reflink/1_2: 3 extents found

It shows that 1_2 is now made up from the same extents as 1 and 2 combined, as expected.

Less of the urandom

These synthetic demonstrations using a handful of 1GiB blobs of data from /dev/urandom are all very well, but what about something a little more like the real world?

Okay well let’s see what happens when I take ~30GiB of backup data created by rsnapshot on another host.

rsnapshot is a backup program which makes heavy use of hardlinks. It runs periodically and compares the previous backup data with the new. If they are identical then instead of storing an identical copy it makes a hardlink. This saves a lot of space but does have a lot of limitations as discussed previously.

This won’t be the best example because in some ways there is expected to be more duplication; this data is composed of multiple backups of the same file trees. But on the other hand there shouldn’t be as much because any truly identical files have already been hardlinked together by rsnapshot. But it is a convenient source of real-world data.

So, starting state:

(I deleted all the reflink files)

$ df -h /mnt/xfs; sudo du -csh /mnt/xfs/rsnapshot
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdc        50G   30G   21G  59% /mnt/xfs
29G     /mnt/xfs/rsnapshot
29G     total

A small diversion about how rsnapshot lays out its backups may be useful here. They are stored like this:

  • rsnapshot_root / [iteration a] / [client foo] / [directory structure from client foo]
  • rsnapshot_root / [iteration a] / [client bar] / [directory structure from client bar]
  • rsnapshot_root / [iteration b] / [client foo] / [directory structure from client foo]
  • rsnapshot_root / [iteration b] / [client bar] / [directory structure from client bar]

The iterations are commonly things like daily.0, daily.1daily.6. As a consequence, the paths:

rsnapshot/daily.*/client_foo

would be backups only from host foo, and:

rsnapshot/daily.0/*

would be backups from all hosts but only the most recent daily sync.

Let’s first see what the savings would be like in looking for duplicates in just one client’s backups.

Here’s the backups I have in this blob of data. The names of the clients are completely made up, though they are real backups.

Client Size (MiB)
darbee 14,504
achorn 11,297
spader 2,612
reilly 2,276
chino 2,203
audun 2,184

So let’s try deduplicating all of the biggest one’s—darbee‘s—backups:

$ df -h /mnt/xfs
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdc        50G   30G   21G  59% /mnt/xfs
# time duperemove -hdr --hashfile=/var/tmp/dr.hash /mnt/xfs/rsnapshot/*/darbee
Using 128K blocks
Using hash: murmur3
Gathering file list...
Kernel processed data (excludes target files): 8.8G
Comparison of extent info shows a net change in shared extents of: 6.8G
9.85user 78.70system 3:27.23elapsed 42%CPU (0avgtext+0avgdata 23384maxresident)k
50703656inputs+790184outputs (15major+20912minor)pagefaults 0swaps
$ df -h /mnt/xfs
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdc        50G   25G   26G  50% /mnt/xfs

3m27s of run time, somewhere between 5 and 6.8GiB saved. That’s 35%!

Now to deduplicate the lot.

# time duperemove -hdr --hashfile=/var/tmp/dr.hash /mnt/xfs/rsnapshot
Using 128K blocks
Using hash: murmur3
Gathering file list...
Kernel processed data (excludes target files): 5.4G
Comparison of extent info shows a net change in shared extents of: 3.4G
29.12user 188.08system 5:02.31elapsed 71%CPU (0avgtext+0avgdata 34040maxresident)k
34978360inputs+572128outputs (18major+45094minor)pagefaults 0swaps
$ df -h /mnt/xfs
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdc        50G   23G   28G  45% /mnt/xfs

5m02 used this time, and another 2–3.4G saved.

Since the actual deduplication does take some time (the kernel having to read the extents, mainly), and most of it was already done in the first pass, a full pass would more likely take the sum of the times, i.e. more like 8m29s.

Still, a total of about 7GiB was saved which is 23%.

It would be very interesting to try this on one of my much larger backup stores.

Why Not Just Use btrfs?

Using a filesystem that already has all of these features would certainly seem easier, but I personally don’t think btrfs is stable enough yet. I use it at home in a relatively unexciting setup (8 devices, raid1 for data and metadata, no compression or deduplication) and I wish I didn’t. I wouldn’t dream of using it in a production environment yet.

I’m on the btrfs mailing list and there are way too many posts regarding filesystems that give ENOSPC and become unavailable for writes, or systems that were unexpectedly powered off and when powered back on the btrfs filesystem is completely lost.

I expect the reflink feature in XFS to become non-experimental before btrfs is stable enough for production use.

ZFS?

ZFS is great. It doesn’t have out-of-band deduplication or reflinks though, and they don’t plan to any time soon.

by Andy at January 10, 2017 08:45 PM

Roger Bell-West

Horniman Museum, Forest Hill

I'd driven past it many times when living in Lee Green, but had never been inside; over Christmas I remedied this. Images follow: cc-by-sa on everything.

January 10, 2017 09:04 AM

January 09, 2017

Roger Bell-West

January 08, 2017

Roger Bell-West

Death and the Dancing Footman, Ngaio Marsh

1941 (some sources say 1942) classic English detective fiction; eleventh of Marsh's novels of Inspector Roderick Alleyn. For his own amusement, Jonathan Royal invites six mortal enemies to a party at his house in Dorset. Really, the first surprise is that he isn't murdered.

January 08, 2017 09:01 AM

January 07, 2017

Roger Bell-West

Musaigen no Phantom World

2016 contemporary fantasy novel adaptation in 13 episodes: Anidb, vt "Myriad Colors Phantom World". Phantoms, weird magical creatures created by the human subconscious, are everywhere; Haruhiko and his school friends seal them away when they cause trouble.

January 07, 2017 09:00 AM

January 06, 2017

Roger Bell-West

Lost Things, Melissa Scott and Jo Graham

2012 historical fantasy, first in the Order of the Air series. In 1929, as Lake Nemi is drained in search of its archaeological treasures, an ancient evil is loosed. Only four heroic aviator-magicians will be able to fight it.

January 06, 2017 09:02 AM

Jonathan McDowell

2016 in 50 Words

Idea via Roger. Roughly chronological order. Some things were obvious inclusions but it was interesting to go back and look at the year to get to the full 50 words.

Speaking at BelFOSS. Earthlings birthday. ATtiny hacking. Speaking at ISCTSJ. Dublin Anomaly. Co-habiting. DebConf. Peak Lion. Laura’s wedding. Christmas + picnic. Engagement. Car accident. Car write off. Tennent’s Vital. Dissertation. OMGWTFBBQ. BSides. New job. Rachel’s wedding. Digital Privacy talk. Graduation. All The Christmas Dinners. IMDB Top 250. Shay leaving drinks.

(This also serves as a test to see if I’ve correctly updated Planet Debian to use https and my new Hackergotchi that at least looks a bit more like I currently do.)

January 06, 2017 08:03 AM

January 05, 2017

Roger Bell-West

Royal Navy School

2016 reality television, 7 episodes. The series follows recruits through the ten weeks of selection for the Royal Navy at HMS Raleigh.

January 05, 2017 09:04 AM