Planet Uknot

January 16, 2019

Roger Bell_West

Matricide at St Martha's, Ruth Dudley Edwards

1994 mystery, fifth in the Robert Amiss series. Amiss is recruited by his old friend "Jack" Troutbeck, Bursar of St Martha's College in Cambridge, to help her sort out the politics of how a bequest will be used. Of course, some people have strong and murderous feelings about that.

January 16, 2019 09:03 AM

January 15, 2019

Roger Bell_West

Doctor Who 2/11.11: Resolution

Chibnall continues to do far more than his share of the writing, but manages a thing that hasn't happened for a while: an end-of-year special with an actual story, rather than a sequence of scenes showing off how cool the protagonists are.

January 15, 2019 09:02 AM

January 14, 2019

Jonathan Dowland

Amiga/Gotek boot test

This is the fourth part in a series of blog posts. The previous post was part 3: preliminaries.

A500 mainboard

A500 mainboard

In 2015 Game 2.0, a Retro Gaming exhibition visited the Centre for Life in Newcastle. On display were a range of vintage home computers and consoles, rigged up so you could play them. There was a huge range of machines, including some Arcade units and various (relatively) modern VR systems that were drawing big crowds but something else caught my attention: a modest little Amiga A500, running the classic puzzle game, "Lemmings".

A couple of weeks ago I managed to disassemble my Amiga and remove the broken floppy disk drive. The machine was pretty clean under the hood, considering its age. I fed new, longer power and data ribbon cables out of the floppy slot in the case (in order to attach the Gotek Floppy Emulator externally) and re-assembled it.

Success! Lemmings!

Success! Lemmings!

I then iterated a bit with setting up disk images and configuring the firmware on the Gotek. It was supplied with FlashFloppy, a versatile and open source firmware that can operate in a number of different modes and read disk images in a variety of formats. I had some Amiga images in "IPF" format, others in "ADF" and also some packages with "RP9" suffixes. After a bit of reading around, I realised the "IPF" ones weren't going to work, the "RP9" ones were basically ZIP archives of other disk images and metadata, and the "ADF" format was supported.

Amiga & peripherals on my desk

Amiga & peripherals on my desk

For my first boot test of the Gotek adaptor, the disk image really had to be Lemmings. Success! Now that I knew the hardware worked, I spent some time re-arranging my desk at home, to try and squeeze the Amiga, its peripherals and the small LCD monitor alongside the equipment I use for my daily work. It was a struggle but they just about fit.

The next step was to be planning out and testing a workflow for writing to virtual floppies via the Gotek. Unfortunately, before I could get as far as performing the first write test, my hardware developed a problem…

January 14, 2019 07:42 PM

Amiga floppy recovery project, part 3: preliminaries

This is the third part in a series of blog posts, following Amiga floppy recovery project, part 2. The next part is Amiga/Gotek boot test.

The first step for my Amiga project was to recover the hardware from my loft and check it all worked.

When we originally bought the A500 (in, I think, 1991) we bought a RAM expansion at the same time. The base model had a whole 512KiB of RAM, but it was common for people to buy a RAM expander that doubled the amount of memory to a whopping 1 MiB. The official RAM expander was the Amiga 501, which fit into a slot on the underside of the Amiga, behind a trapdoor.

The 501 also featured a real-time clock (RTC), which was powered by a backup NiCad battery soldered onto the circuit board. These batteries are notorious for leaking over a long enough time-frame, and our Amiga had been in a loft for at least 20 years. I had heard about this problem when I first dug the machine back out in 2015, and had a vague memory that I checked the board at the time and could find no sign of leakage, but reading around the subject more recently made me nervous, so I double-checked.

AMRAM-NC-issue 2 RAM expansion

AMRAM-NC-issue 2 RAM expansion

Lo and behold, we don't have an official Commodore RAM expander: we were sold a third-party one, an "AMRAM-NC-issue 2". It contains the 512KiB RAM and a DIP switch, but no RTC or corresponding battery, so no leakage. The DIP switch was used to enable and disable the RAM expansion. Curiously it is currently flicked to "disable". I wonder if we ever actually had it switched on?

The follow-on Amiga models A500+ and A600 featured the RTC and battery directly on the machine's mainboard. I wonder if that has resulted in more of those units being irrevocably damaged from leaking batteries, compared to the A500. My neighbours had an A600, but they got rid of it at some point in the intervening decades. If I were looking to buy an Amiga model today, I'd be quite tempted by the A600, due to its low profile, lacking the numpad, and integrated composite video output and RF modulator.

Kickstart 1.3 (firmware) prompt

Kickstart 1.3 (firmware) prompt

I wasn't sure whether I was going to have to rescue my old Philips CRT monitor from the loft. It would have been a struggle to find somewhere to house the Amiga and the monitor combined, as my desk at home is already a bit cramped. Our A500 was supplied with a Commodore A520 RF adapter which we never used in the machine's heyday. Over the Christmas break I tested it and it works, meaning I can use the A500 with my trusty 15" TFT TV (which has proven very useful for testing old equipment, far outlasting many monitors I've had since I bought it).

A520 RF modulator and external FDD

A520 RF modulator and external FDD

Finally I recovered my old Amiga external floppy disk drive. From what I recall this had very light usage in the past, so hopefully it still works, although I haven't yet verified it. I had partially disassembled this back in 2015, intending to re-purpose the drive into the Amiga. Now I have the Gotek, my plans have changed, so I carefully re-assembled it. Compared to enclosures I've used for PC equipment, it's built like a tank!

The next step is to remove the faulty internal floppy disk drive from the A500 and wire up the Gotek. I was thwarted from attempting this over the Christmas break. The Amiga enclosure's top part is screwed on with T9 Torx screws, and I lacked the right screwdriver part to remove it. I've now obtained the right screwdriver bit and can proceed.

January 14, 2019 07:42 PM

Roger Bell_West

The Old English Baron, Clara Reeve

1778 gothic horror, and another prototype for the gothic novel. (Also published in 1777 in a very limited edition as The Champion of Virtue.) Some time in the 1430s, Sir Philip Harclay returns from the wars to find his old friend mysteriously dead, and the friend's castle with a new lord. But there is a Suspiciously Superior Peasant being raised in the household…

January 14, 2019 09:04 AM

January 13, 2019

Roger Bell_West

Entrapment

1999 caper film, dir. Jon Amiel, Sean Connery, Catherine Zeta-Jones; IMDb / allmovie. Gin the insurance investigator tracks down Mac the art thief, but are either of them really what they seem? Of course not.

January 13, 2019 09:04 AM

January 12, 2019

Roger Bell_West

Cop Hater, Ed McBain

1956 police procedural, first in the 87th Precinct series. When an off-duty detective is murdered, it might have been for any number of reasons. Then his partner is shot with the same gun…

January 12, 2019 09:04 AM

January 11, 2019

Roger Bell_West

How to Choose a GURPS Magic System

GURPS 4th edition has a lot of magic systems, as well as rules for designing your own. How can you choose which one(s) you should use in your new campaign?

January 11, 2019 09:02 AM

January 10, 2019

Jonathan Dowland

Amiga floppy recovery project, part 2

This is the second part in a series of blog posts. The first part is Amiga floppy recovery project. The next part is Amiga floppy recovery project, part 3: preliminaries.

Happy New Year!

Nearly four years ago I wrote about my ambition to read the data from my old collection of Amiga floppy disks. I'd wanted to do this for a long time prior to writing that. I knew we still had the Amiga but my family were fairly sure we didn't have the floppies. It was the floppies turning up in 2015 that made the project viable.

I haven't done much for this project in the last four years. I began to catalogue the floppies, and the Amiga itself, the floppies and a 2½ foot pile of old Amiga magazines moved from my parent's loft to mine (as they moved house). During the move, we did briefly power on the Amiga to make sure it still worked. It booted fine, but we couldn't read any disks. I tried about 5 before I felt it was more likely the drive was dead than all the floppies we had tried. If that were the case, then the broken drive might damage any disks we tried in it.

The next step is to replace the internal disk drive in the Amiga, so that I could boot software. After that there are a number of different approaches I could take to move data onto a modern machine, including attempting to connect the Amiga to a network for the first time in its life. I have an external floppy disk drive that I considered swapping into the Amiga, but didn't attempt it.

This Christmas my brother bought me a Gotek floppy disk emulator, modified for use with Amiga computers. This is a lightweight device that can be connected to the Amiga in place of the real floppy disk drive, re-using the existing power and data ribbon cables. It features a USB port, two buttons and a small LCD numeric display. The intention is that you insert a USB drive containing Amiga disk images, and the Gotek presents them to the Amiga as if they were real disks.

Thanks to this device, I can now finally embark upon my disk-reading project!

January 10, 2019 04:41 PM

Roger Bell_West

The Graveyard Game, Kage Baker

2001 science fiction, fourth of The Company series. Mendoza has been disappeared by the Company, the same organisation that made her an immortal cyborg slave in the first place. Her recruiter Joseph, and former co-worker Lewis, try to find out what happened to her.

January 10, 2019 09:01 AM

January 09, 2019

Roger Bell_West

Stabcon 2019

This long-running games convention had another instance at the start of January, on a slightly chilly but not cold weekend. With images; cc-by-sa on everything.

January 09, 2019 09:02 AM

January 08, 2019

Jonathan McDowell

MP130 Maple Leaf and Debian

Like any conference one of the nice things about DebConf is the random interesting people you meet. At the DebConf18 conference dinner I had the pleasure of talking to Jia-Bin Huang, from Centrum Embedded Systems. He told me about how he was using Debian as the basis for his MapleBoard (mostly in Chinese, unfortunately) project, which is based on the Allwinner H3 and has thousands of shipped units. The next day I went to take a look at the website (Google Translate proved helpful), which has details of the board including full schematics of the board. I reached out to Jia-Bin to see if he was still at the conference and if he had any boards with him, but he’d already returned to Taipei. He generously offered to send me one and a few weeks later (mostly due to UK customs delays) I had an “Economy Edition” board in my hands.

MapleBoard MP130

Getting up and running was simple; the board came with a pl2303 based USB serial cable, but I found it a little unreliable so ended up using my own cable. The supplied documentation was in Chinese, but the login details were clearly visible - username mpl1, password 1234. After logging in I found it was running a lightly customized Debian Stretch variant, with the following packages not from the main Debian repository:

base-files              9.9+mp4     Maple base system miscellaneous files
debian-archive-keyring  2017.7+mp1  GnuPG archive keys of the Debian archive
distro-info-data        0.36+mp1    information about the distributions' releases (data files)
linux-image-4.15.2a…    4.15.2a…    Linux kernel, version 4.15.2a-02769-g6d0ce60c8d56
maplewebde              0.1~rc4-2   Web interface to communicate with mapleboard

maplewebde seems to be a web interface for interacting with the board, but it’s in Chinese so I wasn’t able to make much sense of it. I was more interested in the kernel - how much customisation had been done, and how much was already mainlined. Happily the Linux sunxi folk have done a lot of great work in getting things upstream, so although the supplied kernel was using its own drivers (largely branded Mapleboard rather than Allwinner) all the necessary pieces were in mainline. I did a little bit of cleanup of the supplied device tree configuration file, and am pleased to say that as of 5.0-rc1 a multi_v7_defconfig will build a kernel and a sun8i-h3-mapleboard-mp130.dtb file which Just Work™ on the device.

What about the board itself? I haven’t thrown a lot at it, but it feels reasonably responsive compared to some other ARM boards I’ve played with. Quad 1GHz cores and 1GB RAM is a nice enough base spec, though it is ARMv7 so 32-bit only. Unfortunately the “Economy Edition” doesn’t have HDMI on board or I’d have seen about trying to use it as a media player - the video decoding engine apparently has Free drivers based on a vdpau backend, which is pretty cool. There’s no SATA, so it can’t be pressed into service as a build machine easily. I suspect in the long run I’ll press it into service as part of my home automation setup, somewhere I need more oompf than an ESP8266 but still want low power consumption - there are a number of GPIOs conveniently brought out to a 40 pin header.

In general terms of the target market my understanding is the board is largely pitched as a development board, with Centrum being prepared to do customisations for sufficiently sized runs. The open nature of the board, and availability of good support upstream (even if it’s come from the community rather than Allwinner) certainly makes it an attractive option if you’re looking for a 32-bit ARM platform, and although I’m not aware of any option other than ordering from Taiwan at present I’ve found Jia-Bin to be helpful with any queries I’ve had. Also I quite like supporting companies that are using Debian. :) Worth a look if you’re in the market for such a thing.

January 08, 2019 07:07 PM

Steve Kennedy

It's time to Blaze ahead (well Beryl)

Beryl (as was Blaze) have started shipping their new front-facing white light/laser combo now called the Laseerlight Core. It's considerably smaller than the older Laserlight and attaches via a silicon strap that goes around the handlebars. There are two inserts for the clip that attaches to the Core to cope with differing tube widths.

The case is now plastic rather than aluminium and it's an altogether smaller design, though the light is 400 lumens along with the green laser that projects the bicycle shape on to the road.

There are 4 modes Day Flash, Pulse, 100% solid, 50% solid and the laser on or off. The battery should last for 41 hours (according to Beryl).

No idea why the iPhone made the green laser look bluefish, but it's really green.

The original Laserlight cost £125, but the new Core is a much more wallet friendly £69.99 and much smaller so it can easily be taken off the bike and put in a rucksack or bag and not take a lot of space.

by Steve Karmeinsky (noreply@blogger.com) at January 08, 2019 07:07 PM

Roger Bell_West

The Dare and the Doctor, Kate Noble

2016 American Regency romance, third and last of a linked series. Dr Gray conducts an academic correspondence with Miss Babcock, whom he met in passing during the events of the previous book. But when she comes to London to show off her hybrid rose, will they recognise their feelings for each other, or will societal pressures get in the way?

January 08, 2019 09:01 AM

January 07, 2019

Jonathan Dowland

ZDBSP

A Debian package of ZDBSP is now available in the unstable distribution.

I mentioned ZDBSP when I wrote about glBSP. It's my hope to remove glBSP from the archive, leaving ZDBSP as the sole Doom nodebuilder in Debian. I might try to do this in time for the next stable release. That means probably actioning the removal before the 12th March.

If you use glBSP for any purpose, please give ZDBSP a look and make sure it works for you.

January 07, 2019 12:24 PM

Roger Bell_West

New Year's Day Boardgames at Home

Late on New Year's Day, we got together for eight-player games, and ended up on the unexpected theme of "games you can't buy right now".

January 07, 2019 09:01 AM

January 06, 2019

Roger Bell_West

Andromeda's Choice, William C. Dietz

2013 military science fiction, eleventh novel in the Legion of the Damned series (and second in the prequel sub-series). "McKee" is still hiding from the usurper Empress, but after winning the fight in the previous book she has to go to Earth to get a medal… and she might well be recognised.

January 06, 2019 09:04 AM

January 05, 2019

Roger Bell_West

December 2018 Trailers

Some trailers I've seen recently, and my thoughts on them. (Links are to youtube. Opinions are thoroughly personal. Calibration: I hate everything.)

January 05, 2019 09:01 AM

Dcember 2018 Trailers

Some trailers I've seen recently, and my thoughts on them. (Links are to youtube. Opinions are thoroughly personal. Calibration: I hate everything.)

January 05, 2019 09:01 AM

January 04, 2019

Roger Bell_West

Wolf in the Shadows, Marcia Muller

1993 mystery, thirteenth in Muller's series about Sharon McCone, private investigator in San Francisco.

January 04, 2019 09:00 AM

January 03, 2019

Roger Bell_West

2018 in boardgames

2018 was another very boardgame-ful year: more playing, less buying.

January 03, 2019 09:01 AM

2017 in boardgames

2018 was another very boardgame-ful year: more playing, less buying.

January 03, 2019 09:01 AM

January 02, 2019

Jonathan Dowland

Maker Faire UK RIP

I'm sad to belatedly witness the apparent demise of Maker Faire UK: an annual event for (mostly) amateur makers of all skill-levels that ran from the Centre for Life in Newcastle.

The Centre was always packed out for the Maker Faire, which attracted a huge range of people from all over the UK, and beyond. I was proud that this was taking place in my city. I always enjoyed attending the Faire, although it was often bitter-sweet: none of my own Making was of the kind of that you could show off at an exhibition like this. I often found myself wondering if I should try my hand at more physical hobbies.

The Centre for Life is remodelling itself this January, and as part of that project or planning, they seem to have decided that the Maker Faire was not a good fit for their new direction. They talk about opening a new space for "crafting, tinkering and creativity" within the Centre in the Spring, but I very much doubt it will attract the breadth and depth of talent that the Maker Faire did.

The decision was announced back in September, but they've already pulled the plug on the website hosting the announcement (https://makerfaireuk.com/), which was up so briefly it was not captured by either https://web.archive.org/ or https://archive.is. Their defunct twitter account points cryptically at this dead website with a final tweet of "We have posted important news about Maker Faire UK on the homepage of the website: https://makerfaireuk.com/".

At the last event in April, 2018, a friend from the Paper Jam Comics Collective demonstrated an incredible Lego Mindstorms-powered Comics drawing robot:

Other things that stuck out to me at the last Faire were some of the non-computer, non white-male-dominated crafts, such as creative stitching, knitting, crocheting and similar. Some of the exhibitors there seemed to feel like they were outsiders, but I felt they were as deserving to be there as anyone else (and more so than some of the purely vendor tables); some of them proudly exclaimed as such, and their work was a refreshing change of pace from the more dominant themes.

I recall enjoying music-related stalls in years gone by. There was usually a table of synthesizers which was fun to noodle with. I'm not sure what making they were exhibiting, perhaps the intention was their own music: there was no sign of DIY synth building. But it was fun. Various other stalls tended to have DIY synths or novel musical input devices wired up to Arduinos, or Raspberry Pis. The Centre itself has a permanently installed reactive table set up as a synthesizer which my daughter and I both enjoy playing with when we visit. One year a company was demonstrating reactive lighting boxes that I'm fairly sure were designed with (or in collaboration with) Brian Eno.

I was also impressed to see a stand demonstrating a mini replica of a PDP-8 minicomputer (it might have been this one), which reminded me of the Newcastle University Computing History Committee, and made me ponder whether any of our activities in that committee would be worthy of a stall at the next Maker Faire. For the last couple of years I've also weighed up whether or not my daughter was old enough to enjoy attending. I was fairly sure that 2019 would have been the first year that she might be, but alas, it will not come to pass.

January 02, 2019 11:41 AM

Roger Bell_West

2018 in Books

In 2018 I read 169 books, slightly more than in the last few years.

January 02, 2019 09:04 AM

January 01, 2019

Roger Bell_West

Roger's 2018 in 50 Words

A friend likes to sum up his year in a set number of words, and I copy this fine idea. "Think of it as a short and un-boastful summary of the year, which nobody is expected to understand all of."

January 01, 2019 09:02 AM

December 31, 2018

Jonathan McDowell

Maxcio W-UK007S Power Monitoring Smart Plug notes

The house server I built in 2013 is getting on a bit, so I’d like to replace it. That’s currently held up on availability of Ryzen 7 2700E CPUs, which look to be the best power consumption/performance trade-off available at present. While I wait for stock I figured I should see how the current i3-3220T is doing.

To do so I decided to buy a Smart Plug that advertised energy monitoring, planning to integrate it into my current setup for the monitoring and then being able to use it for general power control once the upgrade comparison is complete. I ended up with a pair of Maxcio Smart Plugs - pricing and availability worked out and I’d found confirmation that the W-US002S was ESP8266 based.

The model I ended up with is externally marked as a W-UK007S. It’s a fairly neat form factor (slightly smaller than the SonOff S26 I already have, which doesn’t do power monitoring). It also turned out to be easy to take apart; there is a circular cover in the middle which can be carefully popped out, revealing the single screw holding the device together.

Front of Maxcio Smart Plug

The back plate has 4 clips holding it together at the corners and can be gently pried off. Inside there’s a main circuit board labelled “W-US0007S-V0.3” which has the relay on it and a perpendicular board with the ESP8266 module and power monitoring chip on it.

Inside of Maxcio Smart Plug

Sadly the layout didn’t match anything I was familiar with, or could find any details of. That meant I had to do some detective work to figure out how to talk to the ESP8266. It was easy enough to work out GND + VCC by following PCB tracks. Likewise the relay, the button and the LED (underneath the button, and separately controlled from the relay, unlike the S26). Next move was to hook up power (just a low voltage supply to GND/VCC, I did not engage in any experimentation involving mains voltages!) and monitor each unknown pin in turn in the hope I’d find TX (even if the supplied firmware didn’t print anything out the ESP8266 prints a message on boot, so I’d definitely see something if it was there).

Thankfully TX was brought out to the module connection to the main PCB, so I had something I could monitor.

Maxcio boot log
 ets Jan  8 2013,rst cause:1, boot mode:(3,7)

load 0x40100000, len 1396, room 16 
tail 4
chksum 0x89
load 0x3ffe8000, len 776, room 4 
tail 4
chksum 0xe8
load 0x3ffe8308, len 540, room 4 
tail 8
chksum 0xc0
csum 0xc0

2nd boot version : 1.4(b1)
  SPI Speed      : 40MHz
  SPI Mode       : QIO
  SPI Flash Size & Map: 8Mbit(512KB+512KB)
jump to run user1 @ 1000

OS SDK ver: 1.4.2(23fbe10) compiled @ Sep 22 2016 13:09:03
phy v[notice]user_main.c:268 SDK version:1.4.2(23fbe10)
[notice]user_main.c:270 tuya sdk version:1.0.6
[notice]user_main.c:275 tuya sdk compiled at Jul 26 2017 15:27:36
[notice]user_main.c:277 BV:5.20 PV:2.1 LPV:3.1
reset reason: 0
epc1=0x00000000, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000,depc=0x00000000
mode : softAP(de:4f:22:2c:76:93)
dhcp server start:(ip:192.168.4.1,mask:255.255.255.0,gw:192.168.4.1)
add if1
bcn 100
bcn 0
del if1
usl
mode : sta(dc:4f:22:2c:76:93)
add if0
scandone

[notice]device.c:1694 fireware info name:esp_hys_qx_single_light_jlplug version:1.0.0
[notice]device.c:1695 PID=gDYvLrWhRoqpgHMj

[notice]device.c:1696 FW Compiled date:Feb  5 2018
[notice]gw_intf.c:240 Authorization success
[notice]device.c:1722 ##  AUTH DONE ##

[err]bl0937.c:1013 ### get_coefficient get err: 28
[err]device.c:1767 get ele data err...
[err]device.c:1772 get tem ele data err...
del if0
usl
mode : null
force slp enable,type: 2
fpm open,type:2 0
[notice]device.c:2041  # STAT_STA_UNCONN STAT_LOW_POWER#

[notice]device.c:2042  # STAT_STA_CONN #

[notice]device.c:648 #####I_VAL:0#####

[notice]device.c:654 ***************************Check Start 1**********************************
[notice]device.c:655 ### cur:0 mA power:0 W vol:0 V
[notice]device.c:656 ****************************Check Stop**********************************

Next I took the remaining unknown pins and tried grounding them on boot, in an attempt to find GPIO0 (which needs to be grounded to access the ROM serial bootloader). I ended up finding GPIO2 first, and then eventually figuring out the LED was using GPIO0 - learning the lesson not to assume pins don’t have multiple uses. Now I had TX + GPIO0 I could hold GPIO0 on boot and look for RX by probing the remaining pins and seeing if esptool could talk to the bootloader. Again, I was successful.

At that point I was able to download the firmware from flash, and poke it in the hope of working out the GPIO assignments (I’m a software guy, I’m happier with an assembly dump than probing randomly around a board in the hope of enlightenment). I generated a crude .elf from the flash dump using esp-bin2elf, hacking it up to cope better with an OTA ROM image and skip the boot loader. I initially used objdump to examine the result, which wasn’t that helpful, and then found ScratchABit, which made things much easier. What would have been ideal is some way to load in the .a static libraries from the SDK and automatically map those to the code; as well as providing some useful symbols it would have avoided work looking at functions that were irrelevant. The ESP8266 seems to want various levels of indirection to access functions and memory locations so it’s not just a simple matter of looking for an I/O request to a specific location, but I was able to figure out that the button was on GPIO13 and the relay on GPIO15.

All that left was the bit I was actually interested in - the power monitoring. The appropriate chip (clearly attached to a low resistance bridge from one of the AC power pins, and also attached to the other pin) on the PCB was marked “HJL-01 / J1749CYH / D797480E”. Whatever you find on the web this is not the same as the HLW8012. It’s very similar in operation but is actually a BL0937. Electrodragon’s Energy Meter page was the best resource I found - it has a link to the Chinese datasheet for the BL0937, which in combination with Google Translate allows the basic principles to be understood. Both devices work by having a pin (CF) which outputs pulses based on monitored power consumption, and a pin (CF1) which can be switched between monitoring current and voltage via a third pin (SEL). For the BL0937 you can just count pulses; the pulse width is fixed at 38µS and it’s just the frequency which varies. I’d found the GPIO interupt handler in my flash disassembly which indicated that GPIO5 was connected to CF and GPIO14 to CF1. Additionally the handler around GPIO14 needs to check which mode the chip is currently in, which let me discover GPIO12 was connected to this pin.

That resulted in the following pin mapping of the daughter PCB; the remaining 4 pins weren’t of interest once I had the ones I needed, so I didn’t do further investigation:

PCB EDGE

                +----+
LED    / GPIO0  |1 12| 3.3V
                |2 11|
BUTTON / GPIO13 |3 10| GPIO2
TX              |4  9| RX
RELAY  / GPIO15 |5  8|
GND             |6  7|
                +----+

PCB CENTER

Of course the frequency values that come out of the BL0937 are not directly usable; there’s a certain amount of conversion/calibration that needs to be done. Thankfully although the datasheet has an equation that includes an odd constant value as well as the internal reference voltage this all boils down to a simple scaling value. I ended up using a multimeter to calibrate the voltage and then a standalone power meter + table lamp to calibrate power/current. Better results could be obtained with a constant voltage source and a known resistance load but this worked out close enough for my needs.

I wrote my own driver to fit within the ESP8266 MQTT framework I use, but if you’re looking for something more off the shelf ESPurna is probably a good start. Ticket #737 talks about adding support for the BL0937 (it’s close enough to the HLW8012 that you can get away with just changing the scaling factors), and the Homecube SP1 seems to use the same GPIOs for the various pieces of hardware.

I’ve put all the images from my teardown into a W-UK007S Teardown Album on Google Photos, in case it’s useful to anyone.

December 31, 2018 05:57 PM

Roger Bell_West

Boxing Day Boardgames 2018

This really is a tradition now – for the fourth year in a row, a bunch of us without family commitments got together for boardgames.

December 31, 2018 09:00 AM

December 30, 2018

Roger Bell_West

The Stone Sky, N K Jemisin

2017 Hugo- and Nebula-award-winning science-fantasy, third in its trilogy. It seems as though there may after all be a chance to save the world; but is it worth saving, and what will it look like afterwards?

December 30, 2018 09:01 AM

December 29, 2018

Roger Bell_West

National Army Museum

On a slightly crisp December day, I went to the National Army Museum in Chelsea.

December 29, 2018 09:04 AM

December 28, 2018

Roger Bell_West

No Enthusiasm for Keyforge

Fantasy Flight Games has recently released Keyforge, a Unique Deck Game in which every deck one buys is different from every other. It's been getting mostly positive reviews. Why am I so comprehensively uninterested?

December 28, 2018 09:43 AM

December 27, 2018

Roger Bell_West

Miss Goodhue Lives For a Night, Kate Noble

2016 American Regency romance novella, side story in a linked series. In her youth, Cecilia Goodhue ran off with a wild young man in the direction of Gretna Green; but they were caught, and she learned he was a fortune-hunter; to escape some of the scandal, she lived by the grace of her married sister and became a teacher in a village school. Now, ten years later, her own younger sister has run off with a soldier, and in an attempt to find her she'll meet that man again.

December 27, 2018 09:04 AM

December 26, 2018

Jonathan McDowell

⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛

We obsess over free speech. It is a club with which we beat anyone who we believe is trying to silence us, or otherwise make our opinion unheard. I live in Northern Ireland, so my legal right flows from the European Convention on Human Rights (ECHR) via a combination of the Human Rights Act 1998 (HRA) and the Northern Ireland Act 1998 s6(2) (NIA). Additionally the Charter of Fundamental Rights of the European Union applies in certain circumstances.

If you look at the ECHR the phrases “Free Speech” and “Freedom of Speech” do not actually appear. The relevant article is article 10, “Freedom of Expression”, which says:

  1. Everyone has the right to freedom of expression. This right shall include freedom to hold opinions and to receive and impart information and ideas without interference by public authority and regardless of frontiers. This Article shall not prevent States from requiring the licensing of broadcasting, television or cinema enterprises.

  2. The exercise of these freedoms, since it carries with it duties and responsibilities, may be subject to such formalities, conditions, restrictions or penalties as are prescribed by law and are necessary in a democratic society, in the interests of national security, territorial integrity or public safety, for the prevention of disorder or crime, for the protection of health or morals, for the protection of the reputation or rights of others, for preventing the disclosure of information received in confidence, or for maintaining the authority and impartiality of the judiciary.

Paragraph 1 is the one we trot out. Paragraph 2 doesn’t get so much airtime. The right to freedom of expression is not unqualified (compare Article 3 which prohibits torture and is short and succinct). There are recognised needs to curtail the freedom of expression. Even if you ignore state decisions such as “national security” or “protection of morals” it is still necessary to respect the right of others. The common tension is between the right to Freedom of Expression and the Right to respect for private and family life (Article 8), but prevention of disclosure of information received in confidence is also explicitly called out.

These qualifications on the right to freedom of expression shouldn’t seem surprising. Society is all about balancing the needs of everyone to try and achieve some sort of harmony. We can be generally in favour of a free press while also accepting that there need to be some limits on how much they can intrude into people’s personal lives. Equally just because one finds doxxing despicable doesn’t mean one is against freedom of expression. We need to stop the black and white view that any attempt to curtail freedom of expression, no matter what the context, is a violation of an unqualified fundamental right. Too much simplification of the details takes place.

The other thing that needs to be remembered is that this right relates to governments - “without interference by public authority”. It doesn’t say anything about what you as a private individual have to do to enable my right to freedom of expression. I can’t force you to read this blog, or provide me with aggregation on your site to help me reach a wider audience. Equally I don’t have to host comments that are spam, or that I find offensive. No one’s right to freedom of expression is impinged here; I can blog what I like if I’m the one hosting it, people can conduct their marketing in other forum without me being able to prevent them doing so.

We also don’t see any statement about what organisations have to do. The HRA talks explicitly about being able to review the actions of public authorities, but it is rightly silent on anything to do with private organisations. If you can’t be seen as acting as an arm of the government, the HRA doesn’t apply. Organisations don’t have to support the views of those they disagree with - just look at the split in newspapers over whether Brexit is a good thing or not - or things unrelated to the organisation (there’s no reason I would be syndicated on Planet GNOME). That might be disappointing to individuals, but trotting out Freedom of Expression as a defence or a justification for why an organisation should help promote our views just fails to understand the nature of the right.

I don’t make any points above that haven’t been made more eloquently by others countless times, but sometimes I feel we need to be reminded of them.

December 26, 2018 01:57 PM

Roger Bell_West

Pyramid 122: All Good Things

Pyramid, edited by Steven Marsh, was the monthly GURPS supplement containing short articles with a loose linking theme – though for this final issue that's mostly "look at all the different genres we've covered".

December 26, 2018 09:02 AM

December 25, 2018

Roger Bell_West

Mendoza in Hollywood, Kage Baker

2000 science fiction, third of The Company series. In 1862, Mendoza is collecting plant samples from what will one day be Los Angeles, working out of a stagecoach stop in the Cahuenga Pass run by fellow immortal cyborg agents of the Company; she's still trying to cope with her post-traumatic stress. That won't go well for her.

December 25, 2018 09:04 AM

December 24, 2018

Roger Bell_West

Marlow Tabletop and Board Games 17 December 2018

This Meetup-based boardgames group continues to meet at the Marlow Donkey.

December 24, 2018 09:04 AM

Andy Smith (strugglers.net)

The Internet of Unprofitable Things

Gather round children

Uncle Andrew wants to tell you a festive story. The NTPmare shortly after Christmas.

A modest proposal

Nearly two years ago, on the afternoon of Monday 16th January 2017, I received an interesting BitFolk support ticket from a non-customer. The sender identified themselves as a senior software engineer at NetThings UK Ltd.

Subject: Specific request for NTP on IP 85.119.80.232

Hi,

This might sound odd but I need to setup an NTP server instance on IP address 85.119.80.232.

wats 85.119.80.232 precious?

85.119.80.232 is actually one of the IP addresses of one of BitFolk’s customer-facing NTP servers. It was also, until a few weeks before this email, part of the NTP Pool project.

Was” being the important issue here. In late December of 2016 I had withdrawn BitFolk’s NTP servers from the public pool and firewalled them off to non-customers.

I’d done that because they were receiving an unusually large amount of traffic due to the Snapchat NTP bug. It wasn’t really causing any huge problems, but the number of traffic flows were pushing useful information out of Jump‘s fixed-size netflow database and I didn’t want to deal with it over the holiday period, so this public service was withdrawn.

NTP?

This article was posted to Hacker News and a couple of comments there said they would have liked to have seen a brief explanation of what NTP is, so I’ve now added this section. If you know what NTP is already then you should probably skip this section because it will be quite brief and non-technical.

Network Time Protocol is a means by which a computer can use multiple other computers, often from across the Internet on completely different networks under different administrative control, to accurately determine what the current time is. By using several different computers, a small number of them can be inaccurate or even downright broken or hostile, and still the protocol can detect the “bad” clocks and only take into account the more accurate majority.

NTP is supposed to be used in a hierarchical fashion: A small number of servers have hardware directly attached from which they can very accurately tell the time, e.g. an atomic clock, GPS, etc. Those are called “Stratum 1” servers. A larger number of servers use the stratum 1 servers to set their own time, then serve that time to a much larger population of clients, and so on.

It used to be the case that it was quite hard to find NTP servers that you were allowed to use. Your own organisation might have one or two, but really you should have at least 3 to 7 of them and it’s better if there are multiple different organisations involved. In a university environment that wasn’t so difficult because you could speak to colleagues from another institution and swap NTP access. As the Internet matured and became majority used by corporations and private individuals though, people still needed access to accurate time, and this wasn’t going to cut it.

The NTP Pool project came to the rescue by making an easy web interface for people to volunteer their NTP servers, and then they’d be served collectively in a DNS zone with some basic means to share load. A private individual can just use three names from the pool zone and they will get three different (constantly changing) NTP servers.

Corporations and those making products that need to query the NTP pool are supposed to ask for a “vendor zone”. They make some small contribution to the NTP pool project and then they get a DNS zone dedicated to their product, so it’s easier for the pool administrators to direct the traffic.

Sadly many companies don’t take the time to understand this and just use the generic pool zone. NetThings UK Ltd went one step further in a very wrong direction by taking an IP address from the pool and just using it directly, assuming it would always be available for their use. In reality it was a free service donated to the pool by BitFolk and as it had become temporarily inconvenient for that arrangement to continue, service was withdrawn.

On with the story…

They want what?

The Senior Software Engineer continued:

The NTP service was recently shutdown and I am interested to know if there is any possibility of starting it up again on the IP address mentioned. Either through the current holder of the IP address or through the migration of the current machine to another address to enable us to lease 85.119.80.232.

Um…

I realise that this is a peculiar request but I can assure you it is genuine.

That’s not gonna work

Obviously what with 85.119.80.232 currently being in use by all customers as a resolver and NTP server I wasn’t very interested in getting them all to change their configuration and then leasing it to NetThings UK Ltd.

What I did was remove the firewalling so that 85.119.80.232 still worked as an NTP server for NetThings UK Ltd until we worked out what could be done.

I then asked some pertinent questions so we could work out the scope of the service we’d need to provide. Questions such as:

  • How many clients do you have using this?
  • Do you know their IP addresses?
  • When do they need to use the NTP server and for how long?
  • Can you make them use the pool properly (a vendor zone)?

Down the rabbit hole

The answers to some of the above questions were quite disappointing.

It would be of some use for our manufacturing setup (where the RTCs are initially set) but unfortunately we also have a reasonably large field population (~500 units with weekly NTP calls) that use roaming GPRS SIMs. I don’t know if we can rely on the source IP of the APN for configuring the firewall in this case (I will check though). We are also unable to update the firmware remotely on these devices as they only have a 5MB per month data allowance. We are able to wirelessly update them locally but the timeline for this is months rather than weeks.

Basically it seemed that NetThings UK Ltd made remote controlled thermostats and lighting controllers for large retail spaces etc. And their devices had one of BitFolk’s IP addresses burnt into them at the factory. And they could not be identified or remotely updated.

Facepalm

Oh, and whatever these devices were, without an external time source their clocks would start to noticeably drift within 2 weeks.

By the way, they solved their “burnt into it at the factory” problem by bringing up BitFolk’s IP address locally at their factory to set initial date/time.

Group Facepalm

I’ll admit, at this point I was slightly tempted to work out how to identify these devices and reply to them with completely the wrong times to see if I could get some retail parks to turn their lights on and off at strange times.

Weekly??

We are triggering ntp calls on a weekly cron with no client side load balancing. This would result in a flood of calls at the same time every Sunday evening at around 19:45.

Yeah, they made every single one of their unidentifiable devices contact a hard coded IP address within a two minute window every Sunday night.

The Senior Software Engineer was initially very worried that they were the cause of the excess flows I had mentioned earlier, but I reassured them that it was definitely the Snapchat bug. In fact I never was able to detect their devices above background noise; it turns out that ~500 devices doing a single SNTP query is pretty light load. They’d been doing it for over 2 years before I received this email.

I did of course point out that they were lucky we caught this early because they could have ended up as the next Netgear vs. University of Wisconsin.

I am feeling really, really bad about this. I’m very, very sorry if we were the cause of your problems.

Bless. I must point out that throughout all of this, their Senior Software Engineer was a pleasure to work with.

We made a deal

While NTP service is something BitFolk provides as a courtesy to customers, it’s not something that I wanted to sell as a service on its own. And after all, who would buy it, when the public pool exists? The correct thing for a corporate entity to do is support the pool with a vendor zone.

But NetThings UK Ltd were in a bind and not allowing them to use BitFolk’s NTP server was going to cause them great commercial harm. Potentially I could have asked for a lot of money at this point, but (no doubt to my detriment) that just felt wrong.

I proposed that initially they pay me for two hours of consultancy to cover work already done in dealing with their request and making the firewall changes.

I further proposed that I charged them one hour of consultancy per month for a period of 12 months, to cover continued operation of the NTP server. Of course, I do not spend an hour a month fiddling with NTP, but this unusual departure from my normal business had to come at some cost.

I was keen to point out that this wasn’t something I wanted to continue forever:

Finally, this is not a punitive charge. It seems likely that you are in a difficult position at the moment and there is the temptation to charge you as much as we can get away with (a lot more than £840 [+VAT per year], anyway), but this seems unfair to me. However, providing NTP service to third parties is not a business we want to be in so we would expect this to only last around 12 months. If you end up having to renew this service after 12 months then that would be an indication that we haven’t charged you enough and we will increase the price.

Does this seem reasonable?

NetThings UK Ltd happily agreed to this proposal on a quarterly basis.

Thanks again for the info and help. You have saved me a huge amount of convoluted and throwaway work. This give us enough time to fix things properly.

Not plain sailing

I only communicated with the Senior Software Engineer one more time. The rest of the correspondence was with financial staff, mainly because NetThings UK Ltd did not like paying its bills on time.

NetThings UK Ltd paid 3 of its 4 invoices in the first year late. I made sure to charge them statutory late payment fees for each overdue invoice.

Yearly report card: must try harder

As 2017 was drawing to a close, I asked the Senior Software Engineer how NetThings UK Ltd was getting on with ceasing to hard code BitFolk’s IP address in its products.

To give you a quick summary, we have migrated the majority of our products away from using the fixed IP address. There is still one project to be updated after which there will be no new units being manufactured using the fixed IP address. However, we still have around 1000 units out in the field that are not readily updatable and will continue to perform weekly NTP calls to the fixed IP address. So to answer your question, yes we will still require the service past January 2018.

This was a bit disappointing because a year earlier the number had been “about 500” devices, yet despite a year of effort the number had apparently doubled.

That alone would have been enough for me to increase the charge, but I was going to anyway due to NetThings UK Ltd’s aversion to paying on time. I gave them just over 2 months of notice that the price was going to double.

u wot m8

Approximately 15 weeks after being told that the price doubling was going to happen, NetThings UK Ltd’s Financial Controller asked me why it had happened, while letting me know that another of their late payments had been made:

Date: Wed, 21 Feb 2018 14:59:42 +0000

We’ve paid this now, but can you explain why the price has doubled?

I was very happy to explain again in detail why it had doubled. The Financial Controller in response tried to agree a fixed price for a year, which I said I would be happy to do if they paid for the full year in one payment.

My rationale for this was that a large part of the reason for the increase was that I had been spending a lot of time chasing their late payments, so if they wanted to still make quarterly payments then I would need the opportunity to charge more if I needed to. If they wanted assurance then in my view they should pay for it by making one yearly payment.

There was no reply, so the arrangement continued on a quarterly basis.

All good things…

On 20 November 2018 BitFolk received a letter from Deloitte:

Netthings Limited – In Administration (“The Company”)

Company Number: SC313913

[…]

Cessation of Trading

The Company ceased to trade with effect from 15 November 2018.

Investigation

As part of our duties as Joint Administrators, we shall be investigating what assets the Company holds and what recoveries if any may be made for the benefit of creditors as well as the manner in which the Company’s business has been conducted.

And then on 21 December:

Under paragraph 51(1)(b) of the Insolvency Act 1986, the Joint Administrators are not required to call an initial creditors’ meeting unless the Company has sufficient funds to make a distribution to the unsecured creditors, or unless a meeting is requested on Form SADM_127 by 10% or more in value of the Company’s unsecured creditors. There will be no funds available to make a distribution to the unsecured creditors of the Company, therefore a creditors’ meeting will not be convened.

Luckily their only unpaid invoice was for service from some point in November, so they didn’t really get anything that they hadn’t already paid for.

So that’s the story of NetThings UK Ltd, a brave pioneer of the Internet of Things wave, who thought that the public NTP pool was just an inherent part of the Internet that anyone could use for free, and that the way to do that was to pick one IP address out of it at random and bake that into over a thousand bits of hardware that they distributed around the country with no way to remotely update.

This coupled with their innovative reluctance to pay for anything on time was sadly not enough to let them remain solvent.

by Andy at December 24, 2018 12:48 AM

December 23, 2018

Mark Goodge

The ghosts of Christmas past

I suppose most of us have memories of childhood Christmas. For me, the classic Christmases past are the times we went to my grandparents’ house for Christmas dinner.

We’d start the day, as children always do, by waking as early as we could to see what Father Christmas had brought us, and then unwrap family presents in the living room. Then it was off to chapel for the Christmas morning service, and then to Grandma’s house for dinner.

My extended family is quite large, and there always seemed to be a houseful. Three generations were there, we children, our parents, aunts and uncles, and then as well as our grandparents, lots of great uncles and great aunts that I rarely saw any other time of year. We weren’t a particularly close family, but we were a congenial one, and these annual get-togethers were an opportunity for the gown-ups to catch up with each other’s news over the past year and, as always, tell us children how much we’d grown.

The routine was always the same. After dinner, the men would go for a walk, leaving the women to do the washing up, and then we’d all congregate in the back room to swap presents and play games. No fancy electronics, of course, or even the latest board games – instead, we’d play “Hunt the thimble” and “How many eggs in the nest?”. The air was smokey, partly from the fireplace and partly from the men’s pipes. Even now, the smell of pipe smoke instantly transports me back to Christmas afternoon at Grandma’s. We’d take a break from games and presents to listen to the Queen’s speech on the radio (not on TV, my grandparents didn’t have one!).

Eventually, the games and presents would be done, and our immediate family would move on – typically, mum, dad, my brother and I would go either to my other grandparents (dad’s side of the family), or to my aunt and uncle’s house, for tea. There would be more presents to swap there, and more games to play. And then home, eventually, to bed. As we got older, the routine varied more, and my teenage memories of Christmas are quite different. But my most precious memories of childhood Christmas are those early ones, where the routine seemed to be fixed in stone – this is how it had always been done, and this is how it ever would be.

What I didn’t know then – and didn’t learn for a long time – was how difficult my grandmother found it, sometimes. Not difficult physically – she enjoyed hosting the Christmas dinner, and with plenty of people to muck in and lend a hand the cooking and cleaning wasn’t an arduous task. But, emotionally, it could be challenging, because of the memories it brought back for her.

One of the things I remember from those early Christmases is sitting on my Grandad’s knee. That was usually the point at which Grandma had to leave the room for a moment to “wipe a speck from her eye”. And why, for a little while, she seemed a bit quiet, and distant.

Because Grandma’s own children – my mum and uncle – never got to sit on their grandfather’s knee. Their grandfather (my great-grandfather, and Grandma’s dad) died when Grandma was still only 13. And, of my grandparents’ generation, there was one missing from those Christmas gatherings that Grandma would dearly have loved to be there. Her older brother Ralph was killed in action at Ecoust-Saint-Mein, France, in August 1918, just six months after her father had died.

Christmas 1918 was a very different occasion for Grandma’s family to how it had been before. The loss of both of the family’s breadwinners caused an upheaval that uprooted them all from their home and not only cost Grandma the people she loved most but shattered her hopes and dreams for the future. I can only imagine how they felt, trying to celebrate Christmas that year. To be honest, I don’t even know if they did. Maybe my great-grandmother tried to keep it going, for the sake of the younger children. Or maybe it was too hurtful even to bother.

Fifty years later, with her own grandchildren sitting at the dinner table on Christmas Day, Grandma still carried the memories and pain. And she carried them with her until she died. My childhood memories of Christmas are all happy ones. So, for that matter, are nearly all my adult ones. But in this, the centenary of Grandma’s first Christmas without her dad and big brother, I’m uncomfortably aware that the ghosts of Christmas past still have power to haunt me.

I don’t have any actual photos of an early family Christmas, but the one on this page is from the same era and location. It’s taken in the vegetable garden of Grandad and Grandma’s house in Lakenheath, Suffolk (more recent owners seem to have tarted it up a bit).

From left to right as we look at it, the adults are my mum, Uncle Peter, Aunty Jill, Grandma and Grandad. The children are my brother Stuart, me, and our cousin Judith. Dad isn’t in the photo because he’s behind the camera.

by Mark at December 23, 2018 05:06 PM

Roger Bell_West

The Flowers of Vashnoi, Lois McMaster Bujold

2018 short science fiction novella in the Vorkosigan universe (set between Captain Vorpatril's Alliance and Cryoburn). Ekaterin, Lady Vorkosigan, is supervising an experiment in cleaning up radioactive contamination round the former city of Vorkosigan Vashnoi; but something's going awry, and old secrets will come to light.

December 23, 2018 09:01 AM

December 22, 2018

Roger Bell_West

GURPS Monster Hunters 6: Holy Hunters, Jason Levine

This supplementary volume in the Monster Hunters series models real-world religions to give their monster-hunting adherents appropriate game effects.

December 22, 2018 09:01 AM

December 21, 2018

Roger Bell_West

Clubbed to Death, Ruth Dudley Edwards

1991 mystery, fourth in the Robert Amiss series. Once more, Amiss' friend Pooley sends him into an institution where murder may have taken place, for a bit of unofficial undercover work. This time it's as a waiter at ffeatherstonehaugh's, originally a club for roués that's since become gentrified, then fallen on distressingly moralistic times.

December 21, 2018 09:04 AM

December 20, 2018

Roger Bell_West

Boardgames at Home, December 2018

A boardgaming evening at home; too long since I did one of these.

December 20, 2018 09:02 AM

December 19, 2018

Roger Bell_West

Andromeda's Fall, William C. Dietz

2012 military science fiction, tenth novel in the Legion of the Damned series (but first in the prequel sub-series, which is why I'm reading it first). Catherine Carletto is the spoiled daughter of industrialist nobles. But when the Emperor's sister assassinates the Emperor and takes over, she purges all his supporters. Cat will have to grow up fast just to stay alive, never mind getting her revenge.

December 19, 2018 09:01 AM

December 18, 2018

Roger Bell_West

GURPS Dungeon Fantasy Monsters 4: Dragons, Sean Punch

This fourth Dungeon Fantasy Monsters book deals with the traditional Ultimate Fantasy Foe.

December 18, 2018 09:01 AM

December 17, 2018

Roger Bell_West

The Lie and the Lady, Kate Noble

2015 American Regency romance, second of a linked series. During the deception of the first book, John Turner took on the persona of the Earl of Ashby, and was pursued by Leticia Herzog, fortune-hunting widow of an Austrian Count. When the game was discovered, she fled in public shame. But they each felt a spark for the other, and now they're going to meet again.

December 17, 2018 09:04 AM

December 16, 2018

Roger Bell_West

Galaxy Quest

1999 science fiction comedy, dir. Dean Parisot, Tim Allen, Sigourney Weaver; IMDb / allmovie. Galaxy Quest was a big SF show in the early 1980s, but by the present day the cast are reduced to convention appearances and store openings. Then some particularly odd fans turn out to be rather more into the show than anyone expected… Spoilers.

December 16, 2018 09:00 AM

December 15, 2018

Roger Bell_West

The Castle of Otranto, Horace Walpole

1764 gothic horror, and the prototype for the gothic novel. Conrad, the sickly son of the Lord of Otranto, is killed on his wedding day by a giant helmet falling from the sky, so his father Manfred plans to marry the intended bride himself; things get more foreboding from there.

December 15, 2018 09:02 AM

December 14, 2018

Liam Proven

"Why is Ubuntu faster than Windows?"

Another recycled Quora answer.

The main reason is fairly simple.

Windows was designed to be easy to use, and to be compatible with older Microsoft operating systems, notably 16-bit Windows and DOS.

So, for example, by design, it treats any file with certain extensions (.EXE, .COM, .CMD, .BAT, etc.) as executable and will try to run them.

In contrast, Unix systems do not do this. They will not run even executable files unless they are specifically _marked_ as executable _and_ you have permissions to run them, and by default, Unix does not look in the current directory for executables.

This makes Unix less friendly, but more secure.

Microsoft also made other mistakes. For instance, it wanted to promote Internet Explorer to prevent Netscape getting control of the nascent browser market. To do this, it bundled IE with all copies of Windows. This was challenged in court as anti-competitive — which it was — and a demonstration was staged, in court, by intellectual property expert Lawrence Lessig, showing that IE could be uninstalled from Windows.

To counter this, MS tied IE in more deeply. So, for instance, Windows 98 has a multi-threaded Explorer, based in part on IE code. Window contents are rendered to HTML and IE then displays that content.

This means that all a hostile party has to do is embed a virus into a suitable image file, such as an icon or a folder wallpaper, and IE will render it. Exploit that IE process and you own the computer.

Ditto, IE is used to render HTML Help files. So, put an exploit into a help file, and you can own the computer.

Because MS wanted to counter Adobe Flash and Sun Java, popular ways of embedding interactive content on web pages, it designed its own: ActiveX. ActiveX is basically OLE (Object Linking and Embedding - Wikipedia) in HTML, but OLE was designed for local MS Office documents, or ones on a trusted local network. ActiveX runs over the Internet, but treated it like a LAN. It fetches binaries off remote servers and executes them. This is disastrously insecure design. The Internet cannot be trusted; all non-local machines should be treated as hostile by default, but MS’ pressured designers failed to realise this.

Because of poor design like this, focussing on ease-of-use not safety, Windows is easy to exploit. It is fairly straightforward to inject hostile code into Windows, whereas it’s much harder on Unix boxes.

Windows apologists will deny this, but it is factually accurate and a demonstrable fact.

So, Windows is less secure than other OSes. As a result, it is necessary to run anti-malware protections at all times on Windows machines. Every file must be scanned before it’s opened. Everything must be treated as suspicious, because the underlying OS design trusts everything.

Unix trusts nothing until it’s told to, so Unix doesn’t need this.

So at all times, on your Windows box, a background process is reading all your files, checking them against a database, making sure they are not in a list of known bad stuff, and then letting Windows run them.

(This model is also flawed: what if they’re not in the bad list *yet*? Oops, too late, someone else now controls your computer. Sorry about that.)

This necessity for ubiquitous antivirus kills Windows performance, but the alternative — to do without it — means your machine could be destroyed by the first website you view or the first USB drive you connect.

And there’s nothing MS can do about this except by making a version of Windows that can’t run any existing Windows software or use any existing Windows-supported hardware. So the company is trapped in a prison it made for itself.

It’s necessary to spell this out because the Internet is full of young people who know nothing else _except_ Windows and know nothing of its design compared to rival systems. If they know any other OS at all, it’s something like Linux, which in many ways is very similar to Windows in construction.

Thus, they deny this charge, because they think it’s someone being unfair to their preferred OS or company.

But every point in this is readily confirmed if you do some research. It is all objective fact. I’ve been working with Windows since v2.01 in 1988 and I can demonstrate all of these statements. Go check it and confirm it for yourself.

by liam_on_linux at December 14, 2018 06:26 PM

Yet another "the future of operating systems" post

I think our current style of rich, full-function, "power user" OSes is really starting to show signs of going away. (Yes, it's another of those handwavey sort of big-picture things.)

(Actually I should be drafting a FOSDEM talk about this right now, but I'm having a £1.50 draught beer & on FB instead. Gotta <3 České dráhy.)

Kids — i.e. in the Douglas Adams sense anyone under about 35 --  are more used to phones and tablets. The next billion people to come online will know nothDouglas Adams sense anyone under about 35ing else.

I reckon OSes will become more like Android, iOS, and ChromeOS — self-updating, without a "desktop" or much in the way of local storage or rich window management (for instance, see what's happening to GNOME) and fairly useless unless connected to the Internet and various invisible servers off in the "cloud" somewhere.

Some "apps" will run locally, some will be web apps, some will be display sessions on remote servers. There will be little visible distinction.

We'll have no local admin rights, just the ability to put on sandboxed apps that only interact via the online connection. No local storage, or just a cache. No software updates; the OS will handle that. No easy way to see what apps are running or where.

What'll drive this will be sky-rocketing costs of supporting rich local traditional OSes.

It will have the side-effect of blurring the lines between a workstation, a laptop, a tablet and a phone.

For some current examples, see the Red Hat Atomic Workstation project, Endless OS, and SUSE Kubic MicroOS. Read-only root FS, updates via a whole-system image pushed out periodically. Only containerised apps are allowed: there's not even a local package manager.

by liam_on_linux at December 14, 2018 06:19 PM

Roger Bell_West

The Case of Charles Dexter Ward

2018 audio drama, adapted in ten parts by Julian Simpson from the story by H. P. Lovecraft. On 6 March 2017, Charles Ward vanished from a secure psychiatric hospital in Providence. Two podcasters, looking for mysterious stories, dig into what happened.

December 14, 2018 09:04 AM

December 13, 2018

Roger Bell_West

The Wind Off the Small Isles, Mary Stewart

1968 thriller, short novella. The children's novelist Cora Gresham has decided to set her latest pirate story in Lanzarote, so she'll have to visit, and her secretary Perdita West has to make the arrangements. But the house that would be the ideal writing retreat is already occupied.

December 13, 2018 09:03 AM

Andy Smith (strugglers.net)

Google App Engine started requiring Content-Length header on POST requests

TL;DR

Update: It’s GoCardless who moved api.gocardless.com to Google Cloud. Google Cloud has behaved this way for years.

I think that Google App Engine may have recently started requiring every POST request to have a Content-Length header, even if there is no request body.

That will cause you problems if your library doesn’t add one for POST requests that have no content. Perl’s HTTP::Request is one such library.

You might be experiencing this if an API has just started replying to you with:

Error 411 (Length Required)!!1

411.That’s an error.

POST requests require a Content-length header. That’s all we know.

(Yes, the title does contain “!!1”.)

You can fix it by adding the header yourself, e.g.:

my $ua = LWP::UserAgent->new;

my $req = HTTP::Request->new(
    POST => 'https://api.example.com/things/$id/actions/fettle'
);

$req->header('Accept' => 'application/json');
$req->content_type('application/json');

my $json;
$json = JSON->new->utf8->canonical->encode($params) if $params;

$req->content($json) if $json;
# Explicitly set Content-Length to zero as HTTP::Request doesn't add one
# when there's no content.
$req->header( 'Content-Length' => 0 ) unless $json;

my $res = $ua->request( $req );

This is a bit far outside of my comfort zone so I’m not sure if I’m 100% correct, but I do know that sending the header fixes things for me.

What happened?

Yesterday a BitFolk customer tried to cancel their Direct Debit mandate, and it didn’t work. The server logs contained the above message.

For Direct Debit payments we use the Perl module Business::GoCardless for integrating with GoCardless, but the additional HTML styling in the message (which I’ve left out for brevity) made clear that the message was coming from Google. api.gocardless.com is hosted on Google App Engine (or some bit of Google cloud anyway).

After a bit of debugging I established that HTTP::Request was only setting a Content-Length header when there was actually request content. The API for cancelling a Direct Debit mandate is to send an empty POST to https://api.gocardless.com//mandates/$id/actions/cancel.

Adding Content-Length: 0 makes it work again.

When did it change?

There was a successful mandate cancellation on 25 October 2018, so some time between then and 12 December 2018. I haven’t looked for any change notice put out by Google as I’m not a Google Cloud user and wouldn’t know where to look.

Who’s to blame

I haven’t yet looked into whether the HTTP standard requires POST requests to have a Content-Length header. I welcome comments from someone who wants to do the digging.

Realistically even if it doesn’t and Google is just being overly strict, other servers might also be strict, so I guess HTTP::Request should always send the header.

by Andy at December 13, 2018 06:20 AM

December 12, 2018

Jonathan Dowland

Game Engine Black Book: DOOM

Fabien's proof copies

Fabien's proof copies

*proud smug face*

proud smug face

Today is Doom's' 25th anniversary. To mark the occasion, Fabien Sanglard has written and released a book, Game Engine Black Book: DOOM.

It's a sequel of-sorts to "Game Engine Black Book: Wolfenstein 3D", which was originally published in August 2017 and has now been fully revised for a second edition.

I had the pleasure of proof-reading an earlier version of the Doom book and it's a real treasure. It goes into great depth as to the designs, features and limitations of PC hardware of the era, from the 386 that Wolfenstein 3D targetted to the 486 for Doom, as well as the peripherals available such as sound cards. It covers NeXT computers in similar depth. These were very important because Id Software made the decision to move all their development onto NeXT machines instead of developing directly on PC. This decision had some profound implications on the design of Doom as well as the speed at which they were able to produce it. I knew very little about the NeXTs and I really enjoyed the story of their development.

Detailed descriptions of those two types of personal computer set the scene at the start of the book, before Doom itself is described. The point of this book is to focus on the engine and it is explored sub-system by sub-system. It's fair to say that this is the most detailed description of Doom's engine that exists anywhere outside of its own source code. Despite being very familiar with Doom's engine, having worked on quite a few bits of it, I still learned plenty of new things. Fabien made special modifications to a private copy of Chocolate Doom in order to expose how various phases of the renderer worked. The whole book is full of full colour screenshots and illustrations.

The main section of the book closes with some detailed descriptions of the architectures of various home games console systems of the time to which Doom was ported, as well as describing the fate of that particular version of Doom: some were impressive technical achievements, some were car-crashes.

I'm really looking forward to buying a hard copy of the final book. I would recommend this to anyone has fond memories of that era, or is interested to know more about the low level voodoo that was required to squeeze every ounce of performance possible out of the machines from the time.

Edit: Fabien has now added a "pay what you want" option for the ebook. If the existing retailer prices were putting you off, now you can pay him for his effort at a level you feel is reasonable. The PDF is also guaranteed not to be mangled by Google Books or anyone else.

December 12, 2018 04:50 PM

Roger Bell_West

Doctor Who 2/11.10: The Battle of Ranskoor Av Kolos

The season ends not with a bang but with a "to be continued". And that's not a bad thing.

December 12, 2018 09:02 AM

December 11, 2018

Roger Bell_West

December 10, 2018

Roger Bell_West

Clarkesworld 147, December 2018

Clarkesworld is a monthly on-line magazine edited by Neil Clarke.

December 10, 2018 09:01 AM

December 09, 2018

Roger Bell_West

GURPS Dungeon Fantasy Adventure 2: Tomb of the Dragon King, Matt Riggsby

This second Dungeon Fantasy Adventure book deals with an expedition to the long-abandoned (yeah, right) ruins of a draconic monarch's palace.

December 09, 2018 09:02 AM

December 08, 2018

Roger Bell_West

Apex 114, November 2018

Apex is a monthly on-line magazine edited by Jason Sizemore among others.

December 08, 2018 09:02 AM

December 07, 2018

Jonathan Dowland

I'm moving to the Red Hat OpenJDK team

I'm very excited to announce that I've moved roles within Red Hat: I am now part of the OpenJDK team!

I've been interested in the theory and practise behind compilers, programming language design and the interaction of the two for a long time¹. Before my undergrad I was fascinated by the work of Wouter van Oortmerssen, who built lots of weird and wonderful experimental languages and systems². During my undergrad, dissatisfied with the available choice of topics for my third year, I petitioned the Computer Science Department at Durham University to revive an older module "Programming Language Design & Compiling". I'm eternally grateful to Dr. Paul Callaghan for being prepared to teach it to us³.

I've spent my time within Red Hat so far in "Cloud Enablement". Our mission was to figure out and develop the tools, techniques and best practises for preparing containerized versions of the Middleware product portfolio to run on OpenShift, Red Hat's enterprise container management platform. The team was always meant to be temporary, the end game being the product teams themselves taking responsibility for building the OpenShift containers for their products, which is where we are today. And so, myself and the other team members are dismantling the temporary infrastructure and moving on to other roles.

Within Cloud Enablement, one of my responsibilities was the creation of the Java Applications for OpenShift container image, which is effectively OpenJDK and integration scripts for OpenShift. I am going to continue maintaining and developing this image (or images) within the OpenJDK team.

Longer term I'm looking forward to getting my teeth into some of the technical work within OpenJDK: such as the JVM, architecture ports, garbage collectors or the JIT or AOT compilers within the runtime.

Earlier this year, I put together a private "bucket list" of things I wanted to achieve in the near-ish future. I recently stumbled across it, having not thought about it for a while, and I was pleasantly surprised to see I'd put on "compilers/lang design" as something to revisit. With my move to OpenJDK I can now consider that ticked off.


  1. I'm straying into this area a little bit with my PhD work (graph rewriting, term rewriting, etc.)
  2. one of which, WadC, I took over maintenance of, ten years ago
  3. Paul has recently had some of his writing published in the book Functional Programming: A PragPub Anthology

December 07, 2018 11:23 AM

Roger Bell_West

Marlow Tabletop and Board Games 3 December 2018

This Meetup-based boardgames group continues to meet at the Marlow Donkey.

December 07, 2018 09:00 AM

December 06, 2018

Liam Proven

Why did Apple's Copland fail when so many early-90s OS projects succeeded & dominate the world today

(Adapted from a Quora answer.)

OS/2 1.x was a clean-sweep, largely legacy-free OS with only limited backwards compatibility with DOS.

OS/2 2.x and later used VMs to do the hard stuff of DOS emulation, because they ran on a chip with hardware-assisted DOS VMs: the 80386’s Virtual86 mode.

NeXTstep was a Unix. It predated FreeBSD, but it was based off the same codebase: BSD 4 Unix. It “only” contained a new display layer, and that itself was based off existing code — Adobe PostScript — and the then-relatively-new technique of object-oriented development. Still substantial achievements, but again, built on existing code, and with no requirement for backwards compatibility.

BeOS was a ground-up new OS which wasn’t backwards or sideways compatible with anything else at all.

NT is based on OS/2 3.x, the planned CPU-independent portable version, with a lot of design concepts from DEC VMS incorporated, because it had the same lead architect, Dave Cutler. Again, the core NT OS isn’t compatible with anything else. This is rarely understood. NT is not a Win32-compatible kernel. NT isn’t compatible with anything else, including VMS. It’s something new. But NT supports personalities, which are like emulation layers running on top of the kernel. When NT shipped, it included 3: OS/2, POSIX and Win32. OS/2 is deprecated now, POSIX has developed into the Linux subsystem, and Win32 is still there, now in 64-bit form.

The point is, none of these OSes were enhanced versions of anything else, and none were constrained by compatibility with existing drivers, extensions, applications, or anything else.

Apple tried to do something much, much harder. It tried to create a successor OS to a single-user, single-tasking (later cooperatively-multitasking, and not very well), OS for the 68000 (not something with hardware memory protection, like the 68030 or 68040), which would introduce those new features: pre-emptive multitasking, virtual memory, memory protection, integrated standards-based networking, etc.

All while retaining the existing base of applications, which weren’t written or designed or planned for any of this. No apps == no market == no use.

Apple took on a far harder project than anyone else, and arguably, with less experience. And the base hardware wasn’t ready for the notion of virtual machines yet.

It’s a great shame it failed, and the company came relatively close — it did have a working prototype.

It’s often said that Apple didn’t take over NeXT, nor did it merge with NeXT — in many important ways, NeXT took over Apple. Most Apple OS developers and project managers left, and were replaced by the NeXT team.

The NeXT management discarded Copland, most Apple technologies — OpenDoc, OpenTransport, GameSprockets, basically everything except QuickTime. It took some very brave, sweeping moves. It took the existing MacOS classic APIs, which weren’t really planned or designed, they just evolved over nearly 1½ decades — and cut out everything that wouldn’t work on a clean, modern, memory-managed, multitasking OS. The resulting cut-down, cleaned-up API was called “Carbon”. This was presented to developers as what they had to target if they wanted their apps to run on the new OS.

Alternatively, they could target the existing, far cleaner and richer NeXT API, now called “Cocoa”.

The NeXT team made no real attempt to be compatible with classic MacOS. Instead, it just ran all of classic MacOS inside a VM — by the timeframe that the new OS was targeting, machines would be high-enough spec to support a complete classic MacOS environment in a window on top of the Unix-based NeXTstep, now rebadged as “Mac OS X”. If you wanted your app to run outside the VM, you had to rebuild for “Carbon”. Carbon apps could run on both late versions of classic MacOS and on OS X.

This is comparable to what NT did: it offered a safe subset of the Win32 APIs inside a “personality” on top of NT, and DOS VMs with most of Win16.

It was a brave move. It’s impressive that it worked so well. It was a fairly desperate, last-ditch attempt to save the company and the platform, and it’s easier to make big, brave decisions when your back is against the wall and there are no alternatives... especially if the mistakes that got you into that corner were made by somebody else.

A lot of old Apple developers left in disgust. People who had put years of work into entire subsystems and APIs that had been thrown in the trash. Some 3rd party developers weren’t very happy, either — but at least there was a good path forwards now.

In hindsight, it’s clear that Apple did have an alternative. It had a rich, relatively modern OS, upon the basis of which it could have moved forwards: A/UX. This was Apple’s Unix for 680x0, basically done as a side project to satisfy a tick-box for US military procurement, which required Unix compatibility. A/UX was very impressive for its time — 1988, before Windows 3.0. It could run both Unix apps and classic MacOS ones, and put a friendly face on Unix, which was pretty ugly in the late 1980s and early 1990s.

But A/UX was never ported to the newer PowerPC Macs.

On the other hand, the NeXT deal got back Steve Jobs. NeXTstep also had world-beating developer tools, which A/UX did not. Nor did BeOS, the other external alternative that Gil Amelio-era Apple considered.

No Jobs, no NeXT dev tools, and no Apple today.

by liam_on_linux at December 06, 2018 12:16 PM