will

Where there is a Will

Will McGugan's Blog

I am a freelance software engineer living in Edinburgh.

I post mostly about tech (particularly Python) and photography.

Background: A code monkey

Apr 10, 2018
in Tech

Stream BTC prices over WebSockets with Python and Lomond

WebSockets is a great technology with many applications beyond the front end. I have a feeling that websocket APIs will become far more common in the web ecosphere.

I really like this example of how to interact with a WebSockets api with just-released Lomond 0.1.5.

Get the code from Lomond repos on Github.

This connects to the gdax websocket server and periodically prints the latest BTC price information. If you change the text "ticker" to "level2" it will spew out information on every trade. You could use this as the foundation for algorithmic trading or to keep an eye on your retirement fund (good luck).

Lomond is a websocket client library sponsored by Dataplicity. It has a focus on robustness and ease of use. continue reading…

Oct 9, 2017
in Tech

Lifting the Curtain on Asyncio Magic - Part 1

The new asyncio module introduced in Python3.4 is a nice addition to the Python standard library, especially when used with the async and await keywords introduced in Python3.5.

If you have read the official docs, you should hopefully be able to be able to write high performance network servers and clients with asyncio. But if something goes wrong, you may find it hard to debug. This is no reflection on asyncio; concurrency and IO are deceptively complex things, and asyncio is the point they meet.

I suspect that the problem with debugging asyncio projects is that async code doesn't run itself. You are dependant on a rather large body of code to run your code for you, and the distinction between your code library code is far more blurry than you may be used to. continue reading…

Sep 27, 2017
in Photography

Wildlife of Saint Ives

I'm currently in St Ives, a town in Cornwall, UK. The weather has taken a turn for the worst, and I'm trapped with my wife in a conservatory attached to the B&B we're staying in. Can't complain, there has been plenty sun and sea food so far, and I got some decent shots this week.

This shot was taken from The Island, which isn't an island at all; its a rocky headland with a lighthouse.

A Northern Wheatear. I liked this shot for the framing. If I stood a few inches left or right I would have missed it.

These two shots of a Ruddy Turnstone were taken from the bay. I bought a rain poncho so I could lie down in the wet sand of the harbour and shoot from the birds eye level. If you want to shoot birds, this is the single best thing you can do to get better shots. continue reading…

Aug 7, 2017
in Tech

Amazon S3 Filesystem for Python

I'd like to announce an new Python module to make working with Amazon S3 files a whole lot easier.

The S3FS class in fs-s3fs wraps an Amazon S3 bucket in a PyFilesystem interface. There was an S3FS class built in to the first version of PyFilesystem, but it had suffered from using an older version of 'boto' (Amazon's S3 interface) and was in need of maintenance. The new version is up-to-date with PyFilesystem2 and boto3, and works with Python2.7 and Python3.X.

If you aren't familiar with PyFilesystem, it is a common interface to anything that resembles a filesystem. Amazon S3 isn't quite a full filesystem, but close enough that it can be made to work in the same way as the files and directories on your local drive (or any other supported filesystem for that matter). continue reading…

Jun 11, 2017
in Tech

PyFilesystem 2.0.4 Released

I've released version 2.0.4 of PyFilesystem.

PyFilesystem is an abstraction layer for filesystems (or anything that resembles a filesystem). See this post for more details.

This version adds support for Python3.6 os.pathlike to the OSFS class. This is in oft-requested feature, but it only affects the OSFS constructor; since FS methods use a standardized path format.

Also in the version is a contribution by Martin Larralde which implements an elegant extension mechanism. Essentially, if you layout your filesystem implementation according to the convention, then PyFilesystem will be aware of it automatically. So open_fs('awesomefs://foo/bar') would work without explicitly adding awesomefs to the core library. continue reading…

Jun 1, 2017
in Tech

Announcing Lomond, a WebSocket Client Library for Python

I'd like to announce the first official release of Lomond, a new WebSocket client library for Python. The development was sponsored by Dataplicity.

Lomond is not the first websocket client for Python, so why go to the effort of building another one? For our purposes, we needed a stand alone client that didn't need a framework to run. So that excludes the websocket client support in Tornado, aiohttp etc. The two libraries that were suitable for our product, websocket-client and ws4py, both had show-stopper bugs with ssl support; websocket-client would sometimes refuse to processes packets until additional data was received, and ws4py could lose entire packets. I'm sure both libraries could be fixed, but neither project appears to be actively maintained. continue reading…

May 29, 2017
in Photography

Costa Rica Trip Report 2017

Wildlife photography can require great hardship on the part of the photographer; days of trekking, battling the elements, biting insects, etc. — all while carrying 20 kilos of camera gear.

This was not the case for my recent trip to Costa Rica, which was as close to luxury you can get in the Rainforest, if you don't mind the lack of air-conditioning and occasional oversize bug in your cabin.

My girlfriend and I stayed in El Romanso, an eco-lodge on the Osa Peninsula. Located in primary rainforest overlooking the Pacific ocean, it is an beautiful place, abundant with wildlife. Most of which I photographed from the lodge restaurant, which had been constructed on stilts to put you at eye level with monkeys and toucans. continue reading…

Apr 22, 2017
in Tech

PyFilesystem 2.0.3 Released

I've just released version 2.0.3 of PyFilesystem.

PyFilesystem is an abstraction layer for filesystems (or anything that resembles a filesystem). See this post for more details.

New in this version is a TarFS filesystem contributed by Martin Larralde, which compliments ZipFS nicely.

Contributed by gpcimino, the copy module was extended new functionality to selectively copy only new files from one filesystem to another. Very useful for backups.

This is in addition to bugfixes, and improved documentation.

Apr 19, 2017
in Tech

Speeding up Websockets 60X

I recently I had the opportunity to speed up some Websocket code that was a major bottleneck. The final solution was 60X (!) faster than the first pass, and an interesting exercise in optimizing inner loops.

When you send data via a websocket it is masked with a randomly generated four byte key. This masking provides protection from certain exploits against proxies (details here). The algorithm is super simple; essentially each byte in the payload is XORed with a corresponding byte from the key.

The first version of my XOR mask function looked this:

That's the kind of unfussy and elegant code you want to write for a job interview. If that's the solution that first came to mind, then you know your Python. Alas, this code is as slow as it is beautiful. continue reading…

Dec 15, 2016
in Tech

Can your Python template do this?

I like to imagine components of a Python web framework as personalities. Here's how I would imagine a conversation between a typical view and template would go down:

View Here's that data I just looked up.

Template Don't like it.

View What's wrong with it?

Template Format's wrong.

View You can change the format if you like.

Template Don't wanna. You change it.

View OK. Fine. I'll change it for you.

Which is why I wanted Moya templates to be very capable in processing data, but still keep them designer friendly. If the template data is sane then the template should be able to deal with it, without going back to the back-end developer. continue reading…