NotesToSelf.Dev

Overlap-preserving type refinement assertion signatures in TypeScript

Mon, 15 Jun 2026 17:16:03 +0000

In the last post I wrote about how TypeScript assertion signatures are more useful when they can incorporate existing type information from the calling scope.

Sometimes the asserted type is less precise than information TypeScript already has in scope at the calling site.

function assertString(value: unknown): asserts value is string {
 if (typeof value !== "string") {
 throw new Error("Expected string");
 }
}

type Foo = 
 | "draft"
 | "published"
 | "archived"
 | 404
 | null;

function getFoo(): Foo {
 return "draft";
}

const foo = getFoo();

assertString(foo);

foo;

// const foo: "draft" | "published" | "archived"

The assertion signature from assertString tells TypeScript that foo is a string, and TypeScript is able to combine that with other type information from the scope to infer that foo must therefore be of a literal union type "draft" | "published" | "archived".

Combining TypeScript assertion signatures and function overloads

Mon, 15 Jun 2026 10:46:07 +0000

TypeScript has a useful feature called assertion signatures that allows you to write functions that can be used to narrow the type of a value. For example, you can write a function that asserts that a value is a string and tells TypeScript that after this function has been called, that value must be a string.

function assertIsString(value: unknown): asserts value is string {
 if (typeof value !== "string") {
 throw new Error("Expected string");
 }
}

This is somewhat similar to type guards, but rather than being used in a condition, it lets TypeScript narrow the type in the current scope without introducing a new variable.

Composable type narrowing test assertion matchers in TypeScript

Sun, 14 Jun 2026 13:23:05 +0000

A lot of tests use the expect().toMatchObject() assertion in Jest or Vitest, because we’re most often dealing with object structures returned from the subject of the test. The toMatchObject assertion is useful because it applies a deep partial match, so you only need to specify the particular property paths you’re interested in. It doesn’t matter if other properties are added to the object structure later.

However, I would often find the need to add explicit type information in addition to the assertion, because the fluent interface in expect() cannot indicate any type information to TypeScript or the IDE.

Preventing type errors in TypeScript test files with tsc -p --noEmit

Sat, 13 Jun 2026 08:23:51 +0000

In TypeScript projects it’s common to have separate tsconfig files covering the entire project and separate build contexts which exclude test files.

This can lead to failing to check test files with tsc, so type errors in test files creep in unnoticed only to be discovered later when the root problem is less clear.

For example the main tsconfig.json file in a project might be similar to this:

{
 "compilerOptions": {
 "target": "ES2023",
 "module": "NodeNext",
 "moduleResolution": "NodeNext",
 "lib": ["ES2023"],
 "types": ["node"],
 "strict": true,
 "outDir": "dist",
 "rootDir": "src"
 },
 "include": ["src/**/*", "test/**/*", "typings/**/*.d.ts"]
}

And then there might be a tsconfig.build.json file that extends it but excludes test files from the build:

Simulating an S3 static website on localhost with Yulin

Sun, 07 Jun 2026 08:42:28 +0000

If you’re using S3 to host a static website and want to test and develop it locally, you can use the @kensio/yulin package.

This might seem pointless when it’s easy to serve a static website on localhost in various other ways (such as with hugo serve), but even for this small use-case Yulin does provide some extra benefits.

One immediate small benefit is that Yulin simulates behaviours of S3 static website serving such as error documents, bucket redirects and routing rules with conditions. These might be important behaviours for your project, so it’s useful to be able to develop locally with simulated equivalents.

AWS CloudFront Functions JS2 tooling: ESLint, Vitest, IDE

Sun, 24 May 2026 14:58:56 +0000

AWS CloudFront Functions (CFF) use a special subset of JavaScript called JS2.

JS2 syntax is simpler than full JavaScript, but it’s still nice to have modern tooling around it, such as linting, testing, and IDE autocomplete. This tooling can also help avoid mistakes related to the minimal JS2 feature set.

Firstly, these docblock comments in the CFF JS2 file allow the IDE to understand more about the function and offer more useful autocompletion and problem highlighting:

Fanalysis

Sun, 01 Jun 2025 00:00:00 +0000

In June 2025 I was the second employee to join Fanalysis, as Staff Software Engineer.

Bash script to run CDK synth only if there are changed TS files

Tue, 08 Apr 2025 10:57:23 +0100

The cdk synth command can be pretty slow, even when running for a single small stack. We also often want to run the CDK synth command regularly as part of local development setups, so it’s nice to skip the re-synth if no relevant files have changed.

Here’s a bash script that checks for changed TypeScript source files by comparing their timestamp to that of the generated template file from the previous synth. This way we only run CDK synth when we find a relevant change.

AWS SAM local (WSGI) drops headers with underscores

Sun, 30 Mar 2025 11:57:23 +0100

TLDR: SAM local drops headers that contain an underscore, due to using Flask and WSGI internally, which differs from the behaviour of a deployed API Gateway.

I’m working on an existing project that uses AWS CDK to provision API Gateway in front of some Lambda functions. The team would like to be able to run this locally for faster development iterations, so I set that up with the sam local start-api command, using the template emitted by cdk synth. So far, so good.

Cache a public Docker image in a GitHub Action

Sun, 23 Mar 2025 11:57:23 +0100

Here’s a reusable GitHub Action definition that caches a public Docker image:

---
name: Docker public image cache
description: Pull and cache a public Docker image for reuse
inputs:
 image_name_tag:
 description: Docker image name and tag to pull and cache
 required: true
runs:
 using: "composite"
 steps:
 - name: Restore image cache
 id: restore_cache
 uses: actions/cache@v4
 with:
 path: 'ci/cache/docker'
 key: 'cache-docker-${{ inputs.image_name_tag }}'

 - name: Update image cache on cache miss
 if: steps.restore_cache.outputs.cache-hit != 'true'
 shell: bash
 run: >
 docker pull '${{ inputs.image_name_tag }}'
 && mkdir -p $(dirname './ci/cache/docker/${{ inputs.image_name_tag }}.tar')
 && docker image save '${{ inputs.image_name_tag }}' --output './ci/cache/docker/${{ inputs.image_name_tag }}.tar'

 - name: Use image cache on cache hit
 if: steps.restore_cache.outputs.cache-hit == 'true'
 shell: bash
 run: >
 docker image load --input './ci/cache/docker/${{ inputs.image_name_tag }}.tar'

If you have this action definition at .github/actions/docker_image_cache in the repo, then you can use it from a workflow like this:

eslint changed TypeScript files only

Sun, 23 Mar 2025 10:57:23 +0100

It’s best if a linter is applied from the beginning of a project, and with default linting rules, as that way the project benefits from the linting with the minimum possible effort.

However, we often join existing projects which have already been developed without the use of a linter, by which point there are possibly thousands of linter violations. In that situation, one option is to incrementally improve the linting by only applying the linter to changed files. New code from then on is compliant, and it’s less of a burden to sort out existing violations in existing files that developers touch as part of a change.

Splitting a text file into multiple separate files by section with csplit

Wed, 26 Feb 2025 10:57:23 +0100

Here’s a bash command to split a text file into multiple separate files by section delimited by a regex pattern matching titled sections in the source file:

csplit \
 -z \
 -f section_ \
 -b "%02d.txt" \
 document.txt \
 '/\nSection [0-9]\+\n/' \
 '{*}'

The key part is the /\nSection [0-9]\+\n/ regex pattern, which matches section titles such as “Section 18” alone on a line.

Here’s an explanation of the the rest of the command:

Python "raises in exception group" context manager for tests

Sun, 05 Jan 2025 17:11:23 +0100

Here’s a Python testing utility function for asserting on an expected Exception type from within an ExceptionGroup:

from contextlib import contextmanager


@contextmanager
def raises_in_group(
 expected_exception: type[Exception],
 match_message: str = "",
):
 """
 Like pytest.raises, but for an Exception inside an ExceptionGroup, for
 example coming from asyncio.TaskGroup.
 """

 matched_type = []
 matched_message = []
 try:
 yield
 except* expected_exception as e:
 assert isinstance(e, ExceptionGroup)
 matched_type = [exp for exp in e.exceptions if isinstance(exp, expected_exception)]
 matched_message = [exp for exp in matched_type if match_message in str(exp)]

 assert len(matched_type) >= 1, f"Expected at least one {expected_exception} exception."
 assert (
 len(matched_message) >= 1
 ), f"Expected at least one {expected_exception} exception matching '{match_message}'."

It can be used in test cases in a similar way to pytest.raises, like this:

HOA Mailers

Wed, 01 Jan 2025 00:00:00 +0000

I helped HOA Mailers with their AWS infrastructure, backend systems and frontend software.

Creating AWS Organizational Units and Accounts for Prod and Dev in CloudFormation

Sat, 05 Oct 2024 15:44:45 +0100

Here’s an AWS CloudFormation template and deployment script to set up Organizational Units and Accounts for Prod and Dev for a new project or department.

Deploying a CloudFormation stack from this template will require a user in the Organization’s management account with the AdministratorAccess permission set.

The created Organizational Unit (OU) and Account structure will look this:

Organization Root [pre-existing]

 ╚═ Management Account [pre-existing]

 ╚═ Project Parent OU

 ╚═ Project Prod OU

 ╚═ Project Prod Account

 ╚═ Project Dev OU

 ╚═ Project Dev Account

The OUs are not strictly necessary, but they help to group the whole project under a single OU, and to allow for other accounts to be added later within the Prod and Dev OUs if necessary.

Reduce pragma no cover comments with tool.coverage.report exclude_also

Sun, 29 Sep 2024 19:45:13 +0100

If you’re using Coverage.py or pytest-cov in your Python project, you probably end up with # pragma: no cover comments all over the place to prevent odd lines from being marked as missing coverage, like this:

class Foobar:

 def do_something(): # pragma: no cover
 raise NotImplementedError()

You can reduce the need for these # pragma: no cover comments by configuring Coverage.py to ignore certain patterns by default. This can be configured in the pyproject.toml file like this:

Github Action to post a Telegram notification with no dependencies

Sun, 29 Sep 2024 19:32:04 +0100

Here’s a Github Action that posts a notification message to a Telegram chat, without using any dependencies.

---
name: Post to Telegram
description: Post a message to a Telegram channel
inputs:
 telegram_bot_token:
 description: Auth token of the Telegram bot via which to post.
 required: true
 telegram_chat_id:
 description: ID of the Telegram channel to which to send the message.
 required: true
 message_content:
 description: String content of the Telegram message to send.
 required: true
runs:
 using: "composite"
 steps:
 - shell: bash
 run: |
 curl "https://api.telegram.org/bot${{ inputs.telegram_bot_token }}/sendMessage" \
 --header "Content-Type: application/json" \
 --request POST \
 --data '{
 "chat_id": "${{ inputs.telegram_chat_id }}",
 "text": "${{ inputs.message_content }}"
 }'

The key part of the action is the bash script that uses curl to post a message to Telegram:

Python Coverage boolean compound condition line exit missing branch coverage

Sat, 28 Sep 2024 13:11:39 +0100

With branch coverage enabled in Coverage.py or pytest-coverage, you might see it report missing coverage on a compound boolean condition, like this:

def foobar() -> bool:
 return (
 condition_1()
 and condition_2()
 and condition_3()
 and condition_4()
 and condition_5()
 )

That might lead to a report that the line where the boolean compound condition begins to the exit of the function are missing coverage. The report will look like this:

---------- coverage: platform linux, python 3.12.1-final-0 -------

Name Stmts Miss Branch BrPart Cover Missing
------------------------------------------------------------------
src/foobar.py 39 0 16 1 98% 37->exit
------------------------------------------------------------------

TOTAL 188 0 68 1 99%

The “line to exit” part is 37->exit in the Missing column on the right.

Python recursive override path function and mixin class

Wed, 25 Sep 2024 13:36:56 +0100

Here’s a recursive utility function in Python that allows overriding nested paths through objects that can be mutable mappings, mutable sequences or dataclass instances.

The function allows doing things like this:

override_paths(
 {
 "foo0": "bar",
 "lvl1": {
 "foo1": "bar1",
 "list1": [1, 2, 3, 4, 5, 6],
 "foobar1": Foobar(foo="hello1"),
 },
 },
 foo0="123",
 lvl1__foo1="456",
 lvl1__list1__3=23,
 lvl1__foobar1__foo="hello2",
)

{
 "foo0": "123",
 "lvl1": {
 "foo1": "456",
 "list1": [1, 2, 3, 23, 5, 6],
 "foobar1": Foobar(foo="hello2"),
 },
}

There is also a OverridableMixin that makes it convenient to do things like this inline:

Foobar().overridden(foo__bar__baz=123)

import dataclasses
from copy import deepcopy
from dataclasses import is_dataclass
from itertools import chain
from typing import (
 TYPE_CHECKING,
 Any,
 ClassVar,
 MutableMapping,
 MutableSequence,
 Protocol,
 Self,
 TypeVar,
)


class Dataclass(Protocol):
 """Identifies an instance of any dataclass."""

 __dataclass_fields__: ClassVar[dict[str, Any]]


if TYPE_CHECKING:
 Overridable = MutableMapping | MutableSequence | Dataclass # pragma: no cover
else:
 # Having to add `| object` to the end of this union to keep PyCharm happy as
 # the PyCharm type checker can't figure out the Dataclass Protocol above.
 # Doing this inside a TYPE_CHECKING condition so that mypy still validates
 # the protocol correctly.
 Overridable = MutableMapping | MutableSequence | Dataclass | object


# Using TypeVar instead of the inline type parameter syntax because pydocstyle
# seems unable to cope with the newer syntax.
T = TypeVar("T", bound=Overridable)


def override_paths(
 target: T,
 *args: tuple[str | list[str], Any],
 **kwargs: Any,
) -> T:
 """
 Override keys, indexes or fields specified via a flat path syntax.

 E.g.

 override_paths({"foo": {"bar": 123}}, foo__bar=456)
 {"foo": {"bar": 456}}

 The override paths can be given as either kwargs, or as args as tuples with
 the path and the value as pairs. The path can be a __ string or a list of
 strings.

 override_paths({"foo": {"bar": 123}}, ("foo__bar", 456))
 {"foo": {"bar": 456}}

 override_paths({"foo": {"bar": 123}}, (["foo, "bar"], 456))
 {"foo": {"bar": 456}}

 Mutable mappings, mutable sequences and dataclass instances can be
 overridden.
 """
 target = deepcopy(target)
 for key_path, value in chain(args, kwargs.items()):
 if not key_path:
 return value
 if isinstance(key_path, str):
 key_path: list[str] = key_path.split("__") # type: ignore[no-redef]
 path_entry: str = key_path[0]
 if isinstance(target, MutableMapping):
 if path_entry not in target:
 raise KeyError(f"Key '{path_entry}' not found in target")
 target[path_entry] = override_paths(
 target[path_entry],
 (key_path[1:], value),
 )
 continue
 if path_entry.isdigit() and isinstance(target, MutableSequence):
 override_index = int(path_entry)
 if len(target) - 1 < override_index:
 raise IndexError(f"index {override_index} out of range in target")
 target[override_index] = override_paths(
 target[int(path_entry)],
 (key_path[1:], value),
 )
 continue
 if has_dataclass_field(target, path_entry):
 setattr(
 target,
 path_entry,
 override_paths(
 getattr(target, path_entry),
 (key_path[1:], value),
 ),
 )
 continue
 raise KeyError(f"Key / index / field '{path_entry}' not found in target")
 return target


def has_dataclass_field(
 target: Any,
 field_name: str,
) -> bool:
 """
 Check if an object is a dataclass that has a dataclass field with the given
 name.
 """
 return is_dataclass(target) and any(
 True for f in dataclasses.fields(target) if f.name == field_name
 )


@dataclasses.dataclass
class OverridableMixin:
 """
 Mixin class which adds a convenience method to override paths on the
 instance and get back an overridden copy.

 This makes it convenient to override paths inline with a new instance
 without needing a variable referencing the instance, e.g.

 Foobar().overridden(foo__bar__baz=123)

 Can be dropped in place to get an instance of Foobar with that path
 overridden with the new value.
 """

 def overridden(
 self,
 **kwargs: Any,
 ) -> Self:
 """
 Get a copy of this instance with the given paths overridden.
 """
 return override_paths(self, **kwargs)

Github Action job to post pytest coverage as PR comment

Mon, 23 Sep 2024 18:07:38 +0100

Here’s a Github Action job that runs pytest coverage and then posts a summary of the coverage report as a PR comment with Markdown formatting. This is handy as part of a PR workflow to make the test results and coverage summary visible on the PR.

 coverage:
 name: coverage
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - uses: ./.github/actions/setup_python
 - uses: ./.github/actions/job_url
 id: job_url
 with:
 step_name: 'pytest'
 - name: pytest
 run: |
 pytest
 - name: Test coverage summary report
 id: coverage_report
 if: always()
 run: |
 {
 echo "COVERAGE_REPORT<<EOF"
 coverage report --format=markdown --show-missing --skip-covered
 echo "EOF"
 } >> "$GITHUB_OUTPUT"
 - name: Remove previous coverage comments
 uses: ./.github/actions/remove_comments
 if: always()
 with:
 identifier: coverage
 - name: Test coverage PR comment
 id: coverage_comment
 if: always()
 uses: actions/github-script@v7
 with:
 script: |
 github.rest.issues.createComment({
 owner: context.repo.owner,
 repo: context.repo.repo,
 issue_number: context.issue.number,
 body: `[coverage](${{steps.job_url.outputs.job_url}})
 \r\n\r\n${{steps.coverage_report.outputs.COVERAGE_REPORT}}`
 })

Note that this uses separate actions to setup the Python env, get the job URL and to remove previous PR comments.

Github Action for PR comment with command output on exit status

Mon, 23 Sep 2024 17:50:45 +0100

Here’s a Github Action that runs a given bash command, captures the stdout and strderr output, and posts a comment with the output back to the PR if the command has a non-zero exit status.

This is useful for running check commands such as linters against a PR, and making a PR comment with the linter output if there are linting errors.

name: Command & Comment
description: Run a bash command and comment failure output back to PR
inputs:
 identifier:
 description: Single word identifier for the check command
 required: true
 bash_command:
 description: Bash command to run and comment from
 required: true
runs:
 using: "composite"
 steps:
 - uses: ./.github/actions/job_url
 id: job_url
 with:
 step_name: ${{inputs.identifier}}
 - id: check_command
 name: ${{inputs.identifier}}
 shell: bash
 run: |
 set +e
 echo "COMMAND_OUT<<EOF" | tee "$GITHUB_OUTPUT"
 ${{inputs.bash_command}} 2>&1 | sed 's/$/\\n/g' | tr -d '\n' | tee -a "$GITHUB_OUTPUT" ; COMMAND_STATUS=$?
 ( echo ; echo "EOF" ) | tee -a "$GITHUB_OUTPUT"
 exit $COMMAND_STATUS
 - name: Remove previous command comments
 uses: ./.github/actions/remove_comments
 if: always()
 with:
 identifier: ${{inputs.identifier}}
 - name: PR comment command failure
 uses: actions/github-script@v7
 if: always() && steps.check_command.outcome == 'failure'
 with:
 script: |
 github.rest.issues.createComment({
 owner: context.repo.owner,
 repo: context.repo.repo,
 issue_number: context.issue.number,
 body: '[${{inputs.identifier}}](${{steps.job_url.outputs.job_url}})\n\n```\n${{steps.check_command.outputs.COMMAND_OUT}}```'
 })

Note that this uses separate actions to get the job URL and to remove previous PR comments.

Github Action to remove previous job PR comments

Mon, 23 Sep 2024 14:03:53 +0100

Here’s a Github Action that finds and removes PR comments based on a string identifier. This is useful for Github Action workflows that post comments to PRs, as it clears up out-of-date comments from earlier workflow runs.

name: Command & Comment
description: Run a bash command and comment failure output back to PR
inputs:
 identifier:
 description: Single word identifier for comments to remove
 required: true
runs:
 using: "composite"
 steps:
 - name: Remove previous job comments
 uses: actions/github-script@v7
 with:
 script: |
 const pr_comments = await github.rest.issues.listComments({
 owner: context.repo.owner,
 repo: context.repo.repo,
 issue_number: context.issue.number,
 })
 await Promise.all(
 pr_comments.data.filter(
 comment => comment.body?.startsWith("[${{inputs.identifier}}]")
 ).map(
 comment => github.rest.issues.deleteComment({
 owner: context.repo.owner,
 repo: context.repo.repo,
 issue_number: context.issue.number,
 comment_id: comment.id,
 })
 )
 )

It removes comments on the PR it is running against if the comment begins with [{identifier}]. If other comments happen to begin with that then they will also be removed. More checks could be added to the filter to avoid this if it’s a problem on a particular project.

Capturing multiline output and exit status in a Github Action with bash

Mon, 23 Sep 2024 13:02:14 +0100

You can capture output from a Github Action by echoing a variable name to the file specified by the "$GITHUB_OUTPUT" variable:

echo "foobar='here is some data'" | tee -a "$GITHUB_OUTPUT"

This is a bit trickier with multiline output. You can use the bash EOF syntax for multiline strings:

FOOBAR<<EOF
here
is
some
multiline
data
EOF

The easiest way to use bash multiline strings in a Github Action is to use a bash command group with curly braces.

Github Action to get the current job URL

Mon, 23 Sep 2024 12:13:40 +0100

Here’s a Github Action to get the URL to the current job and specific step in the job to open. This is useful for giving users a direct link to specific jobs.

name: Get current job URL
description: Output the current Github Workflow Job URL
inputs:
 step_name:
 description: Specific job step to include in the URL
 required: true
outputs:
 job_url:
 description: "URL to the calling job and specific input step"
 value: ${{steps.get_job_url.outputs.job_url}}
runs:
 using: "composite"
 steps:
 - id: get_job_url
 env:
 GH_TOKEN: ${{github.token}}
 shell: bash
 run: |
 echo "job_url=$(gh run --repo ${{github.repository}} view ${{github.run_id}} \
 --json jobs \
 --jq '.jobs[]
 | select(.name == "${{github.job}}")
 | .url,
 (
 .steps[]
 | select(.name == "${{inputs.step_name}}")
 | "#step:\(.number):1"
 )' \
 | tr -d "\n")" | tee "$GITHUB_OUTPUT"

The key part is this bash script which uses the gh CLI tool’s --jq option to extract the URL:

Ensure Python async generator and context manager closes with contextlib.aclosing

Wed, 18 Sep 2024 12:42:26 +0100

I noticed a ResourceWarning like this when using aiofiles to iterate lines of a file in an async context manager from aiofiles.open:

ResourceWarning: unclosed file
<_io.TextIOWrapper
	name='.../numbers.tmp'
	mode='r'
	encoding='UTF-8'>

This is only a warning, so by default it won’t error out of the Python program. I try to have pytest filterwarnings set to error whenever possible, so that any warnings during tests will cause the test to error out. That’s how I spotted this issue.

Python asyncio ensure event loop utility function

Sun, 15 Sep 2024 10:51:12 +0100

Here’s a utility function that safely ensures an event loop is running in a Python asyncio application.

This is useful because of the discrepancy in expectations between the asyncio library and how it ends up being used in the real world in a lot of applications. The asyncio library reasonably expects users to be in control of the event loop for the whole application lifecycle, but this becomes complicated in various situations:

Doozy

Sun, 01 Sep 2024 00:00:00 +0000

I’ve been helping to build Doozy’s product as a freelance Software Engineer, using GCP, Firebase and TypeScript.

ElseWhen

Mon, 15 Jul 2024 00:00:00 +0000

I worked as a Software Engineer on a contract basis for ElseWhen, using asyncio, CloudBuild, CloudRun, GCP, Gemini, Google Cloud Functions, LangChain, LLMs, OpenAI, PostgreSQL, Python, SQL, Terraform and other technologies.

Omnipresent

Mon, 01 Jan 2024 00:00:00 +0000

I worked as a Software Engineer on a contract basis for Omnipresent via YLD, using asyncio, AWS, AWS Lambda, AWS SAM, Docker, ECS, FastAPI, PostgreSQL, Python, SNS, SQL, SQS, Terraform and other technologies.

AWS CDK to deny release of Elastic IP in Organization SCP

Sat, 16 Sep 2023 18:23:42 +0000

It’s quite likely that you’ll let other organisations know about the Elastic IPs that you have configured in your AWS VPC. The other organisations could be service providers or your customers, and they might not have automated processes for configuring these static IP addresses on their side (sending configurations by email is surprisingly common).

This creates a risk around the accidental release of Elastic IPs, especially if you manage your Elastic IPs with CloudFormation or CDK. Updating a CloudFormation stack that manages an Elastic IP could accidentally release that Elastic IP and recreate the resource with a different IP address. This would cause an immediate outage or disruption in any service that was configured with the released Elastic IP.

AWS CDK for EC2 bastion host sharing Elastic IP with rest of VPC

Wed, 16 Aug 2023 18:23:42 +0000

Here’s a small CDK stack to set up an EC2 instance as a bastion host inside a VPC, sharing a fixed Elastic IP with other services in the VPC via the NAT Gateway.

A bastion host is useful for connecting to your own services inside the VPC. With this particular setup, you can also use the same bastion host to connect to external services that require a fixed IP address on your side.

Minify JSON clipboard contents back to clipboard with Bash and Python

Tue, 18 Jul 2023 13:18:51 +0100

Here’s a Bash one-liner to minify JSON in the current contents of the clipboard and replace the clipboard contents with the minified JSON:

xclip -selection clipboard -o | python3 -c "import sys;import json;print(json.dumps(json.loads(sys.stdin.read()),separators=(\",\",\":\")))" | tee >(xclip -selection clipboard)

The equivalent on Mac is:

pbpaste | python3 -c "import sys;import json;print(json.dumps(json.loads(sys.stdin.read()),separators=(\",\",\":\")))" | pbcopy

To set it as an alias in Linux (mcj for “minify clipboard json”):

alias mcj='xclip -selection clipboard -o | python3 -c "import sys;import json;print(json.dumps(json.loads(sys.stdin.read()),separators=(\",\",\":\")))" | tee >(xclip -selection clipboard)'

Then you can copy some JSON, run mcj, and then paste the minified JSON back from the clipboard, e.g.

JSON encode clipboard contents back to clipboard with Bash and Python

Tue, 18 Jul 2023 12:38:51 +0100

Here’s a Bash one-liner to JSON-encode the current contents of the clipboard and replace the clipboard contents with the JSON-encoded output:

xclip -o | python -c 'import sys;import json;print(json.dumps(sys.stdin.read().strip()))' | tee >(xclip -selection clipboard)

The equivalent on Mac is:

pbpaste | python -c 'import sys;import json;print(json.dumps(sys.stdin.read().strip()))' | pbcopy

To set it as an alias in Linux (jec for “json encode clipboard”):

alias jec='xclip -selection clipboard -o | python -c "import sys;import json;print(json.dumps(sys.stdin.read().strip()))" | tee >(xclip -selection clipboard)'

Then you can copy some string text, run jec, and then paste the JSON-encoded string back from the clipboard.

AWS EC2 bastion host SSM multi port forwarding bash script

Fri, 16 Jun 2023 18:23:42 +0000

Here’s a small bash script to identify an EC2 instance as a bastion host, and then use SSM Session Manager to forward multiple ports through the bastion on to other hosts.

This is useful when you want to access hosts inside your VPC (such as RDS / Aurora), external services that enforce an IP access list (such as MongoDB Atlas), or a mixture of those.

I’ve used scripts like this to run integration tests that talk to combinations of services like the above.

KrakenFlex

Thu, 01 Jun 2023 00:00:00 +0000

I worked as a Software Engineer on a contract basis for KrakenFlex, using asyncio, AWS, AWS Lambda, AWS SAM, CloudFormation, DynamoDB, Python, SNS, SQS and other technologies.

AWS SAM CloudFormation SSM Parameter secure string not supported workaround

Tue, 16 May 2023 18:23:42 +0000

It’s surprisingly difficult to use AWS SSM Parameter secure strings in CloudFormation templates.

If you try and have CloudFormation fetch the value of the SSM Parameter as a AWS::SSM::Parameter::Value<String> type, you’ll get this error:

An error occurred (ValidationError) when calling the CreateStack operation:
Parameters [/foobar/foo_param] referenced by template have types not supported
by CloudFormation.

If you try and have a CloudFormation parameter of type AWS::SSM::Parameter::Value<SecureString>, you’ll get this error:

An error occurred (ValidationError) when calling the CreateStack operation:
Template format error: Unrecognized parameter type: SecureString

If you try and use the special resolve function in the CloudFormation template, like this:

AWS Lambda Python SAM build with container and CodeArtifact poetry auth

Sun, 16 Apr 2023 18:23:42 +0000

It’s often beneficial to use Docker to build AWS SAM Lambda Function images. Projects using AWS SAM often also use AWS CodeArtifact to manage private libraries, and poetry to manage Python dependencies.

This combination can make it a little bit tricky to get the index auth working during the Docker build.

Your Dockerfile for building the AWS SAM Lambda Function might look like this:

FROM public.ecr.aws/lambda/python:3.12

ENV PIP_DISABLE_PIP_VERSION_CHECK=on \
 PYTHONDONTWRITEBYTECODE=1 \
 PYTHONUNBUFFERED=1

COPY requirements.txt .
RUN python3.12 -m pip install -r requirements.txt
RUN rm requirements.txt

RUN mkdir -p ${LAMBDA_TASK_ROOT}/foobar_lambda

COPY foobar_lambda ${LAMBDA_TASK_ROOT}/foobar_lambda

CMD ["foobar_lambda.aws.lambda_handler"]

You can build the AWS SAM Lambda Function image from that, using an authenticated private repository in AWS CodeArtifact, with a shell script like this:

Converting PDF slides to PNG then CSV to bulk import into Anki

Fri, 17 Mar 2023 10:40:47 +0000

I had a large slide deck in PDF format and wanted to add the individual slides to Anki to revise from. There is some benefit to the learning process in doing this manually, but it was more practical to study the slides first and then automate the import into Anki.

The final bash command to produce an importable CSV file for Anki looks like this:

pdftoppm my_slides.pdf my_slide \
 -progress -png -f 20 -l 923 -rx 60 -ry 60 && \

cp ./*.png ~/.local/share/Anki2/my_username/collection.media/ && \

echo >| my_list.csv && \

ls *.png \
 | awk '{printf("<img src='%s'/>\t\tmy_tag_1 my_tag_2\n",$1)}' \
 >> my_list.csv

First it uses the pdftoppm tool to create a PNG image file for each slide:

Unit testing a Python AWS Lambda handler function that uses asyncio with its own event loop

Fri, 17 Feb 2023 10:40:47 +0000

If you’re using asyncio in an AWS Lambda, the Lambda handler function has to manage the asyncio event loop inside the Lambda, as the handler function itself cannot be an async function.

This is fairly easy to do by having the top-level Lambda handler function use asyncio.run to call an asynchronous function that does the actual work:

import asyncio

async def foobar_handler() -> str:
 return "foobar"

def lambda_handler(event, context) -> str:
 return asyncio.run(main())

Or you might be using AWS Lambda Powertools to do async batch processing:

Python AWS boto moto SNS Invalid Parameter Exception: 'not enough values to unpack (expected 6, got 1)'

Tue, 17 Jan 2023 10:40:47 +0000

I just spent a long time debugging this error I was getting in a unit test using moto to mock out the boto3 SNS client:

botocore.errorfactory.InvalidParameterException: An error occurred
(InvalidParameter) when calling the Publish operation: not enough values to
unpack (expected 6, got 1)

This error comes from the innards of moto, where it splits an ARN on : and assumes there will be six sections to unpack into Python variables there. I was using test_sns_topic_arn as this is just a unit test.

Hyperobjekt

Thu, 01 Sep 2022 00:00:00 +0000

Pytest fixture with optional parameter change inside test case

Mon, 18 Apr 2022 12:34:32 +0000

Here’s a way of setting up a pytest fixture that sets up a default value automatically for any test case that uses the fixture, but allows calling the fixture from inside the test case to update that value where necessary.

The example is for mocking out API responses with httpx_mock, but this fixture arrangement can be used for any situation where you want a default fixture value that can optionally be modified from within a single test case.

Python assert_dict_matches recursive test helper function

Fri, 18 Mar 2022 12:34:32 +0000

The Jest library for Node.js has a helpful expect.objectContaining function that makes it easy to assert only on the keys of interest in a particular test.

Python’s unittest library does have a similar assertDictEqual method, but it asserts on all keys.

I wanted to do be able to easily assert on only the keys of interest in Python tests, so I put together this quick test helper function to do so. It makes use of assertDictEqual, but only after extracting the keys of interest in a recursive way.

Hugo asset pipeline security.exec.allow error

Tue, 15 Mar 2022 16:19:31 +0000

If you hit this error with recent versions of Hugo:

Error: Error building site: BABEL: failed to transform "js/main.js" (application/javascript): access denied: "babel" is not whitelisted in policy "security.exec.allow"; the current security configuration is:

You need to allow babel in Hugo’s security rules in the config file, like this:

[security]
	[security.exec]
		allow = ['^go$', '^npx$', '^postcss$', '^babel$']

Export, clipboard copy and echo a token Bash one-liner

Tue, 15 Feb 2022 16:19:31 +0000

Here’s a handy Bash one-liner to generate a token or some other kind of command output, and do three things at once with it:

Export it as an environment variable in the shell.
Copy it to the clipboard.
Echo it to the terminal.

export FOOBAR_TOKEN=$(foobar-token-command --foobar "foobar") \
 && echo -n $FOOBAR_TOKEN | tee >(xclip -selection clipboard)

I find this is handy for things like security tokens which I often want as an environment variable in the shell as well as being able to paste them into other tools such as HTTP testing apps.

Hugo new post Bash function

Sat, 15 Jan 2022 12:52:06 +0000

Hugo has a hugo new helper command that can generate new content files for you. It doesn’t quite do what I want, though, so I’ve added this quick Bash function to my .profile to make it easier to use:

hnp() {
 hugo new "${2:-posts}/$(date +%Y)/$(date +%m)/${1}/index.md"
}

All it does is create a new post file in its own content directory at a date-based path, which is the structure I use for my Hugo sites.

AWS DocumentDB data source in JetBrains DataGrip

Thu, 06 Jan 2022 00:00:00 +0000

I’m currently working on a project for Kensio Software using AWS DocumentDB, and wanted to access the database for development using DataGrip. Here are some quick notes on how to set up a DocumentDB database as a data source in DataGrip.

Getting DocumentDB connection details from AWS console

It’s probably easiest to get the connection details for the DocumentDB cluster from the AWS Console.

At the time of writing, the URL for the DocumentDB cluster looks like this:

X2X Media

Sat, 01 Jan 2022 00:00:00 +0000

Python iterator chunk generator function

Sat, 23 Oct 2021 00:00:00 +0000

Here’s a versatile Python function that splits any iterator into chunks of size n as a generator. It can take simple iterators such as lists, but also take a generator and chunk it without unwinding the whole thing up front.

This is quite useful when dealing with a database cursor, for example. You can iterate through the cursor in chunks of size n without having to load all of the objects into memory at once.

Simple example of a floating point precision error for monetary amounts

Sat, 23 Oct 2021 00:00:00 +0000

Most software engineers are aware that floating point numbers are not suitable for monetary amounts, or any amount requiring reliable decimal precision.

It’s nice to have a quick and simple example of this, so here’s an operation that will break with floating point numbers in many languages including JavaScript:

1.01 - 0.42

// → 0.5900000000000001

In JavaScript and TypeScript, the big.js library is good for decimal operations.

In SQL, it’s best to store monetary and other decimal amounts as DECIMAL, e.g. DECIMAL(10,2).

Explicit types are a better habit than type inference

Thu, 02 Sep 2021 00:00:00 +0000

TypeScript has a handy type inference feature where the compiler will automatically determine types in most situations.

For example:

function fooBar(): number {
	return 42;
}
const x = fooBar();
// x's type is infered as number

This is quite useful and potentially allows for tidier, more readable code. However, I would actually suggest that avoiding type inference and preferring explicit types is a good habit.

For trivial examples like the above, it probably is overkill to declare x as const x: number = 42. Preferring explicit typing is a valuable habit in larger projects with more complexity. While human readers will lose track of long type chains, the compiler will happily keep infering them to the point where it’s difficult to debug and maintain.

Google Cloud Platform Logging Query Syntax Exploration Fu

Wed, 04 Aug 2021 00:00:00 +0000

Recently I’ve been working in Google Cloud Platform (GCP) in a highly distributed system that is partitioned into many different services with various deployment strategies. A lot of it runs in Google Kubernetes Engine (GKE), a lot of it runs in Google Cloud Functions (GCF), and a lot of it runs in other bits and pieces of GCP infrastructure. Even the areas that are managed via GKE and Cloud Functions are heterogenous, with many idiosyncratic infrastructure configurations.

My Jest mock cheatsheet for TypeScript

Wed, 23 Jun 2021 00:00:00 +0000

I find myself looking up past projects to remember how to mock various things with Jest in TypeScript, so I’ve started writing up notes on this page to gather them in one place.

Mock a function without early referencing

Because of the way Jest hoists mocks, it’s easy to trigger a footgun where Jest complains about referencing something before initialisation.

You can usually avoid that by wrapping the reference in an outer function call, so that it’s not immediately referenced:

Udemy Course Progress JS Bookmarklet

Sat, 05 Jun 2021 12:16:20 +0100

The Udemy video course UI doesn’t give you a lot of information about your progress through a course. I find it helpful to be able to see progress and the amount of time remaining in a course, so I made this JavaScript bookmarklet to display some figures about progress through a Udemy course.

You need to be on the Udemy video player page when you trigger the bookmarklet. It will display progress stats in a browser alert.

Bulb

Tue, 01 Jun 2021 00:00:00 +0000

I worked as a Software Engineer on a contract basis for Bulb via YLD, using Kubernetes, Terraform, GCP, TypeScript, Node.js, Python, MySQL, PostgreSQL and other technologies.

Sweeping complexity under the infrastructure carpet

Sun, 23 May 2021 00:00:00 +0000

Some projects seem to get into the habit of keeping the application code relatively clean and simple by hiding complexity in the infrastructure layer. My made up term for this is sweeping complexity under the infrastructure carpet. It feels appropriate that this is a bit of a mouthful to say.

This anti-pattern seems to creep in via an attitude that complexity in infrastructure “doesn’t count”, so it’s OK to let it grow and fester there while the application code appears to be more healthy. Maybe complex infrastructure is even a goal to aspire to, as working with high-powered cloud architecture components seems more worthy than tinkering with classes and functions in application land.

MySQL timestamp columns default to ON UPDATE CURRENT TIMESTAMP

Tue, 18 May 2021 00:00:00 +0000

A quick note about a footgun in MySQL with timestamp colummns, which I happened to notice when using migrations in the knex.js library.

TLDR: older versions of MySQL (<= 5.7) default to adding a DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP clause to timestamp columns unless otherwise specified.

Documentation here: https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_explicit_defaults_for_timestamp

For example, if you create a table like this in older versions of MySQL:

CREATE TABLE `foobar_table`
(
 `foobar_column` timestamp
)

You’ll actually get a column like this:

TypeScript branded string type utility for type safety

Wed, 28 Apr 2021 00:00:00 +0000

Here’s an easy way to get branded string types in TypeScript. This allows the compiler to enforce minimal type safety on strings with an identity such as a User ID or Post ID.

export type Brand<K, T extends string> = K & { [P in T]: never };

This can be helpful when you have multiple kinds of IDs that are all strings at runtime, and you want to avoid accidentally passing them in the wrong place.

It also lets types in consuming code offer some documentation of interfaces, for example:

Static site full text search with Hugo, S3, Python and JS

Mon, 26 Apr 2021 00:00:00 +0000

Static sites are great for their speed and ease of deployment at low cost. One thing they might seem to miss out on is full-text search, but you can actually achieve reasonable full-text search functionality on a static site with a fairly simple solution.

This approach can be expanded in various ways to offer more functionality, but the basic core of it is:

Use Python to iterate all of the searchable items, and create an index.json file for each term found.
Deploy those index.json files with the rest of the static site to S3.
Create a static file at /search/index.html with a JS script to handle the search behaviour.
That JS script tries to fetch the index.json file for each search term, combines all the items found, scores them by how many index files they appear in and sorts them by that score.
Finally the JS script inserts a DOM element for each item using the data it got from the index.json files.

You can see a demo of this static site search here: https://momentwallart.co.uk/search/?q=japanese+mountains

Selecting the best available social meta data image in Hugo

Fri, 23 Apr 2021 00:00:00 +0000

Here’s a snippet of Hugo template language that I use to try and select the best social meta data image for each page on a static site:

{{ if isset .Params "meta_image" }}
 <meta property="og:image" content="{{ .Params.meta_image | absURL }}"/>
 <meta name="twitter:image" content="{{ .Params.meta_image | absURL }}">
 <link rel="image_src" href="{{ .Params.meta_image | absURL }}"/>
 <meta itemprop="image" content="{{ .Params.meta_image | absURL }}">
{{ else if (.Resources.ByType "image") | len }}
 {{ $firstImage := index (.Resources.ByType "image") 0 }}
 <meta property="og:image" content="{{ $firstImage.Permalink }}"/>
 <meta name="twitter:image" content="{{ $firstImage.Permalink }}">
 <link rel="image_src" href="{{ $firstImage.Permalink }}"/>
 <meta itemprop="image" content="{{ $firstImage.Permalink }}">
{{ else }}
 <meta property="og:image" content="{{"/img/default-image.png" | absURL }}"/>
 <meta name="twitter:image" content="{{"/img/default-image.png" | absURL }}">
 <link rel="image_src" href="{{"/img/default-image.png" | absURL }}"/>
 <meta itemprop="image" content="{{"/img/default-image.png" | absURL }}">
{{ end }}

First it tries using a meta_image key in the page’s frontmatter, then the first image resource if one is available, and finally falls back to a default global meta image if neither of those are present.

Connecting to an old modp1024 L2TP Ipsec VPN on Ubuntu 21.04

Thu, 01 Apr 2021 00:00:00 +0000

A client of Kensio Software is using an L2TP VPN with Ipsec that’s using an obsolete DH2 (modp1024) algorithm. This algorithm was considered insecure as far back as 2001, and has now been deprecated in encryption libraries such as libreswan.

Most of the permanent engineers at the client are on MacOS, which still supports DH2 / modp1024 algorithm out of the box. I was able to get the VPN connected on a Macbook, but still wanted to try and get it working on Ubuntu as that’s my main working OS.

Beware of SQL averages over dates with empty days

Tue, 23 Feb 2021 00:00:00 +0000

It’s pretty common to use SQL to extract sales summaries from tables that represent sales orders as rows. These kinds of queries often use the AVG() aggregate function to summarise the sales volume. There is a major pitfall when doing this, though!

We might run a sub-query like this to get daily revenue totals from the orders table:

SELECT order_date,
 SUM(order_total) AS total_revenue
FROM orders
GROUP BY order_date

Those aggregated revenue query results might look something like this:

Generating a range of dates in MySQL

Sat, 23 Jan 2021 00:00:00 +0000

It’s often useful to have a range of contiguous dates when building SQL queries, for example by using a left join to ensure that data from other tables that might have missing dates will get a null value in those rows instead of skipping the rows entirely.

Postgres has its wonderful generate_series function which can solve a lot of these problems. Unfortunately MySQL doesn’t have the same function.

You can generate a range of contiguous dates in MySQL with a query like this:

Simple Boto3 mock in Python with Pytest monkeypatch

Wed, 20 Jan 2021 17:13:05 +0000

If you’re anything like me, then you might find mocking dependencies in Python tests quite fiddly and unintuitive (it’s certainly less straightforward than Jest in TypeScript).

I needed to mock one part of the Python boto3 library for AWS, and I got it working using Pytest’s monkeypatch.

This shows a test for fetching a value from a JSON-encoded secret string in AWS Secrets Manager, but you can extrapolate from this roughly how to mock out other boto3 APIs in a similar way.

Big Sister, Little Sister, Red Sister

Fri, 01 Jan 2021 00:00:00 +0000

Stubborn Attachments

Fri, 01 Jan 2021 00:00:00 +0000

ESLint custom rules are useful

Wed, 23 Dec 2020 00:00:00 +0000

Here’s an example of a custom eslint rule that detects usage of the knex library in a unit test file and reports it as a linter error.

module.exports = {
 rules: {
 'no-db-unit-tests': {
 create: function (context) {
 return {
 ImportDeclaration(node) {
 const fileName = context.getFilename();
 if (node.specifiers) {
 node.specifiers.map((item) => {
 if (
 item.local.name === 'knex' &&
 fileName.includes('unit.spec')
 ) {
 return context.report(
 node,
 item.loc,
 `Do not use ${item.local.name} in unit test file ${fileName}`,
 );
 }

 return null;
 });
 }
 },
 };
 },
 },
 },
};

Where's My Flying Car?

Tue, 01 Dec 2020 00:00:00 +0000

Jest mock error "ReferenceError Cannot access before initialization"

Wed, 18 Nov 2020 00:00:00 +0000

When using Jest mocks, it’s easy to hit this error:

ReferenceError: Cannot access '...' before initialization

This is a bit vague and difficult to figure out if you don’t know what the issue is.

This error is due to the way Jest hoists mocks to the top.

For example if you have this in a test file:

const fooServiceFooFunction = jest.fn();

jest.mock("@foo-provider/some-library", () => ({
 fooService: () => ({
 foobar: fooServiceFooFunction,
 }),
}));

It looks like it should work fine as the fooServiceFooFunction mock function is initialised before jest.mock() is called to apply the mock.

Ask your interview candidates to ask you an algorithm question in return

Thu, 12 Nov 2020 00:00:00 +0000

Here’s a potential idea to improve the technical interviews that you run: if you use algorithm questions in interviews, ask candidates to ask you an algorithm question as well.

I think it could be beneficial to tell candidates that they should prepare an algorithm question to pose to their interviewer and work through together in the same way. This could have a similar vibe to the general questions at the end of the interview, when we ask candidates if they have any questions for us.

Uploading Etsy listing images in PHP with Oauth

Fri, 23 Oct 2020 00:00:00 +0000

After figuring out how to upload Etsy listing images in Python for one project, I needed to do the same thing in PHP for another. Despite Etsy itself being implemented in PHP (as far as I know), and the examples in their documentation being in PHP, this turned out to be tricky.

First up, the example for uploading listing images in Etsy’s documentation is plain wrong. It doesn’t work. Don’t waste any more time trying to get that example to work.

Easy encrypted git with gcrypt

Sun, 23 Aug 2020 00:00:00 +0000

I’ve been looking at options for keeping personal notes and diary entries stored securely and synced across devices. An encrypted Git repo in Github seems like a good potential option for this.

It turns out this is pretty straightforward to achieve using git-remote-gcrypt. This sets up a Git remote that is encrypted while the local repository is kept in plaintext and used normally.

Check if you have a GPG key already:

Refactoring Terraform state into separate files in GCP GCS

Sun, 23 Aug 2020 00:00:00 +0000

Recently I needed to refactor some Terraform configuration, which had grown into quite a large single state file, into smaller separate directories with their own state files.

This is a little bit tricky but the Terraform CLI tool helps a lot.

The first thing is to get the existing single state file JSON:

terraform state pull | tee terraform.tfstate

This should work for any storage backend.

Alternatively you can use gsutil to copy the Terraform state file from a GCS bucket:

The Great Successor

Sat, 01 Aug 2020 00:00:00 +0000

Raw HTML shortcode in Hugo

Tue, 28 Jul 2020 00:00:00 +0000

Here’s a little Hugo shortcode for including raw HTML within your Markdown content.

Just create a shortcode file at e.g. layouts/shortcodes/rawhtml.html with this content:

{{.Inner}}

Now in your Markdown content you can include whatever raw HTML you want, e.g.:

{{< rawhtml >}}
 <button class="buttonFromMarkdown">Click Me!</button>
 <script>
 document.querySelector(".buttonFromMarkdown").onclick = function (event) {
 console.log("Clickety click");
 event.target.style = "background-color: #77dd77";
 };
 </script>
 <style>
 .buttonFromMarkdown {
 padding: 1em;
 cursor: pointer;
 }
 </style>
{{< /rawhtml >}}

Which results in:

How I get more benefit from technical reading with Anki

Sun, 26 Jul 2020 00:00:00 +0000

Anki is one of my favourite pieces of software ever, and it’s great to see that it seems to be getting more and more attention as a valuable tool.

I often do a kind of learning-and-note-taking exercise with Anki when reading technical content, which I’m writing up here.

It’s not complicated or dogmatic, and the current version boils down to three tactics:

Cloze deletion for easy conversion of text into notes.
Noting short code samples with a dedicated Code note-type.
Use screenshots to turn anything into simple memory prompt notes.

Here’s how each of those works for me. I’m using the article “Some SQL Tricks of an Application DBA” as an example technical text that I recently went through with these note-taking approaches.

Pivot joining 19 million lines of CSV with unix tools

Thu, 11 Jun 2020 00:00:00 +0000

I’m participating in Kodoeba, an open hackathon around the Tatoeba example sentence project.

My plan is to incorporate Mandarin-English example sentence pairs from Tatoeba into the Chinese example sentence search on Chinese Boost, mainly to allow searching them via pinyin with or without tone-marks.

The first challenge was actually to extract the Mandarin-English example sentence pairs from Tatoeba’s data files.

The data is structured in three separate TSV files (tab-separated values, equivalent to CSV):

Productivity with random numbers

Mon, 01 Jun 2020 00:00:00 +0000

One little productivity trick I use is to select tasks or work with a random number. The simplest form is numbering a list of things you could do, then selecting one randomly.

This can be applied to various things you might want to work on:

Items on a todo list.
Chapters or pages in a textbook.
Tickets in a work-tracking system.
A list of projects, with sub-lists of work items.
Options in a decision you’re stalling on.

The random number generator on random.org is quite nice for this.

Simplicity resistance

Mon, 01 Jun 2020 00:00:00 +0000

Here’s another term that ties in with the ideas in “boring considered beneficial”, “fee justification” and “problem maximisation”: simplicity resistance.

When something is too easy, we dislike the sensation, and try to make it more complicated in order to soothe that. This is a kind of cognitive dissonance around finding solutions to problems. We think “it can’t be that easy”, “it’s too good to be true” and so on, and start looking for the more complicated solution that the problem at hand surely justifies.

Deploying a Hugo site with Github Actions, S3, Cloudfront and AWS IAM

Thu, 28 May 2020 00:00:00 +0000

GitHub Actions are a nice way to get “push to deploy” behaviour for a static site. Here’s a quick run-through of how I have it working for Hugo sites being deployed to AWS.

This assumes you have created an S3 bucket and Cloudfront distribution for the site, and have pointed the domain name to the Cloudfront distribution.

It’s best to have an isolated IAM user for deploying with minimal permissions. Create a new policy in AWS IAM, and give it minimal permissions for deploying the site.

Does your code read like your conversations?

Thu, 28 May 2020 00:00:00 +0000

This is a potential rule-of-thumb subjective metric for assessing how well you’re directing your software development resources:

Does your code read like your conversations?

What this means is, does the code you’re writing overlap closely with the conversations you have about the product and the problems it solves?

For example, if you talk about “free month of foobar with the XYZ subscription”, then it’s probably a good sign if those words appear a lot in the code. Hopefully there are classes and functions that overlap with that.

A lack of correlation does imply a lack of causation

Sat, 23 May 2020 00:00:00 +0000

Correlation does not imply causation, but a lack of correlation does imply a lack of causation.

The famous maxim that “correlation does not imply causation” seems to get over-interpreted into some kind of suggestion that correlation and causation can never be related in any way.

One aspect of that is that a lack of correlation does imply that there is no causation effect going on. If there was a causative relationship between two things, we would expect to see correlation.

Engineer pain vs user pain

Mon, 18 May 2020 00:00:00 +0000

A lot of software engineering teams talk about “pain” on a regular basis.

Usually this refers to difficulties around technical debt, design decisions, refactoring and so on. It rarely seems to refer to issues that affect users or the rest of the business.

For example, moving to microservices is often touted as a way to “reduce pain”. Perhaps we can reduce pain by putting HTTP calls in between different parts of the system, but it’s rare that this improves things for users in any way.

Crystal ball development

Sat, 16 May 2020 00:00:00 +0000

There’s a well known software development maxim called YAGNI: “You Ain’t Gonna Need It”. YAGNI is a general rule for deciding whether or not to implement something or add complexity. It’s designed to give some resistance against the temptation to spuriously add features or try to solve hypothetical future problems.

Applying a bit of inversion to YAGNI, you get something that might be called “crystal ball development”. This describes the attempt to make magic predictions about the future and apply them to software development. As with a crystal ball in popular culture, the prediction is an illusion that leads you astray.

Problem maximisation

Fri, 08 May 2020 00:00:00 +0000

A common metaphor about the potential dangers of AI is called “paperclip maximisation”. The idea is that an AI tasked with producing paperclips could figure out that it can produce more paperclips by e.g. taking over the entire galaxy to turn as much matter as possible into paperclips.

Sometimes I wonder if software engineers are prone to a similar issue. We believe we’re tasked with solving as many problems as possible, and realise that if we increase the number of problems, then we can increase the number of solutions.

Python script to automate static Hugo site post images

Sun, 03 May 2020 00:00:00 +0000

It’s good to have a general post image on content sites, as it gives more of an identity to each page, and works well for social media sharing images.

While NotesToSelf.dev doesn’t tend to have images, my Chinese learning site and satirical tech news site have an image for each post.

Unsplash is a nice source of general royalty-free images that you can use for this if you give attribution.

When using Unsplash, I set up the url to the Unsplash page (not the image itself) in the frontmatter of each post:

Easy social media images and markup with Hugo

Sun, 19 Apr 2020 00:00:00 +0000

Twitter Cards and Facebook’s OpenGraph provide semi-standardised markup for describing web content to social media and other aggregators.

This can be quite handy for blogs and other text content, as you can provide default information when your content is shared. This is a nice way to make sure your posts get a reasonable cover image when shared to someone’s social timeline.

Here’s the Hugo / Go HTML template I’m currently using here to add some basic sharing markup to each post as part of the HTML head:

Abstraction castles

Sat, 18 Apr 2020 00:00:00 +0000

“…some service maintained by one person that’s built their little impregnable abstraction castle because nobody else had to work on it so they could just yak-shave and bike-shed with their self.” [sic]

https://news.ycombinator.com/item?id=16201126

That Hacker News comment struck a chord with me, as I think the phrase “impregnable abstraction castle” perfectly captures something that is commonly encountered in software.

An abstraction castle appears as follows. An area of a codebase seems to develop into a fortification that is resistant to change, apparently designed to defend against all envisioned future scenarios and to keep out every encroachment of the outside world that the designers could imagine.

Using Laravel signed routes to improve order confirmation security

Thu, 16 Apr 2020 00:00:00 +0000

The order confirmation page on our 3D cards shop is shown to a customer after they complete checkout, and is also linked to in the confirmation and dispatch emails that they receive.

Importantly, this page is not authenticated; we don’t want a heavy sign up / sign in process for customers as that tends to be annoying on other ecommerce systems. I just want to buy some pop-up cards – why do I need to create an account and remember a password etc?

Specify city timezones, not timezone offsets

Sat, 11 Apr 2020 00:00:00 +0000

Dates and times are notoriously difficult to get right in software. Many of the difficulties come from timezones and how unintuitive they are to think about. This is made worse by timezone offsets that change during the year for the same place, such as British Summer Time.

It’s easier to get timezones right when you specify a region/city timezone, such as Europe/London. Trying to think about and specify timezone offsets such as +01:00 is more likely to go wrong.

A simple, readable, meaningful test style with Jest

Mon, 06 Apr 2020 00:00:00 +0000

I usually write tests with a Given, When, Then structure to keep the test focused on documenting and confirming the important behaviour of whatever is being tested. Simple comments are good for this and provide an easy-to-follow visual structure:

test("some defined behaviour", () => {
 // Given the fandango is combobulated;

 // When we discombobulate the fandango;

 // Then the fandango should be discombobulated.

})

It’s important to keep tests clear and readable. Reducing the number of lines under each step and making each line as meaningful as possible are the main ways to do that. You can use test factories and other extractions to reduce line-count and make test code more meaningful.

Fastmail is a nice multi-domain custom email service

Sun, 05 Apr 2020 00:00:00 +0000

I run a few side projects on various domains, and often want to be able to send and receive email on those domains. Over the years I’ve used a few different solutions for this: Gmail aliases, Mailgun just for automated sending, AWS SES just for automated sending and AWS Workmail for manual sending and receiving.

AWS Workmail has been fine, but ends up being a bit pricey for this kind of usage at $4 USD / user / month. You have to have at least one “user” per domain, so with multiple domains this adds up quite quickly and gets prohibitive for random side projects and experiments.

Random element choice in a Hugo HTML template

Fri, 03 Apr 2020 00:00:00 +0000

I run a couple of sites related to my Chinese studies: an East Asian Studies blog, and a Chinese learning resource site.

The content is all free so I have some simple affiliate image links on the pages to cover the bills. These are just straight forward <a><img></a> elements with no tracking or anything obnoxious at all. Even then, I don’t want to have too many adverts on a single page as it would still be a bit much.

(Cross-post) End to end serverless testing with the Driver Pattern

Thu, 02 Apr 2020 00:00:00 +0000

(Cross-posted from https://blog.freetrade.io/end-to-end-serverless-testing-with-the-driver-pattern-bcf2d22bf73a)

In September 2019, Freetrade’s test code consisted of a large number of fast unit tests and a smaller number of end-to-end API tests. These make real HTTP requests against the platform and assert on the results.

The unit tests confirm that individual classes are behaving correctly and improve the design of interfaces, while the API tests confirm that higher-level business logic is working correctly. This is not a bad combination and at the time already gave us decent test coverage of a large proportion of our platform.

Choosing dates and times in tests

Mon, 30 Mar 2020 00:00:00 +0000

A little note to avoid bugs in tests, or missing bugs: don’t use the current date or time when writing unit tests.

When making up data for tests, it’s common to just use today’s date as it’s easy to think up. This can catch you out when there is actually some hidden dependency on the real time and your test starts failing the next day, month or year. The test is currently passing by coincidence, which is hard to distinguish from a real pass if you’re not careful.

Do software engineers have the same "fee justification" problem as investment managers?

Thu, 12 Mar 2020 00:00:00 +0000

It’s widely known (although somehow still controversial) that investment managers who try to manually pick stocks cannot beat the market the vast majority of the time:

That is even without the fees that active investment managers charge, which are usually 1% or more (which is horrendous long-term when subtracted from compound interest).

Decision fatigue and job interviewing

Mon, 09 Mar 2020 00:00:00 +0000

I was reading about decision fatigue and wondering if it accounts for more major events than we give it credit for. The idea is at least spreading a little bit in certain circles, but doesn’t get the wider attention it should.

“One research study found that the decisions judges make are strongly influenced by how long it has been since their last break. “We find that the percentage of favorable rulings drops gradually from ≈65% to nearly zero within each decision session and returns abruptly to ≈65% after a break.””

SQL Puzzle 2: Absentees

Mon, 02 Mar 2020 00:00:00 +0000

The second puzzle in Joe Celko’s SQL Puzzles is about tracking employee absenteeism.

The initial table structure is like this:

CREATE TABLE personnel (
	emp_id integer PRIMARY KEY
);

CREATE TABLE excuse_list (
	reason_code varchar(40) UNIQUE
);

CREATE TABLE absenteeism (
	emp_id integer NOT NULL REFERENCES personnel (emp_id),
	absent_date date NOT NULL,
	reason_code varchar(40) NOT NULL REFERENCES excuse_list (reason_code),
	severity_points integer NOT NULL
		CONSTRAINT severity_points_range CHECK (severity_points BETWEEN 1 AND 4),
	PRIMARY KEY (emp_id, absent_date)
);

There are already some nice properties to notice in this schema:

All columns are NOT NULL, which is ideal.
The list of valid absence reasons is extracted into an explicit table that will be easier to work with than an enum column within the absenteeism table.
There’s a simple but useful check constraint to ensure severity_points is valid (this could also have been a foreign key to a separate table).
There’s a natural primary key of (emp_id, absent_date) on the absenteeism table. This will improve consistency and let the database optimise queries more effectively.

The goal in this SQL puzzle is to make (kind of nasty!) decisions about the consequences of employee absenteeism using this schema, with fiddly rules around how to count and rate absence:

You are not doing Test Driven Development

Sun, 01 Mar 2020 00:00:00 +0000

Test Driven Development can be a business process, not just a development workflow. It can start much earlier in the software engineering process than when implementation starts and code is first written.

Quite a few engineering teams confuse TDD for doing any testing at all, but even teams that know TDD is a stronger process than that still limit its potential by confining it to an implementation workflow detail that is trivial at the level of the business or software team.

Rendering HTML email version of articles with Hugo

Wed, 26 Feb 2020 00:00:00 +0000

I recently started using Mailcoach, mainly for sending emails to the subscriber list for Chinese Boost.

Mailcoach’s HTML email templating abilities are pretty basic, but that’s fine. I’m happy to use simple HTML emails, and found this HTML email template that looks like it will do the job nicely.

As the content on Chinese Boost is managed with Hugo, it would be nice to have Hugo also be responsible for generating HTML email content for each article that can then just be copy-pasted into Mailcoach for sending to subscribers. Hugo has an output formats feature that lets you do exactly that.

SQL Puzzle 1: Fiscal Year Tables

Wed, 26 Feb 2020 00:00:00 +0000

The first puzzle in SQL Puzzles is about adding strong constraints to ensure correctness of data in a table that looks like this:

CREATE TABLE fiscal_years (
	fiscal_year integer,
	start_date date,
	end_date date
)

Making all the columns NOT NULL is a good default first step as it avoids a large source of errors.

CREATE TABLE fiscal_years (
	fiscal_year integer NOT NULL,
	start_date date NOT NULL,
	end_date date NOT NULL
)

We can prevent other errors by adding UNIQUE constraints to each of the three columns, as we don’t want any shared values across rows for them: each fiscal year should only be described once, and no fiscal year can share a start date or end date with another.

Setting up a Mailcoach app on AWS with Laravel Forge

Sun, 23 Feb 2020 00:00:00 +0000

I’ve been looking forward to the release of Mailcoach, which is a self-hosted mailing list manager app. I have a few mailing lists, most of which are tiny, but one of which has over 3000 subscribers. This puts it over the threshold of expensive subscription options in MailChimp. MailChimp wants over 70 USD / month to run this single list, whereas a Mailcoach license is a one-off payment of 99 USD. Plus it seems like it would be interesting to self-host the mailing list manager and learn a few things along the way.

Better alive than right

Wed, 19 Feb 2020 00:00:00 +0000

“Better alive than right” is a kind of proverb I try to remind myself of when cycling in London on a daily basis. A sizeable minority of drivers in London have a disregard for people on bikes and break traffic rules in various ways around them.

A common one is drivers pulling out from a junction in front of someone on a bike, which requires the person cycling to stop to avoid a collision, even though the driver is at fault. The idea of “better alive than right” is to focus on keeping yourself safe above all, even when it’s frustrating as the driver is in the wrong in this situation.

Eating the recipe

Mon, 10 Feb 2020 00:00:00 +0000

“Eating the recipe” is a term I’ve made up to try and describe the kind of mistakes that get made by mixing up levels of abstraction.

Humans do not struggle with the distinction between a recipe and the food that the recipe describes how to make. We would not make the mistake of eating a sheet of paper that describes how to make spaghetti, instead of eating the spaghetti. We do, however, design and use systems that make this kind of mistake a lot, or make it easy for humans to make this kind of mistake.

Personal bike shedding

Sun, 09 Feb 2020 00:00:00 +0000

The concept of bike shedding is one of my favourites, as once you’ve learnt about it, it won’t be long before you see it in the wild at most organisations.

The standard explanation (from Wikipedia), goes like this:

“Parkinson provides the example of a fictional committee whose job was to approve the plans for a nuclear power plant spending the majority of its time on discussions about relatively minor but easy-to-grasp issues, such as what materials to use for the staff bike shed, while neglecting the proposed design of the plant itself, which is far more important and a far more difficult and complex task.”

Write code that is easy to delete

Sat, 08 Feb 2020 00:00:00 +0000

It’s hard to write software that is easy to maintain and adapt to change. There’s a multitude of maxims to try and tease out general solutions to this, or at least approaches that can mitigate the problem.

This point did the rounds on Hacker News and elsewhere:

“Write code that is easy to delete, not easy to extend.”

– programmingisterrible

This is one of the better maxims for producing maintainable and adaptable software. One of the most important aspects that distinguishes it is that it acknowledges that your code is probably bad. At some point, in some situation, in someone’s perception, the code you are writing now is going to be bad and difficult. The person that encounters that difficulty is also likely to be you.

Generating business name ideas with Bash and Linux

Tue, 04 Feb 2020 00:00:00 +0000

I’m setting up a small side-business selling wall art prints in the UK, and needed to think of a short, reasonably unique name.

After Googling for word lists and name generators, I thought it would actually work best to get Bash to generate a list of name ideas for me, keeping the words to a limited length and randomising them to avoid fatigue when reading the list.

The above can be done with this command on Ubuntu Linux:

Some easy ways to improve dev workflows

Mon, 03 Feb 2020 00:00:00 +0000

Your dev workflow is a business process too.

There’s a funny phenomenon in a lot of companies that hire software engineers, where people draw a dividing line between “engineering” and “the business”. This is even reinforced in a lot of software engineering books and other media, which talk about “business people” and so on. I always think of engineering as very much a part of this apparently separate “business”, but that’s at least a whole other blog post.

Simple way to use ts-firebase-driver-testing with a Jest mock

Thu, 23 Jan 2020 00:00:00 +0000

While working at Freetrade, I made the ts-firebase-driver-testing library to allow rapid testing and debugging of Firebase applications (other engineers have contributed to it since then).

The initial plan with the library was to abstract out the Firebase SDK as a set of plain interfaces, and then provide a single-process in-memory implementation of those interfaces. This worked fine and it does let you swap out Firebase when running tests, which was the main goal.

Extract interfaces, don't pre-plan them

Mon, 20 Jan 2020 00:00:00 +0000

The Interface Segration Principle is one of the more overlooked parts of SOLID. I think this might be because it gets seen as a restatement of the Single Responsibility Principle, and not as the separate idea that it actually is.

Interface segration is about keeping interfaces lean and specific to the smallest possible purpose, as suggested by the formal definition:

“No client should be forced to depend on methods it does not use.”

The code should be a sequence diagram too

Sat, 18 Jan 2020 00:00:00 +0000

I love sequence diagrams and other kinds of visual documentation of a system, especially ones that are committed in the repo alongside the application code.

PlantUML is a nice tool for this as there’s a simple mark-up language for creating and editing diagrams which sits nicely in a Git repo. You can commit and review changes to diagrams as you do with the rest of the code.

Diagrams are a kind of comment, though, and so really they should only supplement the code and not be a substitute for readability or clarity of the application code itself.

Python script to convert an Etsy sales export CSV into a Xero import CSV

Sat, 11 Jan 2020 00:00:00 +0000

We handle the accounting for our pop out cards shop using Xero. There are convenient pre-built integrations that handle Stripe and PayPal data in Xero, but nothing for Etsy, so I had to do a little Python script to generate a Xero import CSV file from the Etsy orders export CSV.

Here’s the script for future reference:

#!/usr/bin/python3

# https://www.etsy.com/uk/your/shops/YourShopName/download
# Sold Orders
# e.g. EtsySoldOrders2020.csv

import argparse
import csv
from datetime import datetime
from io import TextIOWrapper
from typing import Dict

SALES_ACCOUNT_CODE = "200"
NOW = datetime.now()
XERO_FIELDS = [
 "*ContactName",
 "EmailAddress",
 "POAddressLine1",
 "POAddressLine2",
 "POAddressLine3",
 "POAddressLine4",
 "POCity",
 "PORegion",
 "POPostalCode",
 "POCountry",
 "*InvoiceNumber",
 "Reference",
 "*InvoiceDate",
 "*DueDate",
 "Total",
 "InventoryItemCode",
 "*Description",
 "*Quantity",
 "*UnitAmount",
 "Discount",
 "*AccountCode",
 "*TaxType",
 "TaxAmount",
 "TrackingName1",
 "TrackingOption1",
 "TrackingName2",
 "TrackingOption2",
 "Currency",
 "BrandingTheme",
]


def convert_etsy_to_xero(etsy_file: TextIOWrapper):
 output_file = f"etsy_xero_orders_{etsy_file.name}"
 with open(output_file, "w", newline="") as output_file:
 etsy_reader = csv.DictReader(etsy_file, delimiter=",", quotechar='"')
 xero_writer = csv.DictWriter(output_file, XERO_FIELDS,
 delimiter=",", quotechar='"')
 xero_writer.writeheader()
 for etsy_order in etsy_reader:
 xero_writer.writerow(
 etsy_order_to_xero_row(etsy_order)
 )


def etsy_order_to_xero_row(etsy_order: Dict[str, str]) -> Dict[str, str]:
 return {
 "*ContactName": etsy_order.get("Full Name"),
 "EmailAddress": "",
 "POAddressLine1": etsy_order.get("Street 1"),
 "POAddressLine2": etsy_order.get("Street 2"),
 "POAddressLine3": etsy_order.get(""),
 "POAddressLine4": etsy_order.get(""),
 "POCity": etsy_order.get("Delivery City"),
 "PORegion": etsy_order.get("Delivery State"),
 "POPostalCode": etsy_order.get("Delivery Zipcode"),
 "POCountry": etsy_order.get("Delivery Country"),
 "*InvoiceNumber": etsy_order.get("Order ID"),
 "Reference": etsy_order.get(""),
 "*InvoiceDate": etsy_date_to_xero_date(etsy_order.get("Sale Date")),
 "*DueDate": etsy_date_to_xero_date(etsy_order.get("Sale Date")),
 "Total": etsy_order.get("Order total"),
 "InventoryItemCode": "",
 "*Description": f"Etsy Order {etsy_order.get('Order ID')}",
 "*Quantity": "1",
 "*UnitAmount": etsy_order.get("Order total"),
 "Discount": "",
 "*AccountCode": SALES_ACCOUNT_CODE,
 "*TaxType": etsy_order.get("Sales"),
 "TaxAmount": "",
 "TrackingName1": "",
 "TrackingOption1": "",
 "TrackingName2": "",
 "TrackingOption2": "",
 "Currency": etsy_order.get("Currency"),
 "BrandingTheme": "",
 }


def etsy_date_to_xero_date(etsy_date: str) -> str:
 return datetime.strptime(etsy_date, "%m/%d/%y").strftime("%d %b %Y")


if __name__ == "__main__":
 parser = argparse.ArgumentParser(
 description="Convert an Etsy sales export CSV into a xero invoices CSV")
 parser.add_argument("etsy_file", default=f"EtsySoldOrders{NOW.year}.csv",
 type=argparse.FileType("r", encoding="UTF-8"),
 help="Path to Etsy sales export CSV file")
 args = parser.parse_args()

 convert_etsy_to_xero(args.etsy_file)
 args.etsy_file.close()

Be careful with Node file globbing vs bash globbing

Thu, 02 Jan 2020 00:00:00 +0000

There’s a common gotcha when using commandline tools that have their own file globbing, such as Prettier in Nodejs.

This command is supposed to run prettier against all the .ts files in all folders under src recursively;

prettier --write src/**/*.ts

What actually happens though is that bash expands the src/**/*.ts first, which will only go one folder deep instead of the intended unlimited recursion.

The fix is simply to quote the globbing express so that bash passes it to prettier unscathed:

Atlas Shrugged

Wed, 01 Jan 2020 00:00:00 +0000

Conversations with Friends

Wed, 01 Jan 2020 00:00:00 +0000

Drink

Wed, 01 Jan 2020 00:00:00 +0000

How Fascism Works

Wed, 01 Jan 2020 00:00:00 +0000

If This is a Man · The Truce

Wed, 01 Jan 2020 00:00:00 +0000

My Brilliant Friend

Wed, 01 Jan 2020 00:00:00 +0000

One Hundreds Years of Solitude

Wed, 01 Jan 2020 00:00:00 +0000

The Almanack of Naval Ravikant

Wed, 01 Jan 2020 00:00:00 +0000

The Chinese Language: Fact and Fantasy

Wed, 01 Jan 2020 00:00:00 +0000

The Paper Menagerie

Wed, 01 Jan 2020 00:00:00 +0000

Upheaval

Wed, 01 Jan 2020 00:00:00 +0000

Utopia for Realists

Wed, 01 Jan 2020 00:00:00 +0000

丰乳肥臀

Wed, 01 Jan 2020 00:00:00 +0000

Setting up a Hugo static site inside Laravel on Forge

Mon, 23 Dec 2019 00:00:00 +0000

Making a hybrid static + dynamic site can be quite a useful combination. This might sound like an oxymoron, but it simply means serving static pages from a site that also runs a dynamic web application. You can get some of the best of both worlds with this approach.

My Chinese-learning site ChineseBoost uses this combination – a lot of the content pages, e.g. for Chinese grammar, are statically generated with Hugo, while dynamic parts of the site are served by a Laravel application. Similarly on our pop out cards shop, we have a static blog and an ecommerce application on the same server.

Song of Solomon

Sun, 01 Dec 2019 00:00:00 +0000

Easy typed test mocks with Jest and TypeScript

Wed, 20 Nov 2019 00:00:00 +0000

Here’s a quick and simple way to create type-safe test mocks with Jest in TypeScript:

const fooMock: jest.Mocked<IFoo> = {
	fooMethod: jest.fn()
}

const testSubject = new ClientClass(fooMock);

This has a few useful properties:

The type system will enforce that the dependency is the correct type, i.e. that ClientClass is being passed the instance of IFoo that it requires.
The type system will enforce that the mock implements the required interface, i.e. that it has stubs for all the required methods.
The type system can still see that the methods on the mock are jest.fn, allowing things like fooMock.fooMethod.mockReturnValue({}) without complaining.
The IntelliJ IDE (e.g. WebStorm or IDEA) can automatically fill in the method stubs for you once it sees the jest.Mocked<IFoo> type annotation.

You can also use TypeScript’s handy Pick utility type to get more specific type-safe mocks:

Linking to the previous and next page in a taxonomy term in Hugo

Sat, 02 Nov 2019 00:00:00 +0000

Something that can be slightly tricky to achieve within Hugo’s taxonomy system is linking to the previous and next page within a particular taxonomy term. For example, if you have a Series taxonomy, you might want to link to the previous and next page in that series on each page in it.

One way to solve this is to loop through all the pages in the taxonomy term and grab the previous and next pages on the way.

Flexible test model factories in TypeScript

Wed, 23 Oct 2019 00:00:00 +0000

Tests are improved by being more readable, and concise tests are more readable.

It’s good when a test has clear given, when, then steps, and the important details are plain to see in the test code.

This is often obstructed by large amounts of set-up and boilerplate code in the test that distracts from the important details.

Test model factories can reduce some of that boilerplate.

For example, if we have a Client model that contains quite a lot of detail, setting one up in a test will be verbose. We might only be interested in testing the effect of a single field in a single test, but setting up all the other required fields obscures that intention.

Make a deep object delta diff in TypeScript

Wed, 25 Sep 2019 00:00:00 +0000

Building on the object path enumeration, we can also make a deep object diff or delta.

E.g. given these two objects:

{ "a": { "b": { "c": "123" } }, "foo": "hello" }

{ "a": { "b": { "c": "123", "d": "456" } }, "foo": "hello" }

We would make this delta diff:

{ "a": { "b": { "d": "456" } } }

This uses a few utility functions aside from enumeratePaths, but it should be relatively clear what they do here:

import _ from "lodash"

import { enumeratePaths } from "./enumeratePaths"
import { objGet, objSet } from "./objPath"

export function makeDelta(
 oldObject: IStringToUnknown,
 newObject: IStringToUnknown,
): IStringToUnknown {
 if (typeof newObject !== typeof oldObject) {
 return newObject
 }
 if (!_.isObject(newObject)) {
 return newObject
 }

 const paths: string[] = _.uniq(
 enumeratePaths(oldObject).concat(enumeratePaths(newObject)),
 )
 let pathSplits: string[][] = paths.map((path: string): string[] =>
 path.split("/"),
 )
 pathSplits = pathSplits.filter((thisPath: string[]): boolean => {
 return !pathSplits.find((otherPath: string[]): boolean => {
 return (
 otherPath[0] === thisPath[0] &&
 otherPath.length > thisPath.length
 )
 })
 })

 const delta: IStringToUnknown = {}

 pathSplits.forEach((pathSplit: string[]): void => {
 const oldValue = objGet(oldObject, pathSplit)
 const newValue = objGet(newObject, pathSplit)
 if (newValue !== oldValue) {
 objSet(delta, pathSplit, newValue)
 }
 })

 return delta
}

We get the unique set of all possible paths in both objects (O(n) for each leaf key), and then compare each of those paths between the two objects so overall this should be O(2n) for each unique leaf key between the two objects.

Enumerating all object paths in TypeScript

Mon, 23 Sep 2019 00:00:00 +0000

Here’s a small recursive TypeScript function to enumerate all the paths through an object.

E.g. given this object:

{
 "animals": {
 "lion": { 
 "sound": "roar"
 },
 "zebra": { 
 "appearance": "stripes"
 },
 "elephant": {
 "size": "large",
 "features": {
 "trunk": "yes",
 "tusks": "yes",
 "wings": "no"
 },
 },
 },
},

It enumerates these paths:

[
 "animals",
 "animals/lion",
 "animals/zebra",
 "animals/elephant",
 "animals/lion/sound",
 "animals/zebra/appearance",
 "animals/elephant/size",
 "animals/elephant/features",
 "animals/elephant/features/trunk",
 "animals/elephant/features/tusks",
 "animals/elephant/features/wings",
]

The function:

function enumeratePaths(value: any): string[] {
 let paths: string[] = []
 if (value && typeof value === "object") {
 paths = Object.getOwnPropertyNames(value)
 for (const key of Object.getOwnPropertyNames(value)) {
 paths = [
 ...paths,
 ...enumeratePaths(value[key]).map(
 (subKey) => `${key}/${subKey}`,
 ),
 ]
 }
 }
 return paths
}

Can you have multiple instances of your application in a single process?

Sun, 15 Sep 2019 00:00:00 +0000

Here’s a question that might give some insight into the quality of your codebase:

Can you have multiple instances of your application in a single process?

By that I mean that you could easily create and run parts of the application and/or the whole thing in one process, without this causing bad behaviour. If so, this probably suggests that the design is quite well factored.

It’s a similar idea to the questions in the Joel test, but for internal application structure.

Deepin Screenshot is a nice screenshot tool for Ubuntu Linux

Sun, 18 Aug 2019 00:00:00 +0000

If you want a good screenshot tool on Ubuntu, Deepin Screenshot is probably the best one.

It lets you drag-select a rectangular area of the screen, and then re-arrange the area until you’re happy with it. You can then annotate the screenshot selection with highlights, arrows, boxes and text. Finally, you can also choose how to save the resulting screenshot image, or copy it to your clipboard.

Deepin made it super easy to make the personalised cards FAQ on our card shop, with a series of clear screenshots showing the process with the key parts highlighted. This took a few minutes to make with Deepin.

Showing other pages with the same taxonomy term in Hugo

Thu, 01 Aug 2019 00:00:00 +0000

Here’s a quick HTML template snippet to do something I find is quite a common task in a Hugo site: show a list of other pages that share a taxonomy term with the current page.

It’s probably a good idea to refresh your memory of how taxonomies work in Hugo if you’re feeling rusty.

The snippet is just:

{{ if isset .Params "tags" }}
 {{ $tagTitle := index .Params.tags 0 }}
 {{ $currentLink := .Permalink }}
 <h2><small>Tag:</small>&nbsp;{{ $tagTitle | title }}</h2>
 <ol>
 {{ range (index .Site.Taxonomies.tags ($tagTitle | urlize)).Pages.ByTitle }}
 <li>
 {{ if eq $currentLink .Permalink }}
 {{ .Title }} <small>(this page)</small>
 {{ else }}
 <a href="{{ .Permalink }}">
 {{ .Title }}
 </a>
 {{ end }}
 </li>
 {{ end }}
 </ol>
{{ end }}

The example taxonomy here is “tags”, but it could be any taxonomy. We check if tags are set on this page, and then go ahead with showing other pages that share a tag term with the current one if so.

Generating perspective-skewed mock-up images for printed products with Python

Sun, 21 Jul 2019 00:00:00 +0000

For Moment Wall Art (and other printed product sites in future), we need to show the customer not just the main print design itself but also mock-up images demonstrating how it would look in various settings.

Applying a perspective skew to images like this can be handled by the Python Wand library, specifically with the perspective distortion function.

I have a Python script that iterates through a glob of all product files in parallel, and generates these perspective-skewed mock up images where necessary.

Easier interface segregation in TypeScript (even better than Go!)

Mon, 15 Jul 2019 00:00:00 +0000

Preamble: Introduction to interface segregation

(Skip this if you’re aware of what the Interface Segration Principle is.)

The Interface Segration Principle is the I in the SOLID mnemonic. It’s conceptually one of the simplest, but is often overlooked, which is a shame as it’s a highly beneficial principle to follow in software design.

The Interface Segration Principle is defined as “no client should be forced to depend on methods it does not use”. More simply, it suggests that a client should be able to depend on the smallest possible interface, i.e. exactly what it needs and nothing more.

Start with "Given, When, Then" tests

Sat, 13 Jul 2019 00:00:00 +0000

Whether or not you’re practicing test-driven development, it’s helpful to begin writing tests with the “Given, When, Then” structure.

// Given we have a queued order;

// When we cancel it;

// Then the order should be cancelled.

Getting the structure in with comments first has several benefits.

Firstly, it lowers the mental effort to required to start. You only need to think about the desired behaviour in the simplest terms. What situation is this relevant to; what happens; what should the result be? Once you’ve got that written out it’s often easier to go on to the next steps.

The attacker really does know the system when someone leaves your company

Sat, 13 Jul 2019 00:00:00 +0000

A common maxim of computer security is “the attacker knows the system”. In other words, the system should still be secure even given a hypothetical attacker who knows exactly how it works. It’s the flip-side of relying on “security by obscurity”.

In cryptography, the same concept is referred to as Kerckhoffs’s principle, but can be applied to systems in general.

Most engineers are aware of this concept, but it’s also common to find a vague attitude that you can still “get away with” relying on little details that a hypothetical attacker won’t actually know, e.g.

Fan-in from multiple routes to a single VM with iptables, for FIX on Quickfix

Mon, 08 Jul 2019 00:00:00 +0000

Recently we were trying to set-up some network configuration to route FIX traffic from a single Docker container on a VM instance through four of our VPN gateways, two of the other side’s VPN gateways and on to two destination machines on the other side. It’s somewhat complicated, but necessary due to some constraints being out of our control.

The ultimate goal is to have failover redundancy in the system so that a different route can be used if one route is down.

Get to a failing test ASAP

Mon, 08 Jul 2019 00:00:00 +0000

The “Specification driven development” post mentions the idea of a “a low-resistance workflow” that can be achieved with test-driven development.

Overall that post is probably a bit dogmatic and over-eager about the idea of test-driven development, but the point about achieving a low-resistance workflow still seems valuable.

A more trivial way of doing that is to use a simple workflow like this:

Do you have a failing test? -> Make the test pass.
No failing test? -> Make a failing test case.

A lot of the time, no further thought is required. This is a simpler distillation of the idea of a low-resistance workflow. There is rarely a hard decision to make, and you can keep moving forwards by doing either 1. or 2. in a loop.

Pretty print JSON on the command line with Python

Wed, 26 Jun 2019 00:00:00 +0000

I’m often dealing with little bits of JSON, and sometimes it’s helpful to pretty-print them for readability.

Python has a handy tool to make that easy on the command line: json.tool.

E.g.

echo '{"foo": "bar", "thing": { "baz": 42 } }' | python -m json.tool

prints:

{
 "foo": "bar",
 "thing": {
 "baz": 42
 }
}

If you’re on Mac, you can convert JSON in the clipboard to its pretty-printed form with:

Be careful with truthy promises in TypeScript

Sun, 23 Jun 2019 00:00:00 +0000

There’s a type of small “gotcha” bug that is easy to hit with promises for boolean values in JavaScript.

The simplest example might be this:

async function checkSomeCondition(): Promise<boolean> {
	return false
}

if (checkSomeCondition()) {
	// do something.
}

The if branch will always be executed, because the promise object returned by checkSomeCondition() will be evaluated as true, regardless of what it would resolve to. It’s a typo-level mistake, but is easy to make.

It’s easier to miss this mistake when the condition has more parts, e.g.

Loop try catch

Wed, 19 Jun 2019 00:00:00 +0000

A simple cause of failure in software systems is a single problematic item in a set causing the entire set to fail.

Trivially, it looks like this:

for (const item of items) {
 processItem(item)
}

When something goes wrong in processItem() and throws an error, either the whole set fails to process, or all the subsequent items fail.

The trivial example has a trivial improvement:

for (const item of items) {
 try {
 processItem(item)
 } catch (err) {
 // Do something useful with the error.
 }
}

It might be desirable that the whole set fails if some items fail (like a circuit breaker), but often it’s better that the other items go through.

(Cross-post) A Day in the Life of a Software Engineer at Freetrade

Sat, 01 Jun 2019 00:00:00 +0000

(Cross-posted from https://blog.freetrade.io/a-day-in-the-life-of-a-software-engineer-at-freetrade-a39a590d7ffa)

I joined Freetrade as Senior Software Engineer in October 2018. We’ve got a huge amount done in that time — the company’s moving fast!

But behind the bustle of getting big features shipped, I wanted to give people an insight into what an individual day’s work looks like on the engineering team.

Here’s my typical day as a Software Engineer at Freetrade.

(My) commute

I commute by bike 90% of the time. The Freetrade office in Shoreditch is about 10km away, so the showers at the office are essential. There’s also a space to lock bikes safely off the street (no-one’s getting my bike)!

A bad mix of bash and sh with [[

Thu, 23 May 2019 00:00:00 +0000

If you accidentally run a bash script with sh, you’re probably going to have a bad time.

One little gotcha is the use of [[ (double square brackets) in bash.

E.g.:

if [[ -z "${PROJECT_ID}" ]]; then
 echo 'PROJECT_ID is empty, exiting'
 exit 1
fi

Bash will check the condition as expected. Sh, on the other hand, will reject the double-brackets syntax that it is not aware of with a message like this:

[[: not found

That wouldn’t be so bad, but by default it will then continue running the script.

Generate a UUID on the command line with Python

Sat, 18 May 2019 00:00:00 +0000

Sometimes I want to manually generate a UUID for some test or example data.

It’s handy to have this Python snippet in Bash history to quickly do that:

python -c 'import uuid; print(uuid.uuid4())'

Python National Insurance Number (NINo) one-line generator

Tue, 23 Apr 2019 00:00:00 +0000

Here’s a Python one-liner for generating a random UK National Insurance Number (aka NINo) on the command line, which is useful for testing apps that use one:

python -c 'import random; import string; p="ABCEGHJKLMNPRSTWXYZ"; print("".join([random.choice(p),random.choice(p)]+[random.choice(string.digits) for i in range(6)]+[random.choice("ABCD")]))'

To make it easier to use:

alias nino="python -c 'import random; import string; p=\"ABCEGHJKLMNPRSTWXYZ\"; print(\"\".join([random.choice(p),random.choice(p)]+[random.choice(string.digits) for i in range(6)]+[random.choice(\"ABCD\")]))'"

Python random string one-liner for CLI

Tue, 23 Apr 2019 00:00:00 +0000

Here’s a handy Python one-liner for generating a random string of letters and digits on the command line:

python -c 'import random; import string; print("".join(random.choice(string.ascii_letters + string.digits) for i in range(30)))'

To make it easier to use:

alias randomstring="python -c 'import random; import string; print(\"\".join(random.choice(string.ascii_letters + string.digits) for i in range(30)))'"

Extract JSON config variables from the Airflow UI

Mon, 18 Mar 2019 00:00:00 +0000

One way to configure Apache Airflow is with “Variables” that can be modified programmatically or via the Airflow UI.

You can import the variables from a JSON file, again via the API or by uploading the file in the UI. Sensibly, Airflow does not provide a way to export these variables back out to a JSON file.

However, if you find yourself in the situation where Airflow holds the correct values and you’re not confident you’ve got them stored consistently elsewhere, you might end up having to manually copy the variables back out of the Airflow UI.

(Cross-post) Serverless integration testing at Freetrade

Mon, 11 Mar 2019 00:00:00 +0000

(Cross-posted from https://blog.freetrade.io/serverless-integration-testing-at-freetrade-5359fb0b0e57)

Our tech stack at Freetrade is 99% serverless. This means that for the vast majority of our applications, the physical infrastructure, deployment and other operational concerns are largely abstracted away by the cloud provider (Google Cloud in our case).

We focus on building what our users need, and our cloud provider focuses on what we need to keep it running smoothly at scale.

Serverless computing has huge benefits in terms of allowing more resources to be devoted to solving user problems, but it also presents some different challenges. One of those is how you go about applying automated testing to your software systems when you’re in a serverless environment.

qrencode is a nice QR code CLI tool for Ubuntu Linux

Sat, 23 Feb 2019 00:00:00 +0000

There’s something fun about QR codes, and I wanted an easy way generate them as SVGs.

Sure enough there’s a tool in the Ubuntu repos to do that:

sudo apt install qrencode

It’s straightforward to use:

qrencode -t SVG -o qr-code.svg 'whatever you want encoded'

I wanted to generate a QR code to put on little business cards that we pack with the orders for our pop up card shop. The QR code should point to a URL including a discount coupon that gets automatically applied for customers who have arrived via that URL.

Promise batch progress meter in TypeScript

Mon, 18 Feb 2019 00:00:00 +0000

Nodejs has a handy Promise.all function that awaits an entire batch of promises. However, sometimes you want to see how that batch of promises is progressing before the whole thing has finished.

Here’s a rough-and-ready way to print a progress meter to stdout as a batch of promises completes in TypeScript:

const promises = [1, 2, 3, 4, 5].map(async (i: number): Promise<string> => {
 return 'finished some work'
})

let finishedCount = 0
promises.forEach((promise: Promise<string>): void => {
 promise.then((): void => {
 finishedCount++
 process.stdout.clearLine()
 process.stdout.cursorTo(0)
 process.stdout.write(`${finishedCount} / ${promises.length}`)
 }).catch(() => {
 	// Handle errors.
 })
})

await Promise.all(promises)

That just prints and updates a line like 3 / 5 to stdout. You could make it fancier with a percentage or a visual progress meter.

Duolingo Fandom Japanese vocab CSV extractor in JS

Sat, 16 Feb 2019 00:00:00 +0000

I’m casually going through the Japanese course in Duolingo, and found this handy wiki series on Duolingo Fandom Japanese with lists of all of the vocab.

I wanted an easier way to import it into Anki, so I put together this little pastable JS script to export a CSV of the vocab on one page:

(function () {
 const pattern =
 /^([\p{sc=Han}\p{sc=Hiragana}\p{sc=Katakana}？、，。]+)\s*\(([a-z\p{P}-]+)\)\s*=\s*([a-z\p{P}\(\) ?]+)(\s*\(([\p{sc=Han}\p{sc=Hiragana}\p{sc=Katakana}？、，。]+)\))?$/iu;
 const csvRows = ['日本語,ふりがな,ローマ字,英語'];
 document
 .querySelectorAll('h3 ~ ul:not([class]) > li:not([class])')
 .forEach((li) => {
 const match = String(li.innerText).match(pattern);
 console.log({ innerText: String(li.innerText), match });
 if (match) {
 const csvRow = '"'+[
 (match[1] || '').replace(',', ' /').trim() || '',
 (match[5] || '').replace(',', ' /').trim() || '',
 (match[2] || '').replace(',', ' /').trim() || '',
 (match[3] || '').replace(',', ' /').trim() || '',
 ].join('","')+'"';
 console.log(csvRow);
 csvRows.push(csvRow);
 }
 });
 const csv = csvRows.join('\n');
 const download = document.createElement('a');
 download.setAttribute(
 'href',
 'data:text/csv;charset=utf-8,' + encodeURIComponent(csv)
 );
 download.setAttribute('download', String(window.location).split('/').pop());
 document.body.appendChild(download);
 download.click();
 document.body.removeChild(download);
})();

You can paste this into your browser devtools console on one of those pages, and it will trigger a download of the vocab there as a CSV.

Creating a Series taxonomy in Hugo

Wed, 30 Jan 2019 00:00:00 +0000

Hugo’s taxonomies feature is very flexible and covers a lot of use cases for organising static content.

One thing I use it for on several static sites is to create a Series taxonomy that allows an easy grouping of posts into a sequence. Readers can see other posts in the series on each post, like a table of contents, they can see next and previous links within the series, and they can view a dedicated list page for the series as well.

Please comment your code

Wed, 23 Jan 2019 00:00:00 +0000

“Every comment represents a failure to make the code self explanatory.”

&em; Robert C. Martin (Uncle Bob)

One of the better-known sentiments from Uncle Bob’s Clean Code is that comments are bad*. More precisely, the idea is that comments are a last resort option to try and mitigate “bad code”.

Sometimes this seems to lead to a logical fallacy that goes like this: “Comments are a sign of bad code, and my code can’t be bad, so I won’t write comments”. This can progress to an even worse line of thinking: “Comments are for bad code, therefore if I don’t write comments, then my code can’t be bad”.

A Concise Guide to SSL/TLS for DevOps

Tue, 01 Jan 2019 00:00:00 +0000

Antifragile: Things that Gain from Disorder

Tue, 01 Jan 2019 00:00:00 +0000

Children of God

Tue, 01 Jan 2019 00:00:00 +0000

Don Quixote

Tue, 01 Jan 2019 00:00:00 +0000

Exploring CQRS and Event Sourcing

Tue, 01 Jan 2019 00:00:00 +0000

History of the Decline and Fall of the Roman Empire

Tue, 01 Jan 2019 00:00:00 +0000

Hyperion (Hyperion Cantos Book 1)

Tue, 01 Jan 2019 00:00:00 +0000

Jane Eyre

Tue, 01 Jan 2019 00:00:00 +0000

Origin Story: A Big History of Everything

Tue, 01 Jan 2019 00:00:00 +0000

Systemantics: The Systems Bible

Tue, 01 Jan 2019 00:00:00 +0000

The Confidence Game

Tue, 01 Jan 2019 00:00:00 +0000

The Most Human Human

Tue, 01 Jan 2019 00:00:00 +0000

The New Silk Roads: The Present and Future of the World

Tue, 01 Jan 2019 00:00:00 +0000

The Silk Roads: A New History of the World

Tue, 01 Jan 2019 00:00:00 +0000

The Three Body Problem

Tue, 01 Jan 2019 00:00:00 +0000

Why We Sleep

Tue, 01 Jan 2019 00:00:00 +0000

Finding the max branch sum of a sequentially represented binary tree in Python

Sun, 23 Dec 2018 00:00:00 +0000

Given a binary tree represented sequentially as a list of integer node values, we want to identify whether the sum of the left branch or the sum of th right branch is larger. In other words we want to identify which branch has the maximum sum.

For example, this binary tree:

····································
················┌──┐················
00··············│05│················
················└──┘················
··········╱··············╲··········
········┌──┐············┌──┐········
01······│04│············│06│········
········└──┘············└──┘········
······╱······╲······················
····┌──┐····┌──┐····················
02··│16│····│12│····················
····└──┘····└──┘····················
····································

Could be represented sequentially as [5, 4, 6, 16, 12]. Missing nodes are represented as -1.

(Cross-post) Why we use the Firebase Realtime Database at Freetrade

Fri, 21 Dec 2018 00:00:00 +0000

(Cross-posted from https://blog.freetrade.io/coming-to-firebases-nosql-realtime-database-from-a-sql-background-b605b2910c4c)

We use the Firebase Realtime Database at Freetrade, as part of our infrastructure on Google Cloud.

The Firebase Realtime Database is a cloud-hosted NoSQL database that lets you store and sync data between your users in realtime.

Firebase Realtime Database has a markedly different development philosophy to the more traditional SQL databases that many developers are used to. Here’s a rundown of what it’s like coming to Firebase Realtime Database from an SQL background. How the Firebase Realtime Database is different

Spoonerism generator Python script

Thu, 08 Nov 2018 00:00:00 +0000

“What’s the Difference Between” is a childish word-play game where you give clues for other players to guess a spoonerism phrase pair. For example:

Q: “What’s the difference between stress on a group, and an old locomotive?”

A: One is team strain, the other’s a steam train.

I’m part of a WhatsApp group that sends these back and forth on a daily basis (usually with less innocent contents than that example).

Keeping the ElasticSearch service running on an Ubuntu server

Sun, 28 Oct 2018 00:00:00 +0000

A local ElasticSearch instance powers the Chinese example sentence search feature on Chinese Boost. This works fine, but the ElasticSearch instance has a habit of crashing about once a day.

While I’ve been investigating why it crashes and how to prevent that from happening, I also wanted to have the EC2 Ubuntu server it runs on restart the ElasticSearch service automatically as a workaround.

This can be done using systemd.

You can add extra config for the elasticsearch service with this command:

Be careful with the S3 origin URL for a Cloudfront distribution

Tue, 23 Oct 2018 00:00:00 +0000

Be careful with the S3 origin URL for a Cloudfront distribution.

When using an S3 bucket with static website hosting enabled as the origin for a Cloudfront distribution, you must use the static website endpoint, and not the S3 bucket URL. These are different.

The S3 bucket URL looks like this (don’t use this for the Cloudfront origin):

https://s3.eu-west-2.amazonaws.com/foobar-bucket-name/

The S3 static website hosting URL looks like this (use this for the Cloudfront origin):

Making a helpful catalogue data reporting command in Laravel

Tue, 23 Oct 2018 00:00:00 +0000

The catalogue data model in our 3D pop up cards shop includes Product and Tag entities. Tags are a flat structure with a many-to-many relationship to Products, to allow some simple tagging to help customers find cards with particular elements.

The Tag-Product relationship is managed manually in the backoffice UI, which is fine but can be error prone when we miss Products that should have a particular Tag.

To make this easier to work with, I put together a quick command line tool in Laravel that identifies Products that seem to be missing an appriate Tag.

Freetrade

Mon, 08 Oct 2018 00:00:00 +0000

I was Principal Software Engineer at Freetrade, primarily working on the platform but also some client-side work. Freetrade has a serverless architecture running Google Cloud Functions in Firebase using TypeScript.

A quick Python script to help me translate classical Chinese poetry

Sat, 06 Oct 2018 00:00:00 +0000

One of my pastimes is translating Chinese poetry and other text into English on my East Asia Student blog. The translations I make are more like annotations, as I try to adhere as closely as I can to the structure and literal meaning of the original text. The idea is that the translation is helpful for reading the original Chinese text, rather than being a full rendering into English.

One aspect of this is the inclusion of pinyin and literal glosses for each line of the poem. For example:

Island counting algorithm in Typescript

Sun, 23 Sep 2018 00:00:00 +0000

A friend mentioned a code screen interview question he likes to ask based on counting the number of discrete “islands” in a grid:

Given a square grid of cells, some of which are sea (0) and some of which are land (1), implement an island counter that identifies each contiguous area of land cells (an island) and counts the total number of such islands.

This seems like quite a fun problem to solve so I had a go at it in TypeScript.

AlgoDaily 22: Count the Planes

Wed, 05 Sep 2018 00:00:00 +0000

https://algodaily.com/challenges/count-the-planes

This is the same as the island counting problem.

function* enumerateGrid(grid) {
	if (!grid || grid.length < 1 || !grid[0].length) {
		return;
	}
	for (let x = 0; x < grid[0].length; x++) {
		for (let y = 0; y < grid[x].length; y++) {
			yield {x, y};
		}
	}
}

function* enumerateNeighbours(grid, x, y) {
	for (const p of [{x:-1,y:0},{x:0,y:-1},{x:1,y:0},{x:0,y:1}]) {
		if (grid[x+p.x] && grid[x+p.x][y+p.y]) {
			yield {x: x+p.x, y: y+p.y};
		}
	}
}

function visit(grid, x, y) {
	grid[x][y] = "V";
	for (const n of enumerateNeighbours(grid, x, y)) {
		if (grid[n.x] && grid[n.x][n.y] === "P") {
			visit(grid, n.x, n.y);
		}
	}
}

function numOfPlanes(grid) {
	let planesCount = 0;
	for (const p of enumerateGrid(grid)) {
		if (grid[p.x][p.y] === "P") {
			planesCount++;
			visit(grid, p.x, p.y);
		}
	}
	return planesCount;
}

You iterate over the grid cells. Each time you hit a “plane” cell, increment the counter, then recursively mark it and all its neighbouring “plane” cells as visited.

Unix Fizz-Buzz one-liner

Mon, 27 Aug 2018 00:00:00 +0000

Given that “implement Fizz-Buzz in a language of your choice” is quite a common programming interview task, I was wondering if it would be possible to do it in one line on the command line using Unix tools.

Unsurprisingly it is; here’s my version:

seq 100 | awk '{ if ($0 % 15 == 0) { print "fizzbuzz" } else if ($0 % 3 == 0) { print "fizz" } else if ($0 % 5 == 0) { print "buzz" } else { print $0 } }'

The seq at the beginning generates numbers from 1 to 100, and then the awk command does a basic if-else on each line to decide what to print.

AlgoDaily 21: Contiguous Subarray Sum

Thu, 23 Aug 2018 00:00:00 +0000

https://algodaily.com/challenges/contiguous-subarray-sum

There’s a fairly obvious brute-force O(n^2) way to do this by looping through every possible sub-array:

function sum(numbers) {
	return numbers.reduce((sum, x) => sum + x, 0);
}

function* allSubArrays(numbers) {
	for (let i = 0; i <= numbers.length; i++) {
		for (let j = 0; j <= numbers.length; j++) {
			const slice = numbers.slice(i, j);
			if (slice.length > 0) {
				yield slice;
			}
		}
	}
}

function subarraySum(numbers, targetSum) {
	for (let sub of allSubArrays(numbers)) {
		if (sum(sub) === targetSum) {
			return true;
		}
	}
	return false;
}

This is O(n^2), so not great, but I can’t think of a different way to do it. We could make some small optimisations by skipping a subarray as soon as possible, but the worst case would still be O(n^2).

AlgoDaily 20: Uniqueness of Arrays

Wed, 08 Aug 2018 00:00:00 +0000

https://algodaily.com/challenges/uniqueness-of-arrays

This is pretty trivial with the Set class in ECMAScript 2015.

function uniques(items) {
	return [...new Set(items)];
}

If that seems like cheating, then we can achieve this with a hash map and a new array:

function uniques(items) {
	const seen = {};
	const unique = [];
	for (let x of items) {
		if (seen[x] === undefined) {
			unique.push(x);
			seen[x] = true;
		}
	}
	return unique;
}

This is O(n).

AlgoDaily 19: Fast Minimum In Rotated Array

Sun, 05 Aug 2018 00:00:00 +0000

https://algodaily.com/challenges/fast-minimum-in-rotated-array

It seems we have to identify an element that is lower than the preceding one, as this must be the original beginning of the array and thus the lowest. If we get to the end of the array without finding a drop, then it must be the first element of the array, as the pivot was at the end.

This still has a worst case running time of O(n), though, as we might have to iterate through the whole array.

AlgoDaily 18: Sum All Primes

Mon, 23 Jul 2018 00:00:00 +0000

https://algodaily.com/challenges/sum-all-primes

I remember there’s an algorithm called the Sieve of Eratosthenes for finding prime numbers up to a limit, which seems like it would be useful here.

To apply the Sieve of Eratosthenes, you first allocate an object with each integer up to n as the keys.

You then iterate on integers up to $\sqrt{n}$. In other words, keep iterating on integers p until p * p would be more than n. E.g. if n = 25, then the iteration should stop at 5, because 5 * 5 = 25.

Fixing cut off stroke edges of shapes in Inkscape

Mon, 23 Jul 2018 00:00:00 +0000

I got stuck with this problem in Inkscape where the stroke edges of shapes seem to get cut off by the bounding box:

After a bit of searching, it turned out that this was caused by having a small blur set on the layer containing the shapes:

Setting that layer blur back to zero restores the full stroke edges of the shapes.

AlgoDaily 17: Find Missing Number in Array

Sun, 08 Jul 2018 00:00:00 +0000

https://algodaily.com/challenges/find-missing-number-in-array

Seems like we should iterate on the array, adding an integer to the result array if it’s not present in the input. We have to backtrack the iteration by one each time we hit a missing number. This is O(n).

function missingNumbers(sequence) {
	const missing = [];
	let expected;
	for (let i = 0; i < sequence.length; i++) {
		if (i === 0) {
			expected = sequence[i] + 1;
			continue;
		}
		if (sequence[i] !== expected) {
			missing.push(expected);
			i--;
		}
		expected += 1;
	}
	return missing;
}

The solution suggested by Algo Daily is similar, but uses an inner loop to insert all of the missing numbers in a gap at once, rather than doing them one iteration at a time as in the solution above. This avoids the need to backtrack.

AlgoDaily 16: Missing Number In Unsorted

Thu, 05 Jul 2018 00:00:00 +0000

https://algodaily.com/challenges/missing-number-in-unsorted

The requirement to do this without sorting makes me think of using a hash map to track each number we do see, and then finding the missing integer there by iterating across the expected range. This isn’t O(n), though; it’s O(m+n).

function missingInUnsorted(numbers, lowerBound, upperBound) {
	const seen = numbers.reduce((carry, num) => {
		carry[num] = true;
		return carry;
	}, {});
	for (let i = lowerBound; i < upperBound; i++) {
		if (seen[i] === undefined) {
			return i;
		}
	}
	return undefined;
}

Algo Daily has a solution using the Gauss sum formula n(n+1)/2 to determine the sum of a range of numbers, and then subtracting the sum of the given numbers from that to find the single missing integer.

AlgoDaily 15: Max of Min Pairs

Sat, 23 Jun 2018 00:00:00 +0000

https://algodaily.com/challenges/max-of-min-pairs

This requires a bit of thought. We want to get the highest possible smaller number in each pair.

The first thought is that we need to sort the numbers before we do anything else, so that we can allocate pairs with the largest possible minimum.

To get the largest possible minimum in the pair, you have to “waste” a larger number to accompany it. This should be the smallest possible number that is larger than the minimum in the pair.

Simple estimated delivery dates in vanilla JS

Wed, 20 Jun 2018 00:00:00 +0000

Displaying estimated delivery dates is a handy little feature in e-commerce that gives useful information to customers and reassures them about the ordering process.

Estimating delivery dates can be quite involved, but it’s easy to chuck a very simple estimated delivery range on any page with these vanilla JavaScript functions, no libraries required:

function showEstimatedDelivery() {
 const estDelEls = document.querySelectorAll(".estimated-delivery");
 if (!estDelEls || !estDelEls.length) {
 return;
 }
 const [minDate, maxDate] = estimatedDeliveryRange();
 const [minFormat, maxFormat] = [formatDate(minDate), formatDate(maxDate)];
 let estimatedDelivery
 if (minFormat === maxFormat) {
 estimatedDelivery = minFormat;
 } else {
 estimatedDelivery = `${minFormat} &mdash; ${maxFormat}`;
 }
 document.querySelectorAll(".estimated-delivery").forEach((el) => {
 el.innerHTML = estimatedDelivery;
 })
}

const SATURDAY = 6
const SUNDAY = 0

function estimatedDeliveryRange(from = new Date(), minWorkDays = 8, maxWorkDays = 10) {
 const time = new Date(from.getTime())
 time.setHours(12, 0, 0, 0)
 let minDate
 let maxDate
 while (minWorkDays > 0 || maxWorkDays > 0) {
 time.setHours(time.getHours() + 24)
 if (time.getDay() !== SATURDAY && time.getDay() !== SUNDAY) {
 minWorkDays--
 maxWorkDays--
 if (minWorkDays <= 0 && !minDate) {
 minDate = new Date(time.getTime())
 }
 if (maxWorkDays <= 0 && !maxDate) {
 maxDate = new Date(time.getTime())
 }
 }
 }
 return [minDate, maxDate]
}

function formatDate(date = new Date()) {
 return new Intl.DateTimeFormat("en-GB", {
 weekday: 'short',
 day: 'numeric',
 month: 'short',
 }).format(date);
}

We use this on our static e-commerce site Moment Wall Art, for example on this Japanese wall art print product page.

Python asyncio concurrent map pool function

Mon, 18 Jun 2018 13:18:51 +0100

Here is a relatively short Python function to map input items concurrently, with a fixed pool size to limit the maximum concurrency. The work function to map each item is a normal synchronous function, which makes this easy to use in projects that are not otherwise using asyncio.

This function should be OK to run with or without a pre-existing event loop in the process. It uses an existing event loop if there is one, otherwise it creates a new event loop and uses that. Either way, the calling code does not need to know that this function is using asyncio internally.

Clipboard hygiene

Tue, 12 Jun 2018 00:00:00 +0000

After using the excellent keepassx password manager for a while, I’ve noticed it has a nice little security feature: once you copy a password to the clipboard, keepassx waits for a short period of time and then clears your clipboard. It also clears the clipboard when you close keepassx.

This is a good idea as it somewhat reduces the chance of some other process being able to steal a password from the clipboard (which any user-level process could do). More usefully, it also reduces the chance of you accidentally pasting the password somewhere you didn’t want to.

AlgoDaily 14: Find First Non-Repeating Character

Fri, 08 Jun 2018 00:00:00 +0000

https://algodaily.com/challenges/find-first-non-repeating-character

This shouldn’t be too hard with a hash map of characters to counts, relying on JavaScript objects having ordered keys to then find the first character with a single occurrence.

function firstNonRepeat(str) {
	const counts = {}
	for (let char of str) {
		counts[char] = counts[char] ? counts[char] + 1 : 1;
	}
	for (let char in counts) {
		if (counts[char] === 1) {
			return char;
		}
	}
	return "";
}

AlgoDaily 13: Dollar Sign Deletion

Tue, 05 Jun 2018 00:00:00 +0000

“You’re given an array of strings containing alphabetical characters and certain $ characters. A $ represents a DELETE action whereby the character before it is deleted.

Each of these strings will be run through a method to operate on the $ DELETE action. We want to find out if the final string is the same for all of them. Let’s take an example:”

const input = ["f$st", "st"]
isDollarDeleteEqual(input);
// true
// true because both become "st"

“Given the below function signature, can you find a solution in O(n) time and constant space?”

Extracting product image main colours in Python

Sat, 19 May 2018 00:00:00 +0000

The main colours of a product are often quite important data in e-commerce systems, and it’s nice to have an automated way to extract these.

Here’s a rough summary of what I ended doing in Python to get this data from product images.

It uses three different colour extraction libraries and merges the results from them, which is somewhat questionable but seems to provide better overall results than a single library alone.

Year name to century name and decade name in Python

Sat, 19 May 2018 00:00:00 +0000

Here are some little helper functions to convert an integer year into a century name and decade name, for example 1989 would be "20th Century" and "1980s".

from math import floor

def year_to_century_name(year: int) -> str:
 century = (floor(int(year) / 100)) + 1
 suffix = ordinal_suffix(century)
 return f"{century}{suffix} Century"


def ordinal_suffix(num: int) -> str:
 return {1: "st", 2: "nd", 3: "rd"}.get(
 0 if 10 > num > 14 else num % 10, "th"
 )


def year_to_decade_name(year: int) -> str:
 decade = floor(int(year) / 10) * 10
 return f"{decade}s"

We use this in Moment Wall Art to extract the century and decade of each piece, which allows for easy taxonomy pages like 20th Century Wall Art or 2020s Wall Art.

Allow a Django command to use a file or stdin / stdout

Sun, 13 May 2018 00:00:00 +0000

Here’s a little approach I’ve started using when writing Python scripts and Django console commands. It allows them to easily handle i/o from the commandline or from a file without much effort.

Rather than using sys.stdin or sys.stdout directly, you define an argparse.FileType argument for the command, and set the default to sys.stdin.

For example, in a Django command:

from sys import stdin, stdout
from argparse import FileType
from django.core.management.base import BaseCommand

class Command(BaseCommand):

 def add_arguments(self, parser: ArgumentParser):
 parser.add_argument('input', nargs='?', type=FileType('r'),
 default=stdin)
 parser.add_argument('output', nargs='?', type=FileType('w'),
 default=stdout)

 def handle(self, *args, **options):
 input = options['input']
 output = options['output']
 # ...

Now the command is flexible as you can pipe data to or from it:

Streaming a CSV file from the database in Laravel

Sun, 13 May 2018 00:00:00 +0000

Exporting data to CSV is a requirement in almost any business software project. We needed it to get a convenient sales data export in our card shop sideproject recently, and it turned out to be straightforward but slightly weird to do in Laravel.

Sales data export query

The sales data export query is about 30 lines of SQL. This was easier to write directly in SQL than wrapping it up in Laravel’s ORM, and it’s also handy to be able to run this query separately anyway.

AlgoDaily 12: Detect Substring in String

Tue, 08 May 2018 00:00:00 +0000

https://algodaily.com/challenges/detect-substring-in-string

“How would you write a function to detect a substring in a string?

If the substring can be found in the string, return the index at which it starts. Otherwise, return -1.”

We can iterate through the string, and increment a counter when we hit that index in the target substring. If the counter gets to the length of the target substring, we can return the iterator index minus the counter.

AlgoDaily 11: Sum Digits Until One

Sun, 06 May 2018 00:00:00 +0000

https://algodaily.com/challenges/sum-digits-until-one

“We’re provided a positive integer num. Can you write a method to repeatedly add all of its digits until the result has only one digit?

Here’s an example: if the input was 49, we’d go through the following steps:”

// start with 49
4 + 9 = 13

// move onto 13
1 + 3 = 4

This seems to mostly be an issue of type juggling. If we’re careful with types, then all we have to do is keep breaking down the string of the number into single digit characters, sum those as numbers and set the original number to the string of that number.

AlgoDaily 10: Binary Tree In-order Traversal

Sat, 05 May 2018 00:00:00 +0000

“Can you write a function to traverse a binary tree in-order, and print out the value of each node as it passes?”

“The example would output [4, 6, 5] since there is no left child for 4, and 6 is visited in-order before 5.

The definition of a tree node is as follows:”

function Node(val) {
 this.val = val;
 this.left = null;
 this.right = null;
}

Another example tree:

Django model field choices with an inner class enum

Wed, 02 May 2018 00:00:00 +0000

It’s common to define the possible choices for a character field on Django models like this (from the docs):

from django.db import models

class Student(models.Model):
 FRESHMAN = 'FR'
 SOPHOMORE = 'SO'
 JUNIOR = 'JR'
 SENIOR = 'SR'
 YEAR_IN_SCHOOL_CHOICES = (
 (FRESHMAN, 'Freshman'),
 (SOPHOMORE, 'Sophomore'),
 (JUNIOR, 'Junior'),
 (SENIOR, 'Senior'),
 )
 year_in_school = models.CharField(
 max_length=2,
 choices=YEAR_IN_SCHOOL_CHOICES,
 default=FRESHMAN,
 )

This works fine and it makes constants available for the values, so that elsewhere you can refer to Student.FRESHMAN. It is quite verbose, though, and doesn’t group the constants together very well.

One way to group the constants more clearly is to use an inner class:

uMed

Mon, 16 Apr 2018 00:00:00 +0000

I was a back-end Software Engineer at uMed in 2018.

PHP Business Time library v1.0.0

Fri, 13 Apr 2018 00:00:00 +0000

I’ve just released v1.0.0 of my [PHP Business Time](https://github.com/hughgrigg /php-business-time) library. I made this to solve my own problem that I had building my 3D pop-up card shop – calculating estimated delivery dates based on business time constraints.

It would have been simpler to just solve the exact use case I had (probably hard-coding timings and combining that with retrieving holidays from a remote source), but I thought it would be fun to generalise it and make it into a separate library.

Fixing a Github Action error "botocore.awsrequest.AWSRequest" "exit code 255"

Mon, 09 Apr 2018 00:00:00 +0000

The other day I suddenly started hitting an error in my Github Actions that deploy static Hugo sites to S3:

<botocore.awsrequest.AWSRequest object at 0x7fbfb444e160>
Error: Process completed with exit code 255.

For some reason setting an env variable called AWS_EC2_METADATA_DISABLED to true fixes this error.

E.g. your Github Action YAML might look like this:

name: Build and Deploy

on: push

jobs:
 build:
 name: Build and Deploy
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v1
 - name: Install Hugo
 run: |
 HUGO_DOWNLOAD=hugo_extended_${HUGO_VERSION}_Linux-64bit.tar.gz
 wget https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/${HUGO_DOWNLOAD}
 tar xvzf ${HUGO_DOWNLOAD} hugo
 mv hugo $HOME/hugo
 env:
 HUGO_VERSION: 0.71.0
 - name: Hugo Build
 run: $HOME/hugo -v
 - name: Deploy to S3
 if: github.ref == 'refs/heads/master'
 run: aws s3 sync public/ s3://{s3-bucket-name}/ --delete --acl=public-read && aws cloudfront create-invalidation --distribution-id={cloudfront-distribution-id} --paths='/*'
 env:
 AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
 AWS_EC2_METADATA_DISABLED: true

The key bit for avoiding this error is:

AlgoDaily 09: Implement a Hash Map

Fri, 06 Apr 2018 00:00:00 +0000

“Can you implement the JS Map class from scratch? Only two methods are necessary – get and set. Do not use JS Objects ({} notation) for this exercise.

For the two methods you’ll define:

get(key: string) should be given a key, and return the value for that key.

set(key: string, val: string) should take a key and a value as parameters, and store the pair.

Additionally, we’ve supplied the below hashing function hashStr. It tries to avoid collision, but is not perfect. It takes in a string value and returns an integer.”

AlgoDaily 08: Lonely Number

Thu, 05 Apr 2018 00:00:00 +0000

https://algodaily.com/challenges/lonely-number

“In a given array of numbers, one element will show up once and the rest will show up twice. Can you find that number in O(n) linear time?”

lonelyNumber([4, 4, 6, 1, 3, 1, 3])
// 6

lonelyNumber([3, 3, 9])
// 9

This seems straightforward.

function lonelyNumber(items) {
	return Object.keys(items.reduce((seen, x) => {
		if (x in seen) {
			delete seen[x];
			return seen;
		}
		seen[x] = x;
		return seen;
	}, {}))[0];
}

We start with an empty hash, then go through the array, removing an item we’ve seen before, and adding an item we haven’t.

AlgoDaily 07: Power of Three

Tue, 03 Apr 2018 00:00:00 +0000

“Given an integer num, write a method to determine if it is a power of 3.”

console.log(powerOfThree(9));
// true

To calculate a power of x, we could keep multiplying by x until we hit the exponent.

That means to see if something is a power of x, we can do the reverse and keep diving by x until we hit 1 or a number that is not a multiple of 3.

URL encoding data Etsy-style in Python

Mon, 26 Mar 2018 00:00:00 +0000

The Etsy API has quite specific requirements for how data should be encoded in requests.

With a bit of trial and error making test requests, I ended up with these Python functions to encode data in the way the Etsy API requires:

from typing import Union, Dict, List
from urllib import parse

def etsy_urlencoded(data: Dict):
 return parse.urlencode({k: etsy_urlencoded_value(v) for k, v in data.items()})


def etsy_urlencoded_value(value: Union[Dict, List, str, bool, int, float]):
 if type(value) is dict:
 return json.dumps({k: etsy_urlencoded_value(v) for k, v in value.items()})
 if type(value) is list or type(value) is tuple:
 return ",".join([etsy_urlencoded_value(i) for i in value])
 if type(value) is bool:
 return int(value)
 return str(value)

This lets us automate some of the management of our wall art prints in our Etsy shop.

Avoid deriving state from props in React

Fri, 23 Mar 2018 00:00:00 +0000

A common footgun in React projects is derived state, which means basing state on props. This can be directly copying props into state, or deriving state from props with some extra logic.

This often causes trouble because once the state is set based on props, it won’t be updated by subsequent updates to props. This is after all the purpose of props and state in React: props are clean and stateless, and state is of course stateful.

AlgoDaily 06: Majority Element

Sat, 03 Mar 2018 00:00:00 +0000

https://algodaily.com/challenges/majority-element

“Suppose we’re given an array of numbers like the following:

[4, 2, 4]

Could you find the majority element? A majority is defined as “the greater part, or more than half, of the total. It is a subset of a set consisting of more than half of the set’s elements.

Let’s assume that the array length is always at least one, and that there’s always a majority element.

In the example above, the majority element would be 4.”

AlgoDaily 05: Validate Palindrome

Fri, 02 Mar 2018 00:00:00 +0000

“Given a string str, can you write a method that will return True if is a palindrome and False if it is not? If you’ll recall, a palindrome is defined as “a word, phrase, or sequence that reads the same backward as forward”. For now, assume that we will not have input strings that contain special characters or spaces, so the following examples hold:”

let str = 'thisisnotapalindrome';
isPalindrome(str);
// false

str = 'racecar';
isPalindrome(str);
// true

“For an extra challenge, try to ignore non-alphanumerical characters. The final solution that we present will handle all edge cases.”

AlgoDaily 04: Is An Anagram

Thu, 01 Mar 2018 00:00:00 +0000

https://algodaily.com/challenges/is-an-anagram

“Here’s the definition of an anagram: a word, phrase, or name formed by rearranging the letters of another, such as cinema, formed from iceman.

We are given two strings like “cinema” and “iceman” as inputs. Can you write a method isAnagram(str1, str2) that will return true or false depending on whether the strings are anagrams of each other?”

The gotcha with this is being careful with multiple instances of the same letter, i.e. you can’t just check that each character is also found in the other string.

AlgoDaily 03: Reverse Only Alphabetical

Tue, 20 Feb 2018 00:00:00 +0000

(https://algodaily.com/challenges/reverse-only-alphabetical)[https://algodaily.com/challenges/reverse-only-alphabetical]

“You are given a string that contains alphabetical characters (a - z, A - Z) and some other characters ($, !, etc.). For example, one input may be:

‘sea!$hells3’

Can you reverse only the alphabetical ones?”

First thought is to use a stack for this: go through the string forwards, pushing only letters on to the stack. Then go through the positions again, popping the stack for a letter position but leaving non-letters alone:

AlgoDaily 02: Fizz Buzz

Mon, 19 Feb 2018 00:00:00 +0000

“We’re given a number n.

Write a function that returns the string representation of all numbers from 1 to n based on the following rules:

If it’s a multiple of 3, represent it as “fizz”.

If it’s a multiple of 5, represent it as “buzz”.

If it’s a multiple of both 3 and 5, represent it as “fizzbuzz”.

If it’s neither, just return the number itself.

As such, fizzBuzz(15) would result in ‘12fizz4buzzfizz78fizzbuzz11fizz1314fizzbuzz’.”

AlgoDaily 01: Array Intersection

Sun, 18 Feb 2018 00:00:00 +0000

I’ve signed up to AlgoDaily as it seems like a fun way to get out some small blog posts on small coding tasks.

This is AlgoDaily Day 1: Array Intersection.

“Can you write a function that takes two inputs and returns their intersection? All element in the final result should be unique.”

const nums1 = [1, 2, 2, 1];
const nums2 = [2, 2];

intersection(nums1, nums2);
// [2]

“Here’s another one:”

const nums1 = [4, 9, 5];
const nums2 = [9, 4, 9, 8, 4];

intersection(nums1, nums2);
// [9, 4]

Here’s my first-thought naive implementation:

Varmilo keyboard F-keys and Super Win key mode on Ubuntu

Thu, 15 Feb 2018 00:00:00 +0000

I got a Varmilo VA109M keyboard recently (it’s great!), which I’m using on Ubuntu 20.10.

This is how I got the F-keys working correctly:

echo options hid_apple fnmode=2 | sudo tee -a /etc/modprobe.d/hid_apple.conf

sudo update-initramfs -u -k all

Then restart.

And to get the Super key (labelled Win on Varmilo PC keyboards) working correctly, make sure your Varmilo keyboard is in PC mode:

Hold down Fn + Esc for 4 seconds.

The CapsLock key light will flash when the mode has been changed successfully.

Cartesian products in Bash

Tue, 23 Jan 2018 00:00:00 +0000

Here’s a handy thing you can do in Bash: calculate the Cartesian product of two sets.

E.g.

echo {A,B,C}{A,B,C}

gives

AA AB AC BA BB BC CA CB CC

i.e. it’s every possible permutation of two elements from the two sets.

A --- A
 \ /
B -x- B
 / \
C --- C

(Thanks to Bill Hails for the diagram!)

This can be a useful tool for quickly figuring things out in various situations. For example, I wanted to write some tests for a feature that allowed users to set two configuration options to one of three values. Bash made it easy to list out the possible permutations I needed to cover in the tests.

Finding if two rectangles overlap in JavaScript

Thu, 18 Jan 2018 00:00:00 +0000

Given the top-left and bottom-right coordinates of two rectangles, determine if they overlap or not. That is, determine if any part of one rectangle overlaps with any part of the other.

The grid has position (0, 0) at the top left. X increases rightwards and Y increases downwards. This is common in 2D grids in video games.


 0,0 ─ x →

 │

 y

 ↓

You get coordinates for the rectangles as tuples of x and y, e.g. (2, 5).

Converting Roman numerals into decimal in JavaScript

Mon, 08 Jan 2018 00:00:00 +0000

Converting Roman numerals into decimal is a fun little problem to solve. Getting the solution is dependent on knowing the rules of Roman numerals, and they’re actually simpler than you might think:

All letters directly map to an integer value.
A letter followed by a letter of higher value is subtracted.
All other letters are added.

Before I looked this up, I thought that the rules were more complicated, e.g. with multiplication in certain situations (which is how Chinese numerals work).

Identifying a potential palindrome in JavaScript

Fri, 05 Jan 2018 00:00:00 +0000

Given an input string, determine whether the string could possibly be a palindrome if rearranged. It is not necessary to return the potential palindrome, just to identify whether one is possible.

Examples:

aabb => true
abc => false
aab => true
ciicv => true
aba => true
abac => false

A palindrome can have any amount of pairs of letters, and optionally one single odd letter with no partner in the middle.

Bloodchild

Mon, 01 Jan 2018 00:00:00 +0000

Dawn

Mon, 01 Jan 2018 00:00:00 +0000

Dune

Mon, 01 Jan 2018 00:00:00 +0000

How to Destroy a Tech Startup

Mon, 01 Jan 2018 00:00:00 +0000

Rationality: From AI to Zombies

Mon, 01 Jan 2018 00:00:00 +0000

The Big Short

Mon, 01 Jan 2018 00:00:00 +0000

The Daily Stoic

Mon, 01 Jan 2018 00:00:00 +0000

The Gervais Principle

Mon, 01 Jan 2018 00:00:00 +0000

The Heart to Start

Mon, 01 Jan 2018 00:00:00 +0000

The Inner Game of Tennis

Mon, 01 Jan 2018 00:00:00 +0000

The Spy Who Came In from the Cold

Mon, 01 Jan 2018 00:00:00 +0000

War is a Racket

Mon, 01 Jan 2018 00:00:00 +0000

Python Tricks

Sun, 17 Dec 2017 00:00:00 +0000

Product Page model abstraction in e-commerce

Fri, 08 Dec 2017 00:00:00 +0000

A lot of e-commerce abstractions get a bit messy around abstracting products, SKUs, product options, product variations and so on. The fact that there doesn’t seem to be a standard or obvious abstraction for this suggests it is tricky to get right.

It might be helpful in a lot of systems to extract a Product Page model as an abstraction. I think this avoids a lot of the trouble as it’s easy to think about and lets your code match your conversations.

Balancing address lines for e-commerce in PHP

Thu, 07 Dec 2017 00:00:00 +0000

Delivery addresses are notoriously difficult to deal with. One issue we’ve encountered with our greetings card shop is long lines in an address that would get cut off when printed on the address label, so we have to manually tweak them to get the address to fit.

To reduce some of that manual work, I added a little helper that tries to balance the address lines where possible, i.e. take words off long lines and add them to neighbouring lines that have some space to spare.

Fluent Python

Tue, 05 Dec 2017 00:00:00 +0000

Drawing sequentially represented binary trees in Python

Thu, 23 Nov 2017 00:00:00 +0000

I’ve been playing around with sequentially represented binary trees recently, i.e. a binary tree given as a list of integers where the node values are in contiguous sections of the list.

For example, a binary tree where the node values are also their indexes would be sequentially represented like this:

0 | 1, 2 | 3, 4, 5, 6 | 7, 8, 9, 10, 11, 12, 13, 14

Each section in the list represents one level of the binary tree, and has length $2^{l}$ where $l$ is the level. At the root, $l=0$ and $2^0=1$, so we know this section has a length of 1. The next section represents $l=1$, so has length $2^1=2$, and so on. At the fourth level, $l=3$, so there are $2^3=8$ nodes in that section.

Managing Humans

Thu, 05 Oct 2017 00:00:00 +0000

On the Trail of Genghis Kahn

Sun, 01 Oct 2017 00:00:00 +0000

The Dispossessed

Tue, 05 Sep 2017 00:00:00 +0000

Shevek, a brilliant physicist, decides to take action. he will seek answers, question the unquestionable, and attempt to tear down the walls of hatred that have isolated his planet of anarchists from the rest of the civilized universe. To do this dangerous task will mean giving up his family and possibly his life. Shevek must make the unprecedented journey to the utopian mother planet, Anarres, to challenge the complex structures of life and living, and ignite the fires of change.

Blackjack CLI game

Mon, 28 Aug 2017 00:00:00 +0000

The first draft of a blackjack cli game I’m making is now finished.

If you have Go installed then you should be able to play the game with:

go get -u github.com/hughgrigg/blackjack
blackjack

It’s a single-deck blackjack betting game written in Go for the command line. I’ve only tried it on Linux, but it uses the termui library so I imagine it would be OK on OSX as well. Not sure about Windows.

Enable PHP assertion errors on Laravel Homestead for development and testing

Wed, 23 Aug 2017 00:00:00 +0000

Laravel’s Homestead vagrant setup is handy to get going quickly on projects and keep them consistent to work on.

I was surprised that it has PHP assertion errors disabled by default. It’s good to have these enabled for local development and testing so that you can catch more problems earlier.

To enable PHP assertion errors in Homestead you can run this:

for phpConf in cli fpm; do
 sudo sed -i 's/zend.assertions = -1/zend.assertions = 1/g' /etc/php/7.*/${phpConf}/php.ini
 sudo sed -i 's/;assert.active = On/assert.active = On/g' /etc/php/7.*/${phpConf}/php.ini
 sudo sed -i 's/;assert.exception = On/assert.exception = On/g' /etc/php/7.*/${phpConf}/php.ini
done

That will enable PHP assertions and assertion errors for cli and web for all versions of PHP 7 in Homestead. I have it on a script so that it’s always run and I won’t forget about it.

Doughnut Economics

Sat, 05 Aug 2017 00:00:00 +0000

Dates for booleans

Sun, 23 Jul 2017 00:00:00 +0000

Here’s a little maxim for data modeling: use dates instead of booleans. In a lot of situations, this lets you capture more information without losing anything compared to a boolean.

A common example is implementing a status column with a datetime foobared_at column instead of a boolean is_foobar. If it’s null, then that row is not foobar. If the column is filled, then you not only know that it is foobar, but also since when.

Early exit boolean expression parser in Python

Tue, 18 Jul 2017 00:00:00 +0000

I wanted to have a go at rush-implementing a boolean expression parser with early exit, i.e. that it will not bother evaluating the right hand side of an expression like 0&(1|0) and just return false once it sees 0&.

Here’s an attempt in Python. It deals with nested expressions by recursion and “collapsing” the evaluation into a 0 or 1 token to continue evaluating from there. It uses a similar approach to handle !.

Superforecasting

Wed, 05 Jul 2017 00:00:00 +0000

Nordic APIs OAuth Workshop, Amsterdam June 2017

Mon, 12 Jun 2017 00:00:00 +0000

In June I attended [Nordic API’s workshop](http://nordicapis.com/events/nordic- apis-workshops-amsterdam/) on OAuth and OpenID in Amsterdam. Here are my notes on the class.

Actors in OAuth

AS: Authorisation Server (OAuth Server), e.g. bank auth server
RS: Resource Server, e.g. bank account server
RO: Resource Owner, e.g. bank customer
C: Client, e.g. mobile app

OAuth is designed for cross-organisational authorisation, i.e. where the client is not trusted (which should always be the case). The AS and RO do not want to let the client nor the resource server have access to the RO’s credentials.

The Corrections

Mon, 05 Jun 2017 00:00:00 +0000

Auto-printing shipping labels with Laravel, Raspberry Pi and AWS

Mon, 29 May 2017 00:00:00 +0000

The most manual part of the e-commerce side project I run with my wife is shipping the orders. We get order notifications by [email and Telegram](https://github.com/hughgrigg/poprobincards/bl ob/a08850cbee203223db309965b6e264e7e951e693/app/Modules/Sales/Notifications/Staf fOrderNotification.php#L52), and have to:

Find the stock.
Package it.
Print the shipping label, cut out and attach it.
Take the order to the post box.

A lot of that is beyond our means to automate, but we did get a Brother QL-570 label printer to speed up step 3 quite a lot. That prints and cuts adhesive labels that look more professional than what we were doing before, and costs less per label.

Algorithm Design Exercise 1-29: Finding the Fastest of 25 Horses

Thu, 25 May 2017 00:00:00 +0000

This is exercise 1-29 from The Algorithm Design Manual.

There are 25 horses. At most, 5 horses can race together at a time. You must determine the fastest, second fastest, and third fastest horses. Find the minimum number of races in which this can be done.

This is quite a famous puzzle question and involves some algorithmic thinking.

The first key to understanding for me was to focus on excluding horses who are slower than at least 3 other horses. This lets you run a process of elimination instead of a process of selection.

Survivorship Thinking

Tue, 23 May 2017 00:00:00 +0000

The survivorship bias is one of the most interesting cognitive biases to know about (cliché XKCD link).

It’s the idea that our impression of a given set of things is biased towards those that have survived some selection pressure. The classic example is when people say “they don’t make ’em like they used to”. With a moment’s thought, you realise that of course all the old things we see today are well made. The poorly-made things of the past don’t survive through to the present.

Rendering a comma-separated list of items in a Hugo template with a dot after the last

Sat, 13 May 2017 00:00:00 +0000

Here’s a Hugo template snippet for rendering a list of items separated by a comma and space after all but the last item, and then a dot or full stop.

For example, if you have a tags taxonomy and some tags in a post’s front-matter like this:

tags:
 - Foo
 - Bar
 - Baz
---

You can render a comma-separated list of those tags in a template like this:

{{ with (.GetTerms "tags") }}
<p>
 {{ range $i, $t := . }}{{ if $i }},&nbsp;{{ end }}<a href="{{ $t.Permalink }}">{{ $t.LinkTitle }}</a>{{ end }}.
</p>
{{ end }}

The $i variable is the numeric range index and the $t variable is the tag object per tag. This uses the false evaluation of the first index 0 to add a comma and space “, ” after all but the first item in the range.

"Permission denied (public key)" error from Laravel Forge after copying ssh key using xclip or pbcopy

Fri, 12 May 2017 00:00:00 +0000

I use Laravel Forge to run several Laravel projects. It has an interface to add SSH keys to a server so that you can SSH into them, but I’ve repeatedly hit a little issue when trying to use xclip on Linux or pbcopy on Mac to copy the public key and paste it into the Forge interface.

Somehow the public key is not copied properly and you’ll get the generic Permission denied (public key) error when trying to SSH into the server.

The Every Computer Performance Book

Mon, 01 May 2017 00:00:00 +0000

The Remains of the Day

Mon, 01 May 2017 00:00:00 +0000

Algorithm Design Exercise 1-28

Sun, 23 Apr 2017 00:00:00 +0000

This is exercise 1-28 from The Algorithm Design Manual.

Write a function to perform integer division without using either the / or * operators. Find a fast way to do it.

Let’s add multiplication as well. We could do a very inefficient multiplication algorithm by using one factor as a counter, and incrementing the other factor by itself until the counter reaches its value:

$a \times b$
$i = 0$
$p = 0$
while $i < a$
	$p = p + b$
	$i = i + 1$
return $p$

Naive integer multiplication in Python

Station Eleven

Sat, 01 Apr 2017 00:00:00 +0000

An audacious, darkly glittering novel about art, fame, and ambition set in the eerie days of civilization’s collapse, from the author of three highly acclaimed previous novels. One snowy night a famous Hollywood actor slumps over and dies onstage during a production of King Lear. Hours later, the world as we know it begins to dissolve. Moving back and forth in time-from the actor’s early days as a film star to fifteen years in the future, when a theater troupe known as the Traveling Symphony roams the wasteland of what remains-this suspenseful, elegiac, spellbinding novel charts the strange twists of fate that connect five people: the actor, the man who tried to save him, the actor’s first wife, his oldest friend, and a young actress with the TravelingSymphony, caught in the crosshairs of a dangerous self-proclaimed prophet. Sometimes terrifying, sometimes tender, Station Eleven tells a story about the relationships that sustain us, the ephemeral nature of fame, and the beauty of the world as we know it–

Stripping unprintable Unicode variation characters in PHP

Fri, 31 Mar 2017 00:00:00 +0000

Emoji characters like 🥳, 🎉 and 💖 can be represented in a string in different ways. The first way is to include a Unicode point as a single character. For example 💖 is Unicode point U+1F496, which can be included in a PHP string either directly ('💖') or using the Unicode code point escape syntax from PHP 7 onwards: "\u{1F496}".

You can try these on the PHP REPL (php -a):

php > echo '💖';
💖
php > echo "\u{1F496}";
💖

Emoji characters might also be represented using a zero width joiner Unicode point, which lets you combine two separate Unicode points together to be displayed as a single character. For example, the rainbow flag emoji 🏳️‍🌈 can be produced by the sequence [Waving white flag] [ZWJ] [Rainbow] (flags are a common use-case for the ZWJ character).

Algorithm Design Exercises 1

Thu, 23 Mar 2017 00:00:00 +0000

These are the exercises for chapter 1 of The Algorithm Design Manual.

1-1

Show that $a + b$ can be less than $min(a,b)$

Where $a,b < 0$, e.g. if $a = -1, b = -2$, then $a + b = -3 < min(a,b) = -5$.

1-2

Show that $a \times b$ can be less than $min(a,b)$

Where one of $\{a,b\}$ is negative and one is positive, e.g. if $a = -1, b = 2$, then $a \times b = -2 < min(a,b) = -1$.

Cracking the Coding Interview 6thE 10.1 Sorted Merge

Thu, 23 Mar 2017 00:00:00 +0000

“Sorted Merge: You are given two sorted arrays, A and B, where A has a large enough buffer at the end to hold B. Write a method to merge B into A in sorted order.”

Arrays are already sorted, so we can just do an insertion sort: iterate both, taking the next sorted item from each and insert it into A which has space for both.

It would be tricky to insert at the beginning, though, as we’d have to shift the existing elements in A. So we can do the insertion sort in reverse order, taking the largest element at a time from the end of each array and adding it to the moving end of A.

Uploading a listing image to the Etsy API in Python

Tue, 21 Mar 2017 00:00:00 +0000

It took a bit of messing around, but I was able to upload product listing images via the Etsy API in Python 3 with requests_oauthlib like this (reduced to the essentials):


from requests_oauthlib import OAuth1Session

session = OAuth1Session(
 "{etsy client key}",
 client_secret="{etsy client secret}",
 resource_owner_key="{etsy resource owner key}",
 resource_owner_secret="{etsy resource owner secret}",
)

response = session.post(
 "/listings/123456789/images",
 params={
 "listing_id": 123456789,
 "rank": 1,
 "overwrite": True,
 "is_watermarked": False,
 },
 files={
 "image": (
 "image123.jpg",
 open("image123.jpg", "rb"),
 "image/jpeg"
 )
 },
)

print(response.text)

I’ve been using this to automate management of the art decor images for our wall art prints shop.

Getting started with Teach Yourself CS: Algorithms and Data Structures

Fri, 17 Mar 2017 00:00:00 +0000

The website TeachYourselfCS.com was recently featured on Hacker News, and after reading through I’ve decided to embark on a new learning journey by going through one of its subjects as properly as I can.

The subject I’ve chosen is Algorithms and Data Structures, as I think that will be most interesting to me personally, and will have the most immediate practical application in the kinds of things I like to do.

I’m very grateful that the recommended lecture series, Skiena’s Algorithms Lectures, is freely available online (YouTube copy), as are course documents for that course and various others. I was also able to find a copy of The Algorithm Design Manual with a quick search.

Deep Survival

Wed, 01 Mar 2017 00:00:00 +0000

Algorithms to Live By

Wed, 01 Feb 2017 00:00:00 +0000

Moment Wall Art

Mon, 23 Jan 2017 00:00:00 +0000

Creating & updating inventory product offerings via the Etsy API in Python

Sun, 08 Jan 2017 00:00:00 +0000

Managing products and product variations in the Etsy API can be a bit tricky, especially when you want to “sync” them, i.e. either create or update in the Etsy API according to your own product data.

Here’s an approach in Python that I got working.

Say your product data looks like this:


my_products = {
 "sku_123": {"style": "foo", "price": "15.00", "quantity": 30},
 "sku_456": {"style": "bar", "price": "25.00", "quantity": 10},
}

One approach is to fetch the existing Etsy listing’s “inventory” (which might be empty for a newly created one), update it to match your local data, and then put it back to the Etsy API.

A History of Modern Britain

Sun, 01 Jan 2017 00:00:00 +0000

Simplicity is a skill

Sun, 23 Oct 2016 00:00:00 +0000

After a lot of peer reviews and looking back at my own work recently, I’ve been considering the point I was trying to get across in “Boring considered beneficial”.

One of the main thoughts there was that approaches and technologies sometimes considered boring or traditional have that perception for a reason: they’ve become mature enough to generally work and solve problems in a straightforward way. The community is used to them, so they’re not as exciting as more novel and exotic ways of solving the same problems.

Scalar type declarations in PHP 7

Fri, 30 Sep 2016 00:00:00 +0000

One of my favourite new features in PHP 7 is [scalar type declarations](http://php.net/manual/en/migration70.new- features.php#migration70.new-features.scalar-type-declarations).

Prior to PHP 7, the language’s type declaration abilities were somewhat inconsistent. You could type hint classes, interfaces and arrays, but nothing else (including callable and self in those). To require types for any scalar values (strings, ints, bools and floats), you had to do your own type checking and coercion.

However, it is worth pointing out that it was and is possible to do deeper type checking in PHP using value objects or interfaces. This is a nice technique and has some added benefits over plain scalar type declarations. For example, declaring Weight, Distance and Length interfaces is more rigorous than just declaring int or float types for those values.

Always lint

Tue, 13 Sep 2016 00:00:00 +0000

“lint (n): undesirable bits of fiber and fluff found in sheep’s wool”

— Wikipedia

Linting is one of the first things I like to set up on a new project. Especially when working with interpreted languages, it’s reassuring to develop with a mechnical helper pointing out mistakes, problems and potential improvements.

Linters exist for pretty much every language out there, and are a great addition to the development process. Here’s why.

Trigger Warning

Mon, 05 Sep 2016 00:00:00 +0000

The Handmaid's Tale

Fri, 05 Aug 2016 00:00:00 +0000

Python unique random function result decorator

Sat, 23 Jul 2016 00:00:00 +0000

I like messing around with random generators for productivity and creativity systems. Usually this involves functions that return a random value on each call. Even with quite a large set of possible values, duplicates occur more often than you might expect.

Here’s a quick Python decorator that keeps calling the function it decorates until it gets a value we haven’t seen so far during the current process:

def unique_results(func: Callable) -> Callable:
 seen = set()

 def wrapper():
 res = func()
 for i in range(100):
 if res not in seen:
 seen.add(res)
 return res
 res = func()
 raise OverflowError("Hit random value limit")

 return wrapper

Then you can decorate your random result function like this:

Boring considered beneficial

Mon, 23 May 2016 00:00:00 +0000

Being boring is a good thing in software design.

There is no benefit to being impressive with a design, or to demonstrate one’s knowledge in code. Doing that is likely to cause problems and reduce the quality of the software.

One aspect of the programming profession is that there is both an urge and plenty of opportunity to try to show off. We are continually learning and improving, as we should be, and this can lead us to actively seek out opportunities to use the interesting knowledge we have acquired, instead of putting it to use when a concrete need for it arises.

Simple pipes with PHP generators

Tue, 17 May 2016 00:00:00 +0000

Pipes are a great structural pattern that can simplify the design of all kinds of software. They’re a central part of the Unix philosophy, allowing users to chain together a series of simple parts to achieve a more complex goal. Similarly, gulpjs brings the benefit of a pipe-based design to asset generation in the nodejs ecosystem.

I haven’t seen pipes used so much in the PHP world, but they’ve been simple to set up since generators were added to the language in 5.4.

Never Let Me Go

Thu, 05 May 2016 00:00:00 +0000

From the Booker Prize-winning author of The Remains of the Day and When We Were Orphans, comes an unforgettable edge-of-your-seat mystery that is at once heartbreakingly tender and morally courageous about what it means to be human. Hailsham seems like a pleasant English boarding school, far from the influences of the city. Its students are well tended and supported, trained in art and literature, and become just the sort of people the world wants them to be. But, curiously, they are taught nothing of the outside world and are allowed little contact with it. Within the grounds of Hailsham, Kathy grows from schoolgirl to young woman, but it’s only when she and her friends Ruth and Tommy leave the safe grounds of the school (as they always knew they would) that they realize the full truth of what Hailsham is.

Using a real facade pattern in Laravel to encapsulate common behaviour

Thu, 05 May 2016 00:00:00 +0000

Whilst working on my side project Pop Robin Cards (source), one of the PHP-MD rules I have configured flagged an issue:

poprobincards/app/Http/Controllers/Staff/ProductController.php:53
The method __construct has 6 parameters. Consider to reduce parameter number
under 5.

PHP-MD is quite rightly pointing out that six parameters for a method is excessive, even for a constructor. But what can be done? The six parameters are all things that the ProductController needs to do its job:

PHP's list() is asking for a class

Sun, 17 Apr 2016 00:00:00 +0000

PHP has a weird mechanism for achieving multiple return values with the list function. It lets you do things like this:

<?php

function severalThings() {
	return ['thing', 'another thing', 'one more thing'];
}

list($foo, $bar, $foobar) = severalThings();

This is quite commonly used in some codebases, but I think it’s bad style.

Firstly, because PHP doesn’t offer a concrete way to let the user of a function know exactly what it’s returning, it may not be clear what you’re getting from that array. The function could even not return an array at all, in which case trying to use list on its return value will silently set your variables to null.

Chinese strftime

Tue, 05 Apr 2016 00:00:00 +0000

I’ve always used some variation of the original GNOME desktop: GNOME 2, Cinnamon and currently MATE. All of those desktops feature a similar calendar applet that lets you enter a strftime string to get the date formatted how you like it.

To get a tiny bit more Chinese into my day, I use this format:

%Y年%m月%e号 %A %p%l点%M分%S秒

(The main reason I’m writing this post is to keep the format easily accessible (somewhere.)

Optimised image handling with AWS and Laravel

Sat, 19 Mar 2016 00:00:00 +0000

A central feature in my side project [Pop Robin Cards](https://github.com/hughgrigg /poprobincards) is the management of product images. In the first iteration, images uploaded by the user were simply stored locally on disk and served by nginx. This approach works fine, but has some problems:

It uses disk space on the EC2 instance hosting the site, where disk space is at a premium.
It makes it harder to load balance between multiple application server instances.
It uses application server resources to serve images.
If the EC2 instance storing the images goes down then the images may be lost.
There are potential security issues around storing user-uploaded files on the application server.

S3 provides an ideal solution to those issues. It’s cheap, durable, fast and centralised. With that in mind, I implemented a simple image processing system to optimise uploaded images and transfer them to S3.

Stranger in a Strange Land

Tue, 08 Mar 2016 00:00:00 +0000

Stranger in a Strange Land, winner of the 1962 Hugo Award, is the story of Valentine Michael Smith, born during, and the only survivor of, the first manned mission to Mars. Michael is raised by Martians, and he arrives on Earth as a true innocent: he has never seen a woman and has no knowledge of Earth’s cultures or religions. But he brings turmoil with him, as he is the legal heir to an enormous financial empire, not to mention de facto owner of the planet Mars. With the irascible popular author Jubal Harshaw to protect him, Michael explores human morality and the meanings of love. He founds his own church, preaching free love and disseminating the psychic talents taught him by the Martians. Ultimately, he confronts the fate reserved for all messiahs. The impact of Stranger in a Strange Land was considerable, leading many children of the 60’s to set up households based on Michael’s water-brother nests. Heinlein loved to pontificate through the mouths of his characters, so modern readers must be willing to overlook the occasional sour note (Nine times out of ten, if a girl gets raped, it’s partly her fault.). That aside, Stranger in a Strange Land is one of the master’s best entertainments, provocative as he always loved to be. Can you grok it? –Brooks Peck

PyCharm Python script run error "ImportError: attempted relative import with no known parent package"

Tue, 23 Feb 2016 00:00:00 +0000

When trying to run a Python script in PyCharm via a run configuration, you might hit an error like this:

Traceback (most recent call last):
 File ".../__main__.py", line 3, in <module>
 from ...
ImportError: attempted relative import with no known parent package

Process finished with exit code 1

You might be able to resolve this by setting up the PyCharm run configuration as a “Module name” rather than a “Script path”.

Pop Robin Cards

Sat, 23 Jan 2016 00:00:00 +0000

Pop Robin Cards is a side project that my wife and I work on in our spare time. It’s a custom e-commerce platform selling 3D pop-up cards.

The system allows management of a product catalogue, stock, images and so on, as well as a sales and order tracking system.

Selecting the last grid row in CSS

Sat, 23 Jan 2016 00:00:00 +0000

Recently I had the need to select the last row of elements in a grid using CSS. The selector needed to be able to handle an arbitrary total number of elements in the grid; in other words, in a grid with x columns, there could be 0 to x elements in the last row.

23 Things They Dont Tell You About Capitalism

Fri, 01 Jan 2016 00:00:00 +0000

59 Seconds

Fri, 01 Jan 2016 00:00:00 +0000

A psychologist and best-selling author gives us a myth-busting response to the self-help movement, with tips and tricks to improve your life that come straight from the scientific community.Richard Wiseman has been troubled by the realization that the self-help industry often promotes exercises that destroy motivation, damage relationships, and reduce creativity: the opposite of everything it promises. Now, in 59 Seconds, he fights back, bringing together the diverse scientific advice that can help you change your life in under a minute, and guides you toward becoming more decisive, more imaginative, more engaged, and altogether more happy.From mood to memory, persuasion to procrastination, resilience to relationships, Wiseman outlines the research supporting the new science of �rapid change� and, with clarity and infectious enthusiasm, describes how these quirky, sometimes counterintuitive techniques can be effortlessly incorporated into your everyday life. Or, as he likes to say: �Think a little, change a lot.�

A Bitter Revolution

Fri, 01 Jan 2016 00:00:00 +0000

Budding Prospects

Fri, 01 Jan 2016 00:00:00 +0000

Fluent Forever

Fri, 01 Jan 2016 00:00:00 +0000

Presents a series of techniques for acquiring a foreign language, including news ways of training the tongue for pronunciation, use of visual imagery for connecting sounds and spelling, and spaced-repetition methods for learning new vocabulary.

Fooled by Randomness

Fri, 01 Jan 2016 00:00:00 +0000

Haunted

Fri, 01 Jan 2016 00:00:00 +0000

Look to Windward

Fri, 01 Jan 2016 00:00:00 +0000

The Twin Novae battle had been one of the last of the Idiran war, and one of the most horrific: desperate to avert their inevitable defeat, the Idirans had induced not one but two suns to explode, snuffing out worlds and biospheres teeming with sentient life. They were attacks of incredible proportion – gigadeathcrimes. But the war ended, and life went on. Now, eight hundred years later, light from the first explosion is about to reach the Masaq’ Orbital, home to the Culture’s most adventurous and decadent souls. There it will fall upon Masaq’s 50 billion inhabitants, gathered to commemorate the deaths of the innocent and to reflect, if only for a moment, on what some call the Culture’s own complicity in the terrible event. Also journeying to Masaq’ is Major Quilan, an emissary from the war-ravaged world of Chel. In the aftermath of the conflict that split his world apart, most believe he has come to Masaq’ to bring home Chel’s most brilliant star and self-exiled dissident, the honored Composer Ziller. Ziller claims he will do anything to avoid a meeting with Major Quilan, who he suspects has come to murder him. But the Major’s true assignment will have far greater consequences than the death of a mere political dissident, as part of a conspiracy more ambitious than even he can know – a mission his superiors have buried so deeply in his mind that even he cannot remember it. Hailed by SFX magazine as an excellent hopping-on point if you’ve never read a Banks SF novel before, Look to Windward is an awe-inspiring immersion into the wildly original, vividly realized civilization that Banks calls the Culture.

Survivor

Fri, 01 Jan 2016 00:00:00 +0000

The Art of Thinking Clearly

Fri, 01 Jan 2016 00:00:00 +0000

The Brothers Karamazov

Fri, 01 Jan 2016 00:00:00 +0000

The Colour of Magic

Fri, 01 Jan 2016 00:00:00 +0000

The Left Hand of Darkness

Fri, 01 Jan 2016 00:00:00 +0000

The Light Fantastic

Fri, 01 Jan 2016 00:00:00 +0000

Only one individual can save the world from a disastrous collision. Unfortunately, the hero happens to be the singularly inept wizard Rincewind, who was last seen falling off the edge of the world.

The Trial

Fri, 01 Jan 2016 00:00:00 +0000

Kafka’s classic novel of the terror state, in which profound unease and sinister uncertainty simmer beneath Joseph K.’s surface world of desperate normality and security. When arrested, he is spun into a panic and slowly, strangely, a helpless sense of guilt emerges in him.

Island counting, max island area and max island perimeter in Python

Wed, 23 Dec 2015 00:00:00 +0000

Here are three similar algorithms in Python for operations on “island problems”. These are problems were the input is a 2D matrix of grid coordinates, where 0 represents sea and 1 represents land.

For example:

[
 [1, 0, 1, 0, 1],
 [1, 0, 1, 0, 0],
 [0, 0, 1, 1, 0],
 [0, 0, 0, 0, 0],
 [1, 1, 0, 0, 1],
]

It’s slightly hard to see, but there are 5 islands in this grid: one in each corner, and a larger contiguous one in the shape of an L.

Keep controllers thin

Mon, 23 Nov 2015 00:00:00 +0000

The world of professional web development is dominated by the MVC pattern. MVC is so common in object oriented web development that it seems fundamental: of course your app has models, views and controllers (although other approaches could be perfectly valid).

One bit of advice that tends to get passed around for implementing with the MVC pattern is “thin controllers and fat models”, or some variation of that. I think this maxim misses the point.

Validating URLs with the DOM

Mon, 05 Oct 2015 00:00:00 +0000

URL validation is a common task in Web development, and thankfully a lot of cases can now be handled by the browser with the url input type:

 <input type="url" name="gimme-a-link"></input>

That’s great and gives you a lot of functionality and good user experience on a range of devices. However, a couple of issues remain once you’ve got your url type input in place:

The browser’s URL validation is fast and accurate, but can be too permissive.
Accessing the result of the validation is currently [a mixed-bag](https:// developer.mozilla.org/en-US/docs/Web/API/ValidityState) across browsers.
You can’t do fine-grained validation according to specific circumstances.

This leads to implementing your own front-end URL validation either to complement or replace that provided by the browser. If you are rolling your own, here’s a request from me as a user: please keep the url type and add a [novalidate attribute](http://www.w3.org/TR/html- markup/form.html#form.attrs.novalidate) if you want to disable the browser’s validation – this keeps the UX benefits which are particularly helpful on touch devices.

Specification driven development

Wed, 23 Sep 2015 00:00:00 +0000

There’s a great free ebook on test driven development by Grzegorz Gałęzowski. I read a lot of books related to programming, and this one stood out as particularly worthwhile.

Aside from a range of useful approaches and gems of understanding I got from the book, the biggest gain for me was an appreciation that test driven development isn’t really about testing as such. It could instead be named “specification- driven development”, which to me gives a better sense of what its main benefits are (and is what TDD actually means).

Making random memorable passwords with bash

Thu, 17 Sep 2015 00:00:00 +0000

Making secure passwords is quite a common task, and there are plenty of random password generators around (I like the one in keepassx). Making memorable passwords can be a bit trickier, though.

A nice way to make random but memorable passwords is to use a sequence of words and perhaps a short number (obligatory link to xkcd). That kind of password is easy for a human to remember and generally hard to brute-force, so long as the pool of possible words is large enough.

Environment variables in the constructor are a code smell

Sun, 06 Sep 2015 00:00:00 +0000

TLDR: don’t use env vars in class constructors.

Seeing environment variables in a class constructor is a code smell and an anti-pattern.

For example in TypeScript for Node.js, you might see something like this:

Anti-pattern:

class Foobar {
	private niceInjectableDependency: IDependency;

	private importantConfig: string;

	constructor(niceInjectableDependency: IDependency) {
		this.niceInjectableDependency = niceInjectableDependency;

		this.importantConfig = process.env.IMPORTANT_CONFIG; // code smell
	}
}

This might seem reasonable, as we’re still kind of able to inject the importantConfig into the class, just not via the constructor. Unfortunately this injectability is nowhere near as good as the usual kind with function call arguments.

Improving Unix-fu with Anki

Wed, 01 Jul 2015 00:00:00 +0000

Anki has helped me maintain and improve my Chinese for years, but I’ve always been intrigued by using it to develop other kinds of knowledge. At university I used it heavily for East Asian history, our modern Chinese literature course and classical Chinese. I’ve now started using it to improve my Unix command-line fluency, and the results have been quite dramatic.

It’s not really possible to learn programming with an SRS system like Anki, but you can certainly use it to reinforce tokenized knowledge (like [this](https://developer.atlassian.com/blog/2015/06/golang-flashcards-and- spaced-repetition/)). Whilst programming ability doesn’t tokenize very well, a lot of related computing skills certainly do.

Minimum required lecture rooms scheduling in Python

Tue, 23 Jun 2015 00:00:00 +0000

Given a schedule of lectures represented as a list of tuples of start and end times like [(15, 16), (13, 15), (14, 15)], return the minimum number of lecture rooms required to run all of the lectures.

We can solve this in $O(n\log_2n)$, due to the need to sort the lecture times.

The overall steps are:

Convert time tuples into “lecture event” tuples of time and type, i.e. start or end.
Sort the lecture events by time and then type.
Iterate lecture events, tracking current active lectures.
Track the max active lectures.

A Python implementation looks like this:

Using Wget for recursive downloads

Sun, 31 May 2015 00:00:00 +0000

I discovered the VimCasts website recently and have been watching a few of their videos. Due to my unreliable internet connection, I prefer to download whole video files and then watch them (this also allows watching them on portable devices).

VimCasts allow directory listing on their storage server, which I believe is permission to bulk download their content. The only issue is that each episode is stored in its own subdirectory and in two formats. I don’t want the folder structure and I only want one of the formats.

That code is ugly for a reason

Wed, 27 May 2015 00:00:00 +0000

Before I really got into development, I went through the entire archive of Joel on Software reading every single article. I like to think that it set me up with good foundational knowledge for professional software development, but that’s pretty hard to prove.

Recently I re-read the article Things You Should Never Do, Part I. With a lot more development experience under my belt, the article spoke to me much more strongly this time around.

Using Gulp asset versioning with Hugo data files

Sat, 23 May 2015 00:00:00 +0000

I recently finished converting four of the websites I run from Wordpress to Hugo (my original blog, a Chinese learning blog, a Chinese grammar site and this blog), and hosting them with S3 and CloudFront.

It’s been a great change to make and has improved everything from my workflow to loading times and hosting costs. I’d recommend anyone running blogs and other text-focused sites to switch away from Wordpress and use a static site generator instead. Hugo would be my recommendation because of its efficiency and flexibility, but there are many options.

Avoid naive string indexing in PHP

Thu, 23 Apr 2015 00:00:00 +0000

PHP allows treating a string as an array, so that you can use indexing syntax to get or set a single position in the string:

<?php

echo 'hello'[0];
// h

$foo = 'bar';
$foo[2] = 'z';
echo $foo;
// baz

This seems handy but you should never do it.

The reason to avoid string indexing like this in PHP is that PHP strings are not multibyte character strings, they’re just bytes.

You won’t notice this with plain ASCII strings like the above, as each character happens to be one byte anyway so they’re equivalent.

Typing arrow and quote characters in Ubuntu with AltGr

Mon, 23 Feb 2015 00:00:00 +0000

A quick Ubuntu Linux tip: you can use the AltGr key to easily type various arrow and quote characters.

The ones you might commonly want to use are:

AltGr + y ← Left arrow	
AltGr + u ↓ Down arrow
AltGr + i → Right arrow
AltGr + Shift + u ↑ Up arrow

AltGr + z « Left double angle quote
AltGr + x » Right double angle quote

AltGr + v “ Left double quotation mark
AltGr + b ” Right double quotation mark

Topological course prerequisites in Python

Fri, 23 Jan 2015 00:00:00 +0000

We want to figure out the sequence we can take university courses in, and some of the courses may have prerequisite courses. In other words, we want a topological sort of the courses.

For this example, we can just ignore any courses that are impossible to take because they depend on non-existent courses, or have a circular dependency.

The input data is like this:

{'C3': {'C2', 'C1'}, 'C2': {'C1'}, 'C1': {}}

And the desired output is like this:

Alexs Adventures in Numberland