https://notestoself.dev/ Recent content on NotesToSelf.Dev Hugo -- gohugo.io en-gb https://notestoself.dev/posts/cdk-synth-on-change-bash-script/ Tue, 08 Apr 2025 10:57:23 +0100 https://notestoself.dev/posts/cdk-synth-on-change-bash-script/ The cdk synth command can be pretty slow, even when running for a single small stack. We also often want to run the CDK synth command regularly as part of local development setups, so it’s nice to skip the re-synth if no relevant files have changed. Here’s a bash script that checks for changed TypeScript source files by comparing their timestamp to that of the generated template file from the previous synth. https://notestoself.dev/posts/sam-local-wsgi-drops-headers-underscore/ Sun, 30 Mar 2025 11:57:23 +0100 https://notestoself.dev/posts/sam-local-wsgi-drops-headers-underscore/ TLDR: SAM local drops headers that contain an underscore, due to using Flask and WSGI internally, which differs from the behaviour of a deployed API Gateway. I’m working on an existing project that uses AWS CDK to provision API Gateway in front of some Lambda functions. The team would like to be able to run this locally for faster development iterations, so I set that up with the sam local start-api command, using the template emitted by cdk synth. https://notestoself.dev/posts/github-action-cache-public-docker-image/ Sun, 23 Mar 2025 11:57:23 +0100 https://notestoself.dev/posts/github-action-cache-public-docker-image/ Here’s a reusable GitHub Action definition that caches a public Docker image: ---name:Dockerpublicimagecachedescription:PullandcacheapublicDockerimageforreuseinputs:image_name_tag:description:Dockerimagenameandtagtopullandcacherequired:trueruns:using:"composite"steps:- name:Restoreimagecacheid:restore_cacheuses:actions/cache@v4with:path:'ci/cache/docker'key:'cache-docker-${{ inputs.image_name_tag }}'- name:Updateimagecacheoncachemissif:steps.restore_cache.outputs.cache-hit!='true'shell:bashrun:> docker pull '${{ inputs.image_name_tag }}'&&mkdir-p$(dirname'./ci/cache/docker/${{ inputs.image_name_tag }}.tar')&&dockerimagesave'${{ inputs.image_name_tag }}'--output'./ci/cache/docker/${{ inputs.image_name_tag }}.tar'- name:Useimagecacheoncachehitif:steps.restore_cache.outputs.cache-hit=='true'shell:bashrun:> docker image load --input './ci/cache/docker/${{ inputs.image_name_tag }}.tar'If you have this action definition at .github/actions/docker_image_cache in the repo, then you can use it from a workflow like this: ---jobs:foo_job:runs-on:ubuntu-lateststeps:- name:LocalStackpublicimagecacheuses:./.github/actions/docker_image_cachewith:image_name_tag:'localstack/localstack:4.2.0'After the first run, you should see output like this from the action, showing that it has loaded the Docker image from the cache: https://notestoself.dev/posts/eslint-changed-ts-files-only/ Sun, 23 Mar 2025 10:57:23 +0100 https://notestoself.dev/posts/eslint-changed-ts-files-only/ It’s best if a linter is applied from the beginning of a project, and with default linting rules, as that way the project benefits from the linting with the minimum possible effort. However, we often join existing projects which have already been developed without the use of a linter, by which point there are possibly thousands of linter violations. In that situation, one option is to incrementally improve the linting by only applying the linter to changed files. https://notestoself.dev/posts/bash-split-text-file-multiple-separate-sections/ Wed, 26 Feb 2025 10:57:23 +0100 https://notestoself.dev/posts/bash-split-text-file-multiple-separate-sections/ Here’s a bash command to split a text file into multiple separate files by section delimited by a regex pattern matching titled sections in the source file: csplit \ -z \ -f section_ \ -b "%02d.txt" \ document.txt \ '/\nSection [0-9]\+\n/' \ '{*}' The key part is the /\nSection [0-9]\+\n/ regex pattern, which matches section titles such as “Section 18” alone on a line. https://notestoself.dev/posts/python-tests-raises-exception-group-matching-context-manager/ Sun, 05 Jan 2025 17:11:23 +0100 https://notestoself.dev/posts/python-tests-raises-exception-group-matching-context-manager/ Here’s a Python testing utility function for asserting on an expected Exception type from within an ExceptionGroup: from contextlib import contextmanager @contextmanager def raises_in_group( expected_exception: type[Exception], match_message: str = "", ): """ Like pytest.raises, but for an Exception inside an ExceptionGroup, for example coming from asyncio.TaskGroup. """ matched_type = [] matched_message = [] try: yield except* expected_exception as e: assert isinstance(e, ExceptionGroup) matched_type = [exp for exp in e.exceptions if isinstance(exp, expected_exception)] matched_message = [exp for exp in matched_type if match_message in str(exp)] assert len(matched_type) >= 1, f"Expected at least one {expected_exception} exception. https://notestoself.dev/posts/aws-cloudformation-create-organizational-units-accounts-prod-dev/ Sat, 05 Oct 2024 15:44:45 +0100 https://notestoself.dev/posts/aws-cloudformation-create-organizational-units-accounts-prod-dev/ Here’s an AWS CloudFormation template and deployment script to set up Organizational Units and Accounts for Prod and Dev for a new project or department. Deploying a CloudFormation stack from this template will require a user in the Organization’s management account with the AdministratorAccess permission set. The created Organizational Unit (OU) and Account structure will look this: Organization Root [pre-existing] ╚═ Management Account [pre-existing] ╚═ Project Parent OU ╚═ Project Prod OU ╚═ Project Prod Account ╚═ Project Dev OU ╚═ Project Dev Account The OUs are not strictly necessary, but they help to group the whole project under a single OU, and to allow for other accounts to be added later within the Prod and Dev OUs if necessary. https://notestoself.dev/posts/py-coverage-pragma-no-cover-pyproject-tool-coverage-report-exclude-also/ Sun, 29 Sep 2024 19:45:13 +0100 https://notestoself.dev/posts/py-coverage-pragma-no-cover-pyproject-tool-coverage-report-exclude-also/ If you’re using Coverage.py or pytest-cov in your Python project, you probably end up with # pragma: no cover comments all over the place to prevent odd lines from being marked as missing coverage, like this: class Foobar: def do_something(): # pragma: no cover raise NotImplementedError() You can reduce the need for these # pragma: no cover comments by configuring Coverage.py to ignore certain patterns by default. This can be configured in the pyproject. https://notestoself.dev/posts/github-action-telegram-notification-no-dependencies/ Sun, 29 Sep 2024 19:32:04 +0100 https://notestoself.dev/posts/github-action-telegram-notification-no-dependencies/ Here’s a Github Action that posts a notification message to a Telegram chat, without using any dependencies. ---name:PosttoTelegramdescription:PostamessagetoaTelegramchannelinputs:telegram_bot_token:description:AuthtokenoftheTelegrambotviawhichtopost.required:truetelegram_chat_id:description:IDoftheTelegramchanneltowhichtosendthemessage.required:truemessage_content:description:StringcontentoftheTelegrammessagetosend.required:trueruns:using:"composite"steps:- shell:bashrun:| curl "https://api.telegram.org/bot${{ inputs.telegram_bot_token }}/sendMessage" \--header"Content-Type: application/json"\--requestPOST\--data'{ "chat_id": "${{ inputs.telegram_chat_id }}", "text": "${{ inputs.message_content }}" }'The key part of the action is the bash script that uses curl to post a message to Telegram: curl "https://api.telegram.org/bot${{inputs.telegram_bot_token}}/sendMessage" \ --header "Content-Type: application/json" \ --request POST \ --data '{ "chat_id": "${{ inputs.telegram_chat_id }}", "text": "${{ inputs. https://notestoself.dev/posts/py-coverage-compound-boolean-line-exit/ Sat, 28 Sep 2024 13:11:39 +0100 https://notestoself.dev/posts/py-coverage-compound-boolean-line-exit/ With branch coverage enabled in Coverage.py or pytest-coverage, you might see it report missing coverage on a compound boolean condition, like this: def foobar() -> bool: return ( condition_1() and condition_2() and condition_3() and condition_4() and condition_5() ) That might lead to a report that the line where the boolean compound condition begins to the exit of the function are missing coverage. The report will look like this: ---------- coverage: platform linux, python 3. https://notestoself.dev/posts/python-recursive-override-paths-function-mixin/ Wed, 25 Sep 2024 13:36:56 +0100 https://notestoself.dev/posts/python-recursive-override-paths-function-mixin/ Here’s a recursive utility function in Python that allows overriding nested paths through objects that can be mutable mappings, mutable sequences or dataclass instances. The function allows doing things like this: override_paths( { "foo0": "bar", "lvl1": { "foo1": "bar1", "list1": [1, 2, 3, 4, 5, 6], "foobar1": Foobar(foo="hello1"), }, }, foo0="123", lvl1__foo1="456", lvl1__list1__3=23, lvl1__foobar1__foo="hello2", ) { "foo0": "123", "lvl1": { "foo1": "456", "list1": [1, 2, 3, 23, 5, 6], "foobar1": Foobar(foo="hello2"), }, } There is also a OverridableMixin that makes it convenient to do things like this inline: https://notestoself.dev/posts/github-action-job-pytest-coverage-pr-comment/ Mon, 23 Sep 2024 18:07:38 +0100 https://notestoself.dev/posts/github-action-job-pytest-coverage-pr-comment/ Here’s a Github Action job that runs pytest coverage and then posts a summary of the coverage report as a PR comment with Markdown formatting. This is handy as part of a PR workflow to make the test results and coverage summary visible on the PR. coverage:name:coverageruns-on:ubuntu-lateststeps:- uses:actions/checkout@v4- uses:./.github/actions/setup_python- uses:./.github/actions/job_urlid:job_urlwith:step_name:'pytest'- name:pytestrun:| pytest- name:Testcoveragesummaryreportid:coverage_reportif:always()run:| {echo"COVERAGE_REPORT<<EOF"coveragereport--format=markdown--show-missing--skip-coveredecho"EOF"}>>"$GITHUB_OUTPUT"- name:Removepreviouscoveragecommentsuses:./.github/actions/remove_commentsif:always()with:identifier:coverage- name:TestcoveragePRcommentid:coverage_commentif:always()uses:actions/github-script@v7with:script:| github.rest.issues.createComment({owner:context.repo.owner,repo:context.repo.repo,issue_number:context.issue.number,body:`[coverage](${{steps.job_url.outputs.job_url}})\r\n\r\n${{steps.coverage_report.outputs.COVERAGE_REPORT}}`})Note that this uses separate actions to setup the Python env, get the job URL and to remove previous PR comments. https://notestoself.dev/posts/github-action-command-comment-exit-status/ Mon, 23 Sep 2024 17:50:45 +0100 https://notestoself.dev/posts/github-action-command-comment-exit-status/ Here’s a Github Action that runs a given bash command, captures the stdout and strderr output, and posts a comment with the output back to the PR if the command has a non-zero exit status. This is useful for running check commands such as linters against a PR, and making a PR comment with the linter output if there are linting errors. name:Command&Commentdescription:RunabashcommandandcommentfailureoutputbacktoPRinputs:identifier:description:Singlewordidentifierforthecheckcommandrequired:truebash_command:description:Bashcommandtorunandcommentfromrequired:trueruns:using:"composite"steps:- uses:./.github/actions/job_urlid:job_urlwith:step_name:${{inputs.identifier}}- id:check_commandname:${{inputs.identifier}}shell:bashrun:| set +eecho"COMMAND_OUT<<EOF"|tee"$GITHUB_OUTPUT"${{inputs.bash_command}}2>&1|sed's/$/\\n/g'|tr-d'\n'|tee-a"$GITHUB_OUTPUT";COMMAND_STATUS=$?(echo;echo"EOF")|tee-a"$GITHUB_OUTPUT"exit$COMMAND_STATUS- name:Removepreviouscommandcommentsuses:./.github/actions/remove_commentsif:always()with:identifier:${{inputs.identifier}}- name:PRcommentcommandfailureuses:actions/github-script@v7if:always()&&steps.check_command.outcome=='failure'with:script:| github.rest.issues.createComment({owner:context.repo.owner,repo:context.repo.repo,issue_number:context.issue.number,body:'[${{inputs.identifier}}](${{steps.job_url.outputs.job_url}})\n\n```\n${{steps.check_command.outputs.COMMAND_OUT}}```'})Note that this uses separate actions to get the job URL and to remove previous PR comments. https://notestoself.dev/posts/github-action-remove-previous-job-pr-comments/ Mon, 23 Sep 2024 14:03:53 +0100 https://notestoself.dev/posts/github-action-remove-previous-job-pr-comments/ Here’s a Github Action that finds and removes PR comments based on a string identifier. This is useful for Github Action workflows that post comments to PRs, as it clears up out-of-date comments from earlier workflow runs. name:Command&Commentdescription:RunabashcommandandcommentfailureoutputbacktoPRinputs:identifier:description:Singlewordidentifierforcommentstoremoverequired:trueruns:using:"composite"steps:- name:Removepreviousjobcommentsuses:actions/github-script@v7with:script:| const pr_comments = await github.rest.issues.listComments({owner:context.repo.owner,repo:context.repo.repo,issue_number:context.issue.number,})awaitPromise.all(pr_comments.data.filter(comment=>comment.body?.startsWith("[${{inputs.identifier}}]")).map(comment=>github.rest.issues.deleteComment({owner:context.repo.owner,repo:context.repo.repo,issue_number:context.issue.number,comment_id:comment.id,})))It removes comments on the PR it is running against if the comment begins with [{identifier}]. If other comments happen to begin with that then they will also be removed. https://notestoself.dev/posts/github-action-multiline-output-bash/ Mon, 23 Sep 2024 13:02:14 +0100 https://notestoself.dev/posts/github-action-multiline-output-bash/ You can capture output from a Github Action by echoing a variable name to the file specified by the "$GITHUB_OUTPUT" variable: echo "foobar='here is some data'" | tee -a "$GITHUB_OUTPUT" This is a bit trickier with multiline output. You can use the bash EOF syntax for multiline strings: FOOBAR<<EOF here is some multiline data EOF The easiest way to use bash multiline strings in a Github Action is to use a bash command group with curly braces. https://notestoself.dev/posts/github-action-job-url/ Mon, 23 Sep 2024 12:13:40 +0100 https://notestoself.dev/posts/github-action-job-url/ Here’s a Github Action to get the URL to the current job and specific step in the job to open. This is useful for giving users a direct link to specific jobs. name:GetcurrentjobURLdescription:OutputthecurrentGithubWorkflowJobURLinputs:step_name:description:SpecificjobsteptoincludeintheURLrequired:trueoutputs:job_url:description:"URL to the calling job and specific input step"value:${{steps.get_job_url.outputs.job_url}}runs:using:"composite"steps:- id:get_job_urlenv:GH_TOKEN:${{github.token}}shell:bashrun:| echo "job_url=$(gh run --repo ${{github.repository}} view ${{github.run_id}} \--jsonjobs\--jq'.jobs[] | select(.name == "${{github.job}}") | .url, ( .steps[] | select(.name == "${{inputs.step_name}}") | "#step:\(.number):1" )'\|tr-d"\n")" | tee "$GITHUB_OUTPUT"The key part is this bash script which uses the gh CLI tool’s --jq option to extract the URL: https://notestoself.dev/posts/python-asyncio-aiofiles-resourcewarning-contextlib-aclosing/ Wed, 18 Sep 2024 12:42:26 +0100 https://notestoself.dev/posts/python-asyncio-aiofiles-resourcewarning-contextlib-aclosing/ I noticed a ResourceWarning like this when using aiofiles to iterate lines of a file in an async context manager from aiofiles.open: ResourceWarning: unclosed file <_io.TextIOWrapper name='.../numbers.tmp' mode='r' encoding='UTF-8'> This is only a warning, so by default it won’t error out of the Python program. I try to have pytest filterwarnings set to error whenever possible, so that any warnings during tests will cause the test to error out. That’s how I spotted this issue. https://notestoself.dev/posts/python-asyncio-ensure-event-loop/ Sun, 15 Sep 2024 10:51:12 +0100 https://notestoself.dev/posts/python-asyncio-ensure-event-loop/ Here’s a utility function that safely ensures an event loop is running in a Python asyncio application. This is useful because of the discrepancy in expectations between the asyncio library and how it ends up being used in the real world in a lot of applications. The asyncio library reasonably expects users to be in control of the event loop for the whole application lifecycle, but this becomes complicated in various situations: https://notestoself.dev/work/doozy/ Sun, 01 Sep 2024 00:00:00 +0000 https://notestoself.dev/work/doozy/ I’ve been helping to build Doozy’s product as a freelance Software Engineer, using GCP, Firebase and TypeScript. https://notestoself.dev/work/elsewhen/ Mon, 15 Jul 2024 00:00:00 +0000 https://notestoself.dev/work/elsewhen/ I worked as a Software Engineer on a contract basis for ElseWhen, using asyncio, CloudBuild, CloudRun, GCP, Gemini, Google Cloud Functions, LangChain, LLMs, OpenAI, PostgreSQL, Python, SQL, Terraform and other technologies. https://notestoself.dev/work/omnipresent/ Mon, 01 Jan 2024 00:00:00 +0000 https://notestoself.dev/work/omnipresent/ I worked as a Software Engineer on a contract basis for Omnipresent via YLD, using asyncio, AWS, AWS Lambda, AWS SAM, Docker, ECS, FastAPI, PostgreSQL, Python, SNS, SQL, SQS, Terraform and other technologies. https://notestoself.dev/posts/ubuntu-bash-python-minify-json-clipboard-cli/ Tue, 18 Jul 2023 13:18:51 +0100 https://notestoself.dev/posts/ubuntu-bash-python-minify-json-clipboard-cli/ Here’s a Bash one-liner to minify JSON in the current contents of the clipboard and replace the clipboard contents with the minified JSON: xclip -selection clipboard -o | python3 -c "import sys;import json;print(json.dumps(json.loads(sys.stdin.read()),separators=(\",\",\":\")))" | tee >(xclip -selection clipboard) The equivalent on Mac is: pbpaste | python3 -c "import sys;import json;print(json.dumps(json.loads(sys.stdin.read()),separators=(\",\",\":\")))" | pbcopy To set it as an alias in Linux (mcj for “minify clipboard json”): alias mcj='xclip -selection clipboard -o | python3 -c "import sys;import json;print(json. https://notestoself.dev/posts/ubuntu-bash-python-json-encode-clipboard-cli/ Tue, 18 Jul 2023 12:38:51 +0100 https://notestoself.dev/posts/ubuntu-bash-python-json-encode-clipboard-cli/ Here’s a Bash one-liner to JSON-encode the current contents of the clipboard and replace the clipboard contents with the JSON-encoded output: xclip -o | python -c 'import sys;import json;print(json.dumps(sys.stdin.read().strip()))' | tee >(xclip -selection clipboard) The equivalent on Mac is: pbpaste | python -c 'import sys;import json;print(json.dumps(sys.stdin.read().strip()))' | pbcopy To set it as an alias in Linux (jec for “json encode clipboard”): alias jec='xclip -selection clipboard -o | python -c "import sys;import json;print(json. https://notestoself.dev/work/krakenflex/ Thu, 01 Jun 2023 00:00:00 +0000 https://notestoself.dev/work/krakenflex/ I worked as a Software Engineer on a contract basis for KrakenFlex, using asyncio, AWS, AWS Lambda, AWS SAM, CloudFormation, DynamoDB, Python, SNS, SQS and other technologies. https://notestoself.dev/posts/aws-sam-cloudformation-ssm-parameter-secure-string-not-supported-workaround/ Tue, 16 May 2023 18:23:42 +0000 https://notestoself.dev/posts/aws-sam-cloudformation-ssm-parameter-secure-string-not-supported-workaround/ It’s surprisingly difficult to use AWS SSM Parameter secure strings in CloudFormation templates. If you try and have CloudFormation fetch the value of the SSM Parameter as a AWS::SSM::Parameter::Value<String> type, you’ll get this error: An error occurred (ValidationError) when calling the CreateStack operation: Parameters [/foobar/foo_param] referenced by template have types not supported by CloudFormation. If you try and have a CloudFormation parameter of type AWS::SSM::Parameter::Value<SecureString>, you’ll get this error: An error occurred (ValidationError) when calling the CreateStack operation: Template format error: Unrecognized parameter type: SecureString If you try and use the special resolve function in the CloudFormation template, like this: https://notestoself.dev/posts/aws-lambda-python-poetry-codeartifact-auth-sam-container-build/ Sun, 16 Apr 2023 18:23:42 +0000 https://notestoself.dev/posts/aws-lambda-python-poetry-codeartifact-auth-sam-container-build/ It’s often beneficial to use Docker to build AWS SAM Lambda Function images. Projects using AWS SAM often also use AWS CodeArtifact to manage private libraries, and poetry to manage Python dependencies. This combination can make it a little bit tricky to get the index auth working during the Docker build. Your Dockerfile for building the AWS SAM Lambda Function might look like this: FROMpublic.ecr.aws/lambda/python:3.12ENV PIP_DISABLE_PIP_VERSION_CHECK=on \ PYTHONDONTWRITEBYTECODE=1 \ PYTHONUNBUFFERED=1 COPY requirements. https://notestoself.dev/posts/pdf-png-csv-anki/ Fri, 17 Mar 2023 10:40:47 +0000 https://notestoself.dev/posts/pdf-png-csv-anki/ I had a large slide deck in PDF format and wanted to add the individual slides to Anki to revise from. There is some benefit to the learning process in doing this manually, but it was more practical to study the slides first and then automate the import into Anki. The final bash command to produce an importable CSV file for Anki looks like this: pdftoppm my_slides.pdf my_slide \ -progress -png -f 20 -l 923 -rx 60 -ry 60 && \ cp . https://notestoself.dev/posts/python-pytest-asyncio-aws-lambda-handler-event-loop/ Fri, 17 Feb 2023 10:40:47 +0000 https://notestoself.dev/posts/python-pytest-asyncio-aws-lambda-handler-event-loop/ If you’re using asyncio in an AWS Lambda, the Lambda handler function has to manage the asyncio event loop inside the Lambda, as the handler function itself cannot be an async function. This is fairly easy to do by having the top-level Lambda handler function use asyncio.run to call an asynchronous function that does the actual work: import asyncio async def foobar_handler() -> str: return "foobar" def lambda_handler(event, context) -> str: return asyncio. https://notestoself.dev/posts/python-aws-boto-moto-sns-invalid-parameter-unpack-6-arn/ Tue, 17 Jan 2023 10:40:47 +0000 https://notestoself.dev/posts/python-aws-boto-moto-sns-invalid-parameter-unpack-6-arn/ I just spent a long time debugging this error I was getting in a unit test using moto to mock out the boto3 SNS client: botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameter) when calling the Publish operation: not enough values to unpack (expected 6, got 1) This error comes from the innards of moto, where it splits an ARN on : and assumes there will be six sections to unpack into Python variables there. https://notestoself.dev/work/hyperobjekt/ Thu, 01 Sep 2022 00:00:00 +0000 https://notestoself.dev/work/hyperobjekt/ https://notestoself.dev/posts/pytest-fixture-optional-change-parameter-inside-test-case/ Mon, 18 Apr 2022 12:34:32 +0000 https://notestoself.dev/posts/pytest-fixture-optional-change-parameter-inside-test-case/ Here’s a way of setting up a pytest fixture that sets up a default value automatically for any test case that uses the fixture, but allows calling the fixture from inside the test case to update that value where necessary. The example is for mocking out API responses with httpx_mock, but this fixture arrangement can be used for any situation where you want a default fixture value that can optionally be modified from within a single test case. https://notestoself.dev/posts/python-assert_dict_matches-recursive-test-helper-function/ Fri, 18 Mar 2022 12:34:32 +0000 https://notestoself.dev/posts/python-assert_dict_matches-recursive-test-helper-function/ The Jest library for Node.js has a helpful expect.objectContaining function that makes it easy to assert only on the keys of interest in a particular test. Python’s unittest library does have a similar assertDictEqual method, but it asserts on all keys. I wanted to do be able to easily assert on only the keys of interest in Python tests, so I put together this quick test helper function to do so. https://notestoself.dev/posts/hugo-asset-pipeline-security.exec.allow-error/ Tue, 15 Mar 2022 16:19:31 +0000 https://notestoself.dev/posts/hugo-asset-pipeline-security.exec.allow-error/ If you hit this error with recent versions of Hugo: Error: Error building site: BABEL: failed to transform "js/main.js" (application/javascript): access denied: "babel" is not whitelisted in policy "security.exec.allow"; the current security configuration is: You need to allow babel in Hugo’s security rules in the config file, like this: [security] [security.exec] allow = ['^go$', '^npx$', '^postcss$', '^babel$'] https://notestoself.dev/posts/export-copy-clipboard-echo-token-bash/ Tue, 15 Feb 2022 16:19:31 +0000 https://notestoself.dev/posts/export-copy-clipboard-echo-token-bash/ Here’s a handy Bash one-liner to generate a token or some other kind of command output, and do three things at once with it: Export it as an environment variable in the shell. Copy it to the clipboard. Echo it to the terminal. export FOOBAR_TOKEN=$(foobar-token-command --foobar "foobar") \ && echo -n $FOOBAR_TOKEN | tee >(xclip -selection clipboard) I find this is handy for things like security tokens which I often want as an environment variable in the shell as well as being able to paste them into other tools such as HTTP testing apps. https://notestoself.dev/posts/hugo-new-post-bash-function/ Sat, 15 Jan 2022 12:52:06 +0000 https://notestoself.dev/posts/hugo-new-post-bash-function/ Hugo has a hugo new helper command that can generate new content files for you. It doesn’t quite do what I want, though, so I’ve added this quick Bash function to my .profile to make it easier to use: hnp() { hugo new "${2:-posts}/$(date +%Y)/$(date +%m)/${1}/index.md" } All it does is create a new post file in its own content directory at a date-based path, which is the structure I use for my Hugo sites. https://notestoself.dev/posts/jetbrains-datagrip-aws-documentdb-connection/ Thu, 06 Jan 2022 00:00:00 +0000 https://notestoself.dev/posts/jetbrains-datagrip-aws-documentdb-connection/ I’m currently working on a project for Kensio Software using AWS DocumentDB, and wanted to access the database for development using DataGrip. Here are some quick notes on how to set up a DocumentDB database as a data source in DataGrip. Getting DocumentDB connection details from AWS console It’s probably easiest to get the connection details for the DocumentDB cluster from the AWS Console. At the time of writing, the URL for the DocumentDB cluster looks like this: https://notestoself.dev/work/x2x-media/ Sat, 01 Jan 2022 00:00:00 +0000 https://notestoself.dev/work/x2x-media/ https://notestoself.dev/posts/python-iterator-chunk-generator-function/ Sat, 23 Oct 2021 00:00:00 +0000 https://notestoself.dev/posts/python-iterator-chunk-generator-function/ Here’s a versatile Python function that splits any iterator into chunks of size n as a generator. It can take simple iterators such as lists, but also take a generator and chunk it without unwinding the whole thing up front. This is quite useful when dealing with a database cursor, for example. You can iterate through the cursor in chunks of size n without having to load all of the objects into memory at once. https://notestoself.dev/posts/simple-example-floating-point-precision-error-monetary-amount/ Sat, 23 Oct 2021 00:00:00 +0000 https://notestoself.dev/posts/simple-example-floating-point-precision-error-monetary-amount/ Most software engineers are aware that floating point numbers are not suitable for monetary amounts, or any amount requiring reliable decimal precision. It’s nice to have a quick and simple example of this, so here’s an operation that will break with floating point numbers in many languages including JavaScript: 1.01 - 0.42 // → 0.5900000000000001 In JavaScript and TypeScript, the big.js library is good for decimal operations. In SQL, it’s best to store monetary and other decimal amounts as DECIMAL, e. https://notestoself.dev/posts/explicit-types-better-habit-type-inference-typescript/ Thu, 02 Sep 2021 00:00:00 +0000 https://notestoself.dev/posts/explicit-types-better-habit-type-inference-typescript/ TypeScript has a handy type inference feature where the compiler will automatically determine types in most situations. For example: function fooBar(): number { return 42; } const x = fooBar(); // x's type is infered as number This is quite useful and potentially allows for tidier, more readable code. However, I would actually suggest that avoiding type inference and preferring explicit types is a good habit. For trivial examples like the above, it probably is overkill to declare x as const x: number = 42. https://notestoself.dev/posts/gcp-google-cloud-platform-logging-query-syntax-exploration-fu/ Wed, 04 Aug 2021 00:00:00 +0000 https://notestoself.dev/posts/gcp-google-cloud-platform-logging-query-syntax-exploration-fu/ Recently I’ve been working in Google Cloud Platform (GCP) in a highly distributed system that is partitioned into many different services with various deployment strategies. A lot of it runs in Google Kubernetes Engine (GKE), a lot of it runs in Google Cloud Functions (GCF), and a lot of it runs in other bits and pieces of GCP infrastructure. Even the areas that are managed via GKE and Cloud Functions are heterogenous, with many idiosyncratic infrastructure configurations. https://notestoself.dev/posts/jest-mock-cheatsheet-typescript/ Wed, 23 Jun 2021 00:00:00 +0000 https://notestoself.dev/posts/jest-mock-cheatsheet-typescript/ I find myself looking up past projects to remember how to mock various things with Jest in TypeScript, so I’ve started writing up notes on this page to gather them in one place. Mock a function without early referencing Because of the way Jest hoists mocks, it’s easy to trigger a footgun where Jest complains about referencing something before initialisation. You can usually avoid that by wrapping the reference in an outer function call, so that it’s not immediately referenced: https://notestoself.dev/posts/udemy-course-progress-js-bookmarklet/ Sat, 05 Jun 2021 12:16:20 +0100 https://notestoself.dev/posts/udemy-course-progress-js-bookmarklet/ The Udemy video course UI doesn’t give you a lot of information about your progress through a course. I find it helpful to be able to see progress and the amount of time remaining in a course, so I made this JavaScript bookmarklet to display some figures about progress through a Udemy course. You need to be on the Udemy video player page when you trigger the bookmarklet. It will display progress stats in a browser alert. https://notestoself.dev/work/bulb/ Tue, 01 Jun 2021 00:00:00 +0000 https://notestoself.dev/work/bulb/ I worked as a Software Engineer on a contract basis for Bulb via YLD, using Kubernetes, Terraform, GCP, TypeScript, Node.js, Python, MySQL, PostgreSQL and other technologies. https://notestoself.dev/posts/sweep-complexity-under-infrastructure-carpet/ Sun, 23 May 2021 00:00:00 +0000 https://notestoself.dev/posts/sweep-complexity-under-infrastructure-carpet/ Some projects seem to get into the habit of keeping the application code relatively clean and simple by hiding complexity in the infrastructure layer. My made up term for this is sweeping complexity under the infrastructure carpet. It feels appropriate that this is a bit of a mouthful to say. This anti-pattern seems to creep in via an attitude that complexity in infrastructure “doesn’t count”, so it’s OK to let it grow and fester there while the application code appears to be more healthy. https://notestoself.dev/posts/mysql-timestamp-default-update-current-timestamp/ Tue, 18 May 2021 00:00:00 +0000 https://notestoself.dev/posts/mysql-timestamp-default-update-current-timestamp/ A quick note about a footgun in MySQL with timestamp colummns, which I happened to notice when using migrations in the knex.js library. TLDR: older versions of MySQL (<= 5.7) default to adding a DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP clause to timestamp columns unless otherwise specified. Documentation here: https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_explicit_defaults_for_timestamp For example, if you create a table like this in older versions of MySQL: CREATE TABLE `foobar_table` ( `foobar_column` timestamp ) You’ll actually get a column like this: https://notestoself.dev/posts/typescript-branded-string-type-utility-safety/ Wed, 28 Apr 2021 00:00:00 +0000 https://notestoself.dev/posts/typescript-branded-string-type-utility-safety/ Here’s an easy way to get branded string types in TypeScript. This allows the compiler to enforce minimal type safety on strings with an identity such as a User ID or Post ID. export type Brand<K, T extends string> = K & { [P in T]: never }; This can be helpful when you have multiple kinds of IDs that are all strings at runtime, and you want to avoid accidentally passing them in the wrong place. https://notestoself.dev/posts/static-site-full-text-search-hugo-s3-python-js/ Mon, 26 Apr 2021 00:00:00 +0000 https://notestoself.dev/posts/static-site-full-text-search-hugo-s3-python-js/ Static sites are great for their speed and ease of deployment at low cost. One thing they might seem to miss out on is full-text search, but you can actually achieve reasonable full-text search functionality on a static site with a fairly simple solution. This approach can be expanded in various ways to offer more functionality, but the basic core of it is: Use Python to iterate all of the searchable items, and create an index. https://notestoself.dev/posts/hugo-default-page-resource-social-image/ Fri, 23 Apr 2021 00:00:00 +0000 https://notestoself.dev/posts/hugo-default-page-resource-social-image/ Here’s a snippet of Hugo template language that I use to try and select the best social meta data image for each page on a static site: {{ if isset .Params "meta_image" }} <meta property="og:image" content="{{ .Params.meta_image | absURL }}"/> <meta name="twitter:image" content="{{ .Params.meta_image | absURL }}"> <link rel="image_src" href="{{ .Params.meta_image | absURL }}"/> <meta itemprop="image" content="{{ .Params.meta_image | absURL }}"> {{ else if (.Resources.ByType "image") | len }} {{ $firstImage := index (. https://notestoself.dev/posts/ubuntu-21.04-connect-old-modp1024-l2tp-ipsec-vpn/ Thu, 01 Apr 2021 00:00:00 +0000 https://notestoself.dev/posts/ubuntu-21.04-connect-old-modp1024-l2tp-ipsec-vpn/ A client of Kensio Software is using an L2TP VPN with Ipsec that’s using an obsolete DH2 (modp1024) algorithm. This algorithm was considered insecure as far back as 2001, and has now been deprecated in encryption libraries such as libreswan. Most of the permanent engineers at the client are on MacOS, which still supports DH2 / modp1024 algorithm out of the box. I was able to get the VPN connected on a Macbook, but still wanted to try and get it working on Ubuntu as that’s my main working OS. https://notestoself.dev/posts/beware-sql-average-empty-days/ Tue, 23 Feb 2021 00:00:00 +0000 https://notestoself.dev/posts/beware-sql-average-empty-days/ It’s pretty common to use SQL to extract sales summaries from tables that represent sales orders as rows. These kinds of queries often use the AVG() aggregate function to summarise the sales volume. There is a major pitfall when doing this, though! We might run a sub-query like this to get daily revenue totals from the orders table: SELECT order_date, SUM(order_total) AS total_revenue FROM orders GROUP BY order_date Those aggregated revenue query results might look something like this: https://notestoself.dev/posts/mysql-generate-date-range/ Sat, 23 Jan 2021 00:00:00 +0000 https://notestoself.dev/posts/mysql-generate-date-range/ It’s often useful to have a range of contiguous dates when building SQL queries, for example by using a left join to ensure that data from other tables that might have missing dates will get a null value in those rows instead of skipping the rows entirely. Postgres has its wonderful generate_series function which can solve a lot of these problems. Unfortunately MySQL doesn’t have the same function. You can generate a range of contiguous dates in MySQL with a query like this: https://notestoself.dev/posts/simple-boto3-mock-pytest-monkeypatch/ Wed, 20 Jan 2021 17:13:05 +0000 https://notestoself.dev/posts/simple-boto3-mock-pytest-monkeypatch/ If you’re anything like me, then you might find mocking dependencies in Python tests quite fiddly and unintuitive (it’s certainly less straightforward than Jest in TypeScript). I needed to mock one part of the Python boto3 library for AWS, and I got it working using Pytest’s monkeypatch. This shows a test for fetching a value from a JSON-encoded secret string in AWS Secrets Manager, but you can extrapolate from this roughly how to mock out other boto3 APIs in a similar way. https://notestoself.dev/reading/big-sister-little-sister-red-sister/ Fri, 01 Jan 2021 00:00:00 +0000 https://notestoself.dev/reading/big-sister-little-sister-red-sister/ https://notestoself.dev/reading/stubborn-attachments/ Fri, 01 Jan 2021 00:00:00 +0000 https://notestoself.dev/reading/stubborn-attachments/ https://notestoself.dev/posts/eslint-custom-rules/ Wed, 23 Dec 2020 00:00:00 +0000 https://notestoself.dev/posts/eslint-custom-rules/ Here’s an example of a custom eslint rule that detects usage of the knex library in a unit test file and reports it as a linter error. module.exports = { rules: { 'no-db-unit-tests': { create: function (context) { return { ImportDeclaration(node) { const fileName = context.getFilename(); if (node.specifiers) { node.specifiers.map((item) => { if ( item.local.name === 'knex' && fileName.includes('unit.spec') ) { return context.report( node, item.loc, `Do not use ${item.local.name}in unit test file ${fileName}`, ); } return null; }); } }, }; }, }, }, }; https://notestoself.dev/reading/where-s-my-flying-car/ Tue, 01 Dec 2020 00:00:00 +0000 https://notestoself.dev/reading/where-s-my-flying-car/ https://notestoself.dev/posts/jest-mock-reference-error-cannot-access-before-initialization/ Wed, 18 Nov 2020 00:00:00 +0000 https://notestoself.dev/posts/jest-mock-reference-error-cannot-access-before-initialization/ When using Jest mocks, it’s easy to hit this error: ReferenceError: Cannot access '...' before initialization This is a bit vague and difficult to figure out if you don’t know what the issue is. This error is due to the way Jest hoists mocks to the top. For example if you have this in a test file: const fooServiceFooFunction = jest.fn(); jest.mock("@foo-provider/some-library", () => ({ fooService: () => ({ foobar: fooServiceFooFunction, }), })); It looks like it should work fine as the fooServiceFooFunction mock function is initialised before jest. https://notestoself.dev/posts/interview-candidate-should-ask-interviewer-algorithm-question/ Thu, 12 Nov 2020 00:00:00 +0000 https://notestoself.dev/posts/interview-candidate-should-ask-interviewer-algorithm-question/ Here’s a potential idea to improve the technical interviews that you run: if you use algorithm questions in interviews, ask candidates to ask you an algorithm question as well. I think it could be beneficial to tell candidates that they should prepare an algorithm question to pose to their interviewer and work through together in the same way. This could have a similar vibe to the general questions at the end of the interview, when we ask candidates if they have any questions for us. https://notestoself.dev/posts/etsy-api-php-oauth-listing-image-upload/ Fri, 23 Oct 2020 00:00:00 +0000 https://notestoself.dev/posts/etsy-api-php-oauth-listing-image-upload/ After figuring out how to upload Etsy listing images in Python for one project, I needed to do the same thing in PHP for another. Despite Etsy itself being implemented in PHP (as far as I know), and the examples in their documentation being in PHP, this turned out to be tricky. First up, the example for uploading listing images in Etsy’s documentation is plain wrong. It doesn’t work. Don’t waste any more time trying to get that example to work. https://notestoself.dev/posts/easy-encrypted-git-gcrypt-gnupg/ Sun, 23 Aug 2020 00:00:00 +0000 https://notestoself.dev/posts/easy-encrypted-git-gcrypt-gnupg/ I’ve been looking at options for keeping personal notes and diary entries stored securely and synced across devices. An encrypted Git repo in Github seems like a good potential option for this. It turns out this is pretty straightforward to achieve using git-remote-gcrypt. This sets up a Git remote that is encrypted while the local repository is kept in plaintext and used normally. Check if you have a GPG key already: https://notestoself.dev/posts/refactor-terraform-state-files-gcp-gcs/ Sun, 23 Aug 2020 00:00:00 +0000 https://notestoself.dev/posts/refactor-terraform-state-files-gcp-gcs/ Recently I needed to refactor some Terraform configuration, which had grown into quite a large single state file, into smaller separate directories with their own state files. This is a little bit tricky but the Terraform CLI tool helps a lot. The first thing is to get the existing single state file JSON: terraform state pull | tee terraform.tfstate This should work for any storage backend. Alternatively you can use gsutil to copy the Terraform state file from a GCS bucket: https://notestoself.dev/reading/the-great-successor/ Sat, 01 Aug 2020 00:00:00 +0000 https://notestoself.dev/reading/the-great-successor/ https://notestoself.dev/posts/raw-html-shortcode-hugo/ Tue, 28 Jul 2020 00:00:00 +0000 https://notestoself.dev/posts/raw-html-shortcode-hugo/ Here’s a little Hugo shortcode for including raw HTML within your Markdown content. Just create a shortcode file at e.g. layouts/shortcodes/rawhtml.html with this content: {{.Inner}} Now in your Markdown content you can include whatever raw HTML you want, e.g.: {{< rawhtml >}} <button class="buttonFromMarkdown">Click Me!</button> <script> document.querySelector(".buttonFromMarkdown").onclick = function (event) { console.log("Clickety click"); event.target.style = "background-color: #77dd77"; }; </script> <style> .buttonFromMarkdown { padding: 1em; cursor: pointer; } </style> {{< /rawhtml >}} Which results in: https://notestoself.dev/posts/how-benefit-technical-reading-anki/ Sun, 26 Jul 2020 00:00:00 +0000 https://notestoself.dev/posts/how-benefit-technical-reading-anki/ Anki is one of my favourite pieces of software ever, and it’s great to see that it seems to be getting more and more attention as a valuable tool. I often do a kind of learning-and-note-taking exercise with Anki when reading technical content, which I’m writing up here. It’s not complicated or dogmatic, and the current version boils down to three tactics: Cloze deletion for easy conversion of text into notes. https://notestoself.dev/posts/pivot-join-19-million-lines-csv-unix-tools/ Thu, 11 Jun 2020 00:00:00 +0000 https://notestoself.dev/posts/pivot-join-19-million-lines-csv-unix-tools/ I’m participating in Kodoeba, an open hackathon around the Tatoeba example sentence project. My plan is to incorporate Mandarin-English example sentence pairs from Tatoeba into the Chinese example sentence search on Chinese Boost, mainly to allow searching them via pinyin with or without tone-marks. The first challenge was actually to extract the Mandarin-English example sentence pairs from Tatoeba’s data files. The data is structured in three separate TSV files (tab-separated values, equivalent to CSV): https://notestoself.dev/posts/productivity-random-numbers/ Mon, 01 Jun 2020 00:00:00 +0000 https://notestoself.dev/posts/productivity-random-numbers/ One little productivity trick I use is to select tasks or work with a random number. The simplest form is numbering a list of things you could do, then selecting one randomly. This can be applied to various things you might want to work on: Items on a todo list. Chapters or pages in a textbook. Tickets in a work-tracking system. A list of projects, with sub-lists of work items. https://notestoself.dev/posts/simplicity-resistance/ Mon, 01 Jun 2020 00:00:00 +0000 https://notestoself.dev/posts/simplicity-resistance/ Here’s another term that ties in with the ideas in “boring considered beneficial”, “fee justification” and “problem maximisation”: simplicity resistance. When something is too easy, we dislike the sensation, and try to make it more complicated in order to soothe that. This is a kind of cognitive dissonance around finding solutions to problems. We think “it can’t be that easy”, “it’s too good to be true” and so on, and start looking for the more complicated solution that the problem at hand surely justifies. https://notestoself.dev/posts/deploy-hugo-site-github-actions-s3-cloudfront-aws-iam/ Thu, 28 May 2020 00:00:00 +0000 https://notestoself.dev/posts/deploy-hugo-site-github-actions-s3-cloudfront-aws-iam/ GitHub Actions are a nice way to get “push to deploy” behaviour for a static site. Here’s a quick run-through of how I have it working for Hugo sites being deployed to AWS. This assumes you have created an S3 bucket and Cloudfront distribution for the site, and have pointed the domain name to the Cloudfront distribution. It’s best to have an isolated IAM user for deploying with minimal permissions. Create a new policy in AWS IAM, and give it minimal permissions for deploying the site. https://notestoself.dev/posts/code-like-conversations/ Thu, 28 May 2020 00:00:00 +0000 https://notestoself.dev/posts/code-like-conversations/ This is a potential rule-of-thumb subjective metric for assessing how well you’re directing your software development resources: Does your code read like your conversations? What this means is, does the code you’re writing overlap closely with the conversations you have about the product and the problems it solves? For example, if you talk about “free month of foobar with the XYZ subscription”, then it’s probably a good sign if those words appear a lot in the code. https://notestoself.dev/posts/lack-correlation-no-causation/ Sat, 23 May 2020 00:00:00 +0000 https://notestoself.dev/posts/lack-correlation-no-causation/ Correlation does not imply causation, but a lack of correlation does imply a lack of causation. The famous maxim that “correlation does not imply causation” seems to get over-interpreted into some kind of suggestion that correlation and causation can never be related in any way. One aspect of that is that a lack of correlation does imply that there is no causation effect going on. If there was a causative relationship between two things, we would expect to see correlation. https://notestoself.dev/posts/engineer-pain-vs-user-pain/ Mon, 18 May 2020 00:00:00 +0000 https://notestoself.dev/posts/engineer-pain-vs-user-pain/ A lot of software engineering teams talk about “pain” on a regular basis. Usually this refers to difficulties around technical debt, design decisions, refactoring and so on. It rarely seems to refer to issues that affect users or the rest of the business. For example, moving to microservices is often touted as a way to “reduce pain”. Perhaps we can reduce pain by putting HTTP calls in between different parts of the system, but it’s rare that this improves things for users in any way. https://notestoself.dev/posts/crystal-ball-development/ Sat, 16 May 2020 00:00:00 +0000 https://notestoself.dev/posts/crystal-ball-development/ There’s a well known software development maxim called YAGNI: “You Ain’t Gonna Need It”. YAGNI is a general rule for deciding whether or not to implement something or add complexity. It’s designed to give some resistance against the temptation to spuriously add features or try to solve hypothetical future problems. Applying a bit of inversion to YAGNI, you get something that might be called “crystal ball development”. This describes the attempt to make magic predictions about the future and apply them to software development. https://notestoself.dev/posts/problem-maximisation/ Fri, 08 May 2020 00:00:00 +0000 https://notestoself.dev/posts/problem-maximisation/ A common metaphor about the potential dangers of AI is called “paperclip maximisation”. The idea is that an AI tasked with producing paperclips could figure out that it can produce more paperclips by e.g. taking over the entire galaxy to turn as much matter as possible into paperclips. Sometimes I wonder if software engineers are prone to a similar issue. We believe we’re tasked with solving as many problems as possible, and realise that if we increase the number of problems, then we can increase the number of solutions. https://notestoself.dev/posts/python-script-automate-static-hugo-site-post-images/ Sun, 03 May 2020 00:00:00 +0000 https://notestoself.dev/posts/python-script-automate-static-hugo-site-post-images/ It’s good to have a general post image on content sites, as it gives more of an identity to each page, and works well for social media sharing images. While NotesToSelf.dev doesn’t tend to have images, my Chinese learning site and satirical tech news site have an image for each post. Unsplash is a nice source of general royalty-free images that you can use for this if you give attribution. https://notestoself.dev/posts/easy-social-media-images-and-markup-with-hugo/ Sun, 19 Apr 2020 00:00:00 +0000 https://notestoself.dev/posts/easy-social-media-images-and-markup-with-hugo/ Twitter Cards and Facebook’s OpenGraph provide semi-standardised markup for describing web content to social media and other aggregators. This can be quite handy for blogs and other text content, as you can provide default information when your content is shared. This is a nice way to make sure your posts get a reasonable cover image when shared to someone’s social timeline. Here’s the Hugo / Go HTML template I’m currently using here to add some basic sharing markup to each post as part of the HTML head: https://notestoself.dev/posts/abstraction-castles/ Sat, 18 Apr 2020 00:00:00 +0000 https://notestoself.dev/posts/abstraction-castles/ “…some service maintained by one person that’s built their little impregnable abstraction castle because nobody else had to work on it so they could just yak-shave and bike-shed with their self.” [sic] https://news.ycombinator.com/item?id=16201126 That Hacker News comment struck a chord with me, as I think the phrase “impregnable abstraction castle” perfectly captures something that is commonly encountered in software. An abstraction castle appears as follows. An area of a codebase seems to develop into a fortification that is resistant to change, apparently designed to defend against all envisioned future scenarios and to keep out every encroachment of the outside world that the designers could imagine. https://notestoself.dev/posts/laravel-signed-routes-order-confirmation-security/ Thu, 16 Apr 2020 00:00:00 +0000 https://notestoself.dev/posts/laravel-signed-routes-order-confirmation-security/ The order confirmation page on our 3D cards shop is shown to a customer after they complete checkout, and is also linked to in the confirmation and dispatch emails that they receive. Importantly, this page is not authenticated; we don’t want a heavy sign up / sign in process for customers as that tends to be annoying on other ecommerce systems. I just want to buy some pop-up cards – why do I need to create an account and remember a password etc? https://notestoself.dev/posts/specify-city-timezones-not-offsets/ Sat, 11 Apr 2020 00:00:00 +0000 https://notestoself.dev/posts/specify-city-timezones-not-offsets/ Dates and times are notoriously difficult to get right in software. Many of the difficulties come from timezones and how unintuitive they are to think about. This is made worse by timezone offsets that change during the year for the same place, such as British Summer Time. It’s easier to get timezones right when you specify a region/city timezone, such as Europe/London. Trying to think about and specify timezone offsets such as +01:00 is more likely to go wrong. https://notestoself.dev/posts/simple-readable-meaningful-jest-test-style/ Mon, 06 Apr 2020 00:00:00 +0000 https://notestoself.dev/posts/simple-readable-meaningful-jest-test-style/ I usually write tests with a Given, When, Then structure to keep the test focused on documenting and confirming the important behaviour of whatever is being tested. Simple comments are good for this and provide an easy-to-follow visual structure: test("some defined behaviour", () => { // Given the fandango is combobulated; // When we discombobulate the fandango; // Then the fandango should be discombobulated. }) It’s important to keep tests clear and readable. https://notestoself.dev/posts/fastmail-multi-domain-custom-email-service/ Sun, 05 Apr 2020 00:00:00 +0000 https://notestoself.dev/posts/fastmail-multi-domain-custom-email-service/ I run a few side projects on various domains, and often want to be able to send and receive email on those domains. Over the years I’ve used a few different solutions for this: Gmail aliases, Mailgun just for automated sending, AWS SES just for automated sending and AWS Workmail for manual sending and receiving. AWS Workmail has been fine, but ends up being a bit pricey for this kind of usage at $4 USD / user / month. https://notestoself.dev/posts/random-choice-hugo-template/ Fri, 03 Apr 2020 00:00:00 +0000 https://notestoself.dev/posts/random-choice-hugo-template/ I run a couple of sites related to my Chinese studies: an East Asian Studies blog, and a Chinese learning resource site. The content is all free so I have some simple affiliate image links on the pages to cover the bills. These are just straight forward <a><img></a> elements with no tracking or anything obnoxious at all. Even then, I don’t want to have too many adverts on a single page as it would still be a bit much. https://notestoself.dev/posts/cross-post-e2e-serverless-testing-driver-pattern/ Thu, 02 Apr 2020 00:00:00 +0000 https://notestoself.dev/posts/cross-post-e2e-serverless-testing-driver-pattern/ (Cross-posted from https://blog.freetrade.io/end-to-end-serverless-testing-with-the-driver-pattern-bcf2d22bf73a) In September 2019, Freetrade’s test code consisted of a large number of fast unit tests and a smaller number of end-to-end API tests. These make real HTTP requests against the platform and assert on the results. The unit tests confirm that individual classes are behaving correctly and improve the design of interfaces, while the API tests confirm that higher-level business logic is working correctly. This is not a bad combination and at the time already gave us decent test coverage of a large proportion of our platform. https://notestoself.dev/posts/choosing-dates-times-tests/ Mon, 30 Mar 2020 00:00:00 +0000 https://notestoself.dev/posts/choosing-dates-times-tests/ A little note to avoid bugs in tests, or missing bugs: don’t use the current date or time when writing unit tests. When making up data for tests, it’s common to just use today’s date as it’s easy to think up. This can catch you out when there is actually some hidden dependency on the real time and your test starts failing the next day, month or year. The test is currently passing by coincidence, which is hard to distinguish from a real pass if you’re not careful. https://notestoself.dev/posts/software-engineers-investment-managers-fee-justification/ Thu, 12 Mar 2020 00:00:00 +0000 https://notestoself.dev/posts/software-engineers-investment-managers-fee-justification/ It’s widely known (although somehow still controversial) that investment managers who try to manually pick stocks cannot beat the market the vast majority of the time: “Active fund managers trail the S&P 500 for the ninth year in a row in triumph for indexing” | CNBC “Stock-Picking Fund Managers Are Even Worse Than We Thought At Beating the Market” | Fortune “More evidence that it’s very hard to ‘beat the market’ over time, 95% of finance professionals can’t do it” | AEI “Your robo-advisor probably won’t be able to beat the stock market. https://notestoself.dev/posts/decision-fatigue-and-job-interviewing/ Mon, 09 Mar 2020 00:00:00 +0000 https://notestoself.dev/posts/decision-fatigue-and-job-interviewing/ I was reading about decision fatigue and wondering if it accounts for more major events than we give it credit for. The idea is at least spreading a little bit in certain circles, but doesn’t get the wider attention it should. “One research study found that the decisions judges make are strongly influenced by how long it has been since their last break. “We find that the percentage of favorable rulings drops gradually from ≈65% to nearly zero within each decision session and returns abruptly to ≈65% after a break. https://notestoself.dev/posts/sql-puzzles-2-absentees/ Mon, 02 Mar 2020 00:00:00 +0000 https://notestoself.dev/posts/sql-puzzles-2-absentees/ The second puzzle in Joe Celko’s SQL Puzzles is about tracking employee absenteeism. The initial table structure is like this: CREATE TABLE personnel ( emp_id integer PRIMARY KEY ); CREATE TABLE excuse_list ( reason_code varchar(40) UNIQUE ); CREATE TABLE absenteeism ( emp_id integer NOT NULL REFERENCES personnel (emp_id), absent_date date NOT NULL, reason_code varchar(40) NOT NULL REFERENCES excuse_list (reason_code), severity_points integer NOT NULL CONSTRAINT severity_points_range CHECK (severity_points BETWEEN 1 AND 4), PRIMARY KEY (emp_id, absent_date) ); There are already some nice properties to notice in this schema: https://notestoself.dev/posts/you-are-not-doing-test-driven-development/ Sun, 01 Mar 2020 00:00:00 +0000 https://notestoself.dev/posts/you-are-not-doing-test-driven-development/ Test Driven Development can be a business process, not just a development workflow. It can start much earlier in the software engineering process than when implementation starts and code is first written. Quite a few engineering teams confuse TDD for doing any testing at all, but even teams that know TDD is a stronger process than that still limit its potential by confining it to an implementation workflow detail that is trivial at the level of the business or software team. https://notestoself.dev/posts/rendering-html-email-version-of-articles-with-hugo/ Wed, 26 Feb 2020 00:00:00 +0000 https://notestoself.dev/posts/rendering-html-email-version-of-articles-with-hugo/ I recently started using Mailcoach, mainly for sending emails to the subscriber list for Chinese Boost. Mailcoach’s HTML email templating abilities are pretty basic, but that’s fine. I’m happy to use simple HTML emails, and found this HTML email template that looks like it will do the job nicely. As the content on Chinese Boost is managed with Hugo, it would be nice to have Hugo also be responsible for generating HTML email content for each article that can then just be copy-pasted into Mailcoach for sending to subscribers. https://notestoself.dev/posts/sql-puzzles-1-fiscal-year-tables/ Wed, 26 Feb 2020 00:00:00 +0000 https://notestoself.dev/posts/sql-puzzles-1-fiscal-year-tables/ The first puzzle in SQL Puzzles is about adding strong constraints to ensure correctness of data in a table that looks like this: CREATE TABLE fiscal_years ( fiscal_year integer, start_date date, end_date date ) Making all the columns NOT NULL is a good default first step as it avoids a large source of errors. CREATE TABLE fiscal_years ( fiscal_year integer NOT NULL, start_date date NOT NULL, end_date date NOT NULL ) We can prevent other errors by adding UNIQUE constraints to each of the three columns, as we don’t want any shared values across rows for them: each fiscal year should only be described once, and no fiscal year can share a start date or end date with another. https://notestoself.dev/posts/setting-up-a-mailcoach-app-on-aws-with-laravel-forge/ Sun, 23 Feb 2020 00:00:00 +0000 https://notestoself.dev/posts/setting-up-a-mailcoach-app-on-aws-with-laravel-forge/ I’ve been looking forward to the release of Mailcoach, which is a self-hosted mailing list manager app. I have a few mailing lists, most of which are tiny, but one of which has over 3000 subscribers. This puts it over the threshold of expensive subscription options in MailChimp. MailChimp wants over 70 USD / month to run this single list, whereas a Mailcoach license is a one-off payment of 99 USD. https://notestoself.dev/posts/better-alive-than-right/ Wed, 19 Feb 2020 00:00:00 +0000 https://notestoself.dev/posts/better-alive-than-right/ “Better alive than right” is a kind of proverb I try to remind myself of when cycling in London on a daily basis. A sizeable minority of drivers in London have a disregard for people on bikes and break traffic rules in various ways around them. A common one is drivers pulling out from a junction in front of someone on a bike, which requires the person cycling to stop to avoid a collision, even though the driver is at fault. https://notestoself.dev/posts/eating-the-recipe/ Mon, 10 Feb 2020 00:00:00 +0000 https://notestoself.dev/posts/eating-the-recipe/ “Eating the recipe” is a term I’ve made up to try and describe the kind of mistakes that get made by mixing up levels of abstraction. Humans do not struggle with the distinction between a recipe and the food that the recipe describes how to make. We would not make the mistake of eating a sheet of paper that describes how to make spaghetti, instead of eating the spaghetti. We do, however, design and use systems that make this kind of mistake a lot, or make it easy for humans to make this kind of mistake. https://notestoself.dev/posts/personal-bike-shedding/ Sun, 09 Feb 2020 00:00:00 +0000 https://notestoself.dev/posts/personal-bike-shedding/ The concept of bike shedding is one of my favourites, as once you’ve learnt about it, it won’t be long before you see it in the wild at most organisations. The standard explanation (from Wikipedia), goes like this: “Parkinson provides the example of a fictional committee whose job was to approve the plans for a nuclear power plant spending the majority of its time on discussions about relatively minor but easy-to-grasp issues, such as what materials to use for the staff bike shed, while neglecting the proposed design of the plant itself, which is far more important and a far more difficult and complex task. https://notestoself.dev/posts/write-code-that-is-easy-to-delete/ Sat, 08 Feb 2020 00:00:00 +0000 https://notestoself.dev/posts/write-code-that-is-easy-to-delete/ It’s hard to write software that is easy to maintain and adapt to change. There’s a multitude of maxims to try and tease out general solutions to this, or at least approaches that can mitigate the problem. This point did the rounds on Hacker News and elsewhere: “Write code that is easy to delete, not easy to extend.” – programmingisterrible This is one of the better maxims for producing maintainable and adaptable software. https://notestoself.dev/posts/generate-business-name-ideas-bash-linux/ Tue, 04 Feb 2020 00:00:00 +0000 https://notestoself.dev/posts/generate-business-name-ideas-bash-linux/ I’m setting up a small side-business selling wall art prints in the UK, and needed to think of a short, reasonably unique name. After Googling for word lists and name generators, I thought it would actually work best to get Bash to generate a list of name ideas for me, keeping the words to a limited length and randomising them to avoid fatigue when reading the list. The above can be done with this command on Ubuntu Linux: https://notestoself.dev/posts/dev-workflow-business-process-too/ Mon, 03 Feb 2020 00:00:00 +0000 https://notestoself.dev/posts/dev-workflow-business-process-too/ Your dev workflow is a business process too. There’s a funny phenomenon in a lot of companies that hire software engineers, where people draw a dividing line between “engineering” and “the business”. This is even reinforced in a lot of software engineering books and other media, which talk about “business people” and so on. I always think of engineering as very much a part of this apparently separate “business”, but that’s at least a whole other blog post. https://notestoself.dev/posts/firebase-driver-testing-simple-jest-mock/ Thu, 23 Jan 2020 00:00:00 +0000 https://notestoself.dev/posts/firebase-driver-testing-simple-jest-mock/ While working at Freetrade, I made the ts-firebase-driver-testing library to allow rapid testing and debugging of Firebase applications (other engineers have contributed to it since then). The initial plan with the library was to abstract out the Firebase SDK as a set of plain interfaces, and then provide a single-process in-memory implementation of those interfaces. This worked fine and it does let you swap out Firebase when running tests, which was the main goal. https://notestoself.dev/posts/extract-interfaces-dont-pre-plan-them/ Mon, 20 Jan 2020 00:00:00 +0000 https://notestoself.dev/posts/extract-interfaces-dont-pre-plan-them/ The Interface Segration Principle is one of the more overlooked parts of SOLID. I think this might be because it gets seen as a restatement of the Single Responsibility Principle, and not as the separate idea that it actually is. Interface segration is about keeping interfaces lean and specific to the smallest possible purpose, as suggested by the formal definition: “No client should be forced to depend on methods it does not use. https://notestoself.dev/posts/code-should-be-sequence-diagram/ Sat, 18 Jan 2020 00:00:00 +0000 https://notestoself.dev/posts/code-should-be-sequence-diagram/ I love sequence diagrams and other kinds of visual documentation of a system, especially ones that are committed in the repo alongside the application code. PlantUML is a nice tool for this as there’s a simple mark-up language for creating and editing diagrams which sits nicely in a Git repo. You can commit and review changes to diagrams as you do with the rest of the code. Diagrams are a kind of comment, though, and so really they should only supplement the code and not be a substitute for readability or clarity of the application code itself. https://notestoself.dev/posts/etsy-sales-csv-xero-import-python-script/ Sat, 11 Jan 2020 00:00:00 +0000 https://notestoself.dev/posts/etsy-sales-csv-xero-import-python-script/ We handle the accounting for our pop out cards shop using Xero. There are convenient pre-built integrations that handle Stripe and PayPal data in Xero, but nothing for Etsy, so I had to do a little Python script to generate a Xero import CSV file from the Etsy orders export CSV. Here’s the script for future reference: #!/usr/bin/python3 # https://www.etsy.com/uk/your/shops/YourShopName/download # Sold Orders # e.g. EtsySoldOrders2020.csv import argparse import csv from datetime import datetime from io import TextIOWrapper from typing import Dict SALES_ACCOUNT_CODE = "200" NOW = datetime. https://notestoself.dev/posts/be-careful-with-node-file-globbing-vs-bash-globbing/ Thu, 02 Jan 2020 00:00:00 +0000 https://notestoself.dev/posts/be-careful-with-node-file-globbing-vs-bash-globbing/ There’s a common gotcha when using commandline tools that have their own file globbing, such as Prettier in Nodejs. This command is supposed to run prettier against all the .ts files in all folders under src recursively; prettier --write src/**/*.ts What actually happens though is that bash expands the src/**/*.ts first, which will only go one folder deep instead of the intended unlimited recursion. The fix is simply to quote the globbing express so that bash passes it to prettier unscathed: https://notestoself.dev/reading/atlas-shrugged/ Wed, 01 Jan 2020 00:00:00 +0000 https://notestoself.dev/reading/atlas-shrugged/ https://notestoself.dev/reading/conversations-with-friends/ Wed, 01 Jan 2020 00:00:00 +0000 https://notestoself.dev/reading/conversations-with-friends/ https://notestoself.dev/reading/drink/ Wed, 01 Jan 2020 00:00:00 +0000 https://notestoself.dev/reading/drink/ https://notestoself.dev/reading/how-fascism-works/ Wed, 01 Jan 2020 00:00:00 +0000 https://notestoself.dev/reading/how-fascism-works/ https://notestoself.dev/reading/if-this-is-a-man-the-truce/ Wed, 01 Jan 2020 00:00:00 +0000 https://notestoself.dev/reading/if-this-is-a-man-the-truce/ https://notestoself.dev/reading/my-brilliant-friend/ Wed, 01 Jan 2020 00:00:00 +0000 https://notestoself.dev/reading/my-brilliant-friend/ https://notestoself.dev/reading/one-hundreds-years-of-solitude/ Wed, 01 Jan 2020 00:00:00 +0000 https://notestoself.dev/reading/one-hundreds-years-of-solitude/ https://notestoself.dev/reading/the-almanack-of-naval-ravikant/ Wed, 01 Jan 2020 00:00:00 +0000 https://notestoself.dev/reading/the-almanack-of-naval-ravikant/ https://notestoself.dev/reading/the-chinese-language-fact-and-fantasy/ Wed, 01 Jan 2020 00:00:00 +0000 https://notestoself.dev/reading/the-chinese-language-fact-and-fantasy/ https://notestoself.dev/reading/the-paper-menagerie/ Wed, 01 Jan 2020 00:00:00 +0000 https://notestoself.dev/reading/the-paper-menagerie/ https://notestoself.dev/reading/upheaval/ Wed, 01 Jan 2020 00:00:00 +0000 https://notestoself.dev/reading/upheaval/ https://notestoself.dev/reading/utopia-for-realists/ Wed, 01 Jan 2020 00:00:00 +0000 https://notestoself.dev/reading/utopia-for-realists/ https://notestoself.dev/reading/%E4%B8%B0%E4%B9%B3%E8%82%A5%E8%87%80/ Wed, 01 Jan 2020 00:00:00 +0000 https://notestoself.dev/reading/%E4%B8%B0%E4%B9%B3%E8%82%A5%E8%87%80/ https://notestoself.dev/posts/setting-up-a-hugo-static-site-inside-laravel-on-forge/ Mon, 23 Dec 2019 00:00:00 +0000 https://notestoself.dev/posts/setting-up-a-hugo-static-site-inside-laravel-on-forge/ Making a hybrid static + dynamic site can be quite a useful combination. This might sound like an oxymoron, but it simply means serving static pages from a site that also runs a dynamic web application. You can get some of the best of both worlds with this approach. My Chinese-learning site ChineseBoost uses this combination – a lot of the content pages, e.g. for Chinese grammar, are statically generated with Hugo, while dynamic parts of the site are served by a Laravel application. https://notestoself.dev/reading/song-of-solomon/ Sun, 01 Dec 2019 00:00:00 +0000 https://notestoself.dev/reading/song-of-solomon/ https://notestoself.dev/posts/easy-typed-test-mocks-with-jest-and-typescript/ Wed, 20 Nov 2019 00:00:00 +0000 https://notestoself.dev/posts/easy-typed-test-mocks-with-jest-and-typescript/ Here’s a quick and simple way to create type-safe test mocks with Jest in TypeScript: const fooMock: jest.Mocked<IFoo> = { fooMethod: jest.fn() } const testSubject = new ClientClass(fooMock); This has a few useful properties: The type system will enforce that the dependency is the correct type, i.e. that ClientClass is being passed the instance of IFoo that it requires. The type system will enforce that the mock implements the required interface, i. https://notestoself.dev/posts/hugo-taxonomy-term-next-prev-page-links/ Sat, 02 Nov 2019 00:00:00 +0000 https://notestoself.dev/posts/hugo-taxonomy-term-next-prev-page-links/ Something that can be slightly tricky to achieve within Hugo’s taxonomy system is linking to the previous and next page within a particular taxonomy term. For example, if you have a Series taxonomy, you might want to link to the previous and next page in that series on each page in it. One way to solve this is to loop through all the pages in the taxonomy term and grab the previous and next pages on the way. https://notestoself.dev/posts/flexible-test-model-factories-in-typescript/ Wed, 23 Oct 2019 00:00:00 +0000 https://notestoself.dev/posts/flexible-test-model-factories-in-typescript/ Tests are improved by being more readable, and concise tests are more readable. It’s good when a test has clear given, when, then steps, and the important details are plain to see in the test code. This is often obstructed by large amounts of set-up and boilerplate code in the test that distracts from the important details. Test model factories can reduce some of that boilerplate. For example, if we have a Client model that contains quite a lot of detail, setting one up in a test will be verbose. https://notestoself.dev/posts/make-a-deep-object-delta-diff-in-typescript/ Wed, 25 Sep 2019 00:00:00 +0000 https://notestoself.dev/posts/make-a-deep-object-delta-diff-in-typescript/ Building on the object path enumeration, we can also make a deep object diff or delta. E.g. given these two objects: { "a": { "b": { "c": "123" } }, "foo": "hello" } { "a": { "b": { "c": "123", "d": "456" } }, "foo": "hello" } We would make this delta diff: { "a": { "b": { "d": "456" } } } This uses a few utility functions aside from enumeratePaths, but it should be relatively clear what they do here: https://notestoself.dev/posts/enumerating-all-object-paths-in-typescript/ Mon, 23 Sep 2019 00:00:00 +0000 https://notestoself.dev/posts/enumerating-all-object-paths-in-typescript/ Here’s a small recursive TypeScript function to enumerate all the paths through an object. E.g. given this object: { "animals": { "lion": { "sound": "roar" }, "zebra": { "appearance": "stripes" }, "elephant": { "size": "large", "features": { "trunk": "yes", "tusks": "yes", "wings": "no" }, }, }, }, It enumerates these paths: [ "animals", "animals/lion", "animals/zebra", "animals/elephant", "animals/lion/sound", "animals/zebra/appearance", "animals/elephant/size", "animals/elephant/features", "animals/elephant/features/trunk", "animals/elephant/features/tusks", "animals/elephant/features/wings", ] The function: function enumeratePaths(value: any): string[] { let paths: string[] = [] if (value && typeof value === "object") { paths = Object. https://notestoself.dev/posts/can-you-have-multiple-instances-of-your-application-in-a-single-process/ Sun, 15 Sep 2019 00:00:00 +0000 https://notestoself.dev/posts/can-you-have-multiple-instances-of-your-application-in-a-single-process/ Here’s a question that might give some insight into the quality of your codebase: Can you have multiple instances of your application in a single process? By that I mean that you could easily create and run parts of the application and/or the whole thing in one process, without this causing bad behaviour. If so, this probably suggests that the design is quite well factored. It’s a similar idea to the questions in the Joel test, but for internal application structure. https://notestoself.dev/posts/deepin-screenshot-ubuntu-linux/ Sun, 18 Aug 2019 00:00:00 +0000 https://notestoself.dev/posts/deepin-screenshot-ubuntu-linux/ If you want a good screenshot tool on Ubuntu, Deepin Screenshot is probably the best one. It lets you drag-select a rectangular area of the screen, and then re-arrange the area until you’re happy with it. You can then annotate the screenshot selection with highlights, arrows, boxes and text. Finally, you can also choose how to save the resulting screenshot image, or copy it to your clipboard. Deepin made it super easy to make the personalised cards FAQ on our card shop, with a series of clear screenshots showing the process with the key parts highlighted. https://notestoself.dev/posts/hugo-showing-other-pages-taxonomy-term/ Thu, 01 Aug 2019 00:00:00 +0000 https://notestoself.dev/posts/hugo-showing-other-pages-taxonomy-term/ Here’s a quick HTML template snippet to do something I find is quite a common task in a Hugo site: show a list of other pages that share a taxonomy term with the current page. It’s probably a good idea to refresh your memory of how taxonomies work in Hugo if you’re feeling rusty. The snippet is just: {{ if isset .Params "tags" }} {{ $tagTitle := index .Params.tags 0 }} {{ $currentLink := . https://notestoself.dev/posts/python-generate-perspective-print-product-images/ Sun, 21 Jul 2019 00:00:00 +0000 https://notestoself.dev/posts/python-generate-perspective-print-product-images/ For Moment Wall Art (and other printed product sites in future), we need to show the customer not just the main print design itself but also mock-up images demonstrating how it would look in various settings. Applying a perspective skew to images like this can be handled by the Python Wand library, specifically with the perspective distortion function. I have a Python script that iterates through a glob of all product files in parallel, and generates these perspective-skewed mock up images where necessary. https://notestoself.dev/posts/easier-interface-segregation-in-typescript-even-better-than-go/ Mon, 15 Jul 2019 00:00:00 +0000 https://notestoself.dev/posts/easier-interface-segregation-in-typescript-even-better-than-go/ Preamble: Introduction to interface segregation (Skip this if you’re aware of what the Interface Segration Principle is.) The Interface Segration Principle is the I in the SOLID mnemonic. It’s conceptually one of the simplest, but is often overlooked, which is a shame as it’s a highly beneficial principle to follow in software design. The Interface Segration Principle is defined as “no client should be forced to depend on methods it does not use”. https://notestoself.dev/posts/start-with-given-when-then-tests/ Sat, 13 Jul 2019 00:00:00 +0000 https://notestoself.dev/posts/start-with-given-when-then-tests/ Whether or not you’re practicing test-driven development, it’s helpful to begin writing tests with the “Given, When, Then” structure. // Given we have a queued order; // When we cancel it; // Then the order should be cancelled. Getting the structure in with comments first has several benefits. Firstly, it lowers the mental effort to required to start. You only need to think about the desired behaviour in the simplest terms. https://notestoself.dev/posts/the-attacker-really-does-know-the-system-when-someone-leaves-your-company/ Sat, 13 Jul 2019 00:00:00 +0000 https://notestoself.dev/posts/the-attacker-really-does-know-the-system-when-someone-leaves-your-company/ A common maxim of computer security is “the attacker knows the system”. In other words, the system should still be secure even given a hypothetical attacker who knows exactly how it works. It’s the flip-side of relying on “security by obscurity”. In cryptography, the same concept is referred to as Kerckhoffs’s principle, but can be applied to systems in general. Most engineers are aware of this concept, but it’s also common to find a vague attitude that you can still “get away with” relying on little details that a hypothetical attacker won’t actually know, e. https://notestoself.dev/posts/fan-in-from-multiple-routes-to-a-single-vm-with-iptables-for-fix-on-quickfix/ Mon, 08 Jul 2019 00:00:00 +0000 https://notestoself.dev/posts/fan-in-from-multiple-routes-to-a-single-vm-with-iptables-for-fix-on-quickfix/ Recently we were trying to set-up some network configuration to route FIX traffic from a single Docker container on a VM instance through four of our VPN gateways, two of the other side’s VPN gateways and on to two destination machines on the other side. It’s somewhat complicated, but necessary due to some constraints being out of our control. The ultimate goal is to have failover redundancy in the system so that a different route can be used if one route is down. https://notestoself.dev/posts/get-failing-test-asap/ Mon, 08 Jul 2019 00:00:00 +0000 https://notestoself.dev/posts/get-failing-test-asap/ The “Specification driven development” post mentions the idea of a “a low-resistance workflow” that can be achieved with test-driven development. Overall that post is probably a bit dogmatic and over-eager about the idea of test-driven development, but the point about achieving a low-resistance workflow still seems valuable. A more trivial way of doing that is to use a simple workflow like this: Do you have a failing test? -> Make the test pass. https://notestoself.dev/posts/pretty-print-json-on-the-command-line-with-python/ Wed, 26 Jun 2019 00:00:00 +0000 https://notestoself.dev/posts/pretty-print-json-on-the-command-line-with-python/ I’m often dealing with little bits of JSON, and sometimes it’s helpful to pretty-print them for readability. Python has a handy tool to make that easy on the command line: json.tool. E.g. echo '{"foo": "bar", "thing": { "baz": 42 } }' | python -m json.tool prints: { "foo": "bar", "thing": { "baz": 42 } } If you’re on Mac, you can convert JSON in the clipboard to its pretty-printed form with: https://notestoself.dev/posts/be-careful-with-truthy-promises-in-typescript/ Sun, 23 Jun 2019 00:00:00 +0000 https://notestoself.dev/posts/be-careful-with-truthy-promises-in-typescript/ There’s a type of small “gotcha” bug that is easy to hit with promises for boolean values in JavaScript. The simplest example might be this: async function checkSomeCondition(): Promise<boolean> { return false } if (checkSomeCondition()) { // do something. } The if branch will always be executed, because the promise object returned by checkSomeCondition() will be evaluated as true, regardless of what it would resolve to. It’s a typo-level mistake, but is easy to make. https://notestoself.dev/posts/loop-try-catch/ Wed, 19 Jun 2019 00:00:00 +0000 https://notestoself.dev/posts/loop-try-catch/ A simple cause of failure in software systems is a single problematic item in a set causing the entire set to fail. Trivially, it looks like this: for (const item of items) { processItem(item) } When something goes wrong in processItem() and throws an error, either the whole set fails to process, or all the subsequent items fail. The trivial example has a trivial improvement: for (const item of items) { try { processItem(item) } catch (err) { // Do something useful with the error. https://notestoself.dev/posts/day-life-software-engineer-freetrade/ Sat, 01 Jun 2019 00:00:00 +0000 https://notestoself.dev/posts/day-life-software-engineer-freetrade/ (Cross-posted from https://blog.freetrade.io/a-day-in-the-life-of-a-software-engineer-at-freetrade-a39a590d7ffa) I joined Freetrade as Senior Software Engineer in October 2018. We’ve got a huge amount done in that time — the company’s moving fast! But behind the bustle of getting big features shipped, I wanted to give people an insight into what an individual day’s work looks like on the engineering team. Here’s my typical day as a Software Engineer at Freetrade. (My) commute I commute by bike 90% of the time. https://notestoself.dev/posts/a-bad-mix-of-bash-and-sh-with/ Thu, 23 May 2019 00:00:00 +0000 https://notestoself.dev/posts/a-bad-mix-of-bash-and-sh-with/ If you accidentally run a bash script with sh, you’re probably going to have a bad time. One little gotcha is the use of [[ (double square brackets) in bash. E.g.: if [[ -z "${PROJECT_ID}" ]]; then echo 'PROJECT_ID is empty, exiting' exit 1 fi Bash will check the condition as expected. Sh, on the other hand, will reject the double-brackets syntax that it is not aware of with a message like this: https://notestoself.dev/posts/generate-a-uuid-on-the-command-line-with-python/ Sat, 18 May 2019 00:00:00 +0000 https://notestoself.dev/posts/generate-a-uuid-on-the-command-line-with-python/ Sometimes I want to manually generate a UUID for some test or example data. It’s handy to have this Python snippet in Bash history to quickly do that: python -c 'import uuid; print(uuid.uuid4())' https://notestoself.dev/posts/python-national-insurance-number-nino-one-line-generator/ Tue, 23 Apr 2019 00:00:00 +0000 https://notestoself.dev/posts/python-national-insurance-number-nino-one-line-generator/ Here’s a Python one-liner for generating a random UK National Insurance Number (aka NINo) on the command line, which is useful for testing apps that use one: python -c 'import random; import string; p="ABCEGHJKLMNPRSTWXYZ"; print("".join([random.choice(p),random.choice(p)]+[random.choice(string.digits) for i in range(6)]+[random.choice("ABCD")]))' To make it easier to use: alias nino="python -c 'import random; import string; p=\"ABCEGHJKLMNPRSTWXYZ\"; print(\"\".join([random.choice(p),random.choice(p)]+[random.choice(string.digits) for i in range(6)]+[random.choice(\"ABCD\")]))'" https://notestoself.dev/posts/python-random-string-one-liner-for-cli/ Tue, 23 Apr 2019 00:00:00 +0000 https://notestoself.dev/posts/python-random-string-one-liner-for-cli/ Here’s a handy Python one-liner for generating a random string of letters and digits on the command line: python -c 'import random; import string; print("".join(random.choice(string.ascii_letters + string.digits) for i in range(30)))' To make it easier to use: alias randomstring="python -c 'import random; import string; print(\"\".join(random.choice(string.ascii_letters + string.digits) for i in range(30)))'" https://notestoself.dev/posts/extract-json-config-variables-from-the-airflow-ui/ Mon, 18 Mar 2019 00:00:00 +0000 https://notestoself.dev/posts/extract-json-config-variables-from-the-airflow-ui/ One way to configure Apache Airflow is with “Variables” that can be modified programmatically or via the Airflow UI. You can import the variables from a JSON file, again via the API or by uploading the file in the UI. Sensibly, Airflow does not provide a way to export these variables back out to a JSON file. However, if you find yourself in the situation where Airflow holds the correct values and you’re not confident you’ve got them stored consistently elsewhere, you might end up having to manually copy the variables back out of the Airflow UI. https://notestoself.dev/posts/serverless-integration-testing-freetrade/ Mon, 11 Mar 2019 00:00:00 +0000 https://notestoself.dev/posts/serverless-integration-testing-freetrade/ (Cross-posted from https://blog.freetrade.io/serverless-integration-testing-at-freetrade-5359fb0b0e57) Our tech stack at Freetrade is 99% serverless. This means that for the vast majority of our applications, the physical infrastructure, deployment and other operational concerns are largely abstracted away by the cloud provider (Google Cloud in our case). We focus on building what our users need, and our cloud provider focuses on what we need to keep it running smoothly at scale. Serverless computing has huge benefits in terms of allowing more resources to be devoted to solving user problems, but it also presents some different challenges. https://notestoself.dev/posts/qrencode-qr-code-cli-tool-ubuntu-linux/ Sat, 23 Feb 2019 00:00:00 +0000 https://notestoself.dev/posts/qrencode-qr-code-cli-tool-ubuntu-linux/ There’s something fun about QR codes, and I wanted an easy way generate them as SVGs. Sure enough there’s a tool in the Ubuntu repos to do that: sudo apt install qrencode It’s straightforward to use: qrencode -t SVG -o qr-code.svg 'whatever you want encoded' I wanted to generate a QR code to put on little business cards that we pack with the orders for our pop up card shop. The QR code should point to a URL including a discount coupon that gets automatically applied for customers who have arrived via that URL. https://notestoself.dev/posts/promise-batch-progress-meter-in-typescript/ Mon, 18 Feb 2019 00:00:00 +0000 https://notestoself.dev/posts/promise-batch-progress-meter-in-typescript/ Nodejs has a handy Promise.all function that awaits an entire batch of promises. However, sometimes you want to see how that batch of promises is progressing before the whole thing has finished. Here’s a rough-and-ready way to print a progress meter to stdout as a batch of promises completes in TypeScript: const promises = [1, 2, 3, 4, 5].map(async (i: number): Promise<string> => { return 'finished some work' }) let finishedCount = 0 promises. https://notestoself.dev/posts/duolingo-fandom-japanese-js-csv-extractor/ Sat, 16 Feb 2019 00:00:00 +0000 https://notestoself.dev/posts/duolingo-fandom-japanese-js-csv-extractor/ I’m casually going through the Japanese course in Duolingo, and found this handy wiki series on Duolingo Fandom Japanese with lists of all of the vocab. I wanted an easier way to import it into Anki, so I put together this little pastable JS script to export a CSV of the vocab on one page: (function () { const pattern = /^([\p{sc=Han}\p{sc=Hiragana}\p{sc=Katakana}？、，。]+)\s*\(([a-z\p{P}-]+)\)\s*=\s*([a-z\p{P}\(\) ?]+)(\s*\(([\p{sc=Han}\p{sc=Hiragana}\p{sc=Katakana}？、，。]+)\))?$/iu; const csvRows = ['日本語,ふりがな,ローマ字,英語']; document .querySelectorAll('h3 ~ ul:not([class]) > li:not([class])') . https://notestoself.dev/posts/hugo-series-taxonomy/ Wed, 30 Jan 2019 00:00:00 +0000 https://notestoself.dev/posts/hugo-series-taxonomy/ Hugo’s taxonomies feature is very flexible and covers a lot of use cases for organising static content. One thing I use it for on several static sites is to create a Series taxonomy that allows an easy grouping of posts into a sequence. Readers can see other posts in the series on each post, like a table of contents, they can see next and previous links within the series, and they can view a dedicated list page for the series as well. https://notestoself.dev/posts/please-comment-your-code/ Wed, 23 Jan 2019 00:00:00 +0000 https://notestoself.dev/posts/please-comment-your-code/ “Every comment represents a failure to make the code self explanatory.” &em; Robert C. Martin (Uncle Bob) One of the better-known sentiments from Uncle Bob’s Clean Code is that comments are bad*. More precisely, the idea is that comments are a last resort option to try and mitigate “bad code”. Sometimes this seems to lead to a logical fallacy that goes like this: “Comments are a sign of bad code, and my code can’t be bad, so I won’t write comments”. https://notestoself.dev/reading/concise-guide-ssl-tls-devops/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/concise-guide-ssl-tls-devops/ https://notestoself.dev/reading/antifragile-things-gain-disorder/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/antifragile-things-gain-disorder/ https://notestoself.dev/reading/children-of-god/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/children-of-god/ https://notestoself.dev/reading/don-quixote/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/don-quixote/ https://notestoself.dev/reading/exploring-cqrs-event-sourcing/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/exploring-cqrs-event-sourcing/ https://notestoself.dev/reading/history-of-the-decline-and-fall-of-the-roman-empire/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/history-of-the-decline-and-fall-of-the-roman-empire/ https://notestoself.dev/reading/hyperion-1/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/hyperion-1/ https://notestoself.dev/reading/jane-eyre/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/jane-eyre/ https://notestoself.dev/reading/origin-story-big-history-everything/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/origin-story-big-history-everything/ https://notestoself.dev/reading/systems-bible/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/systems-bible/ https://notestoself.dev/reading/the-confidence-game/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/the-confidence-game/ https://notestoself.dev/reading/the-most-human-human/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/the-most-human-human/ https://notestoself.dev/reading/new-silk-roads-present-future-world/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/new-silk-roads-present-future-world/ https://notestoself.dev/reading/silk-roads-new-history-world/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/silk-roads-new-history-world/ https://notestoself.dev/reading/the-three-body-problem/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/the-three-body-problem/ https://notestoself.dev/reading/why-we-sleep/ Tue, 01 Jan 2019 00:00:00 +0000 https://notestoself.dev/reading/why-we-sleep/ https://notestoself.dev/posts/finding-the-max-branch-sum-of-a-sequentially-represented-binary-tree-in-python/ Sun, 23 Dec 2018 00:00:00 +0000 https://notestoself.dev/posts/finding-the-max-branch-sum-of-a-sequentially-represented-binary-tree-in-python/ Given a binary tree represented sequentially as a list of integer node values, we want to identify whether the sum of the left branch or the sum of th right branch is larger. In other words we want to identify which branch has the maximum sum. For example, this binary tree: ···································· ················┌──┐················ 00··············│05│················ ················└──┘················ ··········╱··············╲·········· ········┌──┐············┌──┐········ 01······│04│············│06│········ ········└──┘············└──┘········ ······╱······╲······················ ····┌──┐····┌──┐···················· 02··│16│····│12│···················· ····└──┘····└──┘···················· ···································· Could be represented sequentially as [5, 4, 6, 16, 12]. https://notestoself.dev/posts/cross-post-coming-to-firebases-nosql-realtime-database-from-a-sql-background/ Fri, 21 Dec 2018 00:00:00 +0000 https://notestoself.dev/posts/cross-post-coming-to-firebases-nosql-realtime-database-from-a-sql-background/ (Cross-posted from https://blog.freetrade.io/coming-to-firebases-nosql-realtime-database-from-a-sql-background-b605b2910c4c) We use the Firebase Realtime Database at Freetrade, as part of our infrastructure on Google Cloud. The Firebase Realtime Database is a cloud-hosted NoSQL database that lets you store and sync data between your users in realtime. Firebase Realtime Database has a markedly different development philosophy to the more traditional SQL databases that many developers are used to. Here’s a rundown of what it’s like coming to Firebase Realtime Database from an SQL background. https://notestoself.dev/posts/spoonerism-generator-python-script/ Thu, 08 Nov 2018 00:00:00 +0000 https://notestoself.dev/posts/spoonerism-generator-python-script/ “What’s the Difference Between” is a childish word-play game where you give clues for other players to guess a spoonerism phrase pair. For example: Q: “What’s the difference between stress on a group, and an old locomotive?” A: One is team strain, the other’s a steam train. I’m part of a WhatsApp group that sends these back and forth on a daily basis (usually with less innocent contents than that example). https://notestoself.dev/posts/keep-elasticsearch-service-running-ubuntu-server/ Sun, 28 Oct 2018 00:00:00 +0000 https://notestoself.dev/posts/keep-elasticsearch-service-running-ubuntu-server/ A local ElasticSearch instance powers the Chinese example sentence search feature on Chinese Boost. This works fine, but the ElasticSearch instance has a habit of crashing about once a day. While I’ve been investigating why it crashes and how to prevent that from happening, I also wanted to have the EC2 Ubuntu server it runs on restart the ElasticSearch service automatically as a workaround. This can be done using systemd. https://notestoself.dev/posts/careful-cloudfront-s3-origin-url/ Tue, 23 Oct 2018 00:00:00 +0000 https://notestoself.dev/posts/careful-cloudfront-s3-origin-url/ Be careful with the S3 origin URL for a Cloudfront distribution. When using an S3 bucket with static website hosting enabled as the origin for a Cloudfront distribution, you must use the static website endpoint, and not the S3 bucket URL. These are different. The S3 bucket URL looks like this (don’t use this for the Cloudfront origin): https://s3.eu-west-2.amazonaws.com/foobar-bucket-name/ The S3 static website hosting URL looks like this (use this for the Cloudfront origin): https://notestoself.dev/posts/laravel-catalogue-data-report-command/ Tue, 23 Oct 2018 00:00:00 +0000 https://notestoself.dev/posts/laravel-catalogue-data-report-command/ The catalogue data model in our 3D pop up cards shop includes Product and Tag entities. Tags are a flat structure with a many-to-many relationship to Products, to allow some simple tagging to help customers find cards with particular elements. The Tag-Product relationship is managed manually in the backoffice UI, which is fine but can be error prone when we miss Products that should have a particular Tag. To make this easier to work with, I put together a quick command line tool in Laravel that identifies Products that seem to be missing an appriate Tag. https://notestoself.dev/work/freetrade/ Mon, 08 Oct 2018 00:00:00 +0000 https://notestoself.dev/work/freetrade/ I was Principal Software Engineer at Freetrade, primarily working on the platform but also some client-side work. Freetrade has a serverless architecture running Google Cloud Functions in Firebase using TypeScript. https://notestoself.dev/posts/chinese-poetry-python-script/ Sat, 06 Oct 2018 00:00:00 +0000 https://notestoself.dev/posts/chinese-poetry-python-script/ One of my pastimes is translating Chinese poetry and other text into English on my East Asia Student blog. The translations I make are more like annotations, as I try to adhere as closely as I can to the structure and literal meaning of the original text. The idea is that the translation is helpful for reading the original Chinese text, rather than being a full rendering into English. One aspect of this is the inclusion of pinyin and literal glosses for each line of the poem. https://notestoself.dev/posts/island-counting-algorithm-typescript/ Sun, 23 Sep 2018 00:00:00 +0000 https://notestoself.dev/posts/island-counting-algorithm-typescript/ A friend mentioned a code screen interview question he likes to ask based on counting the number of discrete “islands” in a grid: Given a square grid of cells, some of which are sea (0) and some of which are land (1), implement an island counter that identifies each contiguous area of land cells (an island) and counts the total number of such islands. This seems like quite a fun problem to solve so I had a go at it in TypeScript. https://notestoself.dev/posts/algodaily-22-count-the-planes/ Wed, 05 Sep 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-22-count-the-planes/ https://algodaily.com/challenges/count-the-planes This is the same as the island counting problem. function* enumerateGrid(grid) { if (!grid || grid.length < 1 || !grid[0].length) { return; } for (let x = 0; x < grid[0].length; x++) { for (let y = 0; y < grid[x].length; y++) { yield {x, y}; } } } function* enumerateNeighbours(grid, x, y) { for (const p of [{x:-1,y:0},{x:0,y:-1},{x:1,y:0},{x:0,y:1}]) { if (grid[x+p.x] && grid[x+p.x][y+p.y]) { yield {x: x+p.x, y: y+p. https://notestoself.dev/posts/unix-fizz-buzz-one-liner/ Mon, 27 Aug 2018 00:00:00 +0000 https://notestoself.dev/posts/unix-fizz-buzz-one-liner/ Given that “implement Fizz-Buzz in a language of your choice” is quite a common programming interview task, I was wondering if it would be possible to do it in one line on the command line using Unix tools. Unsurprisingly it is; here’s my version: seq 100 | awk '{ if ($0 % 15 == 0) { print "fizzbuzz" } else if ($0 % 3 == 0) { print "fizz" } else if ($0 % 5 == 0) { print "buzz" } else { print $0 } }' The seq at the beginning generates numbers from 1 to 100, and then the awk command does a basic if-else on each line to decide what to print. https://notestoself.dev/posts/algodaily-21-contiguous-subarray-sum/ Thu, 23 Aug 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-21-contiguous-subarray-sum/ https://algodaily.com/challenges/contiguous-subarray-sum There’s a fairly obvious brute-force O(n^2) way to do this by looping through every possible sub-array: function sum(numbers) { return numbers.reduce((sum, x) => sum + x, 0); } function* allSubArrays(numbers) { for (let i = 0; i <= numbers.length; i++) { for (let j = 0; j <= numbers.length; j++) { const slice = numbers.slice(i, j); if (slice.length > 0) { yield slice; } } } } function subarraySum(numbers, targetSum) { for (let sub of allSubArrays(numbers)) { if (sum(sub) === targetSum) { return true; } } return false; } This is O(n^2), so not great, but I can’t think of a different way to do it. https://notestoself.dev/posts/algodaily-20-uniqueness-of-arrays/ Wed, 08 Aug 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-20-uniqueness-of-arrays/ https://algodaily.com/challenges/uniqueness-of-arrays This is pretty trivial with the Set class in ECMAScript 2015. function uniques(items) { return [...new Set(items)]; } If that seems like cheating, then we can achieve this with a hash map and a new array: function uniques(items) { const seen = {}; const unique = []; for (let x of items) { if (seen[x] === undefined) { unique.push(x); seen[x] = true; } } return unique; } This is O(n). https://notestoself.dev/posts/algodaily-19-fast-minimum-in-rotated-array/ Sun, 05 Aug 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-19-fast-minimum-in-rotated-array/ https://algodaily.com/challenges/fast-minimum-in-rotated-array It seems we have to identify an element that is lower than the preceding one, as this must be the original beginning of the array and thus the lowest. If we get to the end of the array without finding a drop, then it must be the first element of the array, as the pivot was at the end. This still has a worst case running time of O(n), though, as we might have to iterate through the whole array. https://notestoself.dev/posts/algodaily-18-sum-all-primes/ Mon, 23 Jul 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-18-sum-all-primes/ https://algodaily.com/challenges/sum-all-primes I remember there’s an algorithm called the Sieve of Eratosthenes for finding prime numbers up to a limit, which seems like it would be useful here. To apply the Sieve of Eratosthenes, you first allocate an object with each integer up to n as the keys. You then iterate on integers up to $\sqrt{n}$. In other words, keep iterating on integers p until p * p would be more than n. https://notestoself.dev/posts/inkscape-shape-stroke-edges-cut-off/ Mon, 23 Jul 2018 00:00:00 +0000 https://notestoself.dev/posts/inkscape-shape-stroke-edges-cut-off/ I got stuck with this problem in Inkscape where the stroke edges of shapes seem to get cut off by the bounding box: After a bit of searching, it turned out that this was caused by having a small blur set on the layer containing the shapes: Setting that layer blur back to zero restores the full stroke edges of the shapes. Note that you might have this layer blur set on your default template by mistake, in which case you’ll want to save a new default template with the layer blur set to zero. https://notestoself.dev/posts/algodaily-17-find-missing-number-in-array/ Sun, 08 Jul 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-17-find-missing-number-in-array/ https://algodaily.com/challenges/find-missing-number-in-array Seems like we should iterate on the array, adding an integer to the result array if it’s not present in the input. We have to backtrack the iteration by one each time we hit a missing number. This is O(n). function missingNumbers(sequence) { const missing = []; let expected; for (let i = 0; i < sequence.length; i++) { if (i === 0) { expected = sequence[i] + 1; continue; } if (sequence[i] ! https://notestoself.dev/posts/algodaily-16-missing-number-in-unsorted/ Thu, 05 Jul 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-16-missing-number-in-unsorted/ https://algodaily.com/challenges/missing-number-in-unsorted The requirement to do this without sorting makes me think of using a hash map to track each number we do see, and then finding the missing integer there by iterating across the expected range. This isn’t O(n), though; it’s O(m+n). function missingInUnsorted(numbers, lowerBound, upperBound) { const seen = numbers.reduce((carry, num) => { carry[num] = true; return carry; }, {}); for (let i = lowerBound; i < upperBound; i++) { if (seen[i] === undefined) { return i; } } return undefined; } Algo Daily has a solution using the Gauss sum formula n(n+1)/2 to determine the sum of a range of numbers, and then subtracting the sum of the given numbers from that to find the single missing integer. https://notestoself.dev/posts/algodaily-15-max-of-min-pairs/ Sat, 23 Jun 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-15-max-of-min-pairs/ https://algodaily.com/challenges/max-of-min-pairs This requires a bit of thought. We want to get the highest possible smaller number in each pair. The first thought is that we need to sort the numbers before we do anything else, so that we can allocate pairs with the largest possible minimum. To get the largest possible minimum in the pair, you have to “waste” a larger number to accompany it. This should be the smallest possible number that is larger than the minimum in the pair. https://notestoself.dev/posts/simple-estimated-delivery-dates-in-vanilla-js/ Wed, 20 Jun 2018 00:00:00 +0000 https://notestoself.dev/posts/simple-estimated-delivery-dates-in-vanilla-js/ Displaying estimated delivery dates is a handy little feature in e-commerce that gives useful information to customers and reassures them about the ordering process. Estimating delivery dates can be quite involved, but it’s easy to chuck a very simple estimated delivery range on any page with these vanilla JavaScript functions, no libraries required: function showEstimatedDelivery() { const estDelEls = document.querySelectorAll(".estimated-delivery"); if (!estDelEls || !estDelEls.length) { return; } const [minDate, maxDate] = estimatedDeliveryRange(); const [minFormat, maxFormat] = [formatDate(minDate), formatDate(maxDate)]; let estimatedDelivery if (minFormat === maxFormat) { estimatedDelivery = minFormat; } else { estimatedDelivery = `${minFormat}— ${maxFormat}`; } document. https://notestoself.dev/posts/python-asyncio-concurrent-map-pool/ Mon, 18 Jun 2018 13:18:51 +0100 https://notestoself.dev/posts/python-asyncio-concurrent-map-pool/ Here is a relatively short Python function to map input items concurrently, with a fixed pool size to limit the maximum concurrency. The work function to map each item is a normal synchronous function, which makes this easy to use in projects that are not otherwise using asyncio. This function should be OK to run with or without a pre-existing event loop in the process. It uses an existing event loop if there is one, otherwise it creates a new event loop and uses that. https://notestoself.dev/posts/clipboard-hygiene/ Tue, 12 Jun 2018 00:00:00 +0000 https://notestoself.dev/posts/clipboard-hygiene/ After using the excellent keepassx password manager for a while, I’ve noticed it has a nice little security feature: once you copy a password to the clipboard, keepassx waits for a short period of time and then clears your clipboard. It also clears the clipboard when you close keepassx. This is a good idea as it somewhat reduces the chance of some other process being able to steal a password from the clipboard (which any user-level process could do). https://notestoself.dev/posts/algodaily-14-find-first-non-repeating-character/ Fri, 08 Jun 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-14-find-first-non-repeating-character/ https://algodaily.com/challenges/find-first-non-repeating-character This shouldn’t be too hard with a hash map of characters to counts, relying on JavaScript objects having ordered keys to then find the first character with a single occurrence. function firstNonRepeat(str) { const counts = {} for (let char of str) { counts[char] = counts[char] ? counts[char] + 1 : 1; } for (let char in counts) { if (counts[char] === 1) { return char; } } return ""; } https://notestoself.dev/posts/algodaily-13-dollar-sign-deletion/ Tue, 05 Jun 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-13-dollar-sign-deletion/ “You’re given an array of strings containing alphabetical characters and certain $ characters. A $ represents a DELETE action whereby the character before it is deleted. Each of these strings will be run through a method to operate on the $ DELETE action. We want to find out if the final string is the same for all of them. Let’s take an example:” const input = ["f$st", "st"] isDollarDeleteEqual(input); // true // true because both become "st" “Given the below function signature, can you find a solution in O(n) time and constant space? https://notestoself.dev/posts/extracting-product-image-main-colours-python/ Sat, 19 May 2018 00:00:00 +0000 https://notestoself.dev/posts/extracting-product-image-main-colours-python/ The main colours of a product are often quite important data in e-commerce systems, and it’s nice to have an automated way to extract these. Here’s a rough summary of what I ended doing in Python to get this data from product images. It uses three different colour extraction libraries and merges the results from them, which is somewhat questionable but seems to provide better overall results than a single library alone. https://notestoself.dev/posts/year-name-century-name-decade-name-python/ Sat, 19 May 2018 00:00:00 +0000 https://notestoself.dev/posts/year-name-century-name-decade-name-python/ Here are some little helper functions to convert an integer year into a century name and decade name, for example 1989 would be "20th Century" and "1980s". from math import floor def year_to_century_name(year: int) -> str: century = (floor(int(year) / 100)) + 1 suffix = ordinal_suffix(century) return f"{century}{suffix} Century" def ordinal_suffix(num: int) -> str: return {1: "st", 2: "nd", 3: "rd"}.get( 0 if 10 > num > 14 else num % 10, "th" ) def year_to_decade_name(year: int) -> str: decade = floor(int(year) / 10) * 10 return f"{decade}s" We use this in Moment Wall Art to extract the century and decade of each piece, which allows for easy taxonomy pages like 20th Century Wall Art or 2020s Wall Art. https://notestoself.dev/posts/django-command-file-stdin-stdout/ Sun, 13 May 2018 00:00:00 +0000 https://notestoself.dev/posts/django-command-file-stdin-stdout/ Here’s a little approach I’ve started using when writing Python scripts and Django console commands. It allows them to easily handle i/o from the commandline or from a file without much effort. Rather than using sys.stdin or sys.stdout directly, you define an argparse.FileType argument for the command, and set the default to sys.stdin. For example, in a Django command: from sys import stdin, stdout from argparse import FileType from django.core.management.base import BaseCommand class Command(BaseCommand): def add_arguments(self, parser: ArgumentParser): parser. https://notestoself.dev/posts/laravel-stream-csv-database/ Sun, 13 May 2018 00:00:00 +0000 https://notestoself.dev/posts/laravel-stream-csv-database/ Exporting data to CSV is a requirement in almost any business software project. We needed it to get a convenient sales data export in our card shop sideproject recently, and it turned out to be straightforward but slightly weird to do in Laravel. Sales data export query The sales data export query is about 30 lines of SQL. This was easier to write directly in SQL than wrapping it up in Laravel’s ORM, and it’s also handy to be able to run this query separately anyway. https://notestoself.dev/posts/algodaily-12-detect-substring-in-string/ Tue, 08 May 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-12-detect-substring-in-string/ https://algodaily.com/challenges/detect-substring-in-string “How would you write a function to detect a substring in a string? If the substring can be found in the string, return the index at which it starts. Otherwise, return -1.” We can iterate through the string, and increment a counter when we hit that index in the target substring. If the counter gets to the length of the target substring, we can return the iterator index minus the counter. https://notestoself.dev/posts/algodaily-11-sum-digits-until-one/ Sun, 06 May 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-11-sum-digits-until-one/ https://algodaily.com/challenges/sum-digits-until-one “We’re provided a positive integer num. Can you write a method to repeatedly add all of its digits until the result has only one digit? Here’s an example: if the input was 49, we’d go through the following steps:” // start with 49 4 + 9 = 13 // move onto 13 1 + 3 = 4 This seems to mostly be an issue of type juggling. If we’re careful with types, then all we have to do is keep breaking down the string of the number into single digit characters, sum those as numbers and set the original number to the string of that number. https://notestoself.dev/posts/algodaily-10-binary-tree-in-order-traversal/ Sat, 05 May 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-10-binary-tree-in-order-traversal/ “Can you write a function to traverse a binary tree in-order, and print out the value of each node as it passes?” 4 \ 5 / 6 “The example would output [4, 6, 5] since there is no left child for 4, and 6 is visited in-order before 5. The definition of a tree node is as follows:” function Node(val) { this.val = val; this.left = null; this. https://notestoself.dev/posts/django-field-choices-inner-class-enum/ Wed, 02 May 2018 00:00:00 +0000 https://notestoself.dev/posts/django-field-choices-inner-class-enum/ It’s common to define the possible choices for a character field on Django models like this (from the docs): from django.db import models class Student(models.Model): FRESHMAN = 'FR' SOPHOMORE = 'SO' JUNIOR = 'JR' SENIOR = 'SR' YEAR_IN_SCHOOL_CHOICES = ( (FRESHMAN, 'Freshman'), (SOPHOMORE, 'Sophomore'), (JUNIOR, 'Junior'), (SENIOR, 'Senior'), ) year_in_school = models.CharField( max_length=2, choices=YEAR_IN_SCHOOL_CHOICES, default=FRESHMAN, ) This works fine and it makes constants available for the values, so that elsewhere you can refer to Student. https://notestoself.dev/work/umed/ Mon, 16 Apr 2018 00:00:00 +0000 https://notestoself.dev/work/umed/ I was a back-end Software Engineer at uMed in 2018. https://notestoself.dev/posts/php-business-time-library-v1/ Fri, 13 Apr 2018 00:00:00 +0000 https://notestoself.dev/posts/php-business-time-library-v1/ I’ve just released v1.0.0 of my [PHP Business Time](https://github.com/hughgrigg /php-business-time) library. I made this to solve my own problem that I had building my 3D pop-up card shop – calculating estimated delivery dates based on business time constraints. It would have been simpler to just solve the exact use case I had (probably hard-coding timings and combining that with retrieving holidays from a remote source), but I thought it would be fun to generalise it and make it into a separate library. https://notestoself.dev/posts/github-action-s3-botocore-awsrequest-process-255-error/ Mon, 09 Apr 2018 00:00:00 +0000 https://notestoself.dev/posts/github-action-s3-botocore-awsrequest-process-255-error/ The other day I suddenly started hitting an error in my Github Actions that deploy static Hugo sites to S3: <botocore.awsrequest.AWSRequest object at 0x7fbfb444e160> Error: Process completed with exit code 255. For some reason setting an env variable called AWS_EC2_METADATA_DISABLED to true fixes this error. E.g. your Github Action YAML might look like this: name:BuildandDeployon:pushjobs:build:name:BuildandDeployruns-on:ubuntu-lateststeps:- uses:actions/checkout@v1- name:InstallHugorun:| HUGO_DOWNLOAD=hugo_extended_${HUGO_VERSION}_Linux-64bit.tar.gzwgethttps://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/${HUGO_DOWNLOAD}tarxvzf${HUGO_DOWNLOAD}hugomvhugo$HOME/hugoenv:HUGO_VERSION:0.71.0- name:HugoBuildrun:$HOME/hugo-v- name:DeploytoS3if:github.ref=='refs/heads/master'run:awss3syncpublic/s3://{s3-bucket-name}/--delete--acl=public-read&&awscloudfrontcreate-invalidation--distribution-id={cloudfront-distribution-id}--paths='/*'env:AWS_ACCESS_KEY_ID:${{secrets.AWS_ACCESS_KEY_ID}}AWS_SECRET_ACCESS_KEY:${{secrets.AWS_SECRET_ACCESS_KEY}}AWS_EC2_METADATA_DISABLED:trueThe key bit for avoiding this error is: https://notestoself.dev/posts/algodaily-09-implement-a-hash-map/ Fri, 06 Apr 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-09-implement-a-hash-map/ “Can you implement the JS Map class from scratch? Only two methods are necessary – get and set. Do not use JS Objects ({} notation) for this exercise. For the two methods you’ll define: get(key: string) should be given a key, and return the value for that key. set(key: string, val: string) should take a key and a value as parameters, and store the pair. Additionally, we’ve supplied the below hashing function hashStr. https://notestoself.dev/posts/algodaily-08-lonely-number/ Thu, 05 Apr 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-08-lonely-number/ https://algodaily.com/challenges/lonely-number “In a given array of numbers, one element will show up once and the rest will show up twice. Can you find that number in O(n) linear time?” lonelyNumber([4, 4, 6, 1, 3, 1, 3]) // 6 lonelyNumber([3, 3, 9]) // 9 This seems straightforward. function lonelyNumber(items) { return Object.keys(items.reduce((seen, x) => { if (x in seen) { delete seen[x]; return seen; } seen[x] = x; return seen; }, {}))[0]; } We start with an empty hash, then go through the array, removing an item we’ve seen before, and adding an item we haven’t. https://notestoself.dev/posts/algodaily-07-power-of-three/ Tue, 03 Apr 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-07-power-of-three/ “Given an integer num, write a method to determine if it is a power of 3.” console.log(powerOfThree(9)); // true To calculate a power of x, we could keep multiplying by x until we hit the exponent. That means to see if something is a power of x, we can do the reverse and keep diving by x until we hit 1 or a number that is not a multiple of 3. https://notestoself.dev/posts/url-encode-etsy-style-python/ Mon, 26 Mar 2018 00:00:00 +0000 https://notestoself.dev/posts/url-encode-etsy-style-python/ The Etsy API has quite specific requirements for how data should be encoded in requests. With a bit of trial and error making test requests, I ended up with these Python functions to encode data in the way the Etsy API requires: from typing import Union, Dict, List from urllib import parse def etsy_urlencoded(data: Dict): return parse.urlencode({k: etsy_urlencoded_value(v) for k, v in data.items()}) def etsy_urlencoded_value(value: Union[Dict, List, str, bool, int, float]): if type(value) is dict: return json. https://notestoself.dev/posts/react-avoid-derived-state-from-props/ Fri, 23 Mar 2018 00:00:00 +0000 https://notestoself.dev/posts/react-avoid-derived-state-from-props/ A common footgun in React projects is derived state, which means basing state on props. This can be directly copying props into state, or deriving state from props with some extra logic. This often causes trouble because once the state is set based on props, it won’t be updated by subsequent updates to props. This is after all the purpose of props and state in React: props are clean and stateless, and state is of course stateful. https://notestoself.dev/posts/algodaily-06-majority-element/ Sat, 03 Mar 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-06-majority-element/ https://algodaily.com/challenges/majority-element “Suppose we’re given an array of numbers like the following: [4, 2, 4] Could you find the majority element? A majority is defined as “the greater part, or more than half, of the total. It is a subset of a set consisting of more than half of the set’s elements. Let’s assume that the array length is always at least one, and that there’s always a majority element. https://notestoself.dev/posts/algodaily-05-validate-palindrome/ Fri, 02 Mar 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-05-validate-palindrome/ “Given a string str, can you write a method that will return True if is a palindrome and False if it is not? If you’ll recall, a palindrome is defined as “a word, phrase, or sequence that reads the same backward as forward”. For now, assume that we will not have input strings that contain special characters or spaces, so the following examples hold:” let str = 'thisisnotapalindrome'; isPalindrome(str); // false str = 'racecar'; isPalindrome(str); // true “For an extra challenge, try to ignore non-alphanumerical characters. https://notestoself.dev/posts/algodaily-04-is-an-anagram/ Thu, 01 Mar 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-04-is-an-anagram/ https://algodaily.com/challenges/is-an-anagram “Here’s the definition of an anagram: a word, phrase, or name formed by rearranging the letters of another, such as cinema, formed from iceman. We are given two strings like “cinema” and “iceman” as inputs. Can you write a method isAnagram(str1, str2) that will return true or false depending on whether the strings are anagrams of each other?” The gotcha with this is being careful with multiple instances of the same letter, i. https://notestoself.dev/posts/algodaily-03-reverse-only-alphabetical/ Tue, 20 Feb 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-03-reverse-only-alphabetical/ (https://algodaily.com/challenges/reverse-only-alphabetical)[https://algodaily.com/challenges/reverse-only-alphabetical] “You are given a string that contains alphabetical characters (a - z, A - Z) and some other characters ($, !, etc.). For example, one input may be: ‘sea!$hells3’ Can you reverse only the alphabetical ones?” First thought is to use a stack for this: go through the string forwards, pushing only letters on to the stack. Then go through the positions again, popping the stack for a letter position but leaving non-letters alone: https://notestoself.dev/posts/algodaily-02-fizz-buzz/ Mon, 19 Feb 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-02-fizz-buzz/ “We’re given a number n. Write a function that returns the string representation of all numbers from 1 to n based on the following rules: If it’s a multiple of 3, represent it as “fizz”. If it’s a multiple of 5, represent it as “buzz”. If it’s a multiple of both 3 and 5, represent it as “fizzbuzz”. If it’s neither, just return the number itself. As such, fizzBuzz(15) would result in ‘12fizz4buzzfizz78fizzbuzz11fizz1314fizzbuzz’. https://notestoself.dev/posts/algodaily-01-array-intersection/ Sun, 18 Feb 2018 00:00:00 +0000 https://notestoself.dev/posts/algodaily-01-array-intersection/ I’ve signed up to AlgoDaily as it seems like a fun way to get out some small blog posts on small coding tasks. This is AlgoDaily Day 1: Array Intersection. “Can you write a function that takes two inputs and returns their intersection? All element in the final result should be unique.” const nums1 = [1, 2, 2, 1]; const nums2 = [2, 2]; intersection(nums1, nums2); // [2] “Here’s another one:” https://notestoself.dev/posts/ubuntu-varmilo-keyboard-f-super-win-key-mode/ Thu, 15 Feb 2018 00:00:00 +0000 https://notestoself.dev/posts/ubuntu-varmilo-keyboard-f-super-win-key-mode/ I got a Varmilo VA109M keyboard recently (it’s great!), which I’m using on Ubuntu 20.10. This is how I got the F-keys working correctly: echo options hid_apple fnmode=2 | sudo tee -a /etc/modprobe.d/hid_apple.conf sudo update-initramfs -u -k all Then restart. And to get the Super key (labelled Win on Varmilo PC keyboards) working correctly, make sure your Varmilo keyboard is in PC mode: Hold down Fn + Esc for 4 seconds. https://notestoself.dev/posts/cartesian-product-bash/ Tue, 23 Jan 2018 00:00:00 +0000 https://notestoself.dev/posts/cartesian-product-bash/ Here’s a handy thing you can do in Bash: calculate the Cartesian product of two sets. E.g. echo {A,B,C}{A,B,C} gives AA AB AC BA BB BC CA CB CC i.e. it’s every possible permutation of two elements from the two sets. A --- A \ / B -x- B / \ C --- C (Thanks to Bill Hails for the diagram!) This can be a useful tool for quickly figuring things out in various situations. https://notestoself.dev/posts/finding-if-two-rectangles-overlap-in-javascript/ Thu, 18 Jan 2018 00:00:00 +0000 https://notestoself.dev/posts/finding-if-two-rectangles-overlap-in-javascript/ Given the top-left and bottom-right coordinates of two rectangles, determine if they overlap or not. That is, determine if any part of one rectangle overlaps with any part of the other. The grid has position (0, 0) at the top left. X increases rightwards and Y increases downwards. This is common in 2D grids in video games. 0,0 ─ x → │ y ↓ You get coordinates for the rectangles as tuples of x and y, e. https://notestoself.dev/posts/converting-roman-numerals-into-decimal-in-javascript/ Mon, 08 Jan 2018 00:00:00 +0000 https://notestoself.dev/posts/converting-roman-numerals-into-decimal-in-javascript/ Converting Roman numerals into decimal is a fun little problem to solve. Getting the solution is dependent on knowing the rules of Roman numerals, and they’re actually simpler than you might think: All letters directly map to an integer value. A letter followed by a letter of higher value is subtracted. All other letters are added. Before I looked this up, I thought that the rules were more complicated, e. https://notestoself.dev/posts/identifying-a-potential-palindrome-in-javascript/ Fri, 05 Jan 2018 00:00:00 +0000 https://notestoself.dev/posts/identifying-a-potential-palindrome-in-javascript/ Given an input string, determine whether the string could possibly be a palindrome if rearranged. It is not necessary to return the potential palindrome, just to identify whether one is possible. Examples: aabb => true abc => false aab => true ciicv => true aba => true abac => false A palindrome can have any amount of pairs of letters, and optionally one single odd letter with no partner in the middle. https://notestoself.dev/reading/bloodchild/ Mon, 01 Jan 2018 00:00:00 +0000 https://notestoself.dev/reading/bloodchild/ https://notestoself.dev/reading/dawn/ Mon, 01 Jan 2018 00:00:00 +0000 https://notestoself.dev/reading/dawn/ https://notestoself.dev/reading/dune/ Mon, 01 Jan 2018 00:00:00 +0000 https://notestoself.dev/reading/dune/ https://notestoself.dev/reading/how-to-destroy-a-tech-startup/ Mon, 01 Jan 2018 00:00:00 +0000 https://notestoself.dev/reading/how-to-destroy-a-tech-startup/ https://notestoself.dev/reading/rationality-from-ai-to-zombies/ Mon, 01 Jan 2018 00:00:00 +0000 https://notestoself.dev/reading/rationality-from-ai-to-zombies/ https://notestoself.dev/reading/the-big-short/ Mon, 01 Jan 2018 00:00:00 +0000 https://notestoself.dev/reading/the-big-short/ https://notestoself.dev/reading/the-daily-stoic/ Mon, 01 Jan 2018 00:00:00 +0000 https://notestoself.dev/reading/the-daily-stoic/ https://notestoself.dev/reading/gervais-principle/ Mon, 01 Jan 2018 00:00:00 +0000 https://notestoself.dev/reading/gervais-principle/ https://notestoself.dev/reading/heart-to-start/ Mon, 01 Jan 2018 00:00:00 +0000 https://notestoself.dev/reading/heart-to-start/ https://notestoself.dev/reading/inner-game-of-tennis/ Mon, 01 Jan 2018 00:00:00 +0000 https://notestoself.dev/reading/inner-game-of-tennis/ https://notestoself.dev/reading/spy-who-came-in-from-the-cold/ Mon, 01 Jan 2018 00:00:00 +0000 https://notestoself.dev/reading/spy-who-came-in-from-the-cold/ https://notestoself.dev/reading/war-is-a-racket/ Mon, 01 Jan 2018 00:00:00 +0000 https://notestoself.dev/reading/war-is-a-racket/ https://notestoself.dev/reading/python-tricks/ Sun, 17 Dec 2017 00:00:00 +0000 https://notestoself.dev/reading/python-tricks/ https://notestoself.dev/posts/e-commerce-model-product-page-abstraction/ Fri, 08 Dec 2017 00:00:00 +0000 https://notestoself.dev/posts/e-commerce-model-product-page-abstraction/ A lot of e-commerce abstractions get a bit messy around abstracting products, SKUs, product options, product variations and so on. The fact that there doesn’t seem to be a standard or obvious abstraction for this suggests it is tricky to get right. It might be helpful in a lot of systems to extract a Product Page model as an abstraction. I think this avoids a lot of the trouble as it’s easy to think about and lets your code match your conversations. https://notestoself.dev/posts/php-balance-address-lines-ecommerce/ Thu, 07 Dec 2017 00:00:00 +0000 https://notestoself.dev/posts/php-balance-address-lines-ecommerce/ Delivery addresses are notoriously difficult to deal with. One issue we’ve encountered with our greetings card shop is long lines in an address that would get cut off when printed on the address label, so we have to manually tweak them to get the address to fit. To reduce some of that manual work, I added a little helper that tries to balance the address lines where possible, i.e. take words off long lines and add them to neighbouring lines that have some space to spare. https://notestoself.dev/reading/fluent-python/ Tue, 05 Dec 2017 00:00:00 +0000 https://notestoself.dev/reading/fluent-python/ https://notestoself.dev/posts/draw-sequential-binary-tree-python/ Thu, 23 Nov 2017 00:00:00 +0000 https://notestoself.dev/posts/draw-sequential-binary-tree-python/ I’ve been playing around with sequentially represented binary trees recently, i.e. a binary tree given as a list of integers where the node values are in contiguous sections of the list. For example, a binary tree where the node values are also their indexes would be sequentially represented like this: 0 | 1, 2 | 3, 4, 5, 6 | 7, 8, 9, 10, 11, 12, 13, 14 Each section in the list represents one level of the binary tree, and has length $2^{l}$ where $l$ is the level. https://notestoself.dev/reading/managing-humans/ Thu, 05 Oct 2017 00:00:00 +0000 https://notestoself.dev/reading/managing-humans/ https://notestoself.dev/reading/on-the-trail-of-genghis-kahn/ Sun, 01 Oct 2017 00:00:00 +0000 https://notestoself.dev/reading/on-the-trail-of-genghis-kahn/ https://notestoself.dev/reading/the-dispossessed/ Tue, 05 Sep 2017 00:00:00 +0000 https://notestoself.dev/reading/the-dispossessed/ Shevek, a brilliant physicist, decides to take action. he will seek answers, question the unquestionable, and attempt to tear down the walls of hatred that have isolated his planet of anarchists from the rest of the civilized universe. To do this dangerous task will mean giving up his family and possibly his life. Shevek must make the unprecedented journey to the utopian mother planet, Anarres, to challenge the complex structures of life and living, and ignite the fires of change. https://notestoself.dev/posts/blackjack-cli-game/ Mon, 28 Aug 2017 00:00:00 +0000 https://notestoself.dev/posts/blackjack-cli-game/ The first draft of a blackjack cli game I’m making is now finished. If you have Go installed then you should be able to play the game with: go get -u github.com/hughgrigg/blackjack blackjack It’s a single-deck blackjack betting game written in Go for the command line. I’ve only tried it on Linux, but it uses the termui library so I imagine it would be OK on OSX as well. Not sure about Windows. https://notestoself.dev/posts/enable-php-assertion-errors-on-laravel-homestead-for-development-and-testing/ Wed, 23 Aug 2017 00:00:00 +0000 https://notestoself.dev/posts/enable-php-assertion-errors-on-laravel-homestead-for-development-and-testing/ Laravel’s Homestead vagrant setup is handy to get going quickly on projects and keep them consistent to work on. I was surprised that it has PHP assertion errors disabled by default. It’s good to have these enabled for local development and testing so that you can catch more problems earlier. To enable PHP assertion errors in Homestead you can run this: for phpConf in cli fpm; do sudo sed -i 's/zend. https://notestoself.dev/reading/doughnut-economics/ Sat, 05 Aug 2017 00:00:00 +0000 https://notestoself.dev/reading/doughnut-economics/ https://notestoself.dev/posts/dates-for-booleans/ Sun, 23 Jul 2017 00:00:00 +0000 https://notestoself.dev/posts/dates-for-booleans/ Here’s a little maxim for data modeling: use dates instead of booleans. In a lot of situations, this lets you capture more information without losing anything compared to a boolean. A common example is implementing a status column with a datetime foobared_at column instead of a boolean is_foobar. If it’s null, then that row is not foobar. If the column is filled, then you not only know that it is foobar, but also since when. https://notestoself.dev/posts/python-early-exit-boolean-expression-parser/ Tue, 18 Jul 2017 00:00:00 +0000 https://notestoself.dev/posts/python-early-exit-boolean-expression-parser/ I wanted to have a go at rush-implementing a boolean expression parser with early exit, i.e. that it will not bother evaluating the right hand side of an expression like 0&(1|0) and just return false once it sees 0&. Here’s an attempt in Python. It deals with nested expressions by recursion and “collapsing” the evaluation into a 0 or 1 token to continue evaluating from there. It uses a similar approach to handle ! https://notestoself.dev/reading/superforecasting/ Wed, 05 Jul 2017 00:00:00 +0000 https://notestoself.dev/reading/superforecasting/ https://notestoself.dev/posts/nordic-apis-oauth-workshop/ Mon, 12 Jun 2017 00:00:00 +0000 https://notestoself.dev/posts/nordic-apis-oauth-workshop/ In June I attended [Nordic API’s workshop](http://nordicapis.com/events/nordic- apis-workshops-amsterdam/) on OAuth and OpenID in Amsterdam. Here are my notes on the class. Actors in OAuth AS: Authorisation Server (OAuth Server), e.g. bank auth server RS: Resource Server, e.g. bank account server RO: Resource Owner, e.g. bank customer C: Client, e.g. mobile app OAuth is designed for cross-organisational authorisation, i.e. where the client is not trusted (which should always be the case). https://notestoself.dev/reading/the-corrections/ Mon, 05 Jun 2017 00:00:00 +0000 https://notestoself.dev/reading/the-corrections/ https://notestoself.dev/posts/label-print-laravel-raspberry-pi-aws/ Mon, 29 May 2017 00:00:00 +0000 https://notestoself.dev/posts/label-print-laravel-raspberry-pi-aws/ The most manual part of the e-commerce side project I run with my wife is shipping the orders. We get order notifications by [email and Telegram](https://github.com/hughgrigg/poprobincards/bl ob/a08850cbee203223db309965b6e264e7e951e693/app/Modules/Sales/Notifications/Staf fOrderNotification.php#L52), and have to: Find the stock. Package it. Print the shipping label, cut out and attach it. Take the order to the post box. A lot of that is beyond our means to automate, but we did get a Brother QL-570 label printer to speed up step 3 quite a lot. https://notestoself.dev/posts/algorithm-design-exercise-1-29/ Thu, 25 May 2017 00:00:00 +0000 https://notestoself.dev/posts/algorithm-design-exercise-1-29/ This is exercise 1-29 from The Algorithm Design Manual. There are 25 horses. At most, 5 horses can race together at a time. You must determine the fastest, second fastest, and third fastest horses. Find the minimum number of races in which this can be done. This is quite a famous puzzle question and involves some algorithmic thinking. The first key to understanding for me was to focus on excluding horses who are slower than at least 3 other horses. https://notestoself.dev/posts/survivorship-thinking/ Tue, 23 May 2017 00:00:00 +0000 https://notestoself.dev/posts/survivorship-thinking/ The survivorship bias is one of the most interesting cognitive biases to know about (cliché XKCD link). It’s the idea that our impression of a given set of things is biased towards those that have survived some selection pressure. The classic example is when people say “they don’t make ‘em like they used to”. With a moment’s thought, you realise that of course all the old things we see today are well made. https://notestoself.dev/posts/hugo-template-list-comma-separated-last-full-stop/ Sat, 13 May 2017 00:00:00 +0000 https://notestoself.dev/posts/hugo-template-list-comma-separated-last-full-stop/ Here’s a Hugo template snippet for rendering a list of items separated by a comma and space after all but the last item, and then a dot or full stop. For example, if you have a tags taxonomy and some tags in a post’s front-matter like this: tags:- Foo- Bar- Baz---You can render a comma-separated list of those tags in a template like this: {{ with (.GetTerms "tags") }} <p> {{ range $i, $t := . https://notestoself.dev/posts/laravel-forge-ssh-key-xclip-pbcopy-permission-denied-public-key/ Fri, 12 May 2017 00:00:00 +0000 https://notestoself.dev/posts/laravel-forge-ssh-key-xclip-pbcopy-permission-denied-public-key/ I use Laravel Forge to run several Laravel projects. It has an interface to add SSH keys to a server so that you can SSH into them, but I’ve repeatedly hit a little issue when trying to use xclip on Linux or pbcopy on Mac to copy the public key and paste it into the Forge interface. Somehow the public key is not copied properly and you’ll get the generic Permission denied (public key) error when trying to SSH into the server. https://notestoself.dev/reading/the-every-computer-performance-book/ Mon, 01 May 2017 00:00:00 +0000 https://notestoself.dev/reading/the-every-computer-performance-book/ https://notestoself.dev/reading/the-remains-of-the-day/ Mon, 01 May 2017 00:00:00 +0000 https://notestoself.dev/reading/the-remains-of-the-day/ https://notestoself.dev/posts/algorithm-design-exercise-1-28/ Sun, 23 Apr 2017 00:00:00 +0000 https://notestoself.dev/posts/algorithm-design-exercise-1-28/ This is exercise 1-28 from The Algorithm Design Manual. Write a function to perform integer division without using either the / or * operators. Find a fast way to do it. Let’s add multiplication as well. We could do a very inefficient multiplication algorithm by using one factor as a counter, and incrementing the other factor by itself until the counter reaches its value: $a \times b$ $i = 0$ $p = 0$ while $i Naive integer multiplication in Python https://notestoself.dev/reading/station-eleven/ Sat, 01 Apr 2017 00:00:00 +0000 https://notestoself.dev/reading/station-eleven/ An audacious, darkly glittering novel about art, fame, and ambition set in the eerie days of civilization’s collapse, from the author of three highly acclaimed previous novels. One snowy night a famous Hollywood actor slumps over and dies onstage during a production of King Lear. Hours later, the world as we know it begins to dissolve. Moving back and forth in time-from the actor’s early days as a film star to fifteen years in the future, when a theater troupe known as the Traveling Symphony roams the wasteland of what remains-this suspenseful, elegiac, spellbinding novel charts the strange twists of fate that connect five people: the actor, the man who tried to save him, the actor’s first wife, his oldest friend, and a young actress with the TravelingSymphony, caught in the crosshairs of a dangerous self-proclaimed prophet. Sometimes terrifying, sometimes tender, Station Eleven tells a story about the relationships that sustain us, the ephemeral nature of fame, and the beauty of the world as we know it– https://notestoself.dev/posts/stripping-unprintable-unicode-variation-characters-in-php/ Fri, 31 Mar 2017 00:00:00 +0000 https://notestoself.dev/posts/stripping-unprintable-unicode-variation-characters-in-php/ Emoji characters like 🥳, 🎉 and 💖 can be represented in a string in different ways. The first way is to include a Unicode point as a single character. For example 💖 is Unicode point U+1F496, which can be included in a PHP string either directly ('💖') or using the Unicode code point escape syntax from PHP 7 onwards: "\u{1F496}". You can try these on the PHP REPL (php -a): https://notestoself.dev/posts/algorithm-design-1/ Thu, 23 Mar 2017 00:00:00 +0000 https://notestoself.dev/posts/algorithm-design-1/ These are the exercises for chapter 1 of The Algorithm Design Manual. 1-1 Show that $a + b$ can be less than $min(a,b)$ Where $a,b < 0$, e.g. if $a = -1, b = -2$, then $a + b = -3 < min(a,b) = -5$. 1-2 Show that $a \times b$ can be less than $min(a,b)$ Where one of $\{a,b\}$ is negative and one is positive, e. https://notestoself.dev/posts/cracking-coding-interview-6th-10.1-sorted-merge/ Thu, 23 Mar 2017 00:00:00 +0000 https://notestoself.dev/posts/cracking-coding-interview-6th-10.1-sorted-merge/ “Sorted Merge: You are given two sorted arrays, A and B, where A has a large enough buffer at the end to hold B. Write a method to merge B into A in sorted order.” Arrays are already sorted, so we can just do an insertion sort: iterate both, taking the next sorted item from each and insert it into A which has space for both. It would be tricky to insert at the beginning, though, as we’d have to shift the existing elements in A. https://notestoself.dev/posts/etsy-api-listing-image-upload-python/ Tue, 21 Mar 2017 00:00:00 +0000 https://notestoself.dev/posts/etsy-api-listing-image-upload-python/ It took a bit of messing around, but I was able to upload product listing images via the Etsy API in Python 3 with requests_oauthlib like this (reduced to the essentials): from requests_oauthlib import OAuth1Session session = OAuth1Session( "{etsy client key}", client_secret="{etsy client secret}", resource_owner_key="{etsy resource owner key}", resource_owner_secret="{etsy resource owner secret}", ) response = session.post( "/listings/123456789/images", params={ "listing_id": 123456789, "rank": 1, "overwrite": True, "is_watermarked": False, }, files={ "image": ( "image123. https://notestoself.dev/posts/getting-started-teach-yourself-cs-algorithms-data-structures/ Fri, 17 Mar 2017 00:00:00 +0000 https://notestoself.dev/posts/getting-started-teach-yourself-cs-algorithms-data-structures/ The website TeachYourselfCS.com was recently featured on Hacker News, and after reading through I’ve decided to embark on a new learning journey by going through one of its subjects as properly as I can. The subject I’ve chosen is Algorithms and Data Structures, as I think that will be most interesting to me personally, and will have the most immediate practical application in the kinds of things I like to do. https://notestoself.dev/reading/deep-survival/ Wed, 01 Mar 2017 00:00:00 +0000 https://notestoself.dev/reading/deep-survival/ https://notestoself.dev/reading/algorithms-to-live-by/ Wed, 01 Feb 2017 00:00:00 +0000 https://notestoself.dev/reading/algorithms-to-live-by/ https://notestoself.dev/work/moment-wall-art/ Mon, 23 Jan 2017 00:00:00 +0000 https://notestoself.dev/work/moment-wall-art/ https://notestoself.dev/posts/etsy-api-create-update-inventory-product-offerings/ Sun, 08 Jan 2017 00:00:00 +0000 https://notestoself.dev/posts/etsy-api-create-update-inventory-product-offerings/ Managing products and product variations in the Etsy API can be a bit tricky, especially when you want to “sync” them, i.e. either create or update in the Etsy API according to your own product data. Here’s an approach in Python that I got working. Say your product data looks like this: my_products = { "sku_123": {"style": "foo", "price": "15.00", "quantity": 30}, "sku_456": {"style": "bar", "price": "25.00", "quantity": 10}, } One approach is to fetch the existing Etsy listing’s “inventory” (which might be empty for a newly created one), update it to match your local data, and then put it back to the Etsy API. https://notestoself.dev/reading/a-history-of-modern-britain/ Sun, 01 Jan 2017 00:00:00 +0000 https://notestoself.dev/reading/a-history-of-modern-britain/ https://notestoself.dev/posts/simplicity-is-a-skill/ Sun, 23 Oct 2016 00:00:00 +0000 https://notestoself.dev/posts/simplicity-is-a-skill/ After a lot of peer reviews and looking back at my own work recently, I’ve been considering the point I was trying to get across in “Boring considered beneficial". One of the main thoughts there was that approaches and technologies sometimes considered boring or traditional have that perception for a reason: they’ve become mature enough to generally work and solve problems in a straightforward way. The community is used to them, so they’re not as exciting as more novel and exotic ways of solving the same problems. https://notestoself.dev/posts/scalar-type-declarations-php7/ Fri, 30 Sep 2016 00:00:00 +0000 https://notestoself.dev/posts/scalar-type-declarations-php7/ One of my favourite new features in PHP 7 is [scalar type declarations](http://php.net/manual/en/migration70.new- features.php#migration70.new-features.scalar-type-declarations). Prior to PHP 7, the language’s type declaration abilities were somewhat inconsistent. You could type hint classes, interfaces and arrays, but nothing else (including callable and self in those). To require types for any scalar values (strings, ints, bools and floats), you had to do your own type checking and coercion. However, it is worth pointing out that it was and is possible to do deeper type checking in PHP using value objects or interfaces. https://notestoself.dev/posts/always-lint/ Tue, 13 Sep 2016 00:00:00 +0000 https://notestoself.dev/posts/always-lint/ “lint (n): undesirable bits of fiber and fluff found in sheep’s wool” — Wikipedia Linting is one of the first things I like to set up on a new project. Especially when working with interpreted languages, it’s reassuring to develop with a mechnical helper pointing out mistakes, problems and potential improvements. Linters exist for pretty much every language out there, and are a great addition to the development process. https://notestoself.dev/reading/trigger-warning/ Mon, 05 Sep 2016 00:00:00 +0000 https://notestoself.dev/reading/trigger-warning/ https://notestoself.dev/reading/the-handmaids-tale/ Fri, 05 Aug 2016 00:00:00 +0000 https://notestoself.dev/reading/the-handmaids-tale/ https://notestoself.dev/posts/python-unique-random-function-result-decorator/ Sat, 23 Jul 2016 00:00:00 +0000 https://notestoself.dev/posts/python-unique-random-function-result-decorator/ I like messing around with random generators for productivity and creativity systems. Usually this involves functions that return a random value on each call. Even with quite a large set of possible values, duplicates occur more often than you might expect. Here’s a quick Python decorator that keeps calling the function it decorates until it gets a value we haven’t seen so far during the current process: def unique_results(func: Callable) -> Callable: seen = set() def wrapper(): res = func() for i in range(100): if res not in seen: seen. https://notestoself.dev/posts/boring-considered-beneficial/ Mon, 23 May 2016 00:00:00 +0000 https://notestoself.dev/posts/boring-considered-beneficial/ Being boring is a good thing in software design. There is no benefit to being impressive with a design, or to demonstrate one’s knowledge in code. Doing that is likely to cause problems and reduce the quality of the software. One aspect of the programming profession is that there is both an urge and plenty of opportunity to try to show off. We are continually learning and improving, as we should be, and this can lead us to actively seek out opportunities to use the interesting knowledge we have acquired, instead of putting it to use when a concrete need for it arises. https://notestoself.dev/posts/simple-pipes-php-generators/ Tue, 17 May 2016 00:00:00 +0000 https://notestoself.dev/posts/simple-pipes-php-generators/ Pipes are a great structural pattern that can simplify the design of all kinds of software. They’re a central part of the Unix philosophy, allowing users to chain together a series of simple parts to achieve a more complex goal. Similarly, gulpjs brings the benefit of a pipe-based design to asset generation in the nodejs ecosystem. I haven’t seen pipes used so much in the PHP world, but they’ve been simple to set up since generators were added to the language in 5. https://notestoself.dev/reading/never-let-me-go/ Thu, 05 May 2016 00:00:00 +0000 https://notestoself.dev/reading/never-let-me-go/ From the Booker Prize-winning author of The Remains of the Day and When We Were Orphans, comes an unforgettable edge-of-your-seat mystery that is at once heartbreakingly tender and morally courageous about what it means to be human. Hailsham seems like a pleasant English boarding school, far from the influences of the city. Its students are well tended and supported, trained in art and literature, and become just the sort of people the world wants them to be. But, curiously, they are taught nothing of the outside world and are allowed little contact with it. Within the grounds of Hailsham, Kathy grows from schoolgirl to young woman, but it’s only when she and her friends Ruth and Tommy leave the safe grounds of the school (as they always knew they would) that they realize the full truth of what Hailsham is. https://notestoself.dev/posts/real-facade-laravel-encapsulate-common-behaviour/ Thu, 05 May 2016 00:00:00 +0000 https://notestoself.dev/posts/real-facade-laravel-encapsulate-common-behaviour/ Whilst working on my side project Pop Robin Cards (source), one of the PHP-MD rules I have configured flagged an issue: poprobincards/app/Http/Controllers/Staff/ProductController.php:53 The method __construct has 6 parameters. Consider to reduce parameter number under 5. PHP-MD is quite rightly pointing out that six parameters for a method is excessive, even for a constructor. But what can be done? The six parameters are all things that the ProductController needs to do its job: https://notestoself.dev/posts/php-list-asking-for-class/ Sun, 17 Apr 2016 00:00:00 +0000 https://notestoself.dev/posts/php-list-asking-for-class/ PHP has a weird mechanism for achieving multiple return values with the list function. It lets you do things like this: <?php function severalThings() { return ['thing', 'another thing', 'one more thing']; } list($foo, $bar, $foobar) = severalThings(); This is quite commonly used in some codebases, but I think it’s bad style. Firstly, because PHP doesn’t offer a concrete way to let the user of a function know exactly what it’s returning, it may not be clear what you’re getting from that array. https://notestoself.dev/posts/chinese-strftime/ Tue, 05 Apr 2016 00:00:00 +0000 https://notestoself.dev/posts/chinese-strftime/ I’ve always used some variation of the original GNOME desktop: GNOME 2, Cinnamon and currently MATE. All of those desktops feature a similar calendar applet that lets you enter a strftime string to get the date formatted how you like it. To get a tiny bit more Chinese into my day, I use this format: %Y年%m月%e号 %A %p%l点%M分%S秒 (The main reason I’m writing this post is to keep the format easily accessible (somewhere. https://notestoself.dev/posts/optimised-image-handling-aws-laravel/ Sat, 19 Mar 2016 00:00:00 +0000 https://notestoself.dev/posts/optimised-image-handling-aws-laravel/ A central feature in my side project [Pop Robin Cards](https://github.com/hughgrigg /poprobincards) is the management of product images. In the first iteration, images uploaded by the user were simply stored locally on disk and served by nginx. This approach works fine, but has some problems: It uses disk space on the EC2 instance hosting the site, where disk space is at a premium. It makes it harder to load balance between multiple application server instances. https://notestoself.dev/reading/stranger-in-a-strange-land/ Tue, 08 Mar 2016 00:00:00 +0000 https://notestoself.dev/reading/stranger-in-a-strange-land/ Stranger in a Strange Land, winner of the 1962 Hugo Award, is the story of Valentine Michael Smith, born during, and the only survivor of, the first manned mission to Mars. Michael is raised by Martians, and he arrives on Earth as a true innocent: he has never seen a woman and has no knowledge of Earth’s cultures or religions. But he brings turmoil with him, as he is the legal heir to an enormous financial empire, not to mention de facto owner of the planet Mars. With the irascible popular author Jubal Harshaw to protect him, Michael explores human morality and the meanings of love. He founds his own church, preaching free love and disseminating the psychic talents taught him by the Martians. Ultimately, he confronts the fate reserved for all messiahs. The impact of Stranger in a Strange Land was considerable, leading many children of the 60’s to set up households based on Michael’s water-brother nests. Heinlein loved to pontificate through the mouths of his characters, so modern readers must be willing to overlook the occasional sour note (Nine times out of ten, if a girl gets raped, it’s partly her fault.). That aside, Stranger in a Strange Land is one of the master’s best entertainments, provocative as he always loved to be. Can you grok it? –Brooks Peck https://notestoself.dev/posts/pycharm-python-script-import-error-attempted-relative-import/ Tue, 23 Feb 2016 00:00:00 +0000 https://notestoself.dev/posts/pycharm-python-script-import-error-attempted-relative-import/ When trying to run a Python script in PyCharm via a run configuration, you might hit an error like this: Traceback (most recent call last): File ".../__main__.py", line 3, in <module> from ... ImportError: attempted relative import with no known parent package Process finished with exit code 1 You might be able to resolve this by setting up the PyCharm run configuration as a “Module name” rather than a “Script path”. https://notestoself.dev/work/poprobincards/ Sat, 23 Jan 2016 00:00:00 +0000 https://notestoself.dev/work/poprobincards/ Pop Robin Cards is a side project that my wife and I work on in our spare time. It’s a custom e-commerce platform selling 3D pop-up cards. The system allows management of a product catalogue, stock, images and so on, as well as a sales and order tracking system. https://notestoself.dev/posts/css-select-last-grid-row/ Sat, 23 Jan 2016 00:00:00 +0000 https://notestoself.dev/posts/css-select-last-grid-row/ Recently I had the need to select the last row of elements in a grid using CSS. The selector needed to be able to handle an arbitrary total number of elements in the grid; in other words, in a grid with xcolumns, there could be 0 to xelements in the last row. With a fixed or predictable number of elements in the grid, this would be quite straightforward. You can select on :nth-last-child(-n+x), with xbeing the known number of elements in the last row. https://notestoself.dev/reading/23-things-they-dont-tell-you-about-capitalism/ Fri, 01 Jan 2016 00:00:00 +0000 https://notestoself.dev/reading/23-things-they-dont-tell-you-about-capitalism/ https://notestoself.dev/reading/59-seconds/ Fri, 01 Jan 2016 00:00:00 +0000 https://notestoself.dev/reading/59-seconds/ A psychologist and best-selling author gives us a myth-busting response to the self-help movement, with tips and tricks to improve your life that come straight from the scientific community.Richard Wiseman has been troubled by the realization that the self-help industry often promotes exercises that destroy motivation, damage relationships, and reduce creativity: the opposite of everything it promises. Now, in 59 Seconds, he fights back, bringing together the diverse scientific advice that can help you change your life in under a minute, and guides you toward becoming more decisive, more imaginative, more engaged, and altogether more happy.From mood to memory, persuasion to procrastination, resilience to relationships, Wiseman outlines the research supporting the new science of �rapid change� and, with clarity and infectious enthusiasm, describes how these quirky, sometimes counterintuitive techniques can be effortlessly incorporated into your everyday life. Or, as he likes to say: �Think a little, change a lot.� https://notestoself.dev/reading/a-bitter-revolution/ Fri, 01 Jan 2016 00:00:00 +0000 https://notestoself.dev/reading/a-bitter-revolution/ https://notestoself.dev/reading/budding-prospects/ Fri, 01 Jan 2016 00:00:00 +0000 https://notestoself.dev/reading/budding-prospects/ https://notestoself.dev/reading/fluent-forever/ Fri, 01 Jan 2016 00:00:00 +0000 https://notestoself.dev/reading/fluent-forever/ Presents a series of techniques for acquiring a foreign language, including news ways of training the tongue for pronunciation, use of visual imagery for connecting sounds and spelling, and spaced-repetition methods for learning new vocabulary. https://notestoself.dev/reading/fooled-by-randomness/ Fri, 01 Jan 2016 00:00:00 +0000 https://notestoself.dev/reading/fooled-by-randomness/ https://notestoself.dev/reading/haunted/ Fri, 01 Jan 2016 00:00:00 +0000 https://notestoself.dev/reading/haunted/ https://notestoself.dev/reading/look-to-windward/ Fri, 01 Jan 2016 00:00:00 +0000 https://notestoself.dev/reading/look-to-windward/ The Twin Novae battle had been one of the last of the Idiran war, and one of the most horrific: desperate to avert their inevitable defeat, the Idirans had induced not one but two suns to explode, snuffing out worlds and biospheres teeming with sentient life. They were attacks of incredible proportion – gigadeathcrimes. But the war ended, and life went on. Now, eight hundred years later, light from the first explosion is about to reach the Masaq’ Orbital, home to the Culture’s most adventurous and decadent souls. There it will fall upon Masaq’s 50 billion inhabitants, gathered to commemorate the deaths of the innocent and to reflect, if only for a moment, on what some call the Culture’s own complicity in the terrible event. Also journeying to Masaq’ is Major Quilan, an emissary from the war-ravaged world of Chel. In the aftermath of the conflict that split his world apart, most believe he has come to Masaq’ to bring home Chel’s most brilliant star and self-exiled dissident, the honored Composer Ziller. Ziller claims he will do anything to avoid a meeting with Major Quilan, who he suspects has come to murder him. But the Major’s true assignment will have far greater consequences than the death of a mere political dissident, as part of a conspiracy more ambitious than even he can know – a mission his superiors have buried so deeply in his mind that even he cannot remember it. Hailed by SFX magazine as an excellent hopping-on point if you’ve never read a Banks SF novel before, Look to Windward is an awe-inspiring immersion into the wildly original, vividly realized civilization that Banks calls the Culture. https://notestoself.dev/reading/survivor/ Fri, 01 Jan 2016 00:00:00 +0000 https://notestoself.dev/reading/survivor/ https://notestoself.dev/reading/the-art-of-thinking-clearly/ Fri, 01 Jan 2016 00:00:00 +0000 https://notestoself.dev/reading/the-art-of-thinking-clearly/ https://notestoself.dev/reading/the-brothers-karamazov/ Fri, 01 Jan 2016 00:00:00 +0000 https://notestoself.dev/reading/the-brothers-karamazov/ https://notestoself.dev/reading/the-colour-of-magic/ Fri, 01 Jan 2016 00:00:00 +0000 https://notestoself.dev/reading/the-colour-of-magic/ https://notestoself.dev/reading/the-left-hand-of-darkness/ Fri, 01 Jan 2016 00:00:00 +0000 https://notestoself.dev/reading/the-left-hand-of-darkness/ https://notestoself.dev/reading/the-light-fantastic/ Fri, 01 Jan 2016 00:00:00 +0000 https://notestoself.dev/reading/the-light-fantastic/ Only one individual can save the world from a disastrous collision. Unfortunately, the hero happens to be the singularly inept wizard Rincewind, who was last seen falling off the edge of the world. https://notestoself.dev/reading/the-trial/ Fri, 01 Jan 2016 00:00:00 +0000 https://notestoself.dev/reading/the-trial/ Kafka’s classic novel of the terror state, in which profound unease and sinister uncertainty simmer beneath Joseph K.‘s surface world of desperate normality and security. When arrested, he is spun into a panic and slowly, strangely, a helpless sense of guilt emerges in him. https://notestoself.dev/posts/python-island-counting-max-area-perimeter-algorithms/ Wed, 23 Dec 2015 00:00:00 +0000 https://notestoself.dev/posts/python-island-counting-max-area-perimeter-algorithms/ Here are three similar algorithms in Python for operations on “island problems”. These are problems were the input is a 2D matrix of grid coordinates, where 0 represents sea and 1 represents land. For example: [ [1, 0, 1, 0, 1], [1, 0, 1, 0, 0], [0, 0, 1, 1, 0], [0, 0, 0, 0, 0], [1, 1, 0, 0, 1], ] It’s slightly hard to see, but there are 5 islands in this grid: one in each corner, and a larger contiguous one in the shape of an L. https://notestoself.dev/posts/keep-controllers-thin/ Mon, 23 Nov 2015 00:00:00 +0000 https://notestoself.dev/posts/keep-controllers-thin/ The world of professional web development is dominated by the MVC pattern. MVC is so common in object oriented web development that it seems fundamental: of course your app has models, views and controllers (although other approaches could be perfectly valid). One bit of advice that tends to get passed around for implementing with the MVC pattern is “thin controllers and fat models”, or some variation of that. I think this maxim misses the point. https://notestoself.dev/posts/validating-urls-with-the-dom/ Mon, 05 Oct 2015 00:00:00 +0000 https://notestoself.dev/posts/validating-urls-with-the-dom/ URL validation is a common task in Web development, and thankfully a lot of cases can now be handled by the browser with the url input type: <input type="url" name="gimme-a-link"></input> That’s great and gives you a lot of functionality and good user experience on a range of devices. However, a couple of issues remain once you’ve got your url type input in place: The browser’s URL validation is fast and accurate, but can be too permissive. https://notestoself.dev/posts/specification-driven-development/ Wed, 23 Sep 2015 00:00:00 +0000 https://notestoself.dev/posts/specification-driven-development/ There’s a great free ebook on test driven development by Grzegorz Gałęzowski. I read a lot of books related to programming, and this one stood out as particularly worthwhile. Aside from a range of useful approaches and gems of understanding I got from the book, the biggest gain for me was an appreciation that test driven development isn’t really about testing as such. It could instead be named “specification- driven development”, which to me gives a better sense of what its main benefits are (and is what TDD actually means). https://notestoself.dev/posts/random-memorable-passwords-bash/ Thu, 17 Sep 2015 00:00:00 +0000 https://notestoself.dev/posts/random-memorable-passwords-bash/ Making secure passwords is quite a common task, and there are plenty of random password generators around (I like the one in keepassx). Making memorable passwords can be a bit trickier, though. A nice way to make random but memorable passwords is to use a sequence of words and perhaps a short number (obligatory link to xkcd). That kind of password is easy for a human to remember and generally hard to brute-force, so long as the pool of possible words is large enough. https://notestoself.dev/posts/code-smell-env-vars-in-constructor/ Sun, 06 Sep 2015 00:00:00 +0000 https://notestoself.dev/posts/code-smell-env-vars-in-constructor/ TLDR: don’t use env vars in class constructors. Seeing environment variables in a class constructor is a code smell and an anti-pattern. For example in TypeScript for Node.js, you might see something like this: Anti-pattern: class Foobar { private niceInjectableDependency: IDependency; private importantConfig: string; constructor(niceInjectableDependency: IDependency) { this.niceInjectableDependency = niceInjectableDependency; this.importantConfig = process.env.IMPORTANT_CONFIG; // code smell } } This might seem reasonable, as we’re still kind of able to inject the importantConfig into the class, just not via the constructor. https://notestoself.dev/posts/unix-fu-anki/ Wed, 01 Jul 2015 00:00:00 +0000 https://notestoself.dev/posts/unix-fu-anki/ Anki has helped me maintain and improve my Chinese for years, but I’ve always been intrigued by using it to develop other kinds of knowledge. At university I used it heavily for East Asian history, our modern Chinese literature course and classical Chinese. I’ve now started using it to improve my Unix command-line fluency, and the results have been quite dramatic. It’s not really possible to learn programming with an SRS system like Anki, but you can certainly use it to reinforce tokenized knowledge (like [this](https://developer. https://notestoself.dev/posts/minimum-required-lecture-rooms-scheduling-python/ Tue, 23 Jun 2015 00:00:00 +0000 https://notestoself.dev/posts/minimum-required-lecture-rooms-scheduling-python/ Given a schedule of lectures represented as a list of tuples of start and end times like [(15, 16), (13, 15), (14, 15)], return the minimum number of lecture rooms required to run all of the lectures. We can solve this in $O(n\log_2n)$, due to the need to sort the lecture times. The overall steps are: Convert time tuples into “lecture event” tuples of time and type, i.e. start or end. https://notestoself.dev/posts/wget-recursive-downloads/ Sun, 31 May 2015 00:00:00 +0000 https://notestoself.dev/posts/wget-recursive-downloads/ I discovered the VimCasts website recently and have been watching a few of their videos. Due to my unreliable internet connection, I prefer to download whole video files and then watch them (this also allows watching them on portable devices). VimCasts allow directory listing on their storage server, which I believe is permission to bulk download their content. The only issue is that each episode is stored in its own subdirectory and in two formats. https://notestoself.dev/posts/ugly-code/ Wed, 27 May 2015 00:00:00 +0000 https://notestoself.dev/posts/ugly-code/ Before I really got into development, I went through the entire archive of Joel on Software reading every single article. I like to think that it set me up with good foundational knowledge for professional software development, but that’s pretty hard to prove. Recently I re-read the article Things You Should Never Do, Part I. With a lot more development experience under my belt, the article spoke to me much more strongly this time around. https://notestoself.dev/posts/gulp-asset-versioning-hugo-data-files/ Sat, 23 May 2015 00:00:00 +0000 https://notestoself.dev/posts/gulp-asset-versioning-hugo-data-files/ I recently finished converting four of the websites I run from Wordpress to Hugo (my original blog, a Chinese learning blog, a Chinese grammar site and this blog), and hosting them with S3 and CloudFront. It’s been a great change to make and has improved everything from my workflow to loading times and hosting costs. I’d recommend anyone running blogs and other text-focused sites to switch away from Wordpress and use a static site generator instead. https://notestoself.dev/posts/avoid-naive-string-indexing-php/ Thu, 23 Apr 2015 00:00:00 +0000 https://notestoself.dev/posts/avoid-naive-string-indexing-php/ PHP allows treating a string as an array, so that you can use indexing syntax to get or set a single position in the string: <?php echo 'hello'[0]; // h $foo = 'bar'; $foo[2] = 'z'; echo $foo; // baz This seems handy but you should never do it. The reason to avoid string indexing like this in PHP is that PHP strings are not multibyte character strings, they’re just bytes. https://notestoself.dev/posts/ubuntu-altgr-type-arrow-quote-characters/ Mon, 23 Feb 2015 00:00:00 +0000 https://notestoself.dev/posts/ubuntu-altgr-type-arrow-quote-characters/ A quick Ubuntu Linux tip: you can use the AltGr key to easily type various arrow and quote characters. The ones you might commonly want to use are: AltGr + y ← Left arrow AltGr + u ↓ Down arrow AltGr + i → Right arrow AltGr + Shift + u ↑ Up arrow AltGr + z « Left double angle quote AltGr + x » Right double angle quote AltGr + v “ Left double quotation mark AltGr + b ” Right double quotation mark https://notestoself.dev/posts/python-topological-course-prerequisites/ Fri, 23 Jan 2015 00:00:00 +0000 https://notestoself.dev/posts/python-topological-course-prerequisites/ We want to figure out the sequence we can take university courses in, and some of the courses may have prerequisite courses. In other words, we want a topological sort of the courses. For this example, we can just ignore any courses that are impossible to take because they depend on non-existent courses, or have a circular dependency. The input data is like this: {'C3': {'C2', 'C1'}, 'C2': {'C1'}, 'C1': {}} And the desired output is like this: https://notestoself.dev/reading/alexs-adventures-in-numberland/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/alexs-adventures-in-numberland/ https://notestoself.dev/reading/bash-cookbook/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/bash-cookbook/ https://notestoself.dev/reading/blink/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/blink/ Blink is about the first two seconds of looking–the decisive glance that knows in an instant. Gladwell, the best-selling author of The Tipping Point, campaigns for snap judgments and mind reading with a gift for translating research into splendid storytelling. Building his case with scenes from a marriage, heart attack triage, speed dating, choking on the golf course, selling cars, and military maneuvers, he persuades readers to think small and focus on the meaning of thin slices of behavior. The key is to rely on our adaptive unconscious–a 24/7 mental valet–that provides us with instant and sophisticated information to warn of danger, read a stranger, or react to a new idea. Gladwell includes caveats about leaping to conclusions: marketers can manipulate our first impressions, high arousal moments make us mind blind, focusing on the wrong cue leaves us vulnerable to the Warren Harding Effect (i.e., voting for a handsome but hapless president). In a provocative chapter that exposes the dark side of blink, he illuminates the failure of rapid cognition in the tragic stakeout and murder of Amadou Diallo in the Bronx. He underlines studies about autism, facial reading and cardio uptick to urge training that enhances high-stakes decision-making. In this brilliant, cage-rattling book, one can only wish for a thicker slice of Gladwell’s ideas about what Blink Camp might look like. –Barbara Mackoff https://notestoself.dev/reading/consider-phlebas/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/consider-phlebas/ https://notestoself.dev/reading/david-and-goliath/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/david-and-goliath/ In the past decade, Malcolm Gladwell has written three books that have radically changed how we understand our world and ourselves: The Tipping Point, Blink, and Outliers. Regarded by many as the most gifted and influential author and journalist in America today, Gladwell’s rare ability to connect with audiences of such varied interests has ensured that each title become a phenomenal bestseller with more than ten million copies in print combined. Now, Gladwell’s landmark investigations into the world around us are collected together for the first time. Beautifully repackaged and redesigned, including for the first time illustrations throughout each book, MALCOLM GLADWELL: COMPLETE is a perfect treasury of prose and provocation for Gladwell fans old and new. https://notestoself.dev/reading/dead-air/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/dead-air/ https://notestoself.dev/reading/distrust-that-particular-flavour/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/distrust-that-particular-flavour/ https://notestoself.dev/reading/effective-programming-more-than-writing-code/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/effective-programming-more-than-writing-code/ ABOUT THE BOOK Jeff Atwood began the Coding Horror blog in 2004, and is convinced that it changed his life. He needed a way to keep track of software development over time - whatever he was thinking about or working on. He researched subjects he found interesting, then documented his research with a public blog post, which he could easily find and refer to later. Over time, increasing numbers of blog visitors found the posts helpful, relevant and interesting. Now, approximately 100,000 readers visit the blog per day and nearly as many comment and interact on the site. Effective Programming: More Than Writing Code is your one-stop shop for all things programming. Jeff writes with humor and understanding, allowing for both seasoned programmers and newbies to appreciate the depth of his research. From such posts as The Programmer’s Bill of Rights and Why Cant Programmers… Program? to Working With the Chaos Monkey, this book introduces the importance of writing responsible code, the logistics involved, and how people should view it more as a lifestyle than a career. TABLE OF CONTENTS - Introduction - The Art of Getting Shit Done - Principles of Good Programming - Hiring Programmers the Right Way - Getting Your Team to Work Together - The Batcave: Effective Workspaces for Programmers - Designing With the User in Mind - Security Basics: Protecting Your Users’ Data - Testing Your Code, So it Doesn’t Suck More Than it Has To - Building, Managing and Benefiting from a Community - Marketing Weasels and How Not to Be One - Keeping Your Priorities Straight EXCERPT FROM THE BOOK As a software developer, you are your own worst enemy. The sooner you realize that, the better off you’ll be.I know you have the best of intentions. We all do. We’re software developers; we love writing code. It’s what we do. We never met a problem we couldn’t solve with some duct tape, a jury-rigged coat hanger and a pinch of code. But Wil Shipley argues that we should rein in our natural tendencies to write lots of code: The fundamental nature of coding is that our task, as programmers, is to recognize that every decision we make is a trade-off. To be a master programmer is to understand the nature of these trade-offs, and be conscious of them in everything we write.In coding, you have many dimensions in which you can rate code: Brevity of codeFeaturefulnessSpeed of executionTime spent codingRobustnessFlexibility Now, remember, these dimensions are all in opposition to one another. You can spend three days writing a routine which is really beautiful and fast, so you’ve gotten two of your dimensions up, but you’ve spent three days, so the time spent coding dimension is way down.So, when is this worth it? How do we make these decisions? The answer turns out to be very sane, very simple, and also the one nobody, ever, listens to: Start with brevity. Increase the other dimensions as required by testing. I couldn’t agree more. I’ve given similar advice when I exhorted developers to Code Smaller. And I’m not talking about a reductio ad absurdum contest where we use up all the clever tricks in our books to make the code fit into less physical space. I’m talking about practical, sensible strategies to reduce the volume of code an individual programmer has to read to understand how a program works. Here’s a trivial little example of what I’m talking about: if (s == String.Empty)if (s == ) It seems obvious to me that the latter case is… …buy the book to read more! https://notestoself.dev/reading/everything-i-know/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/everything-i-know/ The authors provide the new teacher with guidance and advice that is full of encouragement, humor, and practical ideas, all based on real first-year experiences. https://notestoself.dev/reading/excession/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/excession/ https://notestoself.dev/reading/head-first-design-patterns/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/head-first-design-patterns/ https://notestoself.dev/reading/high-performance-mysql/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/high-performance-mysql/ https://notestoself.dev/reading/how-to-stop-sucking-and-be-awesome-instead/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/how-to-stop-sucking-and-be-awesome-instead/ https://notestoself.dev/reading/inviting-disaster/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/inviting-disaster/ https://notestoself.dev/reading/learning-php-design-patterns/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/learning-php-design-patterns/ With the design patterns in this book, you�ll be able to build PHP applications much more efficiently. Learning PHP Design Patterns shows you how to use patterns through simple examples, and then demonstrates many of them in full-fledged working applications. You�ll learn patterns that help you connect PHP and MySQL, as well as several pattern categories that encapsulate object-oriented programming practices and concepts, such as polymorphism.These patterns allow you to adopt a more sophisticated programming style, focusing on language improvements introduced in PHP 5. You�ll also learn patterns that help you avoid common programming problems. https://notestoself.dev/reading/matter/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/matter/ In a world renowned even within a galaxy full of wonders, a crime within a war. For one man it means a desperate flight, and a search for the one - maybe two - people who could clear his name. For his brother it means a life lived under constant threat of treachery and murder. And for their sister, even without knowing the full truth, it means returning to a place she’d thought abandoned forever.Only the sister is not what she once was; Djan Seriy Anaplian has changed almost beyond recognition to become an agent of the Culture’s Special Circumstances section, charged with high-level interference in civilizations throughout the greater galaxy.Concealing her new identity - and her particular set of abilities - might be a dangerous strategy, however. In the world to which Anaplian returns, nothing is quite as it seems; and determining the appropriate level of interference in someone else’s war is never a simple matter.MATTER is a novel of dazzling wit and serious purpose. An extraordinary feat of storytelling and breathtaking invention on a grand scale, it is a tour de force from a writer who has turned science fiction on its head. https://notestoself.dev/reading/never-let-me-go/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/never-let-me-go/ https://notestoself.dev/reading/predictably-irrational/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/predictably-irrational/ https://notestoself.dev/reading/the-algebraist/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/the-algebraist/ https://notestoself.dev/reading/the-emperor-far-away/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/the-emperor-far-away/ For more than two thousand years, as capital of the powerful Nan Yue kingdom to its present position as the largest city in southern China, Canton has been one of Asia’s most important commercial, political, and cultural centres. Arab merchants first arrived in the Tang dynasty, trading spices, amber, and ivory for silk, ceramics, and precious stones. During the eighteen century, for the next hundred years, it was the only port in China where the emperor allowed merchants from the West to do business. With trade came the interchange of ideas. Missionaries, educators, political leaders, and modernizers, Chinese and non-Chinese alike, all tapped Canton’s energy in their efforts to direct the course of China’s cultural and political development. https://notestoself.dev/reading/the-establishment/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/the-establishment/ Why they cheat… Presidents, religious leaders, CEOs, school teachers, moms, dads, young, old…they all do it and it’s all too common even in what many of us would consider ‘great’ relationships-but why? One recent survey showed that nearly 70% of respondents in self-described monogamous relationships have confessed they’ve either thought about cheating, have cheated, or are currently cheating…and their partner has no idea. In Why They Cheat and How They Get Away With It, J.L. Ford explains why almost every relationship is at risk and exposes that aspect of human nature that compels otherwise responsible men and women to act on the impulse to cheat-regardless of the personal or professional consequences. https://notestoself.dev/reading/the-hydrogen-sonata/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/the-hydrogen-sonata/ https://notestoself.dev/reading/the-player-of-games/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/the-player-of-games/ https://notestoself.dev/reading/the-pragmatic-programmer/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/the-pragmatic-programmer/ Straight from the programming frenches, The Pragmatic Programmer cuts through the increasing specialization and technicalities of modern software development to examine the core process - taking a requirement and producing working, maintainable code that delights its users. It covers topics ranging from personal responsibility and career development to architectural techniques for keeping your code flexible and easy to adapt and reuse.–BOOK JACKET. https://notestoself.dev/reading/the-state-of-the-art/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/the-state-of-the-art/ https://notestoself.dev/reading/the-undercover-economist/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/the-undercover-economist/ https://notestoself.dev/reading/the-way-to-go/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/the-way-to-go/ This book provides the reader with a comprehensive overview of the new open source programming language Go (in its first stable and maintained release Go 1) from Google. The language is devised with Java / C#-like syntax so as to feel familiar to the bulk of programmers today, but Go code is much cleaner and simpler to read, thus increasing the productivity of developers. You will see how Go: simplifies programming with slices, maps, structs and interfaces incorporates functional programming makes error-handling easy and secure simplifies concurrent and parallel programming with goroutines and channels And you will learn how to: make use of Go’s excellent standard library program Go the idiomatic way using patterns and best practices in over 225 working examples and 135 exercises This book focuses on the aspects that the reader needs to take part in the coming software revolution using Go. https://notestoself.dev/reading/thinking-statistically/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/thinking-statistically/ The paperback printing of Thinking Statistically includes all the material from the Kindle original plus innovative new segments giving graphical representations of statistics concepts. Thinking Statistically is the book that shows you how to think like a statistician, without worrying about formal statistical techniques. Along the way we learn how selection bias can explain why your boss doesn�t know he sucks (even when everyone else does); how to use Bayes� Theorem to decide if your partner is cheating on you; and why Mark Zuckerberg should never be used as an example for anything. See the world in a whole new light, and make better decisions and judgements without ever going near a t-test. Think. Think Statistically. https://notestoself.dev/reading/%E4%B8%89%E5%88%86%E9%8D%BE%E5%B0%8F%E8%AA%AA/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/%E4%B8%89%E5%88%86%E9%8D%BE%E5%B0%8F%E8%AA%AA/ https://notestoself.dev/reading/%E9%81%8A%E6%88%B2%E7%8E%A9%E5%AE%B6/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/%E9%81%8A%E6%88%B2%E7%8E%A9%E5%AE%B6/ https://notestoself.dev/reading/%E9%A7%B1%E9%A7%9D%E7%A5%A5%E5%AD%90/ Thu, 01 Jan 2015 00:00:00 +0000 https://notestoself.dev/reading/%E9%A7%B1%E9%A7%9D%E7%A5%A5%E5%AD%90/ https://notestoself.dev/posts/straightforward-recursive-number-to-words-function-typescript/ Fri, 26 Dec 2014 00:00:00 +0000 https://notestoself.dev/posts/straightforward-recursive-number-to-words-function-typescript/ This is a recursive number-to-words algorithm in TypeScript. It tries to be straightforward and just uses if statements to make the logic a bit easier to follow. The key ideas are: Let the recursion do the work. Handle “and” carefully. Don’t move up in factors of ten; move up in word groups. function numberToWords(num: number): string { if (num < 0) { return `minus ${numberToWords(-num)}`; } if (num in DIRECT) { return DIRECT[num]; } if (num < 100) { const units = num % 10; const tens = num - units; return `${DIRECT[tens]}${numberToWords(units)}`; } if (num < 1000) { const sub_hundreds = num % 100; const hundreds = (num - sub_hundreds) / 100; let words = `${numberToWords(hundreds)}hundred`; if (sub_hundreds) { words += ` and ${numberToWords(sub_hundreds)}`; } return words; } if (num < 1000000) { const sub_thous = num % 1000; const thousands = (num - sub_thous) / 1000; let words = `${numberToWords(thousands)}thousand`; if (sub_thous) { if (sub_thous < 100) { words += " and"; } words += ` ${numberToWords(sub_thous)}`; } return words; } if (num < 1000000000) { const sub_mills = num % 1000000; const millions = (num - sub_mills) / 1000000; let words = `${numberToWords(millions)}million`; if (sub_mills) { if (sub_mills < 100) { words += " and"; } words += ` ${numberToWords(sub_mills)}`; } return words; } throw new Error("I can't count that high sorry"); } Jest tests: https://notestoself.dev/posts/typescript-recursive-array-product-function-algorithm/ Fri, 26 Dec 2014 00:00:00 +0000 https://notestoself.dev/posts/typescript-recursive-array-product-function-algorithm/ This is a productOfArray(numbers) function in TypeScript that uses recursion. function productOfArray(numbers: number[]): number { if (numbers.length === 0) { return 0; } if (numbers.length === 1) { return numbers[0]; } return numbers[0] * productOfArray(numbers.slice(1)); } Jest tests: describe(productOfArray.name, () => { test("multiplies array elements together", () => { expect(productOfArray([])).toBe(0); expect(productOfArray([5])).toBe(5); expect(productOfArray([1, 2, 3])).toBe(6); expect(productOfArray([5, 5, 5])).toBe(125); expect(productOfArray([8, 8])).toBe(64); expect(productOfArray([1, 2, 3, 4, 5, 6, 7, 8, 9])).toBe(362880); }); }); https://notestoself.dev/posts/typescript-recursive-factorial-function-algorithm/ Thu, 25 Dec 2014 00:00:00 +0000 https://notestoself.dev/posts/typescript-recursive-factorial-function-algorithm/ This is a factorial(factor) function in TypeScript that uses recursion. function factorial(factor: number): number { if (factor <= 1) { return 1; } return factor * factorial(factor - 1); } Jest tests: describe(factorial.name, () => { test("recursive factorial", () => { expect(factorial(0)).toBe(1); expect(factorial(1)).toBe(1); expect(factorial(2)).toBe(2); expect(factorial(3)).toBe(6); expect(factorial(4)).toBe(24); expect(factorial(5)).toBe(120); expect(factorial(6)).toBe(720); expect(factorial(7)).toBe(5040); }) }); https://notestoself.dev/posts/typescript-recursive-power-function-algorithm/ Wed, 24 Dec 2014 00:00:00 +0000 https://notestoself.dev/posts/typescript-recursive-power-function-algorithm/ This is a power(base, exponent) function in TypeScript that uses recursion instead of the Math.pow() function. function power(base: number, exponent: number): number { if (exponent === 0) { return 1; } return base * power(base, exponent - 1); } Jest tests: describe("power", () => { test("2^x", () => { expect(power(2, 0)).toBe(1); expect(power(2, 1)).toBe(2); expect(power(2, 2)).toBe(4); expect(power(2, 3)).toBe(8); expect(power(2, 4)).toBe(16); }) test("3^x", () => { expect(power(3, 0)).toBe(1); expect(power(3, 1)).toBe(3); expect(power(3, 2)).toBe(9); expect(power(3, 3)). https://notestoself.dev/posts/php-string-obscure-function/ Tue, 23 Dec 2014 00:00:00 +0000 https://notestoself.dev/posts/php-string-obscure-function/ It’s quite common to need to obscure sensitive information when displaying it to a client. For example, it’s safer to display a user’s email address as h***.g****@g****.c**. If someone views the page, the user’s personal information isn’t fully revealed, but the user can still confirm that it’s correct. This post is just a note of a helper function to do this in PHP. <?php /** * @param string|string[] $plain * @param int $revealStart * @param int $revealEnd * @param string $obscuration * * @return string|string[] */ function obscure( $plain, int $revealStart = 1, int $revealEnd = 0, string $obscuration = '*' ) { if (is_array($plain)) { return array_map( function ($plainPart) use ($revealStart, $revealEnd, $obscuration) { return obscure($plainPart, $revealStart, $revealEnd, $obscuration); }, $plain ); } $plain = (string) $plain; return mb_substr($plain, 0, $revealStart) . https://notestoself.dev/posts/observer-pattern-side-effects/ Wed, 05 Nov 2014 00:00:00 +0000 https://notestoself.dev/posts/observer-pattern-side-effects/ The observer pattern is extremely common in a lot of software, particularly in web apps. Most web frameworks, ORMs and other boilerplate systems have a built in structure for events and observers, making it easy to listen to various built-in events and to fire customised events of your own. The observer pattern is one of the classic design patterns – it’s number 7 in the Gang of Four – and is one of the patterns that I deal with most commonly day to day. https://notestoself.dev/posts/initial-commit/ Thu, 23 Oct 2014 00:00:00 +0000 https://notestoself.dev/posts/initial-commit/ This is the first post on my new blog. I’ll be writing about various unrelated things that happen to interest me at the time. This blog used to be on WordPress, then Jekyll hosted on Github pages, and this iteration uses Hugo and AWS for hosting. I believe I’ll stick with Hugo and AWS for some time as I’ve never enjoyed a CMS and hosting setup as much as this. https://notestoself.dev/posts/learning-vocabulary-on-the-commandline/ Thu, 23 Oct 2014 00:00:00 +0000 https://notestoself.dev/posts/learning-vocabulary-on-the-commandline/ I like the idea of adding little bits of learning to my day that don’t take much effort once I’ve set them up. They’re not the most effective way to learn, but they certainly don’t hurt and every now and then you do get something worthwhile out of them. One of the things I’ve set up recently is displaying random Chinese vocabulary items at the start of every command line session in Zsh. https://notestoself.dev/work/lovecrafts/ Mon, 22 Sep 2014 00:00:00 +0000 https://notestoself.dev/work/lovecrafts/ In 2014-2015, I worked on the ecommerce platform, which is based on a heavily customised Magento EE installation with integrations to other business systems. I then worked on the Social API team through to 2018, building a central APIs and microservices for the LoveCrafts social network using code-generation techniques. https://notestoself.dev/work/china-checkup/ Thu, 23 Jan 2014 00:00:00 +0000 https://notestoself.dev/work/china-checkup/ I was the sole developer on ChinaCheckup.com from 2013 to 2014, handling all front-end and back-end development, as well as managing the server. China Checkup sells complex business analysis services to clients outside of China. The site is a custom ecommerce solution based on the Laravel framework. It automates much of the workflow in creating, modifying and selling these services to clients. Some time after I left the company, China Checkup migrated to the Shopify platform. https://notestoself.dev/posts/longest-distinct-substring-algorithm-javascript/ Thu, 09 Jan 2014 00:00:00 +0000 https://notestoself.dev/posts/longest-distinct-substring-algorithm-javascript/ Given a string, find the longest substring of distinct characters. function findLongestDistinctSubstring(text) { const lastPos = {}; let maxLength = 0; let left = 0; for (let right = 0; right < text.length; right++) { const char = text[right]; if (char in lastPos) { // Can't go backwards, so use max() to ensure left = Math.max(left, lastPos[char] + 1); } lastPos[char] = right; maxLength = Math.max(maxLength, (right - left) + 1); } return maxLength; } This solution has $O(n)$ time complexity because we iterate the input string once, and $O(n)$ space complexity because we build up a mapping of input characters to positions. https://notestoself.dev/posts/min-sub-array-length-algorithm-javascript/ Wed, 08 Jan 2014 00:00:00 +0000 https://notestoself.dev/posts/min-sub-array-length-algorithm-javascript/ Given an array of positive intergers and a positive integer target, return the length of the shortest possible contiguos sub-array with a sum greater than or equal to the target. function minSubArrayLen(items, targ) { let left = 0; let right = 0; let currSum = 0; let minLen = Infinity; while (left < items.length) { if (currSum < targ) { if (right === items.length) { break; } currSum += items[right]; right++; } else { minLen = Math. https://notestoself.dev/posts/max-sub-array-sum-algorithm-javascript/ Tue, 07 Jan 2014 00:00:00 +0000 https://notestoself.dev/posts/max-sub-array-sum-algorithm-javascript/ Given an array of numbers and an integer “width”, find the max sum of a contiguous sub-array with that width in positions. function maxSubArraySum(numbers, width) { if (width >= numbers.length) { return numbers.reduce((sum, num) => sum + num, 0) } let subArraySum = numbers.slice(0, width).reduce((sum, num) => sum + num, 0); let maxSubArraySum = subArraySum; for (let t = 0, h = width; h < numbers.length; t++, h++) { subArraySum -= numbers[t]; subArraySum += numbers[h]; maxSubArraySum = Math. https://notestoself.dev/posts/string-is-subsequence-algorithm-javascript/ Mon, 06 Jan 2014 00:00:00 +0000 https://notestoself.dev/posts/string-is-subsequence-algorithm-javascript/ Given two strings needle and haystack, check if needle is a subsequence (not necessarily a substring) of haystack at any point. That means that the letters in needle must appear in haystack in the same order, but not necessarily contiguously. function isSubsequence(needle, haystack) { if (!needle) { return true; } if (haystack.length < needle.length) { return true; } if (haystack === needle) { return true; } let ni = 0; let hi = 0; while (hi < haystack. https://notestoself.dev/posts/sorted-array-average-pair-algorithm-javascript/ Sun, 05 Jan 2014 00:00:00 +0000 https://notestoself.dev/posts/sorted-array-average-pair-algorithm-javascript/ Given a sorted array of numbers and a target average value, return whether or not it is possible to achieve the target average value with a pair of numbers from the array. function canMakeAveragePair(items, target) { let t = 0; let h = items.length - 1; while (t < h) { const avg = (items[t] + items[h]) / 2; if (avg === target) { return true; } else if (avg < target) { t++; } else if (avg > target) { h--; } } return false; } This algorithm is $O(n)$ time and $O(1)$ space. https://notestoself.dev/posts/numbers-digits-same-frequency-algorithm-javascript/ Sat, 04 Jan 2014 00:00:00 +0000 https://notestoself.dev/posts/numbers-digits-same-frequency-algorithm-javascript/ Here’s a super simple duplicate detection algorithm in JavaScript: function hasDuplicate(items) { const seen = {}; for (item of items) { if (item in seen) { return true; } seen[item] = true; } return false; } This algorithm is $O(n)$ time and $O(n)$ space. https://notestoself.dev/posts/count-unique-values-algorithm-javascript/ Fri, 03 Jan 2014 00:00:00 +0000 https://notestoself.dev/posts/count-unique-values-algorithm-javascript/ Given a sorted array of values like [1, 1, 2, 3, 3, 4, 5, 5, 5], count the number of unique values in $O(n)$ time and $O(1)$ space. If there wasn’t a requirement to do this in $O(1)$ space, the most straightforward way to do it would be to use a Set(). We could iterate the input array adding each element to the set, then return set.size. This would be $O(n)$ time but $O(n)$ space, so it wouldn’t meet the space requirement. https://notestoself.dev/posts/valid-anagram-algorithm-javascript/ Thu, 02 Jan 2014 00:00:00 +0000 https://notestoself.dev/posts/valid-anagram-algorithm-javascript/ Given two strings such as "listen" and "silent", return whether or not they are valid anagrams of each other. That is, the second word uses the same letters as the first but potentially in a different order. One approach is to make a frequency count of the first string, and then iterate the second string “checking off” each letter from the frequency count of the first. If we hit a missing letter or reduce the corresponding frequency count to zero, we know it’s not a valid anagram. https://notestoself.dev/posts/numbers-digits-same-frequency-algorithm-javascript/ Wed, 01 Jan 2014 00:00:00 +0000 https://notestoself.dev/posts/numbers-digits-same-frequency-algorithm-javascript/ Given two numbers such as 1987456 and 5796184, check if the two numbers have the same frequency of digits. That is, they have the same digits the same number of times. Another way of putting this would be to check if the digits of the two numbers are anagrams of each other or not. function sameFrequency(a, b) { a = String(a); b = String(b); if (a.length !== b.length) { return false; } const freq_a = {}; const freq_b = {}; for (let i = 0; i < a. https://notestoself.dev/work/hugh-grigg-com/ Wed, 23 Oct 2013 00:00:00 +0000 https://notestoself.dev/work/hugh-grigg-com/ The site you’re on now :-) https://notestoself.dev/work/chinese-boost/ Thu, 23 May 2013 00:00:00 +0000 https://notestoself.dev/work/chinese-boost/ As East Asia Student got increasingly eclectic and unfocused, I decided to branch off the ‘how to learn Chinese’ content to this new site. Like my other text-based sites, Chinese Boost is built using Hugo and hosted with AWS. I get quite a lot of nice feedback from people saying how useful they find Chinese Boost. https://notestoself.dev/work/east-asia-student/ Wed, 05 May 2010 00:00:00 +0000 https://notestoself.dev/work/east-asia-student/ East Asia Student was more active whilst I was at university, but I still add new content from time to time. The site is now built using Hugo and hosted with AWS. https://notestoself.dev/reading/1440-daily-digest/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/1440-daily-digest/ https://notestoself.dev/reading/aeon/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/aeon/ https://notestoself.dev/reading/book-freak/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/book-freak/ https://notestoself.dev/reading/delancey-place/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/delancey-place/ https://notestoself.dev/reading/freek.dev/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/freek.dev/ https://notestoself.dev/reading/hacker-news-digest/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/hacker-news-digest/ https://notestoself.dev/reading/hacker-newsletter/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/hacker-newsletter/ https://notestoself.dev/hire-me/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/hire-me/ https://notestoself.dev/reading/inkstone/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/inkstone/ https://notestoself.dev/reading/james-clear-3-2-1/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/james-clear-3-2-1/ https://notestoself.dev/reading/laravel-news/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/laravel-news/ https://notestoself.dev/reading/marginal-revolution/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/marginal-revolution/ https://notestoself.dev/reading/monevator/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/monevator/ https://notestoself.dev/reading/morning-brew/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/morning-brew/ https://notestoself.dev/reading/mr-money-mustache/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/mr-money-mustache/ https://notestoself.dev/reading/nautilus/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/nautilus/ https://notestoself.dev/reading/news-refinery/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/news-refinery/ https://notestoself.dev/reading/next-draft/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/next-draft/ https://notestoself.dev/reading/ribbonfarm/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/ribbonfarm/ https://notestoself.dev/reading/sinocism/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/sinocism/ https://notestoself.dev/reading/slate-star-codex/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/slate-star-codex/ https://notestoself.dev/reading/the-economist/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/the-economist/ https://notestoself.dev/reading/the-week/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/the-week/ https://notestoself.dev/reading/tim-harford/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/tim-harford/ https://notestoself.dev/reading/typescript-weekly/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/typescript-weekly/ https://notestoself.dev/reading/unintended-consequences/ Mon, 01 Jan 0001 00:00:00 +0000 https://notestoself.dev/reading/unintended-consequences/