
DynamoDB's `begins_with` query function is tremendously useful. It can be used to query records with certain formats (imagine querying based on an ISO date string), or in Single Table Design to match entity types by some canonical prefix. If you're using DynamoDB streams, especially in a Single Table Design setting, you might expect to be able to use the same concept to match events coming through the stream.

AWS's documentation is, to no one's surprise, clear as mud on this subject. Other Lambda triggers have examples detailing how this can be done, but not DynamoDB, which seems to imply one has to perform an exact key match for a trigger to fire.

```json
{
  "dynamodb": {
    "NewImage": {
      "sk": {
        "S": ["SOME_WHOLE_KEY"]
      }
    }
  }
}
```

Thankfully this is not the case, and although a solution isn't obvious, one does exist.

To match records with a specific prefix, you can modify the matching string with an object with key `prefix` and value of whatever prefix you wish to match.

For example, let's say we're storing a ledger of customer payment records. Our key structure looks like this: `pk: CUSTOMER#1234 sk: PAYMENT#2023-02-03`.

Looking at the documentation, you might assume it's impossible to match all new items with an `sk` beginning with `PAYMENT` but fortunately, it's simply a case of altering our filter pattern to something like this:

```json
{
  "dynamodb": {
    "NewImage": {
      "sk": {
        "S": [
          {
            "prefix": "PAYMENT#"
          }
        ]
      }
    }
  }
}
```

Here, we match all new images with an `sk` beginning with `PAYMENT#` — just what we want.


Prefix matching in Lambda DynamoDB triggers


On the subject of generative AI, I bounce back and forth between gleeful childish optimism, and something approaching apocalyptic terror.

I may well write a full egotistical opinion piece on that another time, but for now I'll summarise by saying that the practical applications of neural networks and LLMs in fields like science and medicine will quite likely be a revolutionary force for good in the medium and long term. But, as with almost all technology, there are countless and unforeseeable negative applications too. Although I don't think the AIs are going to turn sentient and decide to exterminate us, I'm more concerned that LLMs will continue to be exploited by dark political actors and authoritarian nation states for anti-democratic purposes. There's a real risk that convincing populist misinformation can be produced so cheaply and so prodigiously that it eventually reduces human society to something subservient and pitiful, or drive us all into solipsistic madness. Though it's unlikely to be _quite_ that bad, the prospect of machines that can write natural language in a convincingly coherent fashion is staggering, and until I actually started using an LLM, I struggled to keep the fear in check.

Society's presumably impending collapse aside (this is sarcasm, if you're in an AI anxiety phase), the other worry about advanced LLMs and generative AI is that it will replace software engineers. As all software engineers have had to do in the last six months, I've considered the possibility that an AI could come along and automate me out of a job. For a while I was almost convinced; the seemingly weekly quantum leaps in capability made this seem inevitable. Now that the novelty shock is wearing off, and the prospect of AI causing mass unemployment seems less likely, especially in the case of software engineers; I no more think AI will make software engineers redundant than I think DAWs will make musicians obsolete, or a brick-laying machine will make builders a thing of the past; you still need someone who understands music, who knows what a house needs to be and do for the tools to have value.

As a seemingly increasing number of engineers seem to have done already, I'm arriving at the conclusion that generative AI is an assistive tool, something that can help us build the things we build by at least partly automating some source code writing. I've experimented with GPT-3 for a little while, and far from being scared into assuming I'll be out of a job in the next few years, I've actually been relieved that it is both more useful than I hoped, and less useful than I feared. (Side-note: I'm aware that GPT-3.5 and 4 are both more powerful than GPT-3, but the fundamentals of the tests I want to do are roughly the same. From what I've read about GPT-4, I'm ready to raise my eyebrows by no more than a couple of millimetres.)

Modern software engineering, for the majority of us, is about far more than just writing source code. There's a tremendous amount of reasoning and planning; architecture and building, which require cognition and reasoning. At a basic level, these are things that generative AIs are simply can't do, no matter how hard the blow-hards blow. This has become clear over the last few months, and I remain sceptical of some of the larger demonstrations of GPT-generated software whose source code remains closed.

Assuming that AI will end up automating some monotony in my job, a brick-laying machine, I wanted to know how useful it would be as a coding tool, something to generate source code for me. I've started small, and asked it to do something that I've had to implement a couple of times in my career: given an input array, split that array into several smaller arrays of a given size. Let's see how the robot got on with that.

> Write a function in JavaScript to divide an array of items into smaller arrays of variable length.

```js
function chunkArray(arr, size) {
  const chunkedArr = [];
  for (let i = 0; i < arr.length; i += size) {
    chunkedArr.push(arr.slice(i, i + size));
  }
  return chunkedArr;
}
```

If I had Googled this same question and landed on Stack Overflow, this is precisely the answer I would expect; a simple `for` loop that does the job. The formatting is a bit weird and clunky (or was, until my editor formatted it), and it's a little hard to grok but, importantly, it _does_ work. Let's see if we can get it to generate something similar using slightly different syntax:

> Write a function in JavaScript to divide an array of items into smaller arrays of variable length. You must use the Array.reduce method, and must not use a for loop.

```js
function chunkArrayReduce(arr, size) {
  return arr.reduce((acc, el, i) => {
    const div = Math.floor(i / size);
    if (!acc[div]) {
      acc[div] = [];
    }
    acc[div].push(el);
    return acc;
  }, []);
}
```

I've implemented this function in exactly the same way for years, so this is a good result. I find it interesting how it changed the name of the function and the names of the variables, and how it added a guard against undefined chunks without being told that was a risk. I mostly use TypeScript these days, so let's ask it to add in some types.

> Refactor the chunkArrayReduce function in TypeScript. The array must use a generic type.

```typescript
function chunkArrayReduce<T>(arr: T[], size: number): T[][] {
  return arr.reduce((acc, el, i) => {
    const div = Math.floor(i / size);
    if (!acc[div]) {
      acc[div] = [];
    }
    acc[div].push(el);
    return acc;
  }, []);
}
```

The fact that it can correctly apply generic types with minimal prompting is quite impressive. This is already a useful tool; it would have taken me a couple of minutes to write that, and there's a good chance I'd make a reasoning error in there or, worse, reach for a third-party library like Lodash.

It's written a bit of code for me, but crucially, it has no idea why it needed that function, or what to do with it next. This function on its own is useless, some coordinating entity needs to know that this function is required at all. There are logical leaps to be made regarding the necessity of a piece of code, and the subsequent utilisation of that code. Generative AI can do neither; it doesn't think, it algorithmically predicts based on the input parameters.

That's a basic bit of code I'd expect any junior developer a little way into a bootcamp to be able to do, to some degree of quality. Let's try something more difficult. Handling timezones is the infamous bane of a dev's professional life; it's sometimes baffling to try to work out what 'now' refers to, and all the variables that go into that equation. Applying that logic practically when localising times between timezones can be an absolute nightmare.

In a previous job, I had the invidious assignment to determine how to represent the same time (which may be months in the future) to three or more parties in different timezones, and then persist that time in a way that didn't fall foul of things like DST changes and legislative changes that altered how a specific jurisdiction understood its own temporal relationship to the sun. The solutions we came up with ended up being robust but fiddly and computaitonally expensive. It took us weeks to get it right, and to the best of my knowledge, is still being improved upon years later.

As a starter for 10, let's ask the LLM if it can write me a function to get the number of minutes offset from UTC for a given city.

> Write a function in JavaScript that returns the number of minutes offset from UTC for a given city.

```js
function getTimezoneOffset(city) {
  let offset = Intl.DateTimeFormat().resolvedOptions().timeZone;
  if (city) {
    offset = Intl.DateTimeFormat(undefined, { timeZone: city }).resolvedOptions().timeZone;
  }

  const tz = new Date().getTimezoneOffset();
  return tz + Intl.DateTimeFormat().resolvedOptions().timeZone.offset;
}
```

This took me by surprise at first; it looks pretty convincing, it's using the datetime methods available to us through `Intl` and everything, and I can imagine an engineer just whacking this into their work, not giving it any thought, and merrily merging a PR because 'the AI knows what it's doing.' That dev is in for a rude awakening (possibly literally) when they find out that this doesn't work.

There are a couple of misconceptions, misappropriations, and outright errors here, any one of which would make this useless.

The `offset` property is bizarrely set to the name of a timezone (the IANA name, like 'Europe/London', not the name of a city; if we provide a city name, it will not be able to resolve the TZ name). It is then disregarded and never referenced again. The LLM then tries to use a non-existent property in a calculation with the current timezone offset; `timeZone` is a string, so `offset` will always be undefined. The output of this function will always be `NaN`. The AI failed us there; it's not even remotely salvageable.

Maybe I'm being unfair - I deliberately asked it to only take the name of a city to see how it would handle the ambiguity, but that was clearly a step too far. Let's try again.

> Write a function in JavaScript that takes the name of a timezone as its argument, and returns the number of minutes offset from UTC for that timezone.

```js
function getUTCOffset(timezone) {
  let offset = 0;
  if (timezone) {
    const tz = new Date().getTimezoneOffset();
    const timezoneOffset = Intl.DateTimeFormat(undefined, { timeZone: timezone }).resolvedOptions()
      .timeZone.offset;
    offset = tz + timezoneOffset;
  }
  return offset;
}
```

I really don't know where it's getting this from; this is simply incorrect. The result will always be 0 at best, and `NaN` at worst.

Let's try giving it an example.

> Write a function in JavaScript that takes the name of a timezone as its argument, and returns the number of minutes that timezone is offset from UTC. The function should be compatible with Node.js 16 and up.

> For example:
> Input: 'Europe/London'
> Output: -60
> Input 'America/New_York'
> Output: 300

This produced exactly the same result as above, which I'll assume is the end of this particular journey; GPT-3 simply can't do it. There are plenty of correct-ish solutions, including some particularly hacky ones involving Regex-ing the offset out of a localised date using Intl, which is probably what I'd try to do at first!

How about simply localising a given UTC time?

> Write a function in JavaScript that takes the name of a timezone and an ISO8601 string date in UTC as its arguments, and localises the date to the given timezone.

```ts
function localiseDate(timezone, dateString) {
  let date = new Date(dateString);
  let offset = getTimezoneOffset(timezone);
  date.setMinutes(date.getMinutes() + offset);
  return date;
}
```

It tried to use its own non-functional function. Mutating the time alone is also not ideal for many reasons.

This is the kind of task I would expect any professional engineer to be able to do without much thought; it made several glaring errors which any human engineer would just not make, or at least would not be incapable of debugging and solving with use of some documentation or a more experienced colleague.

In a previous attempt at this (a session now lost, sadly) I presented the LLM with a similar prompt, and its response was to give me a static map of North American cities and static offsets from UTC with a basic lookup function attached. Of course, that's next to useless for several reasons, but at least it would return something.

I'm hoping to make this a series. There are many subjects I want to explore, such as getting an LLM to create an integration for third party data sources, write test suites, write larger and more impactful implementations and so on. As a software engineer, I remain unconvinced that AI will replace our jobs, but will instead eventually come to augment our toolkit and, in a best-case scenario, lead to less time wasted on fiddly little functions that can often by the lynchpin of a business-critical application. I particularly want to see if GPT-4 is any better at some of these challenges.


Can AI Code? Part 1


I often need a random ID for some reason or other, whether it's a unique non-sequential ID for a data store, a way of differentiating otherwise identical items in the DOM, or a cryptographically secure token.

The JS/TS ecosystem provides us with many tools to do this.

## UUIDs

A Unique Universal Identifier is a 32-digit code, recognisable by its 8-4-4-4-12 format. UUID v4, the most common random format, is a fully random ID that makes some bold claims about how unlikely a collision – two IDs being the same – is to occur. You can determine the 'version' of a UUID by looking at the first digit of the third block of the ID; you're most likely to see a 4.

Creating a UUID in Node.js is easy. There's the excellent, lightweight [node-uuid]() library that's been around for years, but (spoiler) there is an even easier way that I'll get to next.

```ts
import { v4: uuid } from 'uuid';

const id = uuid();
// id => 'ddcace95-1db2-4a05-9249-d42e8c9c19a3'
```

As of Node 14-ish, the Node Crypto API exposes a function for generating v4 UUIDs:

```ts
import { randomUUID } from "node:crypto";

const id = randomUUID();
// id => 'ddcace95-1db2-4a05-9249-d42e8c9c19a3'
```

## Long random strings

UUIDs won't work for everything, especially if you're required to provide a single-use code for some authentication protocols, an initialisation vector, or some other cryptographically useful random code that needs to be a certain length and complexity.

At the root of a lot of these solutions is the `crypto.randomBytes` function, which provides pseudo-random data that can be used for cryptographic purposes.

We can then use the returned `Buffer` instance (essentially an array of 8-bit integers) to yield a useful string. For example:

### 32-digit hexadecimal

We can use the `randomBytes` function, along with some string processing functions as follows:

```ts
import { randomBytes } from "node:crypto";

const id = randomBytes(16).toString("hex");
// id => '4d79a9b797c69ee1a8c6d9fecf3f6a3b'
```

You can use this to generate strings of almost any length. You can also use a variety of encodings with the Buffer.toString() method, though the most useful are generally `hex` and `base64`/`base64url`.

```ts
const id = randomBytes(16).toString("base64url");
// id => 'eh7swNTsSu1euChfLea5Sw'
```

Bear in mind that the length of your final string and the number of bytes you generate are not the same. hexadecimal strings will require 2x the number of bytes (eg. 16 for a 32-digit string).

Base64 is slightly less predictable; although it theoretically encodes 6 bits per character, this leaves some remaindered bits which will require padding. If you require a base64 string of a certain length and don't want to do the maths (or want to avoid trailing padding characters,) you can request more bytes than necessary, then trim the excess:

```ts
const id = randomBytes(32).toString("base64url").slice(0, 32);
// id => 'uAomv50rGywlVmhzyAEpm5TkeWTSeinS'
```

## Hard Mode (JS built-ins only)

Let's say the Node Crypto library is not available, and we have to rely on JS built-in objects only. This method can be particularly useful in front-end contexts where the Node APIs are not available, and you wish to avoid importing additional libraries into your bundle.

Using the `Math.Random()` method, we can create random numbers between 0 and 1, which may not sound helpful at first, but we can coerce that number into a useful ID string by casting it as a hexatridecimal (a fancy word for base36) string and slicing off the leading `0.`:

```ts
const id = Math.random().toString(36).slice(2);
// id => jprznc7hjxg
```


Generating Random IDs with Node.js


I should start by saying I'm not proud of this. This got me out of a tight spot with a production server that would be terminated within minutes, leaving not a lot of time for solutionising.

The server in question held some invaluable logs for debugging a production issue; if I didn't get them off the server immediately, I'd have to wait for another bug, which is not acceptable.

If you can, you should consider uploading the data to AWS S3, or rsync-ing to another server with easier access. Proper log transport (to CloudWatch or some other log aggregator) is always ideal, but that's not a helpful thing to realise when you're in this sort of situation.

## Prerequisites

In this case, I was in an AWS Systems Manager Session Manager session (try saying that five times quickly after a few drinks), which is a web-based shell for EC2 instances, in a shell on an EC2 instance that was about to be terminated. It had minimal permissions, no SSH access at all, bulletproof security, and no automated log transport, hence the manual intervention.

The situation:

- You need to offload some files from the server,
- You have some form of shell access to a server,
- You do _not_ have direct (or even indirect) SSH access to the server,
- The server has no access to an appropriate AWS storage solution, such as S3, OR...
- You can't trust the logs or data to a third-party for compliance or security reasons,
- You don't have time to think of anything better.

## Solution

... or at least, _one possible_ solution: Tar, Base64, and good ol' fashioned copy-and-paste.

On the server:

```bash
tar cj /var/log/whatever | base64
```

This will create an archive containing the specified path, encode the compressed archive as base64, making it trivial to copy-and-paste from the terminal session.

On your local machine:

### macOS

```bash
pbpaste | base64 -d | tar xjv
```

### Linux

```bash
# pbpaste doesn't exist in most Linux distros. First paste the contents to a file, then ...

base64 -d logs.tar.bz2.base64 | tar xjv
```

## How does this work?

Tar is a tool for creating archives. It's ubiquitous in \*nix systems, tremendously useful, and is well worth getting familiar with.

The flags `cj` mean:

- **C**reate a new archive,
- **j** compresses the archive with Bzip2, Gzip's slower but smaller cousin,
- **v**erbose; log file names,

The `x` flag is the opposite of `c`, and will decompress and expand an archive.

Base64 is a very common encoding format, and can be used as a means of encoding binary data (such as a bzip2 archive) in easily representable characters: ie ASCII or low-value Unicode characters. You will often encounter it as a means of encoding small images for storage in things like databases, to avoid having to use some other method of object storage that might be less efficient.

The `-d` flag, as you might expect, reverses the process and decodes the data.

Good luck - may you never need this.


Copying Files off a Restricted Remote System


IDEs and text editors usually give us a quick and easy find-and-replace function. If, like me, you're a masochist who uses things like Vim and spends lots of time staring at command prompts, it might seem as though such ease isn't something we deserve.

Fortunately, that's not the case. The solution devides nicely into two steps:

```bash
grep -lr '<phrase or regex to match>' ./some/path | xargs sed -i 's/<regex to match>/<replacement value>/g'
```

This is for GNU SED. **If you're using a Mac**, you have a slightly different version of SED, so you'll need the following command instead:

```bash
grep -lr '<phrase or regex to match>' ./some/path | xargs sed -i '' -e 's/<regex to match>/<replacement value>/g'
```

Cheers Apple...

## What does it do?

This command searches a directory for files with contents matching a certain string, then replaces the string using a regex substitution.

`grep` searches file contents and outputs matches to the terminal.

The `-l` flag tells it to only output the file name if a match occurs, not every line where a match is found. The `-r` command, in combination with the path in the final position in the arguments list, tells Grep to search **r**ecursively.

`sed`, or **S**tream **Ed**itor is a core Unix tool for... editing streams. It can also be used to edit files in-place, as we're doing here. `-i` tells the interpreter to execute the following command against every line in a file.

`xargs` is a hugely powerful and useful Unix tool for supplying the output from one command to an arbitrary number of commands. In this case, each line of output from `grep` is being piped into xargs, which then runs a `sed` command for each; it's the same command, just with the filename from `grep` at the end.

## It's slow — how can I speed it up?

There's a good chance the recursive `grep` is traversing more directories than it needs to.

You can be more specific with your path, or you can use the `--exclude-dir <some big dir>` option to skip a directory.

```bash
grep -lr '<search pattern>' --exclude-dir node_modules --exclude_dir __tests__ ./some/path/specifically | # ...

```


Find-and-Replace in the Terminal


Most folks who work with a terminal will be familar with this kind of incantation:

```sh
cat file.txt | pbcopy
```

In this example, we get the contents of file.txt and _pipe_ it (using the | operator) to the clipboard (on macOS); we transfer the output of one command into the input of another.

This is a super-useful tool, and it's not unusual for \*nix nerds to pipe data from command to command to get what they need.

For some jobs, it's tempting to create a Node script to do the high-level processing of data in a pipeline. So how do we access that data getting piped our way?

## Reading stdin in Node.js

In Node.js, we can use the `process.stdin` Readable stream.

The older, more commonly found way to handle stdin data in Node is to use a `while` loop compiling data from stdin until it's empty.

```js
let data = "";

process.stdin.on("readable", () => {
  let chunk;
  while (null !== (chunk = process.stdin.read())) {
    data += chunk;
  }
});

process.stdin.on("end", () => {
  // process all the data and write it back to stdout
  process.stdout.write(data);
});
```

This is fine, but a little clunky. Thankfully, `process.stdin` now implements an async iterator that makes this much easier to do.

```js
let data = "";

async function main() {
  for await (const chunk of process.stdin) data += chunk;

  // process all the data and write it back to stdout
  process.stdout.write(data);
}

main();
```

Much more concise.

You'll notice that once we're done with the processing, we write the output to `process.stdout`; this allows the data to be used in the next command, eg. `pbcopy` or `grep`.

Avoid `console.log()`, as it appends a newline.

## Reading data line-by-line

The `chunk` in the above example is usually a linear segment of bytes up to a certain arbitrary limit defined by the runtime. This isn't useful if we have a known line-by-line input coming into our app, say as an output from `grep`, or a file.

In this case, we can use the `readline` API available in Node.js, and hook up stdin as an input stream.

```js
const readline = require("node:readline");

const rl = readline.createInterface({
  input: process.stdin,
});

rl.on("line", (line) => {
  // process a line at a time
  process.stdout.write(`Line: ${line.slice(0, 64)}...\n`);
});
```

The readline interface also implements an async iterator, so just as we did with the Readable stream above, we can refactor this to use a `for..of` loop:

```js
const readline = require("node:readline");

async function main() {
  const rl = readline.createInterface({
    input: process.stdin,
  });

  for await (const line of rl) {
    // process a line at a time
    process.stdout.write(`line: ${line}\n`);
  }
}

main();
```

Happy piping!


How to pipe data into a Node.js script


I was challenged on my use of the Spread operator in the return value of a reducer today. The colleague in question raised a concern over performance when spreading a potentially unknown number of elements, simply to add an element to that array.

Being the professional, mature adult that I am, I immediately set out to prove them wrong.

It was, of course, me who was wrong.

## TL;DR

It gets exponentially slower the more elements it has to cover.

`Array.concat()` is significantly more performant at scale than a spread-merge. Keep reading to find out how I tested this, and what else I covered along the way.

## What is the Spread operator?

It iterates iterables.

> What?

Basically, it expands an object (such as an array) into an available space.

Some (highly contrived) examples:

```js
// 1. Shallow-cloning an array;
const arr = [1, 2, 3, 4];

const clonedArr = [...arr];
// clonedArr now contains all the elements of arr, but is a brand new array.

// 2. Creating a object that inherits some but not all properties from another:
const person = {
  name: "Jon",
  age: 30,
  city: "London",
};

const personAfterHouseMove = {
  ...person,
  city: "Manchester",
};
// => { name: 'Jon', age: 30, city: 'Manchester' }

// 3. Expanding an array into the arguments of a function call:
const args = ["20", 10];

const number = parseInt(...args);
// => 20
```

We also use it _a lot_ for conditional merges in reducer return values. If you've ever written a Redux reducer to handle replacing an object from an array of objects by a specific property, this will look very familiar to you:

```js
const people = [
  { id: 1, name: "Safi" },
  { id: 2, name: "Francis" },
  { id: 3, name: "Sam" },
];

return [...people.filter((person) => person.id !== 2), { id: 2, name: "Fran" }];

// => [
//      { id: 1, name: "Safi" },
//      { id: 3, name: "Sam" },
//      { id: 2, name: "Fran" },
//    ];
```

In this example, we're returning a new array containing all items in `people` with the exception of item with id `2`, and a new item.

## So how slow is it?

Let's do some science and find out.

In this experiment, our aim is to combine two arrays of equal length into a new array.

To test this, we'll create the arrays and populate them with some random dummy data, start a stopwatch, then call a function that merges them. When that function has returned, we stop the stopwatch.

Here's the code we'll use:

```js
const arraySize = 64;

const randomString = () => (Math.random() * 1.2e17).toString(16);

// A little helper function to make the result of hrtime() a bit nicer to look at.
const hrtimeToMilliseconds = ([ms, ns]) => ms * 1e9 + ns / 1e6

function mergeArrays(a, b) {
  // Our implementation will go here
  // ...
}

// Create two arrays of size n (64 to start with) populated with random strings
const itemsA = [...new Array(arraySize)]
  .map(randomString);
const itemsB = [...new Array(arraySize)]
  .map(randomString);

// GO!
const start = process.hrtime());

mergeArrays(itemsA, itemsB);

const end = process.hrtime();

console.log(`${(hrtimeToMilliseconds(end) - hrtimeToMilliseconds(start)).toPrecision(4)}ms`);
```

### Control case

First, let's establish a control case: how long does it take `mergeArrays` to simply return all its arguments?

```js
function mergeArrays(a, b) {
  return [a, b];
}
```

Result: **0.03125ms**

So that's our control case, less than a tenth of a millisecond to return the values without operating on them.

### Spread

Let's try the spread operator first:

```js
function mergeArrays(a, b) {
  return [...a, ...b];
}
```

Result: **0.06892ms**

Clearly some computation going on there, but nothing indicating you should go away and refactor all your spread-merges just yet.

### `Array.concat()`

This function on the Array prototype allows you, as you'd expect, to add another array on to the end of an array, returning a new array.

You can read up on it on [MDN](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/concat).

```js
function mergeArrays(a, b) {
  return a.concat(b);
}
```

Result: **0.05266ms**

Some folks will prefer calling the `concat` method on an empty array literal:

```js
function mergeArrays(a, b) {
  return [].concat(a, b);
}
```

Result: **0.05276ms**

This is perfectly valid and probably preferable, especially given there's basically no performance penalty.

One thing to consider is that in TypeScript, the interpreter does not know what kind of array you're defining, and infer a type of `never[]` to the array literal. This will cause it to trip over whatever you're `concat`'ing in. A solution to this is to lead the compiler by the nose:

```ts
function mergeArrays<T>(a: T[], b: T[]): T[] {
  return ([] as T[]).concat(a, b);
}
```

## Hang on a second...

This is nowhere near dramatic enough. If performance issues begin to reveal themselves at scale, then we need to adjust our methodology and crank up those numbers to get some actual results.

```js
const arraySize = 1e6; // 1,000,000.
```

That'll do it.

So now we're trying to join two arrays of a million elements each together. Let's start the test over.

### Spread (1million)

Result: **63.34ms**

Now we're talking — slow code!

63ms is fairly slow. I'd start to get nervous at this scale, given we're working in the region of millions of elements here.

### `Array.concat()` (1million)

Result: **9.833ms**

As different as day and night.

Both methods (calling `concat` on one of the target arrays, and calling it on an empty Array literal) yielded the same result.

That feels like a pretty clear winner to me, but, for the sake of argument, let's give a few other methods a go:

### The good ol' `for` loop (1million)

In this example, we use an incremented pointer to reference each index of the arrays and push them into the result array.

I toyed with using `var` in this example, for the true early JavaScript experience, but I just couldn't bring myself to do it.

```js
function mergeArrays(a, b) {
  const array = [];

  for (let i = 0; i < a.length; i++) array.push(a[i]);

  for (let i = 0; i < b.length; i++) array.push(b[i]);

  return array;
}
```

Result: **73.57ms**

The slowest yet. There are better ways of iterating arrays than this, so let's keep this one in the history books and not in our Git diffs.

### `for..of` iteration (1million)

Another `for` loop, but this time we're using the arrays' [built-in iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/@@iterator) to yield their values for the `array.push()`

```js
function mergeArrays(a, b) {
  const array = [];

  for (const item of a) array.push(item);

  for (const item of b) array.push(item);

  return a, b;
}
```

Result: **76.94ms**

Even slower.

Using the `for..of` method of iteration is extremely useful in some circumstances, but this isn't one of them.

### `Array.reduce()` (silly)

For the sake of argument (and to see if we can get into triple-digit milliseconds), let's implement a wildly inefficient reducer that merges these arrays one element at a time, spreading the previous results as we go to create new arrays on each iteration.

```js
function mergeArrays(...arrays) { // note the use of the spread operator here
  return arrays.reduce(
    (result, currentArray) => [
      ...result,
      currentArray.reduce(
        (innerResult, innerElement) => [...innerResult, innerElement],
        []
      ),
    ],
    []
  );
```

Result: **FAIL**

Our first failure! The limiting factor of my ever-waning patience caused this experiment to conclude without conclusion; the function never returned, and I killed the process after a minute or two.

I dropped the element count down to 100,000, still nothing (within the bounds of my patience).

At 10,000 elements per array, the function clocked in at an absolutely glacial **237ms**.

## The Finals

With the heats over, we have two competitors ready to go head-to-head.

`Array.concat()` has emerged as the best alternative to the all-too-easy spread-merge, but as we're examining the effect at scale, let's pit each method head-to-head and really dial up those numbers.

I'll try array lenghts of 10 through to 10,000,000 and compare the two.

Numbers are boring, so here's a graph of the results:

![A line graph comparing performance of Array.concat and spread-merge. The vertical axis is the number of records (from 10 to 10,000,000), the horizontal axis is the number of milliseconds elapsed while merging the arrays using the relevant method. Array.concat() is fastest by a wide margin; merging 20,000,000 elements in 99.69 milliseconds, a spread-merge took 465ms. {823x509}](/blog/images/array-performance.png)

Oh have the numbers anyway:

```
[items]         10    100   1000  10000  1e5    1e6    1e7
Spread-merge    0.03  0.03  0.09  1.00   11.75  64.09  465.20 [ms]
Array.concat()  0.03  0.03  0.03  0.19   0.94   9.84   99.69  [ms]
```

`Array.concat()` wins by a mile. 🥇

Both methods are level-pegging up to around 1000 elements, at which point a spread-merge begins to lag behind. But the real problems come when we get into five-digits.

By eight-digits, the difference is enormous; `Array.concat()` merged 20,000,000 items into the same array 365ms faster than a spread-merge.

## Why is it so much faster?

This is down to how these two methods process data under-the-hood.

I can't say for certain, and I'll be damned if I'm going to do any research that involves reading the native C++ implementation of Array prototype functions.

My semi-educated guess, given the disparity in timings we see, is that the spread operator is iterating one-by-one through each element, assigning each one to a new space in memory in sequence.

`Array.concat()`, however, I would expect to do some lower level memory manipulation to duplicate and stack the arrays next to each other. This would explain the very slight increase in time-complexity with increased elements; the number of elements matters to an extent, but isn't as big a dent in performance as iterating through every one of them.

## Conclusion

A spread-merge is probably fine when you're sure you're dealing with no more than a few thousand items, but if you want to make sure your application scales beyond that, give `Array.concat()` a try.


How slow is the Spread operator in JavaScript?


In 2009, I started a blog for writing film reviews. It was, as was the trend at the time, a WordPress site hosted on some VPS somewhere. The reviews were a little tawdry and the kind of thing you'd expect someone to write when they're barely an adult and don't have a wealth of empathy to draw upon. Because try-hard edgelordiness was entirely acceptable in the late Naughties, I named it _User Unfriendly_.

When I set off for Africa on my Gap Year (pronounced _gap yah_), I repurposed the site for travel blogging, as a many-to-one method of updating family and friends to what I'd been up to. Looking back on the content now, I should have renamed it to _Smug White Tosser_.

It suffered the same fate when I worked in France during my undergrad, during which I basically blogged about cheese and being a bad teacher.

Now, eight years later, as a relatively stable adult in a relatively stable career — at which I'm not entirely useless, — I'm ditching the locally hosted MySQL database and butchered PHP backend of WordPress, and basking in the joyous developer experience of Next.js.

I've been finding myself in need of some sort of outlet for code tips, learning notes, music and film reviews of course, and maybe the odd travel article.

Naturally, this being the long-awaited **grand relaunch** (🎉), I'm expecting it to immediately be greeted with rapturous attention, and be a vital source of information without which my fellow engineers and lovers of the arts will be unable to live meaningful lives.

I added a Google Analytics tracker, but if I _ever_ see so much as a view, I will be more confused than elated.

Crucially, there will be code, so I'm leaving this block here as a little formatting test:

```ts
const test = "Hello I am Test!";

const add = (x) => (y) => x + y;

const add3 = add(3);

console.log(add3(5)); // => 7 (I think... needs a unit test.)

function stringifyRecord(obj: Record<string, any>): string {
  return Object.entries(obj).reduce(
    (acc, [key, value]) => `${acc}${acc ? "," : ""}${key}=${JSON.stringify(value)}`,
    ""
  );
}
```


Recent Posts 🧻

☛ Prefix matching in Lambda DynamoDB triggers

☛ Can AI Code? Part 1

☛ Generating Random IDs with Node.js

All Posts

May 2023

April 2023

March 2023

November 2022

October 2022

August 2022

July 2022