Can AI Code? Part 1
On the subject of generative AI, I bounce back and forth between gleeful childish optimism, and something approaching apocalyptic terror.
I may well write a full egotistical opinion piece on that another time, but for now I'll summarise by saying that the practical applications of neural networks and LLMs in fields like science and medicine will quite likely be a revolutionary force for good in the medium and long term. But, as with almost all technology, there are countless and unforeseeable negative applications too. Although I don't think the AIs are going to turn sentient and decide to exterminate us, I'm more concerned that LLMs will continue to be exploited by dark political actors and authoritarian nation states for anti-democratic purposes. There's a real risk that convincing populist misinformation can be produced so cheaply and so prodigiously that it eventually reduces human society to something subservient and pitiful, or drive us all into solipsistic madness. Though it's unlikely to be quite that bad, the prospect of machines that can write natural language in a convincingly coherent fashion is staggering, and until I actually started using an LLM, I struggled to keep the fear in check.
Society's presumably impending collapse aside (this is sarcasm, if you're in an AI anxiety phase), the other worry about advanced LLMs and generative AI is that it will replace software engineers. As all software engineers have had to do in the last six months, I've considered the possibility that an AI could come along and automate me out of a job. For a while I was almost convinced; the seemingly weekly quantum leaps in capability made this seem inevitable. Now that the novelty shock is wearing off, and the prospect of AI causing mass unemployment seems less likely, especially in the case of software engineers; I no more think AI will make software engineers redundant than I think DAWs will make musicians obsolete, or a brick-laying machine will make builders a thing of the past; you still need someone who understands music, who knows what a house needs to be and do for the tools to have value.
As a seemingly increasing number of engineers seem to have done already, I'm arriving at the conclusion that generative AI is an assistive tool, something that can help us build the things we build by at least partly automating some source code writing. I've experimented with GPT-3 for a little while, and far from being scared into assuming I'll be out of a job in the next few years, I've actually been relieved that it is both more useful than I hoped, and less useful than I feared. (Side-note: I'm aware that GPT-3.5 and 4 are both more powerful than GPT-3, but the fundamentals of the tests I want to do are roughly the same. From what I've read about GPT-4, I'm ready to raise my eyebrows by no more than a couple of millimetres.)
Modern software engineering, for the majority of us, is about far more than just writing source code. There's a tremendous amount of reasoning and planning; architecture and building, which require cognition and reasoning. At a basic level, these are things that generative AIs are simply can't do, no matter how hard the blow-hards blow. This has become clear over the last few months, and I remain sceptical of some of the larger demonstrations of GPT-generated software whose source code remains closed.
Assuming that AI will end up automating some monotony in my job, a brick-laying machine, I wanted to know how useful it would be as a coding tool, something to generate source code for me. I've started small, and asked it to do something that I've had to implement a couple of times in my career: given an input array, split that array into several smaller arrays of a given size. Let's see how the robot got on with that.
Write a function in JavaScript to divide an array of items into smaller arrays of variable length.
1function chunkArray(arr, size) { 2 const chunkedArr = []; 3 for (let i = 0; i < arr.length; i += size) { 4 chunkedArr.push(arr.slice(i, i + size)); 5 } 6 return chunkedArr; 7}
If I had Googled this same question and landed on Stack Overflow, this is precisely the answer I would expect; a simple for loop that does the job. The formatting is a bit weird and clunky (or was, until my editor formatted it), and it's a little hard to grok but, importantly, it does work. Let's see if we can get it to generate something similar using slightly different syntax:
Write a function in JavaScript to divide an array of items into smaller arrays of variable length. You must use the Array.reduce method, and must not use a for loop.
1function chunkArrayReduce(arr, size) { 2 return arr.reduce((acc, el, i) => { 3 const div = Math.floor(i / size); 4 if (!acc[div]) { 5 acc[div] = []; 6 } 7 acc[div].push(el); 8 return acc; 9 }, []); 10}
I've implemented this function in exactly the same way for years, so this is a good result. I find it interesting how it changed the name of the function and the names of the variables, and how it added a guard against undefined chunks without being told that was a risk. I mostly use TypeScript these days, so let's ask it to add in some types.
Refactor the chunkArrayReduce function in TypeScript. The array must use a generic type.
1function chunkArrayReduce<T>(arr: T[], size: number): T[][] { 2 return arr.reduce((acc, el, i) => { 3 const div = Math.floor(i / size); 4 if (!acc[div]) { 5 acc[div] = []; 6 } 7 acc[div].push(el); 8 return acc; 9 }, []); 10}
The fact that it can correctly apply generic types with minimal prompting is quite impressive. This is already a useful tool; it would have taken me a couple of minutes to write that, and there's a good chance I'd make a reasoning error in there or, worse, reach for a third-party library like Lodash.
It's written a bit of code for me, but crucially, it has no idea why it needed that function, or what to do with it next. This function on its own is useless, some coordinating entity needs to know that this function is required at all. There are logical leaps to be made regarding the necessity of a piece of code, and the subsequent utilisation of that code. Generative AI can do neither; it doesn't think, it algorithmically predicts based on the input parameters.
That's a basic bit of code I'd expect any junior developer a little way into a bootcamp to be able to do, to some degree of quality. Let's try something more difficult. Handling timezones is the infamous bane of a dev's professional life; it's sometimes baffling to try to work out what 'now' refers to, and all the variables that go into that equation. Applying that logic practically when localising times between timezones can be an absolute nightmare.
In a previous job, I had the invidious assignment to determine how to represent the same time (which may be months in the future) to three or more parties in different timezones, and then persist that time in a way that didn't fall foul of things like DST changes and legislative changes that altered how a specific jurisdiction understood its own temporal relationship to the sun. The solutions we came up with ended up being robust but fiddly and computaitonally expensive. It took us weeks to get it right, and to the best of my knowledge, is still being improved upon years later.
As a starter for 10, let's ask the LLM if it can write me a function to get the number of minutes offset from UTC for a given city.
Write a function in JavaScript that returns the number of minutes offset from UTC for a given city.
1function getTimezoneOffset(city) { 2 let offset = Intl.DateTimeFormat().resolvedOptions().timeZone; 3 if (city) { 4 offset = Intl.DateTimeFormat(undefined, { timeZone: city }).resolvedOptions().timeZone; 5 } 6 7 const tz = new Date().getTimezoneOffset(); 8 return tz + Intl.DateTimeFormat().resolvedOptions().timeZone.offset; 9}
This took me by surprise at first; it looks pretty convincing, it's using the datetime methods available to us through Intl and everything, and I can imagine an engineer just whacking this into their work, not giving it any thought, and merrily merging a PR because 'the AI knows what it's doing.' That dev is in for a rude awakening (possibly literally) when they find out that this doesn't work.
There are a couple of misconceptions, misappropriations, and outright errors here, any one of which would make this useless.
The offset property is bizarrely set to the name of a timezone (the IANA name, like 'Europe/London', not the name of a city; if we provide a city name, it will not be able to resolve the TZ name). It is then disregarded and never referenced again. The LLM then tries to use a non-existent property in a calculation with the current timezone offset; timeZone is a string, so offset will always be undefined. The output of this function will always be NaN. The AI failed us there; it's not even remotely salvageable.
Maybe I'm being unfair - I deliberately asked it to only take the name of a city to see how it would handle the ambiguity, but that was clearly a step too far. Let's try again.
Write a function in JavaScript that takes the name of a timezone as its argument, and returns the number of minutes offset from UTC for that timezone.
1function getUTCOffset(timezone) { 2 let offset = 0; 3 if (timezone) { 4 const tz = new Date().getTimezoneOffset(); 5 const timezoneOffset = Intl.DateTimeFormat(undefined, { timeZone: timezone }).resolvedOptions() 6 .timeZone.offset; 7 offset = tz + timezoneOffset; 8 } 9 return offset; 10}
I really don't know where it's getting this from; this is simply incorrect. The result will always be 0 at best, and NaN at worst.
Let's try giving it an example.
Write a function in JavaScript that takes the name of a timezone as its argument, and returns the number of minutes that timezone is offset from UTC. The function should be compatible with Node.js 16 and up.
For example: Input: 'Europe/London' Output: -60 Input 'America/New_York' Output: 300
This produced exactly the same result as above, which I'll assume is the end of this particular journey; GPT-3 simply can't do it. There are plenty of correct-ish solutions, including some particularly hacky ones involving Regex-ing the offset out of a localised date using Intl, which is probably what I'd try to do at first!
How about simply localising a given UTC time?
Write a function in JavaScript that takes the name of a timezone and an ISO8601 string date in UTC as its arguments, and localises the date to the given timezone.
1function localiseDate(timezone, dateString) { 2 let date = new Date(dateString); 3 let offset = getTimezoneOffset(timezone); 4 date.setMinutes(date.getMinutes() + offset); 5 return date; 6}
It tried to use its own non-functional function. Mutating the time alone is also not ideal for many reasons.
This is the kind of task I would expect any professional engineer to be able to do without much thought; it made several glaring errors which any human engineer would just not make, or at least would not be incapable of debugging and solving with use of some documentation or a more experienced colleague.
In a previous attempt at this (a session now lost, sadly) I presented the LLM with a similar prompt, and its response was to give me a static map of North American cities and static offsets from UTC with a basic lookup function attached. Of course, that's next to useless for several reasons, but at least it would return something.
I'm hoping to make this a series. There are many subjects I want to explore, such as getting an LLM to create an integration for third party data sources, write test suites, write larger and more impactful implementations and so on. As a software engineer, I remain unconvinced that AI will replace our jobs, but will instead eventually come to augment our toolkit and, in a best-case scenario, lead to less time wasted on fiddly little functions that can often by the lynchpin of a business-critical application. I particularly want to see if GPT-4 is any better at some of these challenges.