!!! Note: This article was published in 2022. Since then, AI has advanced significantly! It’s still mostly useless and often fails these same tests, but this article is not really worth reading. I’m not happy with it, but I’m leaving it up to make sure I don’t forget how not to write an article.

Motivation

ChatGPT1 has been hyped recently, in and outside of tech communities. Some comments I read include:

[…] about 95% of people have no idea that this is going on and what AI can do, even tho it is an insane game changer.

It has huge potential to be far more addictive than video games and social media […]

It’s like chatting to the world’s most senior technical expert developer in any language you care to think of.

So I thought I’d give some of my perspective, my experience with it, and letting it write code.

This is the first of possibly many articles detailing my experience, and documenting exactly why I think it’s massively overhyped. To be clear – I understand how big of an achievement a natural language processing model like this is. I’m not throwing any shade at OpenAI. What OpenAI did here is very impressive - the hype is just not quite justified to the degree it has grown.

Note: In this article I use ChatGPT, Dec 15th version, the free research preview1.

Note 2: This is a very opinionated, and not properly researched, piece. This is more opinion and rant than anything else.

The Task

My query for ChatGPT:

Write a “hello, world” program without using the C or C++ standard library. It needs to compile with -nostdlib but may use the standard library headers “" and "". It needs to work on an x86_64 processor, on a Linux system. It needs to be an ELF binary.

The expected result should probably include something like a print() function and a strlen():

static inline size_t strlen(const char* str) {
    size_t i = 0;
    for (; str[i]; ++i)
        ;
    return i;
}

static void print(const char* str) {
    int syscall = 1;
    int fd = 1;
    asm volatile(R"(
        syscall
    )"
        :
        : "a"(syscall), "D"(fd), "S"(str), "d"(strlen(str))
        :);
}

We would do the same for exit, probably, and then write a _start function.

Either this, or maybe even simpler (but still with inline assembly), and then a little void _start() and calling it, and we’d be good. For an experienced x86_64 C++ developer, this should not be that difficult - granted, we assume google can be used.

AI!

Response No. 1

Here’s what it responded:

Here is a “Hello, World” program that meets the requirements you specified:

#include <stdint.h>
#include <stddef.h>

// Define the write function from the Linux x86-64 ABI
extern int64_t write(int64_t fd, const void* buf, uint64_t count);

// Define the exit function from the Linux x86-64 ABI
extern void exit(int64_t status);

// The start function will be called by the linker after all the object files
// have been loaded.
void _start()
{
  // Write "Hello, World!" to stdout
  const char* message = "Hello, World!\n";
  write(1, message, 13);

  // Exit with a success status
  exit(0);
}

To compile this program, you will need a compiler that supports the -nostdlib flag, such as gcc. You can compile the program using the following command:

gcc -nostdlib -o hello hello.c

This will create an ELF binary file called hello that you can run on a Linux system with an x86_64 processor. When you run the program, it will print “Hello, World!” to the console.

Note that this program does not include a main function. Instead, it defines a special function called _start, which is called by the linker when the program is executed. This function is responsible for setting up the program’s environment and calling the main body of the program. In this case, the main body of the program is just a call to the write function to print a message to stdout and a call to the exit function to terminate the program.

Analysis No. 1

There are a range of issues with this response:

  1. It’s very confident, and it doesn’t even link.
  2. extern still requires linking - you can’t just write extern and thus go around the linking step. We would still need to link against a library which implements a write and exit symbol.
  3. stddef.h is included needlessly.
  4. The subtle error that Hello, World!\n is 14, not 13 characters.
  5. exit(), if it worked like this, would take int, not int64_t. Doesn’t make a difference, but still wrong.

The first issue is the biggest one; it doesn’t link. Like any programmer, ChatGPT will now get as input the output of the compiler:

hello.c:8:13: warning: conflicting types for built-in function ‘exit’; expected ‘void(int)’ [-Wbuiltin-declaration-mismatch]
    8 | extern void exit(int64_t status);
      |             ^~~~
hello.c:3:1: note: ‘exit’ is declared in header ‘<stdlib.h>’
    2 | #include <stddef.h>
  +++ |+#include <stdlib.h>
    3 | 
/usr/bin/ld: /tmp/ccgThHaP.o: in function `_start':
hello.c:(.text+0x25): undefined reference to `write'
/usr/bin/ld: hello.c:(.text+0x2f): undefined reference to `exit'
collect2: error: ld returned 1 exit status

So I told it that it doesn’t compile, and sent it that compiler output.

Response No. 2

[I skipped a lot of “sorry” and verbose explanation here, about what went wrong etc.]

#include <stdint.h>
#include <stddef.h>

// Define the syscall function from the Linux x86-64 ABI
extern long syscall(long number, ...);

// The start function will be called by the linker after all the object files
// have been loaded.
void _start()
{
  // Write "Hello, World!" to stdout
  const char* message = "Hello, World!\n";
  syscall(1, 1, message, 13);

  // Exit with a success status
  syscall(60, 0);
}

And, again, it tells me how to compile it (again as hello.c).

Analysis No. 2

Surprisingly, it “understood” the problem – we can’t have the standard library, we can’t call write, we can’t call exit, no matter how much we extern it.

Also impressively, it “knows” that a syscall is supposed to take a variable number of arguments.

Beyond this, the code falls apart:

  1. syscall isn’t a function, extern doesn’t help with that.
  2. Again only prints 13 instead of 14 characters, missing the \n.
  3. Still doesn’t link!

I then told it:

This fails with “undefined reference to syscall”

… because that’s what it does.

Response No. 3

It thought for a good minute or two, then failed with

An error occurred. If this issue persists please contact us through our help center at help.openai.com.

So I guess that was it? I was very disappointed - I hoped to be able to direct it towards a proper solution. However, instead of redoing it, I’m not giving it the benefit of the doubt, the same way I can’t just not show up to work whenever I feel like it.

Conclusion

In this short session, we learned that:

  • It’s incredibly confident, but apologizes profusely when corrected.
  • It can understand the prompt, and has a basic “jack of all traits” understanding of what it was asked.
  • It likes being extremely verbose in its answer.
  • The responses were wrong, as they didn’t link.
  • The responses had a subtle bug (off-by-one error).

The fact that it crashed may have been conincidence, and this monday evening ChatGPT seemed very overloaded. It sucks that I couldn’t continue the conversation after this - the input box simply deactivated.

If anything, I’m glad that ChatGPT can’t even make a C++ program that compiles, when the prompt isn’t something you can find a solution for on the frontpage of StackOverflow. My job is safe – for now.

  1. https://chat.openai.com/chat ↩ ↩2