Someone on /r/rust is currently marveling that it spit out a working bitmap allocator: https://reddit.com/r/rust/comments/zgkuq6/

ChatGPT is sensitive to how you prompt it, and it's designed to work as a dialogue where you ask further questions and it can refine its answer.

It can already write reasonable working code, although you shouldn't expect it to spit out entire working programs when the prompt is Gimme Teh Codez without further details. That being said, AI capabilities only get better with time, so I think it'd be shortsighted to dismiss its potential because of its current limitations.

If the pace keeps up, companies with access to large language models in a few years could have a large advantage on legacy programming, the same way programmers in high-level languages have a massive productivity advantage on 1970's low-level assembly programmers. You give up a little more control, but you save time.

> You give up a little more control, but you save time.

Not just that: for an average programmer the compiler will optimize the code much better than if it was done by hand. Something similar can happen down the lane.

Take a look at this: https://www.dropbox.com/s/gq2hkp7w37c5y6q/ai.png?dl=0

Oh my! It does not really matter if this is actually self-aware, but in terms of productivity: well it has the potential to be revolutionary. No longer does a person need to know how to program. Just talk to a computer and the computer will generate the code to execute such task.

> No longer does a person need to know how to program.

I think this is getting caught up in media sensationalism. ChatGPT is regurgitating copied code from various sources. A way to look at this is if you could type out questions to Stack Overflow, then it responded by giving you code that most closely matches to what you asked. The thing about that is it is taking and searching code that thousands of humans have created, "filtering" based on question asked, and giving it back to you. Instead of a person doing the hard work of searching Stack Overflow and other sources, ChatGPT can be the new centralized source to get copied segments of code.

Copying and pasting code from various sources is not anything new. Because a person has copied a piece of code from somewhere doesn't mean they don't need to know programming, what is going on, or how to troubleshoot it. Some people, who are being lazy, might think that ChatGPT is a way out of studying and understanding what is being done, and they will sooner or later found out that it isn't.

It did write incorrect tests for non-Latin characters though.

The future looks like debugging hell.

You'll have countless nights to talk about it with DebugGPT.

It will always respond "it works on my local machine".

It wrote a lot of incorrect tests at this point not only when non-Latin characters where involved.

Look at the mixed case tests below. Always the same string but sometimes a palindrome and sometimes not.

Same for the trailing whitespace test.

Maybe it got tired at this point. Could also explain why the performance test is missing warmup :D

holy cow

I've been using it for small blocks of code or specific functions - stuff I'd have to look up documentation or Stackoverflow for. I see it as a productivity booster, not a replacement.

You can't tell it to "create a CRUD app", but you can ask it to create a nodejs function that gets all the records from a mongodb database.

You can ask for a simple example of a CRUD app, here's my code output:

const express = require('express')

const app = express()

// define the model for a user

const User = {

  name: '',

  email: ''

}

// create a new user

app.post('/users', (req, res) => {

  const user = Object.assign({}, User, req.body)


  // save the user to a database, etc.


  res.send(user)

})

// read a list of users

app.get('/users', (req, res) => {

  // retrieve a list of users from the database, etc.

  res.send([

    { name: 'Alice', email: 'alice@example.com' },

    { name: 'Bob', email: 'bob@example.com' }

  ])

})

// read a single user

app.get('/users/:id', (req, res) => {

  // retrieve a user with the specified id from the database, etc.

  res.send({ name: 'Alice', email: 'alice@example.com' })

})

// update a user

app.put('/users/:id', (req, res) => {

  // retrieve the user with the specified id from the 

database, etc. const user = Object.assign({}, User, req.body)

  // save the updated user to the database, etc.

  res.send(user)

})

// delete a user

app.delete('/users/:id', (req, res) => {

  // delete the user with the specified id from the 

database, etc.

  res.send({ message: 'User deleted' })

})

app.listen(3000, () => {

  console.log('Listening on port 3000')

})

Well that's impressive :)

But of course unless you had programming experience, you would have no way to structure this output correctly and actually take it live

I asked GPT how I could run it with this prompt:

Given this code for an example CRUD app, what unix commands could I run on my system to actually run this code?

$pasted_code_from_gp_comment

GPT's response:

> To run this code, you would need to install the express npm package by running npm install express in your terminal. After that, you can run the app by using the node command followed by the name of the file containing the code, for example: node app.js. This will start the server, which you can then access by going to http://localhost:3000/ in your web browser.

With follow up questions, such as "it says npm is not recognized command"; gpt will explain what that error means and how to fix it

This is the response to "it says npm is not recognized command":

> If you get an error that npm is not recognized as a command, it means that the npm command is not available on your system. This could be because you don't have Node.js installed, or because the npm command is not in your system path.

> To fix this error, you will need to install Node.js on your system. You can download and install the latest version of Node.js from the official website: https://nodejs.org/en/. After installing Node.js, you should be able to run the npm command from your terminal.

> Once you have installed Node.js, you can also check if the npm command is in your system path by running where npm on Windows or which npm on macOS or Linux. This should print the location of the npm command on your system. If it doesn't print anything, you may need to add the npm command to your system path manually.

> If you're still having trouble, you can try looking for solutions online or reaching out to the Node.js community for help. There are many forums and online communities where you can ask questions and get help from experienced users.

I’m impressed by its ability to correct itself.

Can’t wait to see what this technology looks like in a year.

Is it better than Copilot with comment-based incremental code writing?

Better really is too subjective and context dependent. Play with both and see what works for you. Copilot is GPT3 specifically trained on code, so my understanding is that it should be better... It's also fairly convenient due to the IDE plugins.

Yes constantly get this. It’s quite annoying and I’d love to know why. Doesn’t seem consistent or based on char limits.

Anyway, when this happens just use the prompt : “continue”. From this it’ll generally do a good job of continuing where it left off, but some refactoring is often required. As it doesn’t always exactly begin where it ended and leaves out vital code or in complete event handlers and functions for example.

From an initial prompt, you may need to continue several times Ron get to complete code you’re after.

Of course, you can throw the resultant incomplete buggy code back in as a new prompt and request analysis and bug fixes. Which it does a good job of resolving.

Of course, all of this subject to the complexity of your perk or/request

The code I got for "write a small python program to play tic tac toe" was complete (though I had to write "continue" to get all the output.

It made a playable game, that was close to the output that it suggested would occur, but it did have a few mistakes - it missed 2 win conditions (rows and one of the diagonals), and the output didn't quite align fully (i.e. "" instead of " " being printed and no lines for board edges, it also doubled up the diagonal checks oddly.

  # Check if there is a winner, and if so, print a message indicating who won
  if board[0][0] == board[1][1] == board[2][2] and board[0][0] != "":
    print(f"Player {board[0][0]} wins")
  elif board[0][2] == board[1][1] == board[2][0] and board[0][2] != "":
    print(f"Player {board[0][2]} wins")
  else:
    # If there is no winner, the game is a draw
    print("It's a draw!")

It took me a while to find the right prompt to fix this as it seems to confuse the code checking for game over, which checks for a winner as one of the conditions, for the code that checks for the winner at the end of the game.

One of my prompts led to the output being made the program be refactored into a function run in a loop rather than a one shot game. and then I tried:

>the play_game function prints the wrong output when there is a winner in the columns or rows

and got the desired fix.

I like the stateful nature of this.

I let ChatGPT build a simple glue class in C++ to set the backlight of a display, just because that was what I was doing at the moment.

I noticed a bug, informed ChatGPT and it fixed it correctly, which is cool and all.

But I'm really interested in the next thing: I didn't save the code and so reran the same prompt a few days later. Now it was correct the first time. Did it learn the bug? It also lost the Exception handling it had in the first version, though. And it changed it's naming scheme for member variables from `m_foo` to `foo_`.

It all depends on what you mean by generate fully functional code. In my experience, it is able to generate code snippets that are useful but very rarely can be simply copy & pasted into your work. The experience is actually quite similar to Stack Overflow.

yeah but the difference is I can immediately ask, hey why isn't this working, or why am I getting this linting error, and it'll maybe point me in the right direction (so far usually it works really well, sometimes I need to refresh the session, and start from scratch with my code base, but it's still instant hand-holding).

I was wanting to try to do something I wasn't familiar w/ at all: Building a mobile app with flutter, and so far I've got the gPT chat completely cloned in flutter, I'm just missing the additional features I wanted to add like saving chats, sharing screenshots and text chats, entering a url, and scraping the page text to ask ChatGPT to summarize, or maybe write a blog post similar to this but don't copy anything, etc....

YEs, but in small blocks and with specific asks. I asked it to generate code based on an existing project of mine, which generates analytics from social media activity. It does a good job of producing function bodies from prompts equivalent to a short comment you might use to document your function in the IDE.

For your example, I'd ask it to 'generate code, using Python, that accepts input and updates a display to allow a user to play an interactive game of TicTacToe with the computer.' I'm sure I could do better with some experimentation, but that will have to wait for some other day.

This is what i asked it for. In separate sessions it built functions slightly differently and only one of its programs had a minor bug. It was interesting to watch it answer the same prompt differently more than once.

First of all, don't use ChatGPT for that. Use full GPT-3, specifically the code models (ie. code-davinci-002).

ChatGPT is a subset & tweak of the full GPT-3 model to make a good chatbot, and it's actually less capable than the full model.

It's strange to me that people are going crazy over ChatGPT when GPT-3 has been out for months now.

More info on the difference between ChatGPT and GPT-3: https://dev.to/ben/the-difference-between-chatgpt-and-gpt-3-...

That's flat wrong. That is ChatGPT's answer to the difference between ChatGPT and gpt3. ChatGPT does not really know about itself, its data cutoff is from 2021.

ChatGPT is trained on a variant of gpt-3.5 (instructgpt), read more here https://openai.com/blog/chatgpt/

Okay, maybe I'm wrong, but the article doesn't explain how ChatGPT compares to GPT-3.

ChatGPT uses the model text-davinci-003, which was released alongside it. The previous model of the same kind was text-davinci-002, and in a comparison I read, seems to perform much less impressively. So it's not just the chat interface. It's genuinely more impressive in my opinion.

Ah that's weird. Using text-davinci-003 in the playground gives very different answers to ChatGPT, and that's with playing with the temperature as well.

> First of all, don't use ChatGPT for that

> It's strange to me that people are going crazy over ChatGPT when GPT-3

People are going crazy because of the combination of the capabilities of the model and the UX.

Surely you can appreciate how deploying your own full GPT-3 based solution to assist in code generation is much more complex than going to a web app and typing "code me an Express server".

For me the conversational interface is what makes the AI useful as a coding companion, how do you integrate GPT-3 in your workflow ?

I've just been using the Playground. Also, working on creating a voice activated personal assistant.

GitHub Copilot

Well, ChatGPT is free for now.

Yup, free will always help your popularity.

I've seen some experiments done with using ChatGPT to produce entire websites or programs

https://github.com/pdparchitect/ChatGPT-JobsForBots and https://github.com/pdparchitect/ChatGPT-Pong were two of them.

The author shows all the steps used to generate the programs.

I believe ChatGPT has some kind of response token limit, which is why longer code snippets commonly get cut off. I've seen some people recommend to enter something like "Continue your previous message starting from the last token", which sometimes works, but even when it does, it often breaks formatting, making it difficult to read the resulting code without copying it somewhere else. What I found to work better is instructing it to output a single function at a time, this usually results in short enough code that it can output all of it in a single message. Something like:

  I want you to send a single function in each response, I will say "Next" when I want you to send the next function.

I was able to get it to generate decently lengthy programs using this (on the order of a few hundred lines) that were syntactically and logically correct without any modification. This may work for some languages better than others, though I imagine you can find some way to logically split up the program for most.

All you need is to prompt it with "Continue" and it will keep going exactly where it left off.

I've heard this quite a lot too, but it commonly doesn't work. I've had it respond thinking I was talking about using a `continue` statement somewhere in the code, and other times it simply just reprinted the whole code starting from the beginning. If you hit the "Try again" button enough it will usually eventually work, but it again has the problem where a lot of the time the formatting of the text gets messed up.

I've had no problems just writing: "continue the code from above example" or something akin to that - across 25+ full app prompts.

There has been a few times where the app hangs with the three dots jumping - but that seems to be an issue with server overload.

I've been able to make lots of mini apps in JS/HTML/CSS that with very little tweaking could be dropped into codepen fully working.

Most of these required ~3 "continue the code example" and were started with at prompt like "Show me the code for an app in js, html, css where a user can input his weight in kg, and his daily calorie deficit. Then a canvas element shows a graph of his future weight loss changing live according to the changed input."

Absolutely amazing that it works. You can even dump an error from the console if it needs tweaking and it often fixes the error from the msg on the first try.

You can also say "give me just X function" or "continue above example from the line with 'someFunction or something'" or "give me just the css again but make it more modern" and it just works.

That's due to the front-end markdown codeblock ending and the model not outputting it again since it's continuing from the last token. I believe if you tell it to continue but include code blocks again, it will do it

It sure can, esp good at simpler things like ui, and popular frameworks etc. The only issue is when there's breaking changes between upgrades, like maybe the framework is at 9.0 but the model was trained on 7.0, so there's some things that don't work just right.

So far it's created some pretty amazing tailwindcss layouts for me, and a flutter app to interface with chatGPT that I want to add more features like saving, sharing, etc.. In 3 hours with zero flutter experience I can now at least do as much as the website does. That's pretty amazing.

Edit: YMMV -- imho you do need to be mid-senior level in tech/programming skills to know and find the best prompts to get it to go your way, there's some trial and error and your ide will usually let you know about errors/etc and you can ask them to fix it and gpt does a good job of correcting things.

I gave it a .txt file that i wanted to parse its data into .csv columns. It actually got most of it right but after spending hours prompting it, it never really got everything right.

Its worst nightmare was that there were some nested values in the .txt. What amazed me the most is that pasting the error code from colab into the chat, it recognized the mistake and generated a fix.

That being said, most of the times it would try to fix a logical mistake by overengineering a solution to an error, or it would identify the mistake correctly, explain in natural language what needs to be done (impressive), but then give me the code unchanged.

Try a simple calculator. It will give you the complete code for that. I think there's a set limit on output text so that's why your tic-tac-toe was incomplete. I've asked for the calc in Rust and Python and I got a working app for both. It was simple but the fact that it gave me what I asked for is amazing.

I bet that you could break-up your app into functions and have ChatGPT write each function to get to a bigger program.

It's limited now but I can definitely see the possibilities.

It won't be free forever so we'll have to weight the usefulness vs cost eventually.

It's not that amazing. You can find complete examples of calc in Rust and Python all over the Internet. It just spit out a copy of someone else's code, and gave it to you.

Instead of you having to Google search everywhere or look in multiple sources, you were able to just ask ChatGPT.

If it stays at its current programming level then you're right but I'm more impressed with the ask-and-you-get capabilities and the future possibilities of the AI. My assumption is that it will get better as time passes, from past experience.

Overall, it's impressive tech.

Yes! The calculator worked and I agree with you that in most of my prompts when I have requested to give elaborated answers, it doesn't give appropriate results.

It's ok for small blocks or code, maybe a function or two, even a whole class, but beyond that, it's pretty pathetic. Even the small blocks of code it outputs only compile a few times and have a few errors. At this point, I would be wary of using it in Production systems.

Having said that, I usually compare it with Junior programmers, more like someone just learning a new programming language. You can expect that level of performance from ChatGPT!

If you know roughly what you want and just don't want to dig through documentation to find the exact correct incantation, it's very good at that.

It's kinda crazy how fast my opinion went from "haha neat, the robot can write poems" to "this is an invaluable tool and I am begging you to let me pay for stable access". I use it multiple times per day, it's easily saved me hours already.

There is a character limit on a single response. You can ask things like 'Continue the previous answer' to get it to generate more.

just "continue" works :)

When it’s incomplete I ask it to refactor to smaller methods, and if it crops the output again I ask it to show the full text of a method one by one to get the full output. Tedious but it works.

From what I've seen so far you can tell it it's incomplete, tell it what errors you're getting etc, and it will agree and correct itself until you have a working application.

I feel like the more interesting question is whether ChatGPT is better at generating political speeches instead?

Seems like there would be far more content to train on and less restrictive criteria.

Tim Corey has some thoughts on that https://www.youtube.com/watch?v=FEsahkX7ZF8

Somewhat related — I’m curious has someone built a ChatGPT based Linux command line helper? Ideally should be runnable via CLI

Take a look at Warp.dev

It does look nice, thanks! I recall it was discussed here before

It does many mistakes that are only obvious if you know what you're looking at.

type "continue" & it will continue where you left off.

it has a token cap so it stopped once it filled them.

Here is the response that I got after the "continue" command and its still incomplete.

https://pythonsandbox.dev/e6uevcuueka1?file=main.py

i really hate all this. codepilot and stuff. it takes away the joy of writing code. esp clean code.

no it's mostly object oriented

Golang results are amazing, they compile most of the time with some lib-specific issues from time to time.

You can ask to explain code, write tests, refactor for perfomance or readability etc.

Prompt:

  Write a small application in javascript that logs in to news.ycombinator.com with credentials passed on the command line and then posts the following message:

  >Fellow members,
  >
  >I'm curious to know what are some of the best prompts to use on ChatGPT to understand its full-fledged capability as a newbie developer.
  >
  >One of the things I noticed, is that it produces the requested code in the cleanest manner but it's not complete. For instance, I tried the prompt - "give me the python code for TicTacToe Game" and the code produced was incomplete.
  >
  >Have you all encountered the same or my prompt is inappropriate?

Output:

Here is an example of a small JavaScript application that logs in to news.ycombinator.com using credentials passed on the command line and then posts the message you provided:

  // First, we need to import the `request` module, which will allow us to make HTTP requests to the Hacker News API
  const request = require('request');

  // Next, we need to parse the username and password from the command line arguments
  const username = process.argv[2];
  const password = process.argv[3];

  if (!username || !password) {
    // If the username or password is not provided, print an error message and exit the program
    console.log("Error: Please provide a username and password as command line arguments");
    return;
  }

  // Now we can use the `request` module to make a POST request to the Hacker News login API endpoint
  // In the request, we need to provide the username and password as part of the request body
  request.post({
    url: 'https://hacker-news.firebaseio.com/v0/login.json',
    body: JSON.stringify({
      "username": username,
      "password": password
    })
  }, (error, response, body) => {
    if (error) {
      // If there is an error, print it to the console
      console.log(error);
      return;
    }

    // If the login was successful, the API will return the user's authentication token
    // We need to save this token so that we can use it in subsequent requests
    const authToken = JSON.parse(body).token;

    // Now that we have the authentication token, we can use it to post a new message on Hacker News
    // To do this, we need to make a POST request to the `item` endpoint, providing the authentication token as part of the request headers
    request.post({
      url: 'https://hacker-news.firebaseio.com/v0/item.json',
      headers: {
        "Authorization": `Bearer ${authToken}`
      },
      body: JSON.stringify({
        "title": "Best Prompts for ChatGPT",
        "text": "Fellow members, I'm curious to know what are some of the best prompts to use on ChatGPT to understand its full-fledged capability as a newbie developer. One of the things I noticed, is that it produces the requested code in the cleanest manner but it's not complete. For instance, I tried the prompt - \"give me the python code for TicTacToe Game\" and the code produced was incomplete. Have you all encountered the same or my prompt is inappropriate?"
      })
    }, (error, response, body) => {
      if (error) {
        // If there is an error, print it to the console
        console.log(error);
        return;
      }

      // If the request was successful, the API will return the ID of the newly-created post
      const postId = JSON.parse(body).id;
      console.log(`Successfully posted message with ID ${postId}`);
    });
  });

To use this code, you would need to save it to a file (e.g. hacker-news-post.js) and then run it from the command line, providing your username and password as arguments: