Can ChatGPT really be trusted to write a smart contract or to refactor your existing Solidity code?

Abstract

OpenAI’s latest release, GPT-3.5 or ChatGPT, offers significant improvements in natural language processing for tasks such as language translation, text summarization and text generation. Developers are beginning to explore the potential of asking ChatGPT to assist with writing, reviewing or refactoring code, as well as providing technical information for development tasks. However, when put to the test, ChatGPT struggled with writing Solidity code from scratch and struggled with understanding Solidity code written by others, even with the assistance of an experienced developer. More data and computational power is needed before ChatGPT can be a valuable tool for writing code.

This abstract was generated by GPT-3.5

A deep dive into ChatGPT-3.5

You may have noticed that OpenAI recently released an updated version of their GPT model. The latest version, GPT-3.5, also known as ChatGPT, offers significant improvements over its predecessors, including increased accuracy and larger capacity. These improvements make GPT-3.5 even better at natural language processing tasks, such as language translation, text summarization, and text generation.

Social media has been awash with AI-generated blog posts, essays and even poems and recipes created from ChatGPT prompts. But on a serious note, developers are beginning to realise the possibilities of asking ChatGPT to help write, review or even refactor code, as well as providing technical information that can assist with development tasks.

In this post I want to understand what are the implications for Web3 developers. I will focus on two aspects: first, writing Solidity without previous knowledge, and secondly a more challenging task – writing Solidity with some understanding and interacting with a smart contract from a front end.

Writing Solidity without any previous knowledge

Let’s start with our first prompt: “write a smart contract for an NFT sale”.

Below is what GPT gave me as a response to this prompt:

pragma solidity ^0.6.0;

contract TokenSale {

  // The name of the token
  string public name = "MyToken";

  // The symbol of the token
  string public symbol = "MTK";

  // The total supply of tokens
  uint256 public totalSupply;

  // The amount of tokens sold in the current sale
  uint256 public totalSold = 0;

  // The price of each token in wei
  uint256 public tokenPrice;

  // The address of the contract owner
  address public owner;

  // Events for token purchase and refund
  event TokenPurchase(address indexed purchaser, uint256 amount);
  event TokenRefund(address indexed refundee, uint256 amount);

  // Mapping from user address to their token balance
  mapping(address => uint256) public balanceOf;

  // Constructor to set the total supply of tokens and the price per token
  constructor(uint256 _totalSupply, uint256 _tokenPrice) public {
    owner = msg.sender;
    totalSupply = _totalSupply;
    tokenPrice = _tokenPrice;
  }

  // Fallback function to purchase tokens
  function() external payable {
    buyTokens();
  }

  // Function to purchase tokens
  function buyTokens() public payable {
    require(msg.value >= tokenPrice, "Insufficient payment");

    // Calculate the number of tokens to be purchased
    uint256 tokens = msg.value.div(tokenPrice);

    // Update total sold and the balance of the purchaser
    totalSold += tokens;
    balanceOf[msg.sender] += tokens;

    // Emit a token purchase event
    emit TokenPurchase(msg.sender, tokens);
  }

  // Function to refund ether to the sender
  function refund() public {
    require(balanceOf[msg.sender] == 0, "Cannot refund - tokens already purchased");
    msg.sender.transfer(msg.value);
    emit TokenRefund(msg.sender, msg.value);
  }

  // Function to end the token sale
  function endSale() public {
    require(msg.sender == owner, "Only the owner can end the sale");
    require(totalSold == totalSupply, "Cannot end sale - all tokens not sold");
    owner.transfer(address(this).balance);
  }
}

I was interested to note that it used Solidity ^0.6.0 , and also that it had not chosen to use the ERC721 and ERC1155 standards.

I decided to try an experiment. I gave it the same prompt a second time – and to my surprise, this time it generated a contract that used the ERC721 standard. This showed how the system has a random aspect for keeping the innovation moving forward – but it did make me wonder how a developer new to Solidity could possibly understand the difference between the two versions.

// Import ERC-721 interfaces
import "https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/token/ERC721/ERC721.sol";

contract NFTokenSale is ERC721 {
  // ...
}

Still in the persona of someone who knew nothing about Solidity or smart contracts, I entered the prompt “Where can I test this code?” and got the following answer:

You can test this code using a local development environment or a blockchain testnet.
To test the code locally, you can use a Solidity compiler like solc to compile the code and generate the contract ABI (Application Binary Interface). You can then use a local Ethereum client like ganache-cli to deploy the contract to a local development blockchain and interact with it using a web3.js library like truffle-contract.
To test the code on a blockchain testnet, you can use a service like Remix or Bamboo to compile the code and generate the contract ABI. You can then use a wallet like MetaMask to deploy the contract to a testnet like Rinkeby or Ropsten and interact with it using a web3.js library.
Alternatively, you can use a service like OpenZeppelin’s Ethereum Package Registry to deploy a pre-compiled version of the contract to a testnet and interact with it using a web3.js library. This approach can save you time and effort compared to compiling and deploying the contract yourself.

While this was broadly helpful, it did not actually contain any actionable items. I decided to go ahead and try the code with Hardhat – but it failed to compile because the specified version of OpenZeppelin doesn’t have the ERC721 contract with pragma solidity ^0.6.0;

To figure out ChatGPT’s logic, I asked how to fix this issue, and the assistant came back with a statement that make clear it was working from an old dataset:

You can fix this issue by specifying the Solidity version that is supported by the latest version of the OpenZeppelin library. The current version of the OpenZeppelin library supports Solidity versions 0.5.x and 0.6.x, so you can update the pragma statement in your contract to use one of these versions:

// For Solidity 0.5.x
pragma solidity ^0.5.0;

// For Solidity 0.6.x
pragma solidity ^0.6.0;

You should also make sure that you are using the latest version of the OpenZeppelin library in your contract. You can do this by using the import statement to import the contract interfaces from the latest version of the library, which is hosted on the Ethereum Package Registry. For example:

// Import ERC-721 interfaces from the latest version of the OpenZeppelin library
import "@openzeppelin/contracts/token/ERC721/ERC721.sol";

The last part is critical because without changing the http import to the Ethereum package Registry, Hardhat is not able to compile.

I decided to change task and asked ChatGPT: “Does the NFTTokenSale contract compile?” only to receive the following generic and rather frustrating answer:

It is not possible to determine whether the NFTokenSalecontract will compile without actually trying to compile it. The contract may contain syntax errors or other issues that prevent it from being compiled successfully.

Giving up with the assistant for now, I tried to compile by myself and I started to receive a mix of compile errors, including:

• the fallback function wasn’t implemented by using the fallback keywords
• All the public variables are not implemented correctly
• String concatenation doesn’t work like that in Solidity 0.6.x

This made it clear to me that for a user without Solidity know-how, or for a non-developer, ChatGPT is not yet at the stage where it can be useful.

It was now time to move on see how ChatGPT functioned as an assistant to a developer with a working knowledge of Solidity.

Writing Solidity with the AI assisting an experienced developer

This time I decided to start from something existing like the CryptoPunk contract. I picked this one because it’s quite short which helps with ChatGPT timeout issues.

I tried first to understand a portion of the code (the whole contract returns a timeout on ChatGPT) by leveraging the assistant: “Explain the below buyPunk method ”

function buyPunk(uint punkIndex) payable {
        if (!allPunksAssigned) throw;
        Offer offer = punksOfferedForSale[punkIndex];
        if (punkIndex >= 10000) throw;
        if (!offer.isForSale) throw;                // punk not actually for sale
        if (offer.onlySellTo != 0x0 && offer.onlySellTo != msg.sender) throw;  // punk not supposed to be sold to this user
        if (msg.value < offer.minValue) throw;      // Didn't send enough ETH
        if (offer.seller != punkIndexToAddress[punkIndex]) throw; // Seller no longer owner of punk

        address seller = offer.seller;

        punkIndexToAddress[punkIndex] = msg.sender;
        balanceOf[seller]--;
        balanceOf[msg.sender]++;
        Transfer(seller, msg.sender, 1);

        punkNoLongerForSale(punkIndex);
        pendingWithdrawals[seller] += msg.value;
        PunkBought(punkIndex, msg.value, seller, msg.sender);

        // Check for the case where there is a bid from the new owner and refund it.
        // Any other bid can stay in place.
        Bid bid = punkBids[punkIndex];
        if (bid.bidder == msg.sender) {
            // Kill bid and refund value
            pendingWithdrawals[msg.sender] += bid.value;
            punkBids[punkIndex] = Bid(false, punkIndex, 0x0, 0);
        }
    }

This generated the following explanation:

The buyPunk() function is a function that allows users to buy a “punk” token from another user who has offered it for sale. This function takes the index of the punk as an input, and it is expected that the user who wants to buy the punk sends a payment in Ether along with the function call. The function first checks if the punk is actually for sale, and if the caller is allowed to buy it (if it has been offered for sale to a specific user). If either of these checks fail, the function throws an error and stops execution. If the checks pass, the function updates the ownership of the punk by transferring it from the seller to the buyer. It also updates the balances of the seller and buyer to reflect the transfer. Finally, the function checks if the buyer had previously entered a bid for the punk, and if so, it refunds the bid and removes the bid from the record. This is done to ensure that the buyer does not end up paying twice for the punk.

This was interesting because it gave me plenty of context on something that looked abstract at first glance. I found particularly cool the fact it was capable to understand the meaning of BalanceOf[seller]— and BalanceOf[msg.sender]++ without leveraging any comment from the code.

I then tried to change the code by giving it the following prompt: “Change the above code to take a 5% fee from the transaction and keep it in the contract”.

// Calculate the 5% fee and keep it in the contract
uint fee = msg.value / 20;
address contractAddress = this;
contractAddress.transfer(fee);
// few lines later after other logic ...
pendingWithdrawals[seller] += msg.value - fee;

I was impressed, but once again I had to greatly reduce the scope of my query to avoid the timeout and even with that, iI couldn’t get the whole function written.

Now I asked it to test the function:

function testBuyPunk() {
    // Create a new CryptoPunksMarket contract
    CryptoPunksMarket contract = new CryptoPunksMarket();

    // Set up some test data
    address seller = msg.sender;
    address buyer = address(1);
    uint punkIndex = 0;
    uint value = 5;

    // Offer the punk for sale
    contract.offerPunkForSale(seller, punkIndex, value, 0x0);

    // Buy the punk from the seller
    contract.buyPunk(punkIndex).send({ value: value, from: buyer });

    // Check that the punk has been transferred from the seller to the buyer
    assert(contract.balanceOf(seller) == 0);
    assert(contract.balanceOf(buyer) == 1);
    assert(contract.punkIndexToAddress(punkIndex) == buyer);
}

I thought it was an interesting approach, although it failed to take the fee into account. One of the cool things about ChatGPT is that it retains the knowledge of your previous prompts, allowing you to iterate and refine by asking the assistant to perform follow-up tasks.

I pointed out: ”It doesn’t check the fact that the 5% fee has been deducted” and received the following answer:

function testBuyPunk() {
    // Create a new CryptoPunksMarket contract
    CryptoPunksMarket contract = new CryptoPunksMarket();

    // Set up some test data
    address seller = msg.sender;
    address buyer = address(1);
    uint punkIndex = 0;
    uint value = 5;

    // Offer the punk for sale
    contract.offerPunkForSale(seller, punkIndex, value, 0x0);

    // Buy the punk from the seller
    contract.buyPunk(punkIndex).send({ value: value, from: buyer });

    // Check that the 5% fee has been deducted from the transaction and kept in the contract
    uint fee = value / 20;
    assert(address(this).balance == fee);

    // Check that the punk has been transferred from the seller to the buyer
    assert(contract.balanceOf(seller) == 0);
    assert(contract.balanceOf(buyer) == 1);
    assert(contract.punkIndexToAddress(punkIndex) == buyer);
}

It tried to check that the contract has the correct balance but the syntax was wrong because it referred to this – and also, the seller couldn’t get all the funds that were due. So I nudged the assistant with the prompt: The seller can’t receive the full value if the fee has been deducted.

function testBuyPunk() {
// Create a new CryptoPunksMarket contract
CryptoPunksMarket contract = new CryptoPunksMarket();

// Set up some test data
address seller = msg.sender;
address buyer = address(1);
uint punkIndex = 0;
uint value = 5;

// Offer the punk for sale
contract.offerPunkForSale(seller, punkIndex, value, 0x0);

// Buy the punk from the seller
contract.buyPunk(punkIndex).send({ value: value, from: buyer });

// Check that the seller receives the correct amount after the 5% fee has been deducted
uint fee = value / 20;
assert(contract.pendingWithdrawals(seller) == value - fee);

// Check that the punk has been transferred from the seller to the buyer
assert(contract.balanceOf(seller) == 0);
assert(contract.balanceOf(buyer) == 1);
assert(contract.punkIndexToAddress(punkIndex) == buyer);
}

Frustratingly, the pendingWithdrawals now took account of the fees but the balance of the contract now was now no longer checked.

We can see from this that we can indeed update code successfully by prompting ChatGPT, but performance limitations (specifically in terms of timeouts) mean that it is not practical to use it for complex use cases at the moment.

Interacting with a Solidity smart contract from the front end

My final challenge was to understand what happens if rather than using Solidity I switch to Javascript to interact with the existing smart contract. In order to do this, I needed to get the ABI of the contract, converting Solidity into a JSON format which my front end could use.

I prompted ChatGPT to “write the ABI of the transferPunk function” and it worked nicely, including the explanation of the characteristic of the function in a human-readable format.

A snippet below:

{
  "constant": false,
  "inputs": [
    {
      "name": "_punkIndex",
      "type": "uint256"
    }
  ],
  "name": "buyPunk",
  "outputs": [],
  "payable": true,
  "stateMutability": "payable",
  "type": "function"
}

The buyPunk function is not marked as constant, so it can modify the state of the contract. It takes a single input, _punkIndex, which is the index of the punk to be purchased. The function is payable, so it can receive cryptocurrency as payment. It has no return value, so the outputs array is empty.

Note that this is just one possible ABI for the buyPunk function. The specific ABI will depend on the implementation of the function in the smart contract.

The only small issue I found is that the ABI format wraps all functions into an array, which it did not do for this snippet. However, when I asked it to generate the ABI of the whole contract, it indeed correctly enclosed all functions into an array.

The next step was to see how it performed with the front end. I prompted: “Write with React and etherjs a minimal ui with a mint button to buy a punk”

A snippet below:

import React, { useState } from "react";
import ethers from "ethers";

// Set the contract ABI and address
const contractABI = [
  {
    "constant": false,
    "inputs": [
      {
        "name": "_punkIndex",
        "type": "uint256"
      }
    ],
    "name": "buyPunk",
    "outputs": [],
    "payable": true,
    "stateMutability": "payable",
    "type": "function"
  },
  // Other contract ABI goes here...
];
const contractAddress = "0x...";

function App() {
  // Create a state variable to store the status of the transaction
  const [transactionStatus, setTransactionStatus] = useState("");

  // Function to handle the buy punk button click
  async function handleBuyPunkClick() {
    // Create a provider to connect to the Ethereum network
    const provider = new ethers.providers.JsonRpcProvider("http://localhost:8545");

    // Create a wallet to sign transactions
    const privateKey = "0x...";
    const wallet = new ethers.Wallet(privateKey, provider);

    // Get the contract instance
    const contract = new ethers.Contract(contractAddress, contractABI, wallet);

    // Buy a punk by calling the buyPunk function and passing the punk index and the payment amount
    // The payment amount should be in wei
    const punkIndex = 0;
    const paymentAmount = ethers.utils.parseEther("0.01");
    try {
      await contract.buyPunk(punkIndex, { value: paymentAmount });
      setTransactionStatus("Transaction successful");
    } catch (err) {
      setTransactionStatus(`Transaction failed: ${err.message}`);
    }
  }

  return (
    <div>
      <button onClick={handleBuyPunkClick}>Buy Punk</button>
      <p>{transactionStatus}</p>

The example is reasonably good, but as you can see it’s interrupted part of the way through and there is no explanation to contextualize what it is doing.

Nevertheless it’s impressive that ChatGPT could handle the context so that it remembered the ABI it generated, which it then uses to build the example with React and etherjs.

I now had to reset the section and start from scratch. Interestingly, the results were quite different this time and I had to nudge the assistant a few times to recreate the right context, but finally it produced something.

import React, { useState } from 'react';
import { useContract, useWeb3 } from '@ethers-react/web3';

const buyPunkABI = [
  {
    constant: false,
    inputs: [
      {
        name: 'punkIndex',
        type: 'uint256',
      },
      {
        name: 'onlyBuyFrom',
        type: 'address',
      },
    ],
    name: 'buyPunk',
    outputs: [],
    payable: true,
    stateMutability: 'payable',
    type: 'function',
  },
];

function App() {
  const contract = useContract(process.env.REACT_APP_CONTRACT_ADDRESS, buyPunkABI);
  const { account } = useWeb3();
  const [punkIndex, setPunkIndex] = useState(0);

  async function handleMint() {
    await contract.mint(account, punkIndex);
  }

  return (
    <div>
      <input
        type="number"
        value={punkIndex}
        onChange={(e) => setPunkIndex(e.target.value)}
      />
      <button onClick={handleMint}>Buy Punk</button>
    </div>
  );
}

export default App;

This looks good but unfortunately it was not quite as good as it seems – because @ethers-react/web3 doesn’t exist as a library. It seems that ChatGPT simply decided to invent a library based on the concepts and capabilities of others it had seen.

I then tried multiple different prompts such as “rewrite the above example using React and Ethers libraries only“ or “ethers-react libray doesn’t exist, just use import ethers from "ethers"; and use this library only” but none of the outputs gave me what I needed.

Conclusions

Despite the mixed results, ChatGPT, aka GPT-3.5, is a step forward in the direction of writing code with an AI assistant. I actually enjoyed doing these little experiments. However, compared to other experiments I did with JavaScript and other languages, a clear takeaway from my efforts is that when it comes to the Web3 space, GPT doesn’t yet have enough accuracy.

In fairness, there is far less available Solidity and Web3-related JavaScript code in the wild than there is general-purpose JavaScript code. Plus, the web3 industry is constantly changing, which makes the problems of ChatGPT relying on an old dataset much worse. .

On the positive side, generating an ABI from Solidity is something it did well, which shows it can learn from the available snippets the general rules to create something new.

Another aspect I found quite helpful is its capability of explaining the code in plain English. This is an area where I can see it will help developers gain a better understanding of code written by someone else because, most of the time, a developer reads rather than writes new code.

In conclusion, despite the difficulties, I thought the technology was extremely promising, but we are early, and it needs more data and computation power before it can achieve the next breakthrough. Training the models with compilers to verify the output could significantly improve the accuracy of the output when it comes to writing code.

I’m looking forward to seeing what GPT will bring next. Let me know about your own discoveries when exploring ChatGPT and Web3!