Benchmarking AI Code Builders

Complexity of software development is increasing exponentially and consequently is becoming ever more time consuming. Artificial intelligence tools are providing capability for developers to automate areas of software development, such as coding, testing, and debugging. In this article, we will assess different automated AI code generating tools on the market.Artificial intelligence has been around for a long time and the know how to deliver such intelligence. However, until recently there has been missing ingredients. The intersection of cloud computing, big data, increased processing capabilities have all converged, making artificial intelligence tools more powerful and more capable.We have run an analysis of a number of well known, and less well known AI tools for software development, including

Co Pilot

Chat GPT 4o

Gemini Pro 1.5

Sonar-8x7B

SES

Claude 3

Requirement 

"make an apex function to validate a postcode and email then format postcode and insert an account with these details and finally create and attach a file to the Account passed to the function" 

SES Response

See this video Link 

SES re-used code from the system, correctly produced an Account and a File and validated both the postcode and email.

Co Pilot

Asking Co Pilot the same question that was asked to SES in the video 

Response 

The response produced is not what was asked and isn't a working solution and so Co Pilot failed to produce what was required. Even if Co Pilot had produced something that could be used the developer would have to construct the function, something that the developer wouldn't have to do using SES. Secondly, Co Pilot would produce new code instead of re-using existing code and therefore compounding the problem of bloated and duplicated code in the system; whereas SES avoids making the same mistakes.  

Chat GPT 4o Response

Asking Chat GPT 4o the same question provided a better response, but didn't produce any code for 3 of 4 required functions. It also decided to create an Attachment not a File which is a poor choice. 

Gemini Pro 1.5 Response

Asking Gemini Pro 1.5, their AI created a Java function which is not what was asked and could not be adapted for Salesforce 

Sonar-8x7B Response

Asking Sonar inserts an Account and creates a File correctly. Sonar has also validated the postcode and email addresses and formatted the postcode. However, didnt use a try{} catch{} or a database savepoint and the user is not able to pass the File content to the function. 

Claude 3 Response 

Asking Claude 3 inserts an Account and creates a File correctly, but has not validated the postcode or email addresses. Sonar probably performed the best from 5 AI's tested. However, all produced new code which would create bloated and duplicated code in the system and the customer would in the future have to remove most of the code produced by these AI systems and therefore all of the above AI tools create technical debt for customers. Only SES was able to complete the requirements, but also re-use existing code from system, and therefore minimise the creation of new code. This is significant because reduced code size and especially reduced duplicated code will mean fewer defects, faster code and more manageable code.