We’ve asked ChatGPT for general interview advice and tested how well ChatGPT writes cover letters and resumes. This time, we got a little more specific and tested ChatGPT for technical interviews.
Once again, the results were mixed. But this time, they were very mixed! Over several tests of the same question, we ended up with wildly different results!
What Is a Technical Interview?
A technical interview assesses your technical abilities. It’s similar to a case study interview in that you’re given a problem and asked to solve it on the spot. This could be math or coding questions, but technical interview questions can ask about other things, too.
While you may or may not encounter the technical problem you’re being asked to solve on the job, interviewers use technical interviews to see how you put your skills to work.
Technical Interview Prep
Get prepped and ready for your technical interview in this free course from Girls Who Code.
Avg. Time: 5-6 hours
Skills you’ll build: Whiteboard challenge, hash table, array, matrix
ChatGPT’s Technical Interview Tips
Most advice on prepping for technical interviews centers on brushing up on your technical skills, reviewing the basics of the job you’re applying for, and remaining calm during the interview. All good advice! But those are very general tips, and technical interviews often ask specific questions. So, we wondered, could you use ChatGPT to better prepare for technical interviews?
We started with the basics and asked ChatGPT if it had any tips for prepping for a technical interview. Here’s what it said:
Overall, the first five tips, while good, are pretty general, which is OK. Some interview tips are the same, no matter what kind of interview you’re in!
The second half of the answer is also OK in that it offers some general technical interview questions you might encounter. That said, ChatGPT doesn’t give any advice on how to answer the questions, so we asked specifics.
Handling Conflicts and Version Control Systems
We started with one of the technical interview questions ChatGPT mentioned, handling conflicts in a version control system:
The advice is fine, though not optimized for an interview answer. Interestingly, when we Googled the same question, a number of search results talked about handling general conflict in the workplace. While an interview may ask about your conflict resolution skills in a behavioral interview, those have nothing to do with handling conflicts in a version control system. So, we have to hand it to ChatGPT on this one. It clearly understood what we were asking and provided the better answer.
Advanced Software Engineering
Take your software engineering skills to the next level in this free course from Walmart USA.
Avg. Time: 3-4 hours
Skills you’ll build: Java, UML, SQL, Python, data munging
The one nitpick we have is that ChatGPT doesn’t suggest talking about a time you had to handle this situation in real life. That would help the interviewer understand that you already possess the skills they’re asking about and know how to address this situation.
>>MORE: How to Use the STAR Method for Interview Questions
Talking About Technical Projects
Next, we asked ChatGPT a less technical question. We wanted to know how to talk about our side projects in a technical interview:
Overall, this is a pretty good answer. Each tip starts with a general and easy-to-remember tip. After the colon, ChatGPT explains what you should talk about in particular (your skills, the problem you wanted to solve) and, more importantly, explains why this matters to your answer (how you solved the problem, you received positive feedback).
Once again, we have to give it to ChatGPT. This is excellent advice!
Software Engineering
See what it's like to work in software engineering at Hewlett Packard Enterprise in this free virtual job simulation.
Avg. Time: 6-7 hours
Skills you’ll build: Java, JSON, HTTP requests, JUnit, Mockito, RESTful web development
Designing a System for Scale
Then we tried a more complex situation. Could ChatGPT help someone prepare for a technical interview question that you can’t really prepare for? We asked how to prepare for a question that asked us to design a system for scale:
And again, ChatGPT gets it right.
The trick to this question is that you can’t answer it without getting more information first, something ChatGPT points out right from the start. Then it tells you what areas you need to ask questions in (data storage, networking, etc.) and what kinds of questions you need to ask to get the right information. Only then can you formulate your answer.
Well done!
Software Engineering
Test your software engineering skills in this free course from Goldman Sachs.
Avg. Time: 1-2 hours
Skills you’ll build: Cryptography basics, password cracking, password best-practices
Where ChatGPT Confused Us: Code Output
One of the problems with using ChatGPT for technical interview prep is if you don’t understand the basics, you may not know if the answer is right or wrong. Frankly, even when you understand the basics and beyond, ChatGPT’s answers may confuse you.
We asked ChatGPT a technical interview question that we grabbed off the internet and input it exactly as we found it:
At first glance, this answer seems fine. However, it took a few attempts to get a response because ChatGPT kept crashing. The next day, we reentered the question to see what would happen, and this time ChatGPT said the answer was “1, 21.” What’s more, a quick search of the internet said the answer is 21, 21.
Confused, we turned to the engineering team at Forage and asked which answer, if any, was correct.
Software Engineering
See what it's like to work as a software engineer in this free course from Electronic Arts.
Avg. Time: 5-6 hours
Skills you’ll build: Game engine technology, code analysis, class design, object-oriented design
After a lively discussion on Slack, the conclusion was, “it depends.” Part of the problem with the question is that it doesn’t specify which language the question is in, which impacts the answer. This is something ChatGPT missed on our initial inquiries.
Curious, we went back to ChatGPT and asked the question again, but this time specified several languages. ChatGPT said that if the language were Python, Java, JavaScript, R, or C++, the code would not run and produce an error. In each instance, ChatGPT explained the specifics of what was wrong with the code. Here’s one example:
Of the languages we tested, this code would only run in PHP:
So, why then did ChatGPT originally say the output would be 1,21, 1 21 or any of the other answers?
We started by asking when this code would result in an answer of 1,21:
And apparently, the answer is “never.” Then we asked how we could get an output of 1 21, and ChatGPT explained what to do:
Interestingly, when we asked ChatGPT how to get to 2 2 and 21 21, ChatGPT explained that the code was wrong and how to fix it:
Because this was one long conversation, we wondered if ChatGPT would have a different answer to the original question. And, sure enough, this time ChatGPT said the code wouldn’t work at all:
Clearly, ChatGPT gets confused sometimes. And while the initial coding question we asked had problems, the fact that ChatGPT’s answers changed a lot over the course of a few days indicates that relying on ChatGPT as your only source of technical interview preparation may not be your best bet.
Software Engineering
Test-drive a career in software engineering in this free course from Wells Fargo.
Avg. Time: 3-4 hours
Skills you’ll build: Java, JPA, Spring, system design
Should You Use ChatGPT to Prepare for Technical Interviews?
As we’ve noted in our other experiments, ChatGPT can be hit or miss. And, in this case, while the hits were close to home runs, the misses were spectacular strikeouts.
So, should you use ChatGPT to prepare for technical interviews? Maybe. Some of the advice was quite useful and actionable, but some was just plain wrong — and confusing.
Because technical interviews are about your technical skills and abilities, you are often better off (as much of the advice says!) brushing up on those skills by reviewing coding languages or whatever technical skills you’ll need for the role. Then, relax, talk through what you’re doing and why you’d do it, then do your best to solve the problem at hand.
Gain the skills you’ll need to ace a technical interview. Enroll in a free Forage software engineering virtual job simulation today!
Image credit: Canva
