Microsoft subsidiary GitHub announced the new AI-powered Copilot service as a private beta for developers on June 29, 2021, with the “AI pair programmer” embedded in Microsoft’s massively popular Visual Studio Code editor as an extension for beta users. As developers write their code, Copilot will step in to make suggestions, much like the autocomplete feature in email applications like Gmail, but for code written in Python, JavaScript, TypeScript, Ruby, Go, and other programming languages.
Copilot was built in collaboration with OpenAI, the artificial intelligence lab founded by Elon Musk, Sam Altman, and others and that Microsoft invested $1 billion in last year. OpenAI adapted its GPT-3 language-prediction model for computer code for this project in a model it calls Codex. Teaching neural networks to write code is not a new endeavor, with startups like TabNine and Kite working on similar projects. But the heft of Microsoft and OpenAI means Copilot is a strong entrant to the market from Day 1.
What can a $1 billion investment buy?
On Tuesday this week, OpenAI and GitHub answered this question boldly with the preview of a new AI tool — GitHub Copilot. It can write user-compatible code and is much better at the task than its predecessor — GPT-3.
Copilot autocompletes code snippets, suggests new lines of code, and can even write whole functions based on the description provided. According to the GitHub blog, the tool is not just a language-generating algorithm based on user input — it is a virtual pair programmer.
It learns and adapts to the user’s coding habits, analyzes the available codebase, and generates suggestions backed by billions of lines of public code it has been trained on.
It is the first taste of Microsoft’s $1 billion investment into OpenAI — a research and development company specializing in AGI (Artificial General Intelligence).
What works well in Copilot?
Cassidy Williams, director of developer experience at PaaS company Netlify, said that Copilot “is going to be incredibly useful for things like smaller utilities that I have to often rewrite across various projects. The fact that I could use it to do certain capitalization patterns, or write certain pattern-matching functions for me that I would normally have to either just hunker down and write or look up how to do for the 100th time,” she told InfoWorld.
“Instead of autocomplete, it gives me ideas for full-fledged functions, random code items, and can even pull together some written paragraphs for when I blog,” Basile said, based on his early experiences with the tool.
“I think it really works extremely well as a very smart autocompletion tool,” prolific open source maintainer and software development consultant Alexey Golub told InfoWorld. “It can infer the context from your code and finish whatever line you were typing accurately most of the time. It also helps in other scenarios outside of coding too, like writing blog posts, talk abstracts, comments, or documentation. Sometimes, the right word might just barely escape you, but then Copilot suggests it for you.”
Colby Fayock, a developer advocate at test automation company Applitools, stayed up all night playing with the tool once he was given early access. “I was surprised at how well it was able to provide answers for a lot of the input I tried,” he told InfoWorld. “The first thing that came to mind was that this would be most useful for generating common utility functions like sorting by date that one might use in different projects. Typically, that might be a common Google search for a Stack Overflow answer, but now I can just write the function and have that answer given to me right in Visual Studio Code.”
How does it work?
At its core, GitHub Copilot uses a new language-generating algorithm by OpenAI called Codex. OpenAI CEO Greg Brockman describes it as the descendant of GPT-3 but narrowly focused on code generation.
Codex algorithm has been trained on terabytes of public code pulled from GitHub and a selection of English language. This gives the tool the ability to write context-based code with unprecedented accuracy.
Currently, it is a Visual Studio Code extension, and the spots for technical preview are limited.
Under the hood, the extension sends your code and comments to the GitHub Copilot services, which use the Codex algorithm to synthesize and create suggestions.
It speaks virtually any programming language but works best with Python, JavaScript, TypeScript, Ruby, and Go.
According to its selection of users, the tool can generate up to 10 alternatives for a single suggestion. The algorithm consistently improves by recording whether each suggestion is accepted or not.
How good is it?
Developers at GitHub conducted an experiment to measure the tool’s accuracy. It was tested on a set of Python functions that have good test convergence in open-source repositories.
All the function bodies have been deleted, and the only context provided was the function names and docstrings. Copilot was able to fill them in correctly 43% of the time, and the accuracy increased to 57% after 10 attempts.
This is truly a remarkable feat since the tool generates working code you can use in your projects. Given that millions will use it once it goes public, it will significantly speed up development.
To get the most out of it, it is suggested to divide the code into small functions, provide meaningful function names, parameters, and docstrings.
In other words, it truly is your pair programmer — it makes you follow software engineering best practices, and in return, it learns from your code to improve its suggestions.
What is the quality of the generated code?
Even though Copilot is correct almost half the time, the creators say its output should be monitored:
GitHub Copilot tries to understand your intent and generate the best code it can, but the code it suggests may not always work, or even make sense.
The quality of the suggestions also depends on the existing code. Copilot can only use the current file as a context. It cannot test its own code meaning it may not even run or compile.
Besides, the FAQs say you use the tool at your own risk since it can suggest old or deprecated versions of libraries and frameworks.
There are also concerns around Copilot’s training set that contain the code written by millions.
The obvious question is, “Does it ever repeat the code from the training data” to which the answer is yes. It has been observed that there is a 0.1% chance the tool leaks code from the training set.
Framing this probability as a percentage might not show the bigger picture. One in a 1000 chance is actually more serious considering that there will be up to 10 options for each suggestion.
However, an in-depth case study on this problem showed these instances only happen when there is insufficient context to learn from. In particular, the algorithm was more likely to make these mistakes when the current file was short or empty.
Should You Be Worried as a Data Scientist?
Considering its merits and flaws, it is worth asking if GitHub Copilot affects developer jobs in the future.
When GPT-3 was released, the answer to this question was a tentative, faint yes. However, now that Copilot is out and will be a commercially available product that integrates into one of the heavily used IDEs globally, we should reconsider our answer.
The creators claim the tool will only serve to boost productivity and free developers from doing manual tasks and help them focus on more interesting work. It might also be possible it lowers the barriers for beginners to enter the software industry.
Personally, I strongly believe that AI, no matter how advanced, won’t replace humans any time soon. I completely disagree that one day, AI will excel across all programming disciplines and will be able to solve any unique, intricate problem all by itself.
This is especially true for the data science sphere. Regardless of how good GitHub Copilot is, I don’t think it will even beat the simplest Auto-ML libraries.
Data science isn’t just about writing good code — it is a collection of domain knowledge, understanding the data, and beautiful math.
However, I won’t deny that an AI-based tool like this, a technology backed by a multi-billion company, will play a vital role in the daily workflow of millions of programmers in the future.