With Googles scale, Ng thought, neural networks could become not just useful but powerful. Feats of engineering tend to erase themselves. Craig Silverstein, a twenty-seven-year-old with a small frame and a high voice, sat by the far wall. The company had reached an inflection point. They did it with a piece of software called MapReduce. Sanjays brother, Pankaj, became the youngest faculty member ever awarded tenure at Harvard Business School. By signing up, you agree to our User Agreement and Privacy Policy & Cookie Statement. At Google, Sanjay is what is known as an individual contributora coder who works alone and manages no one. In the parlance of Silicon Valley, Google wasnt scalable.. MapReduce handles the details of distributionand, by doing so, hides them. The worlds most robust computer systems, at NASA, financial firms, and the like, used special hardware that could tolerate single bit-flips. Perhaps because hes so private, an air of mystery surrounds him at Google. In the world of software, the best way to lead is with code. The company was in the midst of an unprecedented emergency. Coding together at the same computer, Jeff Dean and Sanjay Ghemawat changed the course of the companyand the Internet. He rarely references an old notebook, but writes in order to think. Strange environmental factors came into play. project: an attempt, Jeff says, to train a giant machine-learning model to do thousands, or millions, of different tasks. His father, Mahipal, was a botany professor; his mother, Shanta, took care of Sanjay and his two older siblings. There were four windows open: on the left, a Web browser and a terminal, for running analysis tools; on the right, two documents in the text editor Emacs, one a combination to-do list and notebook, the other filled with colorful code. Sanjay scrolled, bringing a new section of code into view. Still, they could see the Santa Cruz Mountains to the south and Mission Peak to the east. Before MapReduce, each programmer had to figure out how to divide and distribute data, assign work, and account for hardware failures on her own. In Powers of Two: Finding the Essence of Innovation in Creative Pairs, the writer Joshua Wolf Shenk quotes from a 1971 interview in which John Lennon explained that either he or Paul McCartney would write the good bit, the part that was easy, like I read the news today or whatever it was. One of them would get stuck until the other arrivedthen, Lennon said, I would sing half, and he would be inspired to write the next bit and vice versa. Everyone falls into creative ruts, but two people rarely do so at the same time. Youre always scrolling back and forth to figure out whats going on. Others write code thats too dense: You look at it, youre, like, Ugh. Stern.) Once a week, he meets with a group of Area Tech LeadsGoogles engineering Jedi councilto make technical decisions that affect the entire company. On the kitchenettes counter, a three-foot-wide La Marzocco hummed. After cappuccinos, they walked to their computers. Still, he never dated, and does so only very, very infrequently now. The New Yorker may earn a portion of sales from products that are purchased through our site as part of our Affiliate Partnerships with retailers. Soon they reached a minor milestone, and Sanjay typed a command to test their progress. (Chuck Norris counted to infinity. Its an A.I. I kind of lived in the shadow of my brother, Sanjay said. Until the March index debacle, Googles systems had been rooted in code that its founders had written in grad school, at Stanford. When Jeff and Sanjay put all the missorted words together, they saw a patternthe same sort of glitch in every word. You work on infrastructure, he thought. In graduate school, at M.I.T., Sanjay found a tight-knit group of friends. After years of sharing their working lives, duos sometimes develop a private language, the way twins do. . Getting to Level 4 takes several years, or a Ph.D. Some of their best-known papers have as many as a dozen co-authors. One Monday last summer, after a morning of programming together, Jeff and Sanjay went to lunch at a campus cafeteria called Big Table, which was named for a system theyd helped develop, in 2005, for treating numberless computers as though they were a single database. Sanjay looked at Jeff. Or a slightly different pass in our indexing data, Jeff added. Wherever Jeff works at Google, espresso machines follow. Randy Garrett, a former director of technology at the National Security Agency, remembers demonstrating the technology to the agencys director, General Keith Alexander. Hed followed a colleague of hisa rangy, energetic thirty-one-year-old named Jeff Deanfrom Digital Equipment Corporation. O.K., Jeff said happily, his mood having turned on a dime. When Google was still called BackRub, in 1996, its map was small enough to fit on computers installed in Pages dorm room. In October, its core systems, which crawled the Web to build an index of it, had stopped working. Half a meg?. There was a gelato store in the middle.. So we could make notes about it, right?. Jeff, wearing a faded T-shirt and jeans, looked like a reformed beach bum; Sanjay wore a sweater and gray pants. On his desk, he keeps a stack of Mead composition notebooks going back nearly twenty years, filled with tidy lists and diagrams. One screen of code has very little information on it. Sanjay couldnt understand Jeffs move, either. To revisit this article, select My Account, thenView saved stories, To revisit this article, visit My Profile, then View saved stories. Many machines never worked in the first place; some would unaccountably grow slower. Jeff and Sanjay threaded the needle with software. research lab, Jeff said. 2022 Cond Nast. I thought somebody said this was a pretty flat hike, Sanjay said. A few younger programmers began to trickle in. I dont quite know what threshold we should pick for a unit size, Sanjay said. The hills here are steeper than I thought, Jeff said. Programming this way requires empathy with readers. Failures, which occurred seemingly at random, kept breaking the system. The only way that Google could keep up was by buying consumer machines and wiring them together into a fleet. In 2016, when he was inducted into the American Academy of Arts and Sciences, he didnt tell his parents; their neighbor had to give them the news. Sanjay, close behind, had his own Tesla, a red ModelS. Sanjay had spent the morning reading. Jeff rolled a chair from his own desk, which was messy, to Sanjays, which was spotless. Two heads are better than one! He laughed. I would walk from my D.E.C. Apportioning credit between them becomes impossible. Side by side, Jeff and Sanjay took charge of this effort. Then it became a privilege. Ad targeting became a river of money that the company directed back into its computing infrastructure. While working on the house, he discovered that although he liked designing spaces, he didnt have patience for what he calls the Sanjay-oriented aspects of architecture: the details of beams, bolts, and loads that keep the grand design from falling apart. His spindly fingers moved gently across the keys.

By developing new encoding and compression schemes, they effectively doubled the systems capacity. Even soand perhaps ironicallymany coders see the work done by Jeff and Sanjay, together, as proof that the 10x programmer exists. Jeffs knowledge of neural networks hadnt advanced much since his undergrad years, and Heidi watched as their bathroom filled with textbooks. So were going to ignore a problem? Jeff said indignantly. They soldered upgrades onto the machine, learning every part of it. Sanjay didnt touch a computer until he went to Cornell, at the age of seventeen. In the woods, they reminisced about how quickly Google had grown. On a Monday morning this spring, Jeff and Sanjay stood in the kitchenette of Building 40, home to much of Googles artificial-intelligence division. The idea venerates the individual, when software projects are often vast and collective. Computer hardware rarely failed, until you had enough of itthen it failed all the time. Now the Feds are on the case. He didnt brag about any of that stuff, she said. When a car goes around a turn, more ground must be covered by the outside wheels; likewise, the outer edge of a spinning hard disk moves faster than the inner one.

At M.I.T., his graduate adviser was Barbara Liskov, an influential computer scientist who studied, among other things, the management of complex code bases. Everything was broken, and we didnt know why.. As an adult, he retains a talent for self-effacement. They couldnt find the bug. You need to find someone that youre gonna pair-program with whos compatible with your way of thinking, so that the two of you together are a complementary force, Jeff said. The idea of distributed computation was an old one; concepts like cloud computing and big data predated Googles rise. Use of this site constitutes acceptance of our User Agreement and Privacy Policy and Cookie Statement and Your California Privacy Rights. Of the two, Jeff is more eager to expound, and while they walked he shared his soft-serve strategy. Still, very simple techniques, when you have a lot of data, work incredibly well, Jeff said. Generalizing their solution would mean that they could avoid revisiting the problem again and again. Weve been doing it since before Google, Jeff said. Behind them, a whiteboard was filled with matrix algebra; a paper about unsupervised adversarial networks lay on a table. At thirteen, he skipped the last three months of eighth grade to help them at a refugee camp in western Somalia. research lab two blocks away to his D.E.C. Recently, Jeff and Sanjay were enjoying margaritas and enchiladas at Palo Alto Sol, a Mexican restaurant they frequent, when Jeff pulled out his phone. this is the common pattern, Sanjay said, finishing their thought. Their behavior isnt specified by coders in the usual way; instead, its learned using inputs and feedback. Jeff and Sanjay, by contrast, sometimes seem to be two halves of a single mind.

They began reminiscing about their early projects. Like, all that can be made into a routine, couldnt it? Jeff said. Now Google processed its data. (A lesser programmer watching them would have been impressed, more than anything else, by the fact that they never stopped or got stuck.) The MapReduce paper arrived as a deus ex machina. It was eight-thirty-two. Its standard engineering practice to have your code reviewed by another coder, but Jeff and Sanjay skip this step, entering, in their log, a perfunctory lgtm, for looks good to me. In a sense, they had been occupied by minutiae. As they helped spearhead several rewritings of Googles core software, the systems capacity scaled by orders of magnitude. This was a major new software project related to machine learning, and Jeff and Sanjay were worried that it was bloated; like book editors, they were looking for cuts. Ng had told Jeff that this was changing. We were writing things by hand, Sanjay said. Googles co-founders, Larry Page and Sergey Brin, were negotiating a deal to power a search engine for Yahoo, and theyd promised to deliver an index ten times bigger than the one they had at the timeone capable of keeping up with the World Wide Web, which had doubled in size the previous year. With his feet spread shoulder-width apart, he looked as if he were working on his posture.