The future of programming as we know it

Tyumen State University

Institute of Mathematics, Science and Information Technologies

Foreign Languages and Intercultural Professional Communication Department

 

 

 

 

 

 

 

THE FUTURE OF PROGRAMMING AS WE KNOW IT

 

 

Submitted by: A.S. Fljagin,

Gr.114-2

Supervisor: L.V. Skorokhodova,

Senior Lecturer

 

 

 

 

Tyumen 2013

Оглавление

THE FUTURE OF PROGRAMMING AS WE KNOW IT 0

Abstract 2

Main part 3

Moore’s bounty: software abstractions 3

Software abstractions and hardware mechanisms needed 7

Current State of Programming 10

Основная часть 12

Премия Мура: абстракции программного обеспечения 12

Необходимые абстракции и аппаратные механизмы программного обеспечения 17

Текущее состояние программирования 20

Bibliographic References 23

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Abstract

 

Как мы представляем программирование в будущем

Аннотация: Будущий рост мощности компьютерной техники вытекает из программного параллелизма. Закон Гордона Мура помог описать условия достижения успеха при использовании этой технологии. Исследуя проблемы программирования параллельных систем, были выдвинуты современные тенденции развития компьютерных технологий. Данный проект сначала объясняет, как текущее программное обеспечение пожинало преимущества Закона Гордона Мура, и на сколько, в результате, программное обеспечение не подходит для параллелизма. Также здесь исследуется текущее состояние программирования, которое помогло выявить основную проблему - оценка новых языков и подходов, а не упрощение выражений и практическое использование параллелизма. Таким образом, чтобы разрабатывать параллельные приложения, будущие разработчики должны изобрести новые параллельные алгоритмы и создать новые параллельные приложения.

Ключевые слова: Закон Гордона Мура, параллельные системы, фортран, параллельные приложения, параллельные алгоритмы.

 

The future of programming as we know it

Abstract: Future growth in computing performance follows from software parallelism. Gordon Moore's law helped to describe conditions of success when using this technology. Investigating problems of programming of parallel systems, current trends of development of computer technologies were put forward. This project at first explains how the current software reaped advantages of the Law of Gordon Moore, and why the software isn't suitable for parallelism. Also the current state of programming which helped to reveal the main problem - an assessment of modern languages and approaches, instead of simplification of expressions and practical use of parallelism is investigated. So that to develop parallel applications, future developers have to invent new parallel algorithms and create new parallel applications.

Key words: Gordon Moore's law, parallel systems, Fortran, parallel applications, parallel algorithms.

 

 

Main part

 

Future growth in computing performance will have to come from software parallelism that can exploit hardware parallelism. Programs will need to be expressed by dividing work into multiple computations that execute on separate processors and that communicate infrequently or, better yet, not at all. This chapter first explains how current software reaped the benefits of Moore’s law and how much of the resulting software is not well suited to parallelism. It then explores the challenges of programming parallel systems. The committee explores examples of software and programming parallelism successes and possibilities for leveraging these successes, as well as examples of limitations of parallelism and challenges to programming parallel systems. The sudden shift from single-core to multiple-core processor chips requires a dramatic change in programming, but software developers are also challenged by the continuously widening gap between memory system and processor performance. That gap is often referred to as the “memory wall,” but it reflects a continuous rather than discrete shift in the balance between the costs of computational and memory operations and adds to the difficulty of obtaining high performance. To optimize for locality, software must be written to minimize both communication between processors and data transfers between processors and memory.

Moore’s bounty: software abstractions

 

Moore’s bounty is a portable sequential-programming model. Programmers did not have to rewrite their software—software just ran faster on the next generation of hardware. Programmers therefore designed and built new software applications that executed correctly on current hardware but were often too compute-intensive to be useful (that is, it took too long to get a useful answer in many cases), anticipating the next generation of faster hardware. The software pressure built demand for next-generation hardware. The sequential-programming model evolved

in that ecosystem as well. To build innovative, more capable sophisticated software, software developers turned increasingly to higher-level sequential- programming languages and higher levels of abstraction (that is, the reuse of libraries and component software for common tasks). Moore’s law helped to drive the progression in sequential-language abstractions because increasing processor speed hid their costs. For example, early sequential computers were programmed in assembly language. Assembly-language statements have a one-to-one mapping

to the instructions that the computer executes. In 1957, Backus and his colleagues at IBM recognized that assembly-language programming was arduous, and they introduced the first implementation of a high-level sequential scientific computing language, called FORTRAN, for the IBM 704 computer.2 Instead of writing assembly language, programmers wrote in FORTRAN, and then a compiler translated Fortran into the computer’s assembly language. The IBM team made the following claims for that approach:

1) Programs will contain fewer errors because writing, reading, and understanding FORTRAN is easier than performing the same tasks in assembly language.

2) It will take less time to write a correct program in FORTRAN because each FORTRAN statement is an abstraction of many assembly instructions.

3) The performance of the program will be comparable with that of assembly language given good compiler technology.

The claimed benefits of high-level languages are now widely accepted. In fact, as computers got faster, modern programming languages added more and more abstractions. For example, modern languages—such as Java, C#, Ruby, Python, F#, PHP, and JavaScript—provide such features as automatic memory management, object orientation, static typing, dynamic typing, and referential transparency, all of which ease the programming task. They do that often at a performance cost, but companies chose these languages to improve the correctness and functionality of their software, which they valued more than performance mainly because the progress of Moore’s law hid the costs of abstraction. Although higher levels of abstraction often result in performance penalties, the initial transition away from hand-coded assembly language came with performance gains, in that compilers are better at managing the complexity of low-level code generation, such as register allocation and instruction scheduling, better than most programmers.

That pattern of increases in processor performance coupling with increases in the use of programming-language abstractions has played out repeatedly. High-level programming languages have made it easier to create capable, large sophisticated sequential programs. During the height of the synergy between software and increasing single-processor speeds in

1997, Nathan Myhrvold, former chief technology officer for Microsoft, postulated four “laws of software”:

1) Software is a gas. Software always expands to fit whatever container it is stored in.

2) Software grows until it becomes limited by Moore’s law. The growth of software is initially rapid, like gas expanding, but is inevitably limited by the rate of increase in hardware speed.

3) Software growth makes Moore’s law possible. People buy new hardware because the software requires it.

4) Software is limited only by human ambition and expectation.

 

We will always find new algorithms, new applications, and new users. Myhrvold’s analysis explains both the expansion of existing applications and the explosive growth in innovative applications. Some of the code expansion can be attributed to a lack of attention to performance and memory use: it is often easier to leave old and inefficient code in a system than to optimize it and clean it up. But the growth in performance also enabled the addition of new features into existing software systems and new paradigms for computing. For example, Vincent Maraia reports that in 1993, the Windows NT 3.1 operating system (OS) consisted of 4-5 million lines of code and by 2003, the Windows Server OS had 50 million lines, 10 times as many. Similarly, from 2000 to 2007, the Debian 2.2 Linux OS grew from about 59 to 283 million lines in version 4.0, about 5 times as many. Those operating systems added capabilities, such as better reliability, without slowing down the existing features, and users experienced faster operating system startup time and improvement in overall performance. Furthermore, the improvements described by Moore’s law enabled new applications in such domains as science, entertainment, business, and communication. Thus, the key driver in the virtuous cycle of exploiting Moore’s law is that applications benefited from processor performance improvements without those applications having to be adapted to changes in hardware. Programs ran faster on successive generations of hardware, allowing new features to be added without slowing the application performance.

 

Thus, current programs will not run faster on successive generations of hardware. The shift in the hardware industry has broken the performance- portability connection in the virtuous cycle—sequential programs will not benefit from increases in processor performance that stem from the use of multiple processors. There were and are many problems—for example, in search, Web applications, graphics, and scientific computing—that require much more processing capability than a single processor provides. The developers of the applications and programming systems have made much progress in providing appropriate abstractions but not enough in that most developers and programming systems currently use the sequential model. Conventional sequential programs and programming systems are ill equipped to support parallel programming because they lack abstractions to deal with the problems of extracting parallelism, synchronizing computations, managing locality, and balancing load. In the future, however, all software must be able to exploit multiple processors to enter into a new virtuous cycle with successive generations of parallel hardware that expands software capabilities and generates new applications. To develop parallel applications, future developers must invent new parallel algorithms and build new parallel applications. The applications will require new parallel-programming languages, abstractions, compilers, debuggers, execution environments, operating systems, and hardware virtualization systems. We refer to those tools collectively as a programming system. Future programming systems will need to take advantage of all those features to build applications whose performance will be able to improve on successive generations of parallel hardware that increase their capabilities by increasing the number of processors. In contrast, what we have today are conventional sequential-programming systems based on two abstractions that are fundamentally at odds with parallelism and locality. First, they tie the ordering of statements in a program to a serial execution order of the statements. Any form of parallelism violates that model unless it is unobservable. Second, conventional programs are written on the assumption of a flat, uniform-cost global memory system. Coordinating locality (minimizing the number of expensive main memory references) is at odds with the flat model of memory that does not distinguish between fast and slow memory (for example, on and off chip). Parallelism and locality are also often in conflict in that locality will encourage designs that put all data close to a single processor to avoid expensive remote references, whereas performing computations in parallel requires spreading data among processors.

Software abstractions and hardware mechanisms needed

 

Simplifying the task of parallel programming requires software abstractions that provide powerful mechanisms for synchronization, load balance, communication, and locality, while hiding the underlying details. Most current mechanisms for these operations are low-level and architecture-specific. The mechanisms must be carefully programmed to obtain good performance with a given parallel architecture, and the resulting programs are typically not performance-portable; that is, they do not exhibit better performance with a similar parallel architecture that has more processors. Successful software abstractions are needed to enable programmers to express the parallelism that is inherent in a program and the dependences between operations and to structure a program to enhance locality without being bogged down in low-level architectural details. Which abstractions make parallel programming convenient and result in performance-portable programs is an open research question. Successful abstractions will probably involve global address spaces, accessible ways to describe or invoke parallel operations over collections of data, and constructs for atomic operations. Abstractions may also involve abstract machine models that capture resource costs and locality while hiding details of particular machines. Abstractions for parallelism are typically encapsulated in a programming system and execution model. At the same time, reasonable performance requires efficient underlying hardware mechanisms, particularly in cases that need fine-grained communication and synchronization. Some parallel machines require interactions between processors to occur by means of high-overhead message transfers or by passing data via shared memory locations. Such mechanisms are useful but can be cumbersome and restrict the granularity of parallelism that can be efficiently exploited. Resolving those details will require research, but successful mechanisms will enable low overhead communication and synchronization and will facilitate migration of data and operations to balance load. There are several emerging directions in hardware to support parallel computations. It is too early to know which hardware architecture or architectures will prove most successful, but several trends are evident:

1) Multiple processors sharing a memory. This direction was taken by chip multiprocessors and was the primary approach used by semiconductor companies once they could not continue to increase their single-processor products.

2) Multiple computers interconnected via a high-speed communication network. When very large computation facilities are needed for research or business, it is impractical for all the processors to share a memory, and a high-speed interconnect is used to tie the hundreds or thousands of processors together in a single system. Data centers use this model.

3) A single processor containing multiple execution units. In this architecture, a single processor, or instruction stream, controls an array of similar execution units. This is sometimes termed single instruction stream multiple-data (SIMD) architecture.

4) Array of specialized processors. This approach is effective for executing a specialized task, such as a graphic or video processing algorithm. Each individual processor and its interconnections can be tailored and simplied for the target application.

5) Field-programmable gate arrays (FPGAs) used in some parallel computing systems. FPGAs with execution units embedded in their fabric can yield high performance because they exploit locality and program their on-chip interconnects to match the data flow of the application.

That list of parallel architectures is not exhaustive, and some systems will use a combination of them. We expect current versions of the architectures to evolve substantially to support the most promising programming systems, and we may see entirely new hardware architectures in support of not-yet-developed programming approaches to parallel computing. An encouraging development is that programs of research in parallelism being initiated or revived in a few research universities. Some research efforts already under way are aimed at some of the challenges that this report outlines. For example, in 2008, the University of California, Berkeley, and the University of Illinois at Urbana–Champaign were awarded research grants from Microsoft and Intel to establish Universal Parallel Computing Research Centers. In 2009, Stanford University with industrial funding from Sun, AMD, NVIDIA, and other companies started the Pervasive Parallelism Laboratory. Those centers at leading research universities are a good beginning to address the broad and challenging research agenda. History shows that the development of technology similar to that needed for parallelism often takes a decade or more. The results of such research are needed now, so the research is starting a decade late. Moreover, there is no guarantee that there is an answer to the challenges. If there is not a good answer, we need to know that as soon as possible so that we can push innovation in some other direction in a timely way.

Current State of Programming

 

One view of the current pool of practicing programmers is that there is a large disparity between the very best programmers and the rest in both time to solution and elegance of solution. The conventional wisdom in the field is that the difference in skills and productivity between the average programmer and the best programmers is a factor of 10 or more. Opinions may vary on the specifics, but the pool of programmers breaks down roughly as follows:

A. A few highly trained, highly skilled, and highly productive computer science (CS) system designers.

B. A few highly trained, highly skilled, and highly productive CS application developers.

C. Many moderately well-trained (average), moderately productive CS system developers.

D. Many moderately productive developers without CS training.

The developers who are not CS-trained are domain scientists, business people, and others who use computers as a tool to solve their problems. There are many such people. It is possible that fewer of those people will be able to program well in the future, because of the difficulty of parallel programming. However, if the CS community develops good abstractions and programming languages that make it easy to program in parallel, even more of those types of developers will be productive. There is some chance that we will find solutions in which most programmers still program sequentially. Some existing successful systems, such as databases and Web services, exploit parallelism but do not require parallel programs to be written by most users and developers. For example, a developer writes a single-threaded database query that operates in parallel with other queries managed by the database system. Another more modern and popularly known example is MapReduce, which abstracts many programming problems for search and display into a sequence of Map and Reduce operations. Those examples are compelling and useful, but we cannot assume that such domain-specific solutions will generalize to all important and pervasive problems. In reality, the wizard programmers can have an even far greater effect on the organization. Indeed, as in arts in general, there is a sense in which no number of ordinary people can be combined to accomplish what one gifted person can contribute. Without adequate training, we will not produce enough of the category (A) and (B). Without them, who will build the programming abstraction systems? Parallel computing and thus parallel programming showed great promise in the 1980s with comparably great expectations about what could be accomplished. However, apart from horizontally scalable programming paradigms, such as MapReduce, limited progress resulted in frustration and comparatively little progress in recent years. Accordingly, the focus recently has been more on publishable research results on the theory of parallelism and new languages and approaches and less on simplification of expression and practical use of parallelism and concurrency. There has been much investment and comparatively limited success in the approach of automatically extracting parallelism from sequential code. Thus, much less attention was given to effective expression of parallelism.

 

 

Основная  часть

 

Будущий рост в вычислительной производительности прибудет из параллелизма программного обеспечения, который может использовать аппаратный параллелизм. Программы должны выражаться, деля работу на множество вычислений, которые выполняются на отдельных процессорах и которые редко связываются или, еще лучше, вообще никогда. Эта глава сначала объясняет, как текущее программное обеспечение пожинало преимущества Закона Гордона Мура, и на сколько, в результате, программное обеспечение не подходит для параллелизма. Затем в главе исследуются проблемы программирования параллельных систем. Комитет исследует примеры программного обеспечения, успехи параллелизма в программировании, и возможности усилить эти успехи, а также примеры ограничений параллелизма и вызовов программирования параллельных систем. Внезапный сдвиг от одноядерных до многоядерных микросхем процессора требует значительной перемены в программировании, так как разработчикам программного обеспечения бросает вызов постоянно расширяющийся разрыв между системной памятью и производительностью процессора. Этот разрыв часто упоминается как "стена памяти", так как он отражает непрерывный, а не дискретный сдвиг в соотношении между затратами вычислительных операций и операции с памятью, и в итоге увеличивает сложность получения высокой производительности. Чтобы оптимизироваться для локализации, программное обеспечение должно записываться так, чтобы минимизировать передачу данных между процессорами, и между процессором и памятью.

Премия Мура: абстракции программного обеспечения

 

Премия Мура – портативная модель последовательного программирования. Программисты не должны были переписывать свое программное обеспечение — программное обеспечение просто работало быстрее на следующем поколении аппаратных средств. Поэтому программисты разрабатывали и создавали новые приложения, которые выполнялись правильно на текущих аппаратных средствах, но зачастую делали слишком много интенсивных вычислений, чтобы быть полезными (то есть, в большинстве случаев  прошло бы слишком много времени, чтобы получить нужный ответ), ожидая следующее поколение более быстрых аппаратных средств. Давление программного обеспечения создавало спрос на аппаратные средства следующего поколения. Модель последовательного программирования также развилась в этой экосистеме. Чтобы создать инновационное, более адаптированное, сложное программное обеспечение, разработчики все более и более уходили к высокоуровневому последовательному программированию - языки и более высокие уровни абстракции (т.е. повторное использование библиотек и компонентного программного обеспечения для общих задач). Закон Гордона Мура помог управлять прогрессией в абстракциях последовательного языка, поэтому увеличение скорости процессора скрыло их затраты. Например, рано последовательные компьютеры были запрограммированы в ассемблере. У операторов ассемблера есть непосредственный доступ к инструкциям, которые выполняет компьютер. В 1957 Бекус и его коллеги в IBM поняли, что программировать ассемблер слишком трудно, и они представили первую реализацию высокоуровневого последовательного языка научных вычислений, названного ФОРТРАНОМ, для IBM 704. Вместо того чтобы писать на ассемблере, программисты писали в ФОРТРАНЕ, а затем компилятор переводил ФОРТРАН в ассемблер. Команда IBM предъявила следующие претензии к этому подходу: