2010 January 19
Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.— Greenspun's Tenth Rule of Programming
Lisp was invented in the 1950s by John McCarthy. It is one the oldest languages in widespread use today. Lisp pioneered many radical ideas in programming language design like conditionals, higher-order functions, full recursion, garbage collection and interactive software development. After 52 years of existence, Lisp is still at the forefront of programming language research, with newer languages doing mediocre re-implementation of ideas borrowed from it.
Lisp is a living language. It is also a philosophy and design methodology that enables the creation of dynamic software systems - computer programs that evolve and adapt to changing requirements. Smalltalk, Python, Ruby and many other languages have copied features from Lisp. Even the portly Java and C++ has a hidden, but uncontrollable yearning to be more and more like Lisp. For more than half-a-century, Lisp has been the lofty goal that programming language designers are tirelessly aiming to reach. Then why this "ultimate language" is not popular in the software industry? Why 'enterprise' software companies settle down for something inferior?
The often heard reasons for despising Lisp are:
All of the above concerns, except the fourth one, are wrong. About the fourth point - yes, Lisp programmers are rare. Lisp is simple but it is not a language for the average programmer. Still, if you are able to hire a Lisp programmer, it is very likely that he is good enough for ten (or more) of your average Java programmers.
Now I will try to prove how the first three arguments are wrong:
John McCarthy was an AI researcher and he used Lisp extensively in his work. It doesn't mean that Lisp is only for AI programming. Lisp is a general purpose programming language. Lot of successful software has been written in Lisp, that has got little to do with AI. Emacs is a popular text editor that is almost completely written in Lisp. It also uses Lisp as an extension language. AutoCAD and GIMP are two other famous software applications that have Lisp as an important component. ViaWeb was one of the first web applications and it was written in Lisp. ViaWeb later became Yahoo Stores. Lisp is being actively used in a wide variety of industries - music production, bio-informatics and education. There are commercial software companies like ITA software and Clozure Associates that use Lisp for developing some of the most demanding software in existence.
Programming languages like Python and Perl provide built-in support for fundamental data structures. Some of them even provide syntactic-sugar for initializing and manipulating these structures. For instance, the following python code creates a list of five integers:
[1, 2, 3, 4, 5]The same list can be expressed in Lisp as:
(1 2 3 4 5)Lisp uses parenthesis instead of square-brackets and does away with the superfluous comma separators. But Lisp gives you more than just compact syntax. Lists in Lisp can be executed. Let us try to understand what that means.
In addition to the usual data types, Lisp provide 'symbols'. Any data can be encoded as symbolic information by prefixing it with a 'quote'. The following are all valid symbols:
'abc
'123
'&%^anythig@@#Lists can be quoted too:
'(1 2 3 4 5)
'(+ 1 2 3 4 5)If they are not quoted, lists are treated specially by assuming that the first element is a call to a function and the rest of the elements are its arguments. If we enter the second list unquoted at a Lisp prompt, the addition function (+) is invoked with the five numbers as arguments.
lisp> (+ 1 2 3 4 5)
15Symbols and lists are called symbolic-expressions or s-expressions. S-expressions are evaluated (or executed). In the above case the list was evaluated to the value 15. Lisp uses one of its built-in data structure (i.e, list) to represent code. While other languages segregate between data and the code that work on that data, Lisp has the ability to treat data as code and code as data. This adds immense power to the language. You can write Lisp programs that write programs. Lisp is a programmable programming language. Let us see how this helps in solving a practical problem.
Consider a spreadsheet program written in Java. It has chosen XML as the format to store data on disk. Here is a sample spreadsheet file - "salary.xml":
<?xml version="1.0" encoding="UTF-8" ?>
<book name="Salary">
<sheet number="1" title="SalarySlip01">
<cell id="A1" type="string">Basic Pay</cell>
<cell id="B1" type="float">2450</cell>
<cell id="A2" type="string">HRA</cell>
<cell id="B2" type="float">450</cell>
<cell id="A3" type="string">Prof. Tax</cell>
<cell id="B3" type="float">100</cell>
<cell id="A4" type="string">Gross Pay</cell>
<cell id="B4" type="formula">B1+B2-B3</cell>
</sheet>
</book>
Special Java code is needed to load and parse this XML encoded information. To make use of an embedded formula, like B1+B2-B3, it has to be first parsed into a Java object that represents a tiny language interpreter. The rather heavy task of writing this "formula interpreter" rests on the shoulder of the application programmer. Lisp relieves the programmer from this chore. As there is no difference between data and the code that operates on it, Lisp allows you to represent the spreadsheet as Lisp code that can be straight-away loaded and executed.
Note: In this example, we have used Scheme, one of the two dialects of Lisp that is in wide use today, the other being Common Lisp. Specifically, we have used Spark-Scheme, my own Scheme system. More on the various Lisp implementations later.
There are various ways to load the spreadsheet data to a Lisp system. One straight-forward approach to directly use a list of many lists:
'(book '(name "Salary")
'(sheet '(number 1) '(title "SalarySlip01")
'(cell '(id "A1") '(type "string") "Basic Pay")
'(cell '(id "B1") '(type "float") 2450)
;; and so on ...
))But this representation does not let us exploit the full power of Lisp. We can achieve more. We can represent our spreadsheet as a live object - something that can store information, respond to messages, and perform computations on its own. Lisp helps us to create data representations that are themselves small computing engines, something no other language can do efficiently and with so much ease!
Here is our new representation. Yes, it is a bit longer than the previous one but as I said earlier, it is not just data encoded as text, it is a living, intelligent organism. Once loaded and brought to life by the Lisp system, it can do wonders for you!
(lambda (msg . args)
(let ((sheets
(list (lambda (msg . args)
(let* ((A1 "Basic Pay") (B1 2450)
(A2 "HRA") (B2 450)
(A3 "Prof. Tax") (B3 100)
(A4 "Gross Pay") (B4 (list - (+ B1 B2) B3))
(cells (list
(list (cons 'string A1) (cons 'float B1))
(list (cons 'string A2) (cons 'float B2))
(list (cons 'string A3) (cons 'float B3))
(list (cons 'string A4) (cons 'formula B4)))))
(case msg
((title) "SalarySlip01")
((rows) 4)
((cols) 2)
((cell)
(let ((r (car args)) (c (car (cdr args))))
(list-ref (list-ref cells r) c)))
((eval)
(eval (car args)))
(else #f)))))))
(case msg
((name) "salary")
((sheet-count) 1)
((sheet) (list-ref sheets (car args)))
(else #f))))This is equivalent to a class specification in an Object Oriented Programming language. The way it is loaded and used is extremely dynamic. Only a few lines of code is needed to load and display the spreadsheet:
> (define book (load "salary.lisp"))
> (printf "SpreadSheet Name: ~s~n" (book 'name))
SpreadSheet Name: "salary"
> (define sheet-count (book 'sheet-count))
> (for i in (range sheet-count)
(let* ((sheet (book 'sheet i))
(rows (sheet 'rows))
(cols (sheet 'cols)))
(printf "Worksheet ~a: ~s~n" (add1 i) (sheet 'title))
(for r in (range rows)
(for c in (range cols)
(let* ((cell (sheet 'cell r c))
(type (car cell))
(value (cdr cell)))
(if (eq? type 'formula)
(printf "~s " (sheet 'eval value))
(printf "~s " value))))
(newline)))
(newline))
;; Output
Worksheet 1: "SalarySlip01"
"Basic Pay" 2450
"HRA" 450
"Prof. Tax" 100
"Gross Pay" 2800No parsing, no data transformation, less number of bugs and of course, a shortened development time. This is just one small example of how the "code as data and data as code" philosophy of Lisp can help in writing maintainable, bug-free code.
Lisp offers a unique facility called macros. Lisp macros are very different from macros in other languages, where it means simple text substitution. Lisp macros are different in the sense that they are actually programs that generate programs. This enable the programmer to add new syntax to the language, while the system is still running. Implementing DSLs in Lisp is a snap. Lisp can be molded into a language suitable for solving the problem at hand.
The following Common Lisp macro shows how to implement the 'while' loop, in Lisp itself. Other languages provide such constructs as part of the core language implementation, which the programmer is not allowed to change.
;; This sample is from the book "On Lisp".
(defmacro while (test &body body)
'(do ()
((not ,test))
,@body))Lisp's fully parenthesized syntax and macros make it an extremely adaptable language. When a new paradigm becomes the fashion, Lisp just adapts to it. For example, The Common Lisp Object Oriented Programming system CLOS is all written in Lisp itself!
These might have been true, decades ago, when Lisp was still in its infancy. Note that this was also true for languages like Java. Today there are two popular dialects of Lisp that are both standardized and has actively maintained, high quality implementations with rich libraries. Many of them can compile Lisp into highly optimized native code.
Few of the Common Lisp and Scheme implementations suitable for large scale software development are:
CLISP, Clozure CL, Steel Bank Common Lisp, PLT Scheme, Chicken Scheme, Spark-Scheme
Scheme implementations that can be easily embedded into C/C++ applications:
Guile Scheme, MzScheme, Scheme48
Lisps for the Java Virtual Machine:
Some notable Common Lisp projects that will of interest to developers:
The Lisp community is also blessed with a number of great books and tutorials. In fact, some of the best programming books ever written use Lisp as there implementation language. Many of these books are available for free on the web: