I agree - I really think that when it comes down to it, Lisp isn't popular because it's harder to read. Even if it's more terse and far more elegant, the mental tax of trying to understand another person's code is far higher than C, where there's only "the dumb way" to do something.
If people can't understand the code, they can't contribute, so you don't end up with the network effect of F/OSS contributions that make a language popular (note that I say popular, not better)
As a person who started learning Lisp after years and years of using mostly mainstream languages, I have to at least partially disagree. I will say this, however: Lisp is, at first, difficult to read for many people. But I think it is mainly difficult to read for people who are used to reading programs written in languages with Algol-like syntax (C, C++, Java, Perl, etc). If you've been writing programs for 5, 10, or 15 or more years mostly in languages that all have that common ancestry, it is understandable that your brain will (at first) work against you when reading something like Lisp. However, after using it for a reasonable period of time, I found that it became much easier to read. And now I don't find it any more difficult to read than any of the mainstream languages that I have used in the past.
I think that, in fairness to Lisp, if you take two people who have never programmed in their life, and if each of the two has a reasonable and approximately equal amount of skill and potential for being a decent developer, and taking those two individuals, you teach one to develop in Lisp, and one in, say Java. Given that situation, I don't think the Lisp developer will find his code more difficult to read really at any stage of his learning (more difficult that is, than the Java developer finds his code to read at a similar stage of learning).
It seems to me, from every testimony I've read from developers who have come to Lisp after extensive careers using other more popular languages, that there is a "breaking in" period where it takes some time to get used to things. However, anyone I've read about who put in any reasonable effort to learn a Lisp (or a language related to Lisp, like Scheme) seems to repeat what others like him or her have said -- you get used to the syntax after a while, and then at the very least it no longer bothers you or gets in your way.
I program in C++/etc but also in SQL. SQL is terse logical language. Everyone may has SQL but I think given constraints of being a logical language SQL is far more readable than lisp - but I've only read bits of lisp online.
I'm not clear what point you're making. CSS isn't a logical language, and they don't really have enough in common to compare them. (And FWIW, I find SQL clearer than Nahuatl.)
Lisp isn't popular because it's harder to read. Even if it's more terse and far more elegant, the mental tax of trying to understand another person's code is far higher than C
I am deeply suspicious of explanations that boil down to, Lisp isn't popular because the average programmer isn't smart enough. Let me tell you, anyone smart enough to use C++ in anger is smart enough to use any language.
The real world is highly competitive, and organizations that think they can win by using niche languages, do. There's APL at Morgan Stanley, Ocaml at Jane Street Capital, K at Lehman's (OK, maybe that's not so good an example), F#, Haskell, blah blah. Lisp isn't even on the radar, anywhere. What does that tell you?
Mathematica is widely used in the financial sector. Don't let the square brackets, prefix notation and fugly symbolic shortcuts fool you: Mathematica is lisp.
Well, yes, you could argue that Python is Lisp too, but my point stands: if Lisp, as in, CL, really was a huge productivity boost, organizations that have demonstrated their willingness to adopt niche languages where well-suited to their problem domain would have adopted it, but that isn't actually happening. If anything Haskell and APL are nichier (if that's even a word).
The big Lisp success stories all start from the premise of a bunch of programmers who are already hugely experienced in Lisp happening to do this project in it (e.g. Viaweb, Orbitz). But when people with 20 years experience in C complete a project successfully, it isn't blogworthy news...
Python is a primarily imperative/OO language. Good python code consists of writing loops and building objects. Syntactic abstraction consists primarily of looking up strings in obj.__dict__. Whenever more complicated metaprogramming is needed, strings are parsed.
"obj__parent__name_eq"=foo
is django for
obj.parent.name == foo.
Mathematica is a primarily functional language. The primary abstraction method is patterns (macros), which allow you to modify/build up the parse tree before evaluating it.
I've programmed Python and Lisp, and have begun writing some projects of reasonable size in Mathematica. There are only a couple of major differences between Mathematica and lisp. One is syntax: f[a,b,c] vs (f a b c), as well as some infix syntactic sugar. The other is that arrays are used far more widely in Mathematica than in lisp (but that may be due to the problem domain of mathematica).
But you haven't answered the question. Why Mathematica and not Lisp + some libraries? Or (+ Lisp Libraries). Again, there are many organizations willing to adopt any language that will give them some extra leverage. Lisp just doesn't appear. The explanation that the average programmer just isn't smart enough is easily demolished by pointing out the C++ (and APL, and Haskell) users. So either we're all missing something or the elephant in the corner is that Lisp just isn't that useful for commercially relevant problems. Which is not to say it's a bad language, but that it's not the "secret sauce" everyone's looking for.
Actually, the question you are asking is "Why mathematica and not Common Lisp or Scheme + some libraries?"
Mathematica is lisp + a bunch of a math libraries + vendor support + slightly different syntax + renamed functions. The syntactic differences:
{a b c} -> '(a b c)
f[a,b] -> (f a b)
a == b -> (Equals a b)
(* comment *) -> ;;comment
Since there is no advantage to different syntax and renamed functions, people choose mathematica over other lisps (and grumble about the weird syntax).
i agree with your main point (python isn't lisp because it doesn't have macros). but the language nerd in me can't resist commenting on your example from django. that's a very special case. it's hard to think of any other package that uses that approach. rather, the way people solve problems in python that you would solve in lisp with macros is by late binding, introspection, and higher order functions. none of these replace macros entirely, but together they can be hacked into a solution.
You'll get a productivity boost with good people. Those are rare and expensive.
Lisp is a bit remote from machines and their operating system: GC, data formats, ...
Lisp has many degrees of freedom. It is much harder to control. It is designed for flexibility (even at runtime).
The original Lisp was done for 'symbolic computation with recursive functions': Theorem provers, machine translation, computer algebra systems, knowledge representation, planning systems, ...
If your problem domain fits into above list, then Lisp is likely to be a good candidate language to use. Then there are related domains where Lisp has been used with success: expert systems, algorithmic composition, language implementation (compilers, ...), deductive databases. Again, Lisp may be very useful in these domains.
There are lots of domains where it does not fit that well or is mostly useless. From the domains above you can see that those are not domains that many people work in (compared to web development, database apps, accounting apps, ...). Lisp is quite useful in those domains, but they tend to be a bit remote from the current mainstream.
Viaweb and Orbitz are exceptions. Most Lisp applications are relatively unknown and they are not widely published. Huge successes were for example DART and SPIKE. DART was a logistics system for the military (it is said to have repaid all the government expenses for AI research made) and SPIKE is s scheduling system for telescopes (originally for the Hubble Space Telescope). These applications were not written because some guys knew only Lisp, but because lots of the research applications in planning and scheduling systems were written in Lisp - because in this domain these applications work with symbolic representations of plans.
Maybe that generalizes, to say that it's the code that lisp doesn't turn into "Fungible Cogs", and so it can't be used by others in a standardized way.
Standardization seems to be anathema to the individuals who are attracted to lisp's power and flexibility. If lisp ever achieved code fungibility, lispers would hate it.
6. And the LORD said, Behold, the people is one, and they have all one language; and this they begin to do: and now nothing will be restrained from them, which they have imagined to do.
7. Go to, let us go down, and there confound their language, that they may not understand one another's speech.
I think a bigger factor is the amount of prior knowledge required before you can be productive.
In C (and c-like languages) that barrier is relatively low, a handful of concepts and a very limited subset of library modules and you can go a long long way.
Indeed. Lisp programmers with 20 years experience are more productive than Java programmers with 2 years experience. That doesn't say anything about Lisp vs Java (or, not nearly as much as some would claim it does).
If people can't understand the code, they can't contribute, so you don't end up with the network effect of F/OSS contributions that make a language popular (note that I say popular, not better)