I can't imagine someone who has ever written a Lisp macro coming away thinking "there's nothing special about s-expressions". Honestly, sounds like you're commenting on something you have absolutely no knowledge of.
I have used Scheme, Racket and Common Lisp (SBCL) since the 1990s (before mostly switching to the ML family of languages). I read 'On Lisp' cover to cover. I know what I am talking about. You can still disagree with me, of course. But don't do it of the basis of me not having used enough Lisp.
S-Expressions make it easier to write macros that blend seamlessly into the host language. But there's no magic to them (compared to macros you could do as pre-processing in any other language) beyond that.
S-Expressions are a neat syntax. But homoiconicity is a weird concept; if it is coherent at all, it only pertains to the surface syntax level of the language.
> there's no magic to them (compared to macros you could do as pre-processing in any other language)
The "magic" of s-expressions is that they make it easy to operate on the source code of a program as a hierarchical data structure (i.e. as an AST) rather than as text. That turns out to be an extremely powerful lever. It is is one of the reasons Lisp has lasted as long and been as influential as it has. I'm sorry if this sounds like an ad-hominem, but if you think there is "no magic" to S-expressions the most likely explanation is that you don't really understand them. There is nothing comparable in any other language. There's a reason that they are still in use today. Indeed, there is a reason that they keep getting re-invented again and again. HTML, XML, and JSON are all (badly) reinvented s-expressions.
> homoiconicity is a weird concept
No, it isn't. It is simple and straightforward: homoiconicity is where the primary representation of programs is a data structure in a primitive type of the language itself. It isn't any weirder than (say) recursion. If you think it's weird that is more evidence that you don't actually understand it.
Plenty of languages give you the option of writing macros, however the difference between writing macros in a lisp and those languages is like the difference between using a general purpose programming language and an accidentally Turing-complete one.
No. You can say that Common Lisp and Racket are deliberately designed around macros. In Haskell, Rust, C++, they have been bolted on to the language as an after thought. Unless I am mistaken, those languages do not have the ability to write DSLs interactively, unlike Common Lisp.
>The "magic" of s-expressions is that they make it easy to operate on the source code of a program as a hierarchical data structure
where exactly is the magic? it's just tuple unpacking? like congrats you've constrained yourself to the absolute bare minimum and you've made it work. i mean i guess congrats for getting it to work at all but it's not magical, it's tedious. if you showed me a homoiconic language that did somehow pull off the magic trick of being more expressive than tuples then i would be indeed enchanted (mathematica at least manages to put lipstick on the whole exercise).
> i mean i guess congrats for getting it to work at all but it's not magical, it's tedious.
Magic lies in the eyes of the beholder. So you may not find it magical while I do. But I don't understand how you say that it is tedious.
With the homoiconicity of Lisp, you use the Lisp language itself to write macros and you manipulate the lists that make your Lisp programs directly.
Mainstream languages these days either need you to learn a special syntax for macros or they need you to learn their AST structures/classes and manipulate them or sometimes both. Isn't this more tedious? Isn't the Lisp way simpler and less tedious?
There is a reason I put the word "magic" in scare quotes.
> it's just tuple unpacking?
It's not even that. S-expressions are just a serialization of linked lists. The "magic" happens because the details of the serialization happen to make them particularly good for writing code.
The distinction between a linked list and a binary tree is in the eye of the beholder. One of the brilliant things about s-expression syntax is that (a b c) is equivalent to (a . (b . (c . nil)))) so there is no distinction between binary trees and linked lists (whose elements can be other linked lists) other than that to be a "proper" list it has to be null-terminated.
> The "magic" of s-expressions is that they make it easy to operate on the source code of a program as a hierarchical data structure (i.e. as an AST) rather than as text.
I agree that ergonomics are a big deal when using programming languages, and it shapes their culture.
Just like it's a pain to re-use code in C, so everyone always re-implements linked lists from scratch every time they need one.
> It is simple and straightforward: homoiconicity is where the primary representation of programs is a data structure in a primitive type of the language itself.
Why does it have to be a 'primitive' type?
Eg Haskell supports representing ASTs just fine, but you would use user-defined types for that.
Imagine a variant of Haskell that used S-Expression syntax for the sake of argument. That version of Haskell would still represent S-Expressions internally with a user-defined type. See https://hackage.haskell.org/package/s-expression and mentally translate all the code into S-Expressions (but leave the semantics the same).
Or just imagine a variant of Lisp where cons-cells are a user-defined data-structure. That wouldn't make their S-Expressions any worse, would it?
As long as whatever data structure you use to represent your AST is easy to work with, that's surely good enough?
And what do you mean by 'primary' representation? A Lisp compiler (or interpreter) has many different layers, and the AST is but one representation used in one of the layers.
For many purposes, S-Expressions aren't particularly useful, because eg they don't keep track of which variables are bound where. And, of course, an interpreter that works by walking S-Expressions is really, really slow.
So in what sense are S-Expression a primary representation?
You could imagine a re-skin of C that used S-Expression syntax. It's actually relatively easy to write such a front-end as a pre-processor that translates S-Expression syntax into classic C-syntax before feeding the result to a C compiler. (And allows you to run C programs on your S-Expressions as macros.)
But that change by itself wouldn't change too much about the language. And wouldn't make S-Expression-C as pleasant a language as any old Lisp.
Well, yeah, but that is missing the point rather badly. There is nothing you can do in any high level language that you can't do in assembler. The only reason high level languages exist at all is to make programming less annoying.
> For many purposes, S-Expressions aren't particularly useful, because eg they don't keep track of which variables are bound where.
S-expressions are nothing more than a serialization of binary trees. There is a long, long list of features that they do not provide. Again, if you think that is relevant, you have completely missed the point.
For the record: the point, the thing that makes s-expressions cool, is that they provide a serialization of linked lists, and this serialization is super-simple, to the point where writing code in it is actually practical. They weren't designed for this. The fact that you can actually write practical code in s-expressions was a surprise, a discovery, not an invention. You can serialize binary trees as XML or JSON, and you can even use those to write code if you want to, but no one in their right might would actually do that. You would go nuts typing all those angle brackets, double-quotes, and commas. The reason s-expressions are cool is that they are parsimonious. You don't need all the angle-brackets and quotes and commas. S-expressions are actually a reasonable syntax for writing code whereas XML and JSON are not despite being fundamentally the same thing under the hood. Indeed, once you get accustomed to s-expressions, they are a superior syntax for writing code than conventional languages because they are easy to parse (no precedence rules to deal with).
> Well, yeah, but that is missing the point rather badly. There is nothing you can do in any high level language that you can't do in assembler. The only reason high level languages exist at all is to make programming less annoying.
Oh, there's plenty of stuff you deliberately _can't_ do in high level languages (compared to assembly or low level languages).
Eg Python doesn't let you use naked jumps (also called GOTO). Haskell doesn't let you mutate state willy-nilly.
> You don't need all the angle-brackets and quotes and commas.
Well, you have quoting and quasi-quoting. Very useful, and they use some more interesting syntax.
Yes, S-Expressions are a reasonable syntax. I used to like them a lot. But I also came to appreciate more traditional syntax when switching from Scheme to Haskell.
I was not arguing against S-Expressions at all. Only against 'homoiconicity' being a coherent or useful concept, or saying anything deeper about a language.
As I said earlier, if you start from C, change its syntax to use S-Expressions (rather simple to do), and allow S-Expression based macros (that use the same C-as-S-Expression syntax), you still don't have a language that's as pleasant to work with nor makes macros as easy as your favourite Lisp.
Not natively, but you can do it. It's just "a bit more annoying".
> you have quoting and quasi-quoting
That is obviously not what I meant. What I meant was that you can write (x y z) instead of ("x", "y", "z"). The latter is just "a bit more annoying" but that "bit" makes all the difference.
Also, ' and ` are not really a core part of s-expression syntax, they are just a convenient shorthand. Any s-expression written using ' and ` can also be written without them. The core syntax includes only parens and dots, and of course the constituents of atoms. Modern implementations also include native support for double-quoted strings, but that was not part of the original spec. Everything else is just syntactic sugar.
s-expressions can provide a hierarchical structure, but they are not representing AST. It's just another form of source code.
> And, of course, an interpreter that works by walking S-Expressions is really, really slow.
Common Lisp interpreters work by walking S-expressions. Typically you'll see the s-expressions directly in a debugger -> modifying the s-expression will modify the running program.
