If you liked this, you'll probably like even more "A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux" (http://www.muppetlabs.com/~breadbox/software/tiny/teensy.htm...). Last time I checked, the teensy executable had stopped working around Linux 2.6.26 or 2.6.27, and I don't know why.
Relatedly, I wrote a tiny quasi-Forth in itself that generates ELF executables directly at https://github.com/kragen/stoneknifeforth. Its executables also stopped running around Linux 2.6.27, and I still haven't bothered to figure out why.
That's odd. I'm currently running 2.6.32, and it's working fine for me. Maybe it was a transient problem?
Feel free to email me if you notice any of the teensy executables not working. I don't follow the bleeding edge very closely, but I do make a serious effort to keep all of them working.
Agreed. I would heartily recommend reading the article you linked to, I certainly liked it a lot.
The project you've worked on looks interesting, too! It's a little late for me to be reading too much code, but I'll make sure to take a good look at it tomorrow. Thanks for the link.
If you'd reported that at the time as a regression, then whatever change had stopped them working would most likely have been reverted. They take binary backwards-compatibility pretty seriously.
StoneKnifeForth is a toy, and the executables it generates were probably actually invalid in some way I don't understand because I haven't read the ELF spec. Preventing the kernel from barfing on them would have an insignificant real benefit, I think, and possibly a significant cost. If we were talking about PulseAudio or something, I would agree, but no.
The obvious first step to reporting the bug (after looking at diffs, which I did try) would be to instrument the ELF loading code with printk calls to explain why it's failing to exec a supposedly ELF executable, so that I could report what it was about the executable it was that made the kernel barf. And then I could fix the executable. I just haven't gotten around to recompiling my kernel with a modified ELF loader.
Being "invalid" doesn't nullify the backwards-compatibility rule - if the kernel accepts a certain kind of invalid ELF file, and people have come to rely on it doing so, then that has become a de facto part of the ABI. "But userspace is buggy!" is not considered a valid excuse.
What does nullify the backwards-compatibility rule is no-one noticing or caring that it has changed.
One of the cool things about "personal computers", when they were not synonymous with "IBM PC" or x86 box, was that you had total control over every bit and byte that went into them. Sometimes you had to use switches to manually set memory contents, sometimes there was a hard coded 'boot loader' which would do that for you. The coolest demo I think I've seen in this space was an FPGA computer design where the first download into the FPGA made it a serial device that could write memory, and then you erased that and loaded a computer which would then read that memory and execute it.
I believe that one of the reasons the Arduino has the appeal it does is because you can know everything there is to know about the program it is running. That is a lot of fun for a technologist.
It would be fun to have this blog posting written from within a debugger. I always found it intriguing to 'debug a program into existence' as it were.
So essentially, this is writing an application in machine code?
This is so fascinating I'm starting to get palpitations. I've always wanted to know how to program in machine code. I've looked through the web countless times in search of something like this, and haven't ever found it, until now.
Thanks for the great insight. This kind of stuff makes C/C++ look like stuff for total newbies. Getting this close to the hardware really is quite mind-boggling.
It is pretty fun. If you have a TI-83 or TI-84 calculator you can do the same thing a little bit easier.
In high school, I would print out 2 pages of all the Z80 opcodes and put them on my desk during class. The calculators let you directly input hex and then run your program with asm(), so I would write my program by hand on paper, and then next to each line translate it to machine code and put it into the calculator. A lot of TI routines are easily accessible with bcalls, so you don't have to reinvent the wheel with each program.
Plus Z80 is a lot easier than x86. (I still remember the return code: C9)
I had a lot of fun taking friend's calculators and disabling the LCD and power button in assembly :)
Manually picking out hex opcodes is maybe a bit over the top. A nice assembler like nasm should be "low level enough" while still providing a lot more convenience and flexibility.
Well of course, nobody writes any applications in hex; it's more of a proof of concept, but a very interesting proof of concept.
Asm, at least for me, is still too low-level for me to do any sort of serious work. I make desktop/mobile apps, and don't really do any sort of kernel hacking, so currently, I'll stick with C++ and Java. It is a nice thing to know, however, I don't see myself learning it right now. It's inevitable that I'll have to learn it sometime in the future if I ever want to broaden my prospects, but it's nice to play around with, but isn't all too useful for me.
Hi, I'm the author. In this particular example, I basically skipped the compiler and linker to create an executable directly, yes. It's certainly a lot of fun, but if you're really trying to become proficient at programming in assembly language, I would recommend reading Assembly Language Step-by-Step: Programming with Linux.
> I've always wanted to know how to program in machine code.
1) Decide which processor you want to program. I'm going to use a Z80, since that's one I'm familiar with.
2) Get hold of a good book, making sure it includes a section that exhaustively describes each instruction and the machine code for each. Eg. "Programming the Z80" by Rodnay Zaks. The microprocessor's datasheet will generally do if your familiar with other processors.
3) Read the book, cover to cover.
4) Make three vertical columns on a sheet of paper.
5) Write your program, in assembly language, in the middle column. Use alphanumeric labels (label means variable name) for all addresses.
6) Decide at what address you want your program to start (the origin)
7) Write the numeric address in the left hand column, next to the first instruction.
8) Look up the machine code, for the instruction in the middle column, in the Zaks book.
9) Write the machine code in the right hand column.
10) Add the length of the instruction (# bytes) that you have just written down to the address, and write the answer on the next row of the left hand column. You might have to leave labels in place for instructions that refer to parts of the program that you have not yet assembled.
11) Repeat from step 8, until the entire program has been assembled.
12) Go back and fill in the numerical addresses for any addresses that are still labels.
13) Now you have to get the program in memory. We're going to assume that a programming language, such as BASIC is available. The alternative is a keypad with address/data entry functions and a method to get an initial value into the program counter.
14) Create a big array, in BASIC, with all the numbers from the right hand column.
15) Write a loop which POKES each numeric value into memory, starting from the origin address.
16) Transfer execution to the origin address, using the a USR(nnnn) instruction, where nnnn is the origin address.
17) Hopefully your program works, but if not, make changes and go back to step 4 until your program works.
Enjoy! In time, the process gets quicker, as you eventually remember the machine code for the the most common instructions and don't have to refer to the book.
If you're writing timing critical code, have a fourth column, in which you record the number of clock cycles for each instruction to execute, and sum those numbers to determine execution time. If you have a target execution time, you will need to add/remove instructions to achieve the target.
This is great. I've had low-level stuff on my mind lately. I was thinking about where to start, either writing my own binary to the boot sector of a disk, or booting up MS-DOS and going from there, or from linux or windows and starting with machine code from there. So this would be perfect. (I was thinking of maybe implementing FORTH. Interesting that kragen mentions a "tiny quasi-Forth" that he made.)
My favourite boot-loader story was told to me by one of the old-hands on the Jindalee Radar, when it was still a research project. Jindalee had a manned receiver station and a remote transmitter station, about 100km away. They were connected via a radio link, and to start with, the PDP at the remote site had to be booted by keying in the boot-loader over the radio link. If you lost the radio link, you had to start again...and again... and again...
At some point soonish I intend to have a go at loading dynamic libraries in bare metal on the Raspberry Pi. I'll be on the lookout for stuff like this for .so's. Suggestions welcome.
I am voiceless: why would the f*ck would someone loose such that much time on a trivial, yet annoying task?
That is the first lesson you have in assembly language: program in assembly language, then in C because it spares you the time (and the mistakes involved in the process) to do such a thing.
Thousands of men year efforts and wisdom ruined in a post, showing how people have too much time on their hands, and so few imaginations (well he could have written something interesting on how to display p0rn pictures in ASCII art at least).
Having spent my years climbing up and down the ladder of abstraction, I think your analysis is way off the mark. This is a great way to learn how ELF works (and binaries in general), learn how machine code is formed, etc. Is it an actually 'useful' task, that produces a useful end product? Yes: knowledge. Knowledge that isn't easy to acquire.
You won't ever do this in the Real World (TM), but it's a fantastic way to get into things.
I studied physics, and I learnt actual programming and assembly language: I actually did this in the real world!
What do you learn in Computer Science then?!
Gibbering useless concepts that makes you non-sensical experts in the field of not delivering your software neither in time, nor in the frame of the specification? Or just
in the unlikely case that your question is sincere, this article on wikipedia touches lightly upon some of daeken's "real world" contributions: https://en.wikipedia.org/wiki/Cody_Brocious
Indeed. He's telling us much more about himself than he is anything relevant to the article.
If you see nothing but a huge waste of time in this post, you're probably on the wrong site. HN in its prime would have really pissed you off, no doubt...
Relatedly, I wrote a tiny quasi-Forth in itself that generates ELF executables directly at https://github.com/kragen/stoneknifeforth. Its executables also stopped running around Linux 2.6.27, and I still haven't bothered to figure out why.