As someone who is just starting to dive into Django and Python in general I have to admit being confused by benchmark after benchmark that puts Python near to the bottom of the performance curve when compared to other technologies (Go being one that jumps out at me).
I have to admit that part of me is wondering if I am making the right decision in investing time, effort and money getting up to speed on Python/Django for new projects rather than devoting that time to something like Go. Can't do both. Not enough time and we need to get things done.
Now, I do appreciate that Python has a huge ecosystem and lots of very useful libraries out there. That alone might make the price of admission worth it. I also look at these benchmarks and it seems that PHP (something I am really trying to leave behind) frequently scored above the various Python ecosystems.
Can someone put this into perspective for me? In the end performance can and is hugely important. What am I missing when looking at all of this?
For many web programming tasks, your user-perceptible speed and/or maximum system loads will not be dominated by the performance of your web application but rather by database, I/O (including networking), and client performance (JS performance, aggressive asset caching, CDN usage, etc etc).
Go or Java are wonderful languages if you have serious number-crunching needs or if cutting your server budget in half would save you a billion dollars. My server budget is $500 a month and if you doubled the performance of all of my servers it would be $500 a month. I'll elide the detailed explanation of the whole "User clicks a button" to "User sees something happen" HTTP request waterfall chart but my Rails app is generally less than 5% of it.
Plus, Django/Rails pretty much have a library for anything you'll want to write. You can cobble applications together in days, and I've done it multiple times. I tried Go for web development, but I had to reinvent everything, which was just too slow.
> How about in Round 7 that Python surpasses Nodejs
I can't find what you're referring to. Node was still infront of Django in the JSON benchmark.
Django (until 1.6) hasn't had connection pooling for the database either. Updating those benchmarks with pooling should make significant differences.
But my point was that the framework itself can have a huge impact. Database and other I/O does have overhead, and it may even be significant. But it's not necessarily the biggest factor.
Oh, I see. But "python" wasn't represented, which confused me. There is still a large discrepancy between the different frameworks in the javascript and python set.
But the original point was that database and I/O make the largest difference. After restricting the set of languages and frameworks to python and javascript, I can only (now) agree.
109,000-5700 rps for JSON requests, and 4800-117 rps for multiple database queries. Quite significant.
I am also a novice and not an authority on Python, but can offer three reasons for using Python even if it is not fast as other languages.
1. Raw language speed is an important factor to consider but for some projects language syntax, community, ecosystem and libraries are much more significant. Those are python strong aspects. Due to its simple syntax, Python is very popular at mooc platforms, such as Coursera, edX and Udacity. Really lots of very good educational content and I mean MIT level good.
2. Usually web projects do not need to scale that much. Website traffic distribution follows power law so if you do not expect your project to be in top 1%, Django will usually handle all standard CRUD workloads.
3. There are Python bindings for many C libraries, for example image manipulation with C speeds is available using ImageMagick bindings.
Very often in web development, the kind of performance Go or Java delivers just isn't necessary. You'll often find that the time you spend on developing new features is much more important, and with python I think this can be significantly less.
Here's the advice I was given: if you're going to go with an opinionated framework, go with Rails. If you really want to use Python over Ruby, go with Flask. His thought is that Ruby is more dogmatic than Python by design, so Rails' dogmatic nature lends itself to Ruby more. Python's flexibility lends itself to a more flexible, lightweight framework like Flask.
Choosing a minimalistic web framework like Flask or Sinatra makes sense if you're experienced enough to know what choices Django and Rails make for you and you're comfortable making those decisions on your own. Otherwise, pick Django or Rails and start building things.
Couldn't agree with you more. I know what choices django makes for me, and I'm entirely happy letting it. I'd really miss a lot of what django has to offer if I were to move to something like flask.
If he's concerned with speed, why recommend Rails? It's just the same as Django in that department, if not slower. My advice to someone asking this question is to pick the language they feel the most expressive in, and then pick the framework. Django's a fine choice.
That being said, I'd love to see a microframework like Flask for Lua/openresty gain in popularity.
the perspective is, if you can be 20% more productive in Python vs. Go due to the ecosystem and style of Python, you can spend all the money you save in development costs on a few more servers to offset whatever performance penalties you find. I'm not sure what benchmarks you're looking at, but in the real world Python provides the infrastructure for extremely high volume sites like Reddit, Dropbox, Hulu and a crapload more.
That's not to say runtime performance issues have no effect, for example if you had some kind of algorithmic server at the center of a high frequency trading system or something like that, then benchmarks might be more critical (in which case write that one service in Go or C, then have your Python infrastructure talk to it via REST or as a C extension). It kind of depends on what you're doing but in the vast majority of business cases, it's not.
Pypy is also a great alternative to cPython which I hope starts to become viable for widespread production usage very soon - it features performance a lot more like the JVM.
I'll provide a dissenting opinion from everyone else and say that it's very nice being able to run a large site from a small number of web servers with very snappy requests, and to do this, it helps to not have the framework eat up 100ms. Also, a well tuned database or KV store can be extremely fast - just make sure all data is buffered in RAM and/or on SSDs.
Doing this right saves you from having to go all out on caching, and it can help keep your infrastructure simple, the benefits of which shouldn't be underestimated if you have a small team.
Also, if you're building interactive B2C sites and have some success, server costs can quickly become nontrivial, to the point where a 2-3x speedup can make a large monetary difference, and it's easier to not have to rewrite things for speed. If you're building B2B SAAS sites, server costs are probably a complete nonissue.
I adore Golang, but I would not use it to build a database-backed web application, and would look funny at anyone who did. Golang is a painful language to build conventional web apps in. Stick with Django for now.
I wouldn't take random benchmarks on someone's blog particularly seriously. There are many different performance metrics and many different ways to configure your web server and database. Most of the time these things are IO bound in these types of applications. Ultimately it depends on your particular application (which will probably be a little different from "hello world" and so will have a different performance profile). That said, it's not particularly surprising the Go would go faster for most equivalent (the concept of equivalence between programs in different languages is a tricky on itself) programs given it is compiled ahead of time and has complete type information about programs. I would remain a little dubious about the whether the benchmarks involving PHP were really comparing like with like.
My advice is not to sweat it and know that if/when the time comes that there are plenty of options for making things fast such as load balancers, serving static files from another machine, database on a different machine, various kinds of caching (whole page, database level or somewhere in between), tweaking the number of python threads/processes, etc and that you can choose these based on the results of benchmarks. A lot of the time there are big wins to be found in optimising frontend stuff first anyway.
I think it really centers around ease of use, amount of libraries, and community support. Django is widely adopted, has a long history of use, and is relatively easy to get started with. Performance is obviously important, but there are plenty of major websites using Django, so it can scale. This is a decent thread on that: http://www.quora.com/Django-web-framework/What-is-the-highes... .
Those benchmarks are for CPU-bound tasks. Typical web sites usually have other kinds of bottlenecks.
So, if you're doing heavy calculations in your user interface layer (that's what a web site is) - you're a) supposedly doing something wrong b) probably better with something else than plain CPython.
In many other cases, Python is fine. Not the very best option, performance-wise, but there are many other aspects than performance, too.
I'm certainly no expert, but thought I'd chime in as I have been wrestling with the same issue. I enjoy web development and have done projects with Java with JSP's, a few smaller projects with Rails, a large Django project, and now am playing with Rails again.
Your performance concerns (and I believe the other comments have touched on this) likely don't matter for most real world sites, and then, even if you get big, you can probably work it out (Pintrest is done with Django I believe, Github is rails, etc).
The question you really want to ask yourself is what does your compiler/runtime do for you. I've been pretty much solely a Python developer for the past year or so, but I like Java (... yea, I know), and find Go interesting, too. A compiler is no excuse to not write tests, but you know what you've written is going to run after it compiles. Added datetime's wrong in Python by adding one that has a timezone marked and one that doesn't? Runtime error. In fact, everything is a runtime error. Tests should not be optional regardless of language, but with Python and Ruby, they really take the place of the compiler in statically typed languages.
I don't want to discourage you from Python by any means, the Django community is fantastic, and Rails 4 with it's Russian doll caching seems pretty sweet. And nothing can beat the libraries for these two projects. The other options that I've seen (ASP.NET, Java Play, Go Revel, all for which, admittedly, I've only done preliminary reading about) just don't seem to have the libraries to get you off the ground as fast (though, if you've never written all the tedious stuff like user creation, email password resets, it is a good exercise in web development basics).
If you stick with Python I encourage you to:
1. Tackle learning the best practices for testing as you go.
2. Check out Two Scoops of Django [1] after having used Django for a bit.
3. Learn good server management skills (maybe look into Puppet [2])
The third seemingly is an offshoot, and would be necessary for really any framework, but once you're ready to deploy your app, you'll likely spend a great deal of time comparing your various options for stack choices and will get your sever set up just how you want it. Make sure that process is easily repeatable. If you don't like how Puppet does it, look into Chef or Salt Stack or whatever; these will save you tons of time down the line.
I would also suggest checking out the TwoScoops book, very well written best practices. Barring that, just using the template defined in the book to start new projects will get you on the right path to great project organization.
I have to admit that part of me is wondering if I am making the right decision in investing time, effort and money getting up to speed on Python/Django for new projects rather than devoting that time to something like Go. Can't do both. Not enough time and we need to get things done.
Now, I do appreciate that Python has a huge ecosystem and lots of very useful libraries out there. That alone might make the price of admission worth it. I also look at these benchmarks and it seems that PHP (something I am really trying to leave behind) frequently scored above the various Python ecosystems.
Can someone put this into perspective for me? In the end performance can and is hugely important. What am I missing when looking at all of this?