The NLP is pretty basic and takes a ratio of the original article, so you do get some longer listings.

Big thanks to Wayne Larsen of hckrnews.com for providing me with some insight on tracking top stories and letting me use his ranking data. Also, I recommend http://www.hackernewsletter.com/ for a human-curated version.

Bookshrink has some pretty amusing summaries... it reminded me of this meme a while back where people would paste books into Microsoft Word and AutoSummarize it down to 6 words :)

Effectively, unsuccessful sign-up locks user out. You could provide some options (resend? remove?) where 'Email address has already been taken' is now.

Edit: 'unsuccessful sign-up' - seems like success is fractional here.

--------------------------------------------------------

Count how many times each word appears in the document into a dictionary or map structure

Also make sure you track the total words.

document |> splitBySpace |> if dictionary has word then +1 else 1; totalwords++

Then split the document into sentences.

Okay, now for each sentence

==========================================

score = 0.

split sentence by space and

for each word score+= -(dictionary[word]/sum) * log(dictionary[word]/sum)

dictionaryScore.Add(sentence, score)

==========================================

So now each sentence has a score. You can sort by best and lose order. Or if you want to limit (0 - 1) based on score:

findbestScore and filter each sentence by if limit < docscore / bestscore.

As I said this is only a start off point and is susceptible to list of random words (guess why) there are many ways to make it better. Here is a portion of code I dug up from a while ago:

  let inline sumMap m = m |> Map.fold (curryfst (+)) 0. 

  let inline internal countsAndSum n doc =
    let counts = splitstr [|" "|] doc |> filterStop n |> Array.fold mapAdd Map.empty
    counts, sumMap counts

  let ent m sum k = 
    let p = (mapGet m k 0.)/sum
    if p = 0. then 0. else -p * log2 (p)

  let eScore doc =      
    let counts , sum = countsAndSum 0 doc    
    splitSentenceRegEx doc |> Array.map (fun str -> str, splitstr [|" "|] str |> Array.fold (flip ((+) << (ent counts sum))) 0.)

I'd never seen that technique before, but (like a lot of algorithms) it is quite obvious once you've seen it.

Edit: actually, it's slightly different to my technique because I just used liner scoring (ie, counting popular words). I'm not sure which technique would work best.

[1] https://groups.google.com/d/topic/nltk-dev/qV9e5TsCBHg/discu...

Anyways, it is really hard to judge these things (statistical recommenders) since the metric is inherently subjective and there really is no wrong or right answer. But the way I like to defend it is: if you are going to just skim you should at least use a statistical based approach. Better than just jumping about randomly.

These days I'm more interested in abstract summarization without cheating (no templates).

Why is that?

I did not write the SO post which also is based on just word frequencies. I have found at least that in terms of picking the most relevant words with respect to the topic, the method I wrote which was inspired by ideas of entropy gives what I deem to be better results. Its robust against stop words and commonly repeated words that are not part of the topic. The summaries though, I cannot say are better or worse.

My algorithm is something I made up, and from memory it works like this:

1) Remove HTML, stem, remove stopwords etc

2) Sort unique words by popularity in the text

3) Split the original text on sentence boundaries.

4) Include each sentence that first mentions the next most popular word, until the summary is the maximum length requested.

http://news.ycombinator.com/item?id=1803020

Googling turns up http://sujitpal.blogspot.com.au/2009/02/summarization-with-l... which compares a few approaches.

Edit: Also http://stackoverflow.com/questions/2829303/given-a-document-..., which I think is by Dn_Ab who wrote the OP.

Then HNsummaries.com could fetch those when available instead of or in addition to the auto-summaries.

Would love to get an index of headlines on top of the email with anchors to actual stories below.

Would love to see shorter summaries and maybe some of the top comments for each story (summarized, if possible).

I'll take a crack and maybe add it as an option.

Quality feels at least as good as an open source summarizer I played around with a while back; good work!

I spend most of my time in Asia-Pacific timezones, so most of my automated emails arrive at awkward times. I'm glad that this one won't be staring at me from my inbox first thing in the morning -- helping me to produce first, consume second.

"9 year old Caine sets up an arcade in his father’s used car parts store in East L.A., using only cardboard boxes his dad had lying around and a ton of ingenuity. Watch his dreams come true when this filmmaker sets up a flash mob to come and play. Just watching this may make you a better person. $82,000 has already been raised for Caine’s scholarship fund! little behind on the bandwagon, but...film just had me in tears."

For more general text, the first problem that comes up is that out of context sentences with pronouns that point nowhere end up being unintelligible. The second sentence above only worked because the "he" was completely unambiguous in this summary.

Ideally I think you would do it client-side, so readers could adjust the shrinkage to the amount of time they have to peruse. I was also thinking about a scenario where you could browse at say 100-words and then dive-deep if you found anything that interests you. A more interactive approach. You might want to consider this.

But I really like the idea. Would love to hear how the project goes!

You should make sure that the summaries don't scale linearly with the size of the content--just because an article is 10x as long, doesn't mean I want a summary to be 10x longer. Maybe scale logarithmically?

I didn't find any of the summaries to be high quality or any better than I could get from briefly skimming HN myself.

I've unsubscribed.

One small suggestion...could you make the "76 comments" under the title clickable through to the HN comments?

One other option (maybe a user preference), include some noteworthy excerpts from the HN comments in the email as well?

It would be great to get a weekly or monthly summary.

Nice work.

I'd say the only time summaries could be a bad thing is for fiction, where you don't want to give things away.

For non-fiction giving things away is whole point. :)