Automation of download of chapter list page

[naraht]

I've written a script that will download the information on the
http://www.apo.org/show/How_to_Start...r/Chapter_List page (after you click on go) and boil it down to a vertical bar delimited file suitable for importing into spreadsheets and databases. This should work on any Unix/Linux/Mac machine with nc (netcat) on it. It might be called nc or netcat on different machines.

#!/bin/bash
(nc www.apo.org 80 < nc.apo.in) | grep "<table>"| sed -e 's#</tr><tr>#</tr>+<tr>#g'| tr "+" "\n"| grep -v colspan | grep -v "td width="| sed -e 's/Send Email//g'|sed -e 's#</b><br>#|#g' | sed -e 's#<br>Region:#|Region:#g' |sed -e 's#<i>#|#g'| perl -pe 's/<[^>]*>//g' > apo`date '+%y%m%d'`
cut -f 4 -d \| apo`date '+%y%m%d'`| sort | uniq -c > apo`date '+%y%m%d'`.count

All of the line breaks except the one before the word 'cut' are simply from wordwrap and should not be in the program.

In addition the file nc.apo.in needs to exist which contains

POST /show/How_to_Start_a_Chapter/Chapter_List HTTP/1.0
Content-Length: 71
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: www.apo.org

bystatus=0&byregion=0&bysection=0&bycity=&bystate= 0&bysort=S&submit=Go

[GMUAPhiOAdvisor]

Be careful, my brother.....your inner geek is showing!!
Love and LFS,
Elyssa

[naraht]

Originally Posted by GMUAPhiOAdvisor

Be careful, my brother.....your inner geek is showing!!
Love and LFS,
Elyssa

Really, if you want, I can be geekier.

Sincerely
Randolph Finder
National Media & Tech Committee.

[GMUAPhiOAdvisor]

Originally Posted by naraht

Really, if you want, I can be geekier.

Sincerely
Randolph Finder
National Media & Tech Committee.

Is that possible? I showed that to my husband, who I call my "turbo-geek" and even HE said it was way up there on the Geek-o-Meter!!

You DO know I'm just playing, right???
In LFS,
Elyssa

[naraht]

Originally Posted by GMUAPhiOAdvisor

Is that possible? I showed that to my husband, who I call my "turbo-geek" and even HE said it was way up there on the Geek-o-Meter!!

You DO know I'm just playing, right???
In LFS,
Elyssa

Sure, I haven't imported it into a mysql database yet. I just wish I had access to the database that they actually use rather than grabbing what I can from what the website shows.

And besides, this isn't even as complicated as some of the shell scripts that I actually get paid to write. An example: (This is one of the uglier ones)

#!/bin/bash
AAA=`echo $TERM`
if [ $AAA == 'xterm' ]; then
COLOR_START='^[[0;1;34m'
COLOR_FINISH='^[[0m'
fi
(cat /home/DIR1/DIR2/html/tutorials/vulnerability/*.html | perl -p -0777 -e 's#<pre>.+?</pre># #sg'|perl -p -e 's#<tt>.+?</tt># #g'| perl -p -e 's/<.+?>/ /g' | perl -p -e 's/[A-Za-z]*\.[A-Za-z]*/ /g'|tr -c "[:alnum:]" "\n"| sort | grep -v "[0-9]"| grep -v "^[A-Z]*$" | uniq -s 1| tee /tmp/ccc |uniq -c -i | grep -v "^ 1" | cut -f 2 | tr 'A-Z' 'a-z'| comm -23 - /usr/local/bin/check_capsconsistencyexceptions |xargs -i grep -i "^{}$" /tmp/ccc | xargs -i echo "${COLOR_START}check_capsconsistency:${COLOR_FINIS H} Capitalization difference: {}");rm /tmp/ccc

check_capsconsistency is a sanity check for capitalization in our tutorials, making sure that we don't have cases where we use 'KeyView' and 'Keyview' by mistake so that we can correct one.


The initial test (from the beginning to the 'fi') is to make sure that the name of the sanity check shows up in color (purple, I think) when we run it from a terminal and doesn't when we run it automated overnight.

The main statement outputs all of our tutorials into the check, then takes anything in pre tags and deletes them and the tags they came in, deletes all tags, erases any words that have a dot in the middle, transforms anything that's left with a non alpha-numeric to a carriage return, sorts it, gets rid of anything with a number in it, gets rid of anything entirely made of capital letters, takes only unique values ignoring the first letter, drops a copy of the file in /tmp/ccc and keeps going, does another run of unique, ignoring case and counting how many of each it has, getting rid of anything where there is only one, cutting off the second field (the first is the count), makes all uppercase letters lower case, gets the lines in the stream that aren't in check_capsconsistencyexceptions, looks for each of those lines ignoring case in /tmp/ccc to get the non-lowercased versions in the stream at that point, and then outputs the results of that back to the screen for the user.

When I ran it just now I got back
check_capsconsistency: Capitalization difference: Keyview
check_capsconsistency: Capitalization difference: KeyView

Oh, you're playing? Never mind.

[GMUAPhiOAdvisor]

Originally Posted by naraht

And besides, this isn't even as complicated as some of the shell scripts that I actually get paid to write. An example: (This is one of the uglier ones)

You get PAID to write that kind of stuff??? WOW.....I mean no offense, but it kind of looks like what comes up on the screen when Dori (my 23 month old) tells her daddy "Email, write email!!" and her lets her!!

In all seriousness, I am truly impressed by people who write code. I could never do it.....and without it, this exchange would have never taken place. So for that skill, I am truly, humbly impressed.

[naraht]

Originally Posted by GMUAPhiOAdvisor

You get PAID to write that kind of stuff??? WOW.....I mean no offense, but it kind of looks like what comes up on the screen when Dori (my 23 month old) tells her daddy "Email, write email!!" and her lets her!!

In all seriousness, I am truly impressed by people who write code. I could never do it.....and without it, this exchange would have never taken place. So for that skill, I am truly, humbly impressed.

Strangely enough we have an infinite number of monkeys writing code for us, but right now they are behind schedule...

I don't have much of a clue on how to write the code that Greekchat uses for it's bulletin boards.

Randy

[GMUAPhiOAdvisor]

Regardless, as a former English teacher, the technical world is well out of my realm of knowledge and ANYONE, primate or human, who can take letters and symbols and turn them into something that does the functions you make them do is truly skilled - IMHO.

Elyssa

[naraht]

Originally Posted by GMUAPhiOAdvisor

Regardless, as a former English teacher, the technical world is well out of my realm of knowledge and ANYONE, primate or human, who can take letters and symbols and turn them into something that does the functions you make them do is truly skilled - IMHO.

Elyssa

True, but it is much easier to get machines to do you want than 16 year old High School students.

Randy

[GMUAPhiOAdvisor]

Originally Posted by naraht

True, but it is much easier to get machines to do you want than 16 year old High School students.

Randy

As I butcher the English language.....

Ain't THAT the truth?!

[naraht]

Originally Posted by GMUAPhiOAdvisor

As I butcher the English language.....

Ain't THAT the truth?!

*English rant*
I've *never* understood the issue with "ain't" in the English language. It stands as a perfectly reasonable contraction of "am not". Now the statement you've used should use "isn't", but that isn't *that* much of a stretch in English. Also, in English, I can't think of *any* word that can go in a sentence before "am" *other* than I since it is the 1st person singular of "to be". So taking ain't as the contaction of "am not", the I before it can *quite* reasonably be assumed, so "Ain't going" is nearly as proper as "I ain't going" which *should* be just as proper as "I am not going".
*rant off*

[GMUAPhiOAdvisor]

Originally Posted by naraht

*English rant*
I've *never* understood the issue with "ain't" in the English language. It stands as a perfectly reasonable contraction of "am not". Now the statement you've used should use "isn't", but that isn't *that* much of a stretch in English. Also, in English, I can't think of *any* word that can go in a sentence before "am" *other* than I since it is the 1st person singular of "to be". So taking ain't as the contaction of "am not", the I before it can *quite* reasonably be assumed, so "Ain't going" is nearly as proper as "I ain't going" which *should* be just as proper as "I am not going".
*rant off*

I just teach it the way the county tells me to.....I like my job.
Personally, I use y'all in class all the time....and I get dinged for that, too.....

[naraht]

Originally Posted by GMUAPhiOAdvisor

I just teach it the way the county tells me to.....I like my job.
Personally, I use y'all in class all the time....and I get dinged for that, too.....

Would a Math teacher get equally dinged?

Also, for the Spanish teachers, where is the Spanish that they teach from? Argentine, Mexican, Columbia and Castillian spanish are all somewhat different and unlike the French, they have no concept of a group trying to keep it pure...


Randy

[GMUAPhiOAdvisor]

As for math teachers, as long as the students can pass the SOL, the county is happy....doesn't matter how they get the info across.

As for foreign language teachers, most of them are *not* native speakers (this goes for ASL, too) Most of them are second language learners and they teach the way they were taught.....vocab, vocab, vocab.....very little grammar, very little culture behind the languages unless you get a REALLY enthusiastic.

It's all about the credentials....no longer is being a fluent, native person enough......now there is the evil PRAXIS and NCLB!! (do NOT get me started on that one!)

[naraht]

Originally Posted by GMUAPhiOAdvisor

As for math teachers, as long as the students can pass the SOL, the county is happy....doesn't matter how they get the info across.

As for foreign language teachers, most of them are *not* native speakers (this goes for ASL, too) Most of them are second language learners and they teach the way they were taught.....vocab, vocab, vocab.....very little grammar, very little culture behind the languages unless you get a REALLY enthusiastic.

It's all about the credentials....no longer is being a fluent, native person enough......now there is the evil PRAXIS and NCLB!! (do NOT get me started on that one!)

What's the SOL?

Sort of sad for foreign languages....

I've heard of the PRAXIS, NCLB is "No Child Left Behind", right?

Randy