Archive for the ‘Python’ Category

Checking automatically your server status from Android device

marzo 6, 2013

After a while without writing anything related to computer/scripting stuff I am back presenting a script for Android devices that will help you notifying when your personal server is down.

I have a personal server running in a virtual machine that during some weeks was a bit buggy, as my personal webpage is hosted there and I needed it to be up 24/7 when system failed the host was no longer available but it took me some days to realize as I wasn’t checking it everyday. As I wanted to apply some Python knowledge under Android I came out with the idea: I have a phone that is always connected so let’s create a script that periodically (using already-available tools) check the status of the server and in case of problem notifies it, thus, I don ‘t have to worry about checking the availability of my server.

The concept is quite simple and will work as follows:

  • Small script in Python that will run under SL4A.
  • A tool (TaskBomb) that will run the script periodically.
  • A server to be checked, http://hoyhabloyo.com in this case.

So, according to that steps, the process for creating the automatic check was:
First I installed TaskBomb (the scheduler), SL4A (scripting layer), the Python interpreter and SL4A Script Launcher which allows TaskBomb to run scripts as it can’t handle them directly. All these tools can be downloaded for free in the Android Market.

Second, I created the following script in Python:

import android
import urllib2

# Web page to check
web_page = 'http://hoyhabloyo.com'

try:
	urllib2.urlopen(web_page)
	# print 'Page was found!'
	# Ok, nothing to do
except (urllib2.HTTPError,urllib2.URLError) as e:
	# Error? Notifiy!
	droid = android.Android()
	droid.notify("Server seems down!", web_page)

Third, and finally, I configured the TaskBomb to run the script (through ScriptLauncher) everyday at 6.15:

And that is it! Now, every morning your phone will try to connect to the server, if everything is ok nothing will happen but if anything fails, it will display a small notification on the status bar (I established the rule on IPTABLES to make it fail).

Notification when failure

Notification when failure

In SL4A page you can find examples and a book to start developing some powerful scripts. Enjoy!

Basic WordPress crawler in Python

enero 29, 2012

I’m developing a new personal web where I will probably include some links to this, my blog, I thought it would be a good idea to have an automatic way to extract them from this site. As I was a bit bored last Monday and wanted to remember my Python skills, I thought “Let’s do a really simple WP crawler to make that work for me!”.

The next Python crawler it’s a really simple script. I know it could have been coded showing a beautiful menu with options and some methods having default params and all that, but just wanted to code it quickly. Every tricky part it’s commented but I will explain it in a few lines.

  1. WordPress blogs show only the last entries and at the bottom, there’s a link which allows you to go to older entries (linking to something like …/page/number)
  2. The crawler will start seeking for every link in the webpage indicated as source
  3. For every link found, if it’s an ‘h2’ type, will consider it as a blog entry so will store it in the list wp_entries
  4. If it’s an internal link will crawl it as long as it contains the word ‘page’
  5. There’s a depth limit so it doesn’t crawl through every page if desire
  6. When it has finished crawling, it will show every entry found

Here is the code:

import urllib2
import urlparse
import re

class Crawler:
    _source = ''
    _depth = 0
    _links = []
    _wp_entries = []
    _debug = False

    def __init__(self, source, depth):
        self._source = source
        self._depth = depth
        self._links = [] 
        self._wp_entries = []
    
    def get_childs(self, url, level):
        if (level <= self._depth):
            if self._debug : print 'Crawling ', url
            try:
                page = urllib2.urlopen(url)

                # Discard non html files
                page_info = page.info()['content-type']
                if not(re.search('text\/html', page_info)):
                    if self._debug: print 'Found a non-html file ', url, ' :', page.info()['content-type']

                else:
                    for line in page:
                        # Find every link in source code
                        if ((re.search('a href=', line) != None)):
                            # If several links in a line -->  Iterate through all
                            links_in_line = re.findall('a href="(.*?)"', line)
                            for link in links_in_line:
                                # For each link found
                                #   0 - Check the link its internal (regex source or a page in same dir)
                                #   1 - Create the new page (only if it's without source in link)
                                #   2 - Check it doesnt been crawled (check self._links)
                                #   3 - Add to the list (self._links)
                                #   4 - Crawl the new one according to the rules (match 'page' in this case)
                                new_link = ''
                                crawl_link = False
                                if (re.match(self._source, link)):
                                    # Subpage sharing source (source/sthing) -> use that link
                                    if self._debug: print 'Found internal link ', link
                                    new_link = link
                                    crawl_link = True

                                elif (re.match('http:\/\/', link)):
                                    # External link -> do nothing
                                    if self._debug: print 'Found external link ', link

                                elif (re.match('\w|\_|\.', link)):
                                    # Internal link -> construct full url
                                    if self._debug: print 'Found internal link', link
                                    new_link = urlparse.urljoin(url, link)
                                    crawl_link = True

                                else:
                                    # Weird link
                                    if self._debug: print  'Found weird link ', link

                                if ((self._links.count(new_link) == 0) & crawl_link):
                                    # Found a new link, store & crawl it
                                    self._links.append(new_link)

                                    # WP specific actions
                                    # Store only h2 links (meaning entries)
                                    # Just crawl whenever link contains 'page' 
                                    if(re.search('h2', line)): 
                                        if self._debug: print 'WP entry ', new_link
                                        self._wp_entries.append(new_link)

                                    level += 1
                                    if(re.search('page', new_link)): self.get_childs(new_link, level)
                                    level -= 1

            except urllib2.HTTPError as e:
                if self._debug: print 'Found error while crawling ', url, e

            except urllib2.URLError as e:
                if self._debug: print 'Found error while crawling ', url, e
        else:
            if self._debug: print 'Maximum depth level reached, skipping'
                
    def show_childs(self):
        print len(self._links),' links were found, listing:'
        for link in self._links:
            print link

    def show_wp_entries(self):
        print len(self._wp_entries),' WP entries were found, listing:'
        for head in self._wp_entries:
            print head


# Create a new crawler with page and maximu depth level
crawler = Crawler('https://hoyhabloyo.wordpress.com/', 5)

# Get childs for that page (could have used defaults params in method)
crawler.get_childs('https://hoyhabloyo.wordpress.com/', 0)

# Show me what you found
crawler.show_wp_entries()

And now, let’s check the output where I’m asking the crawler to list this blog 6 pages of entries:

kets@ExoduS:~/programacion/sources/python$ python wordpres_crawl.py 
30  WP entries were found, listing:
https://hoyhabloyo.wordpress.com/2012/01/24/mitos-y-verdades-sobre-las-becas-icex-en-informatica/
https://hoyhabloyo.wordpress.com/2012/01/18/xfce-display-switching-dual-single-monitor/
https://hoyhabloyo.wordpress.com/2012/01/08/ano-nuevo-vida-nueva/
https://hoyhabloyo.wordpress.com/2011/12/31/adios-2011/
https://hoyhabloyo.wordpress.com/2011/12/23/100-000-visitas/
https://hoyhabloyo.wordpress.com/2011/12/19/de-dibujos-animados/
https://hoyhabloyo.wordpress.com/2011/12/01/hablemos-de-los-rumanos-vorbim-despre-romanii/
https://hoyhabloyo.wordpress.com/2011/12/01/reencuentro-de-becarios-ic3x-en-navaluenga/
https://hoyhabloyo.wordpress.com/2011/11/29/sigo-vivo/
https://hoyhabloyo.wordpress.com/2011/10/07/va-de-despedidas/
https://hoyhabloyo.wordpress.com/2011/09/27/los-rincones-de-bucarest-piata-matache/
https://hoyhabloyo.wordpress.com/2011/09/22/los-rincones-de-bucarest-parcul-carol-i/
https://hoyhabloyo.wordpress.com/2011/09/13/los-rincones-de-bucarest-piata-universitatii/
https://hoyhabloyo.wordpress.com/2011/09/12/receta-hummus/
https://hoyhabloyo.wordpress.com/2011/09/08/viaje-por-los-balcanes/
https://hoyhabloyo.wordpress.com/2011/09/01/receta-gazpacho/
https://hoyhabloyo.wordpress.com/2011/08/31/los-rincones-de-bucarest-parcul-herestrau/
https://hoyhabloyo.wordpress.com/2011/08/16/viaje-expres-a-espana/
https://hoyhabloyo.wordpress.com/2011/08/02/ruta-por-la-rumania-profunda-valaquia/
https://hoyhabloyo.wordpress.com/2011/07/17/de-paseo-por-los-carpatos/
https://hoyhabloyo.wordpress.com/2011/07/10/guia-para-vivir-en-bucarest/
https://hoyhabloyo.wordpress.com/2011/06/30/viaje-por-asia/
https://hoyhabloyo.wordpress.com/2011/06/28/receta-pescado-blanco-al-microondas/
https://hoyhabloyo.wordpress.com/2011/05/20/la-revolucion-espanola-spanishrevolution/
https://hoyhabloyo.wordpress.com/2011/05/17/viaje-a-belgrado/
https://hoyhabloyo.wordpress.com/2011/04/29/semana-santa-por-la-rumania-profunda/
https://hoyhabloyo.wordpress.com/2011/04/14/bruselas-y-amsterdam/
https://hoyhabloyo.wordpress.com/2011/04/04/roma/
https://hoyhabloyo.wordpress.com/2011/03/17/las-1000-grullas/
https://hoyhabloyo.wordpress.com/2011/03/02/receta-torretas-de-berenejena-queso-y-tomate-vegetarianas/

It works!! As you can see, with just a hundred lines of Python where typed to achieve that, and much more could be done just modifying some parameters. Hope it helps!

Programming PyS60 in Linux

mayo 17, 2010

Some time ago I wrote about how to control your PC using your Nokia S60 mobile phone (see it here). It consisted in an application written in Python which stablished a connection between the phone an the computer and according to user inputs in the phone it performed one or another action in the computer.

I was really interested in programming my mobile phone to develop some applications I have in mind and as I also wanted to learn some Python this past weeks I’ve spent some spare time doing my first applications. I’m going to explain how can you program your S60 phone with Python under Linux and in following posts I’ll try to write some tutorials about how to develop some applications.

In order to program your S60 phone with Python, first of all you should get the interpreter and install it into your phone. You can find it in sourceforge (current version is 1.4.5) and where you should download the one according to your phone. As mine is a Nokia N70 I’m using PythonForS60_1_4_5_SDK_2ndEdFP3. Download both the interpreter and the script shell and install them into your phone. I also recommend to download the latest API documentation.

At this point you should be able to execute Python scripts in your phone, by the way, some examples are provided with the interpreter try to run them in order to check that the installation was succesfully done.

Lets go now with the funny part, programming our own applications under Linux. The following options are available:

  • Programming the application directly on the phone (which I consider very cumbersome so lets discard it).
  • Programming the application on your computer and transfering and running it in the phone (which is not a bad option but, in my opinion, it is not really the best option).
    In order to do this in a more easy way you could use the PUTools utility which will help you transferring the files, interacting with the phone, taking snapshots… in an easier way through a terminal running in your computer.
    In the PUTools page, the instructions are writter for Windows users, but it’s not hard to adapt them for Linux users, basically what you have to do is:
    • Edit the pcfiles/sync.config to specify the connction port (I’ll use rfcomm0)


      #COM_PORT = 11
      COM_PORT = '/dev/rfcomm0'

    • Adding a Serial Port service for the bluetooth:


      sdptool add service=SP --channel=3

    • Getting ready for incomming connections on rfcomm0:


      $> rfcomm listen /dev/rfcomm0
      Waiting for connection on channel 1

    • Considering that you have previously installed the push.py script in your phone, run it and choose your computer in the connect dialog. If everything was done correctly you should see a message on your computer like the following:


      $> rfcomm listen /dev/rfcomm0
      Waiting for connection on channel 1
      Connection from 00:1C:9A:40:CB:BB to /dev/rfcomm0
      Press CTRL-C for hangup

    • And now, just run pcfiles/push script to get the terminal which will let you directly interact with the phone!
      Putools connected

      PUTools working console

  • Programming in your computer, testing your applications on it using an emulator, and once the applications are working correctly transferring them to the phone (which for me is the best option).
    In order to do this, you’ll have to download the PyS60 emulation library (S60-compact) available in sourceforge and which will let you test a lot of your applications directly in your computer without needing to transfer them to your phone. (Notice that modules like camera or some fonts are not available but will be more than enough for letting us introduce in the world of PyS60 programming!).
    What I usually do is programming the application on my phone, test that it works correctly and later on transferring to my phone with the single following command, and later on run the interpreter in the phone and selecting the transferred script.


    $> obexftp -b 00:1C:9A:40:CB:BB -c C:\\System\\Apps\\Python -p script.py

Now it’s time to start coding our first applications in Python. I recommend reading the API and the following tutorial which is one of the best I could find:

In following posts I’ll show you how to code step by step a very simple eyecandy-but-useless application covering several aspects of the PyS60 API.

Happy coding!