Search Directory Trees with Python

Here is a simple but powerful way to use Python to search and find all files with a specific extension within a given folder and all of its sub-folders using os.walk(). I use it a lot to find map documents with broken data sources, images that need to be organized or just to get a quick count of certain files in a directory.

import os

directorypath = raw_input("Enter a directory path: ")
extension = raw_input("enter an extension: .")

#Loop through all folders and subfolders in your target directory. 
for root, dirs, files in os.walk(directorypath):
    fileList = [os.path.join(root, f) for f in files if f.endswith(extension)]             
    for item in fileList:
        print item #or do something else with each file found.

 

When Django Met IIS

The Problem:

The county I work for had a one page geography quiz with outdated questions and a poorly structured user interface. It was just an unstyled list of select dropdowns and a submit button. The answer page that was returned just listed each question with all the answer choices under them. One answer under each question was highlighted in yellow but it wasn’t made clear to the user whether this was the correct answer or just the answer they chose.

Old Mesa County Quiz AppTo compound things, the quiz was stuck in an enterprise CMS that gave it a really ugly url. I wanted to change that so it would look good and be really easy to understand. Thankfully I wasn’t restricted to any particular language or technology stack to build the new quiz.

The Solution:

A few years back I had played around with Django and thought it was a cool framework but I had never really applied it to any project. So I said to myself “why not?” and set out to build a new geography quiz app with Django. Over the next couple of days I put together the components of the new application:

  • A PostgreSQL database to store the questions and answers
  • Models to define the data in the database
  • Views to process and send the data and to route user answers to the answer page
  • Templates to render the data

Everything seemed to be going along smoothly although I wasn’t very happy with the way I wrote my views to handle the user submitted answers. The app basically builds a form on the fly (questions with radio button answers) then does a POST when the quiz taker clicks the submit button. The view then takes the POST data and turns it into a Python list. The rest of the view just slices and dices the list and uses offsets to pull out the matching questions and answers.

I know there are cleaner ways of doing this. Especially since the returned POST data is a querydict object which is basically just a Python dictionary. Manipulating key/value pairs seems neater. But the way I did it worked and I got it up and running fast. Maybe a project next time I’m bored will be to make the code cleaner and more maintainable.

Another Problem:

Remember earlier when I said that I asked myself “why not?” when considering using Django? Well, I was about to answer that question and the answer wasn’t pretty.

While I was developing my quiz app I used Django’s built-in server which is a lightweight “please don’t use in production” server. It worked great. Then I decided it was time to port the app to my production server and actually use it in the real world. At that point I remembered we use IIS on a Windows server.

IIS isn’t all bad, especially if you work in a place that uses .NET components heavily which I do. Unfortunately, Django was never really developed to run on IIS. Django was really designed to live in a Unix world and be served out with something like Apache. I remembered that from my prior experience with the platform – I just forgot.

Another Solution:

There are several tutorials on the web (most of them several years old) that discuss running Django under IIS but none of them were very straight-forward. The solution that seemed like the quickest route to a running app was to use Helicon’s Python Hosting Package (part of the Helicon Zoo repository available through the IIS Web Package Installer). The hosting package basically loads all of your dependencies and does all of the complicated work of getting IIS to run an antagonistic technology. You then load a Python Project module which builds everything for you including:

  • A virtual Python install specific to your app
  • A web config file with needed environment variables
  • Permissions and application pool settings

The only pain point I had with using Helicon’s solution was discovering it doesn’t work with Django 1.7. I had developed in 1.7 and then when I migrated into the Helicon environment everything broke. This really threw me for a loop for a while until I found a post suggesting using Django 1.6. This didn’t turn out to be a big deal as it didn’t affect my apps functionality. I just had to remove a couple of middleware classes from my settings file and I was good-to-go.

New Mesa County Quiz AppConclusion:

I love working with Django. If I wasn’t in a Windows environment I might be trying to use it throughout my office GIS site. But I can’t see trying to force uncommitted technologies into a relationship they don’t even seem to want. I guess Django is just going to be a hobby framework for me for now. Fortunately there are plenty of others out there just waiting to be learned and implemented.

Installing Setuptools and PIP for Python

Python logoI’ve installed a lot of Python packages over the years using Distutils, SetupTools/easy_install and PIP. Distutils is Python’s built-in package distribution module and is pretty easy to use. However, it has some limitations, primarily that you have to manually download the package dependencies and there is no method to uninstall packages.

The Setuptools easy_install script takes care of downloading packages and package dependencies but still lacks certain features you would want from a fully functioning package manager. It doesn’t provide version control support, package tracking and uninstallation. There is a lot more to the Python package discussion but there is no point in bringing it up.

Anyway, while I use package distribution tools I rarely have to install the tools themselves since they only get loaded once. When I do have to set up a new machine or upgrade someone elses, I always forget the steps to get Setuptools and PIP installed. So I thought I would document the steps here. Now I just have to remember to come back here when I need them.

 Installing Setuptools:

1. Right click on this ez_setup.py link and save the file to your Python Scripts folder (If you have ArcGIS loaded you will usually find this at C:\Python27\ArcGIS10.x\Scripts).

2. Open a command prompt and change into the SCRIPTS directory.

3. Type

python ez_setup.py

then hit enter to execute the code. This will run the script which will download and install setuptools on your system.

For the official installation instructions for setuptools, which includes instructions for installing on Windows 8 with Powershell visit https://pypi.python.org/pypi/setuptools.

Installing PIP:

1. Open a command prompt and change into the C:\Python27 directory.

2. Type

easy_install pip

then hit enter to execute. Pip should now be installed on your system.

To actually install a package using PIP from a command prompt you simply type

pip install "PackageName"

and everything will be taken care of for you. To explore the more than 54,000 packages that are available for Pip to load visit PyPI – the Python Package Index.

Use Python to Keep Your Brain Sharp

“Use it or lose it” certainly applies where brain function is concerned. The experts tell you to exercise your brain to keep it in shape and ward off forgetfulness and possibly dementia when you are older.

One way to exercise your brain is to do computations in your head. Not long ago I read a book called The Power of Forgetting by Mike Byster that introduced several tricks and methods for doing these mental computations. One that stuck with me was how to multiply two, two-digit numbers in your head. Here’s an excerpt from the book explaining how to do it:

addnumbersThere are probably easier ways to do mental multiplication but Byster’s method is meant to be an exercise in remembering and forgetting numbers at will to make your brain stronger.

I wanted to find a way to challenge myself with the above multiplication method on a regular basis. My solution was to write a very small Python script that would generate random two-digit by two-digit multiplication problems and display them on screen. Here is what I came up with:

from random import randint
from ctypes import windll

firstNumber = randint(10,99)
secondNumber = randint(10,99)

problem = str(firstNumber) + " x " + str(secondNumber)
answer = str(firstNumber*secondNumber)

windll.user32.MessageBoxA(0, problem, "Can you multiply these in your head?", 0)
windll.user32.MessageBoxA(0, "The answer is: " + answer, "How did you do?", 0)

The script simply generates two random two-digit numbers, displays them to the user, then displays the correct answer when the user closes the first message box. I used Python’s ctypes library to create Windows message boxes,so you would need to make some adjustments if you wanted to use it on a different operating system. I also set up my task scheduler to run the script automatically every hour. Now, throughout the day at work and at home I’m reminded to exercise my mind in a way I wouldn’t normally exercise it.