tommy's scratchpad

Fresh off the press, check it out.

I’m in Boulder, Colorado for the OGC meetings. I had a chance to check out Denver earlier today (16th Street Mall, as well stumbling upon a Latin-American festival), which was very nice. The mountains are absolutely breathtaking!

I’m going to try to hook up with Sean sometime this week, as he is close by. In the meantime, check out some photos from today — I’ll be updating these as the week goes on.

Saw this via digg earlier today, and couldn’t help but appreciate the “magic” of UNIX for the nth time.

My first foray into the UNIX world (years ago) was the result of a colleague (Mike Adair, of mapbuilder fame) who advised me to take UNIX training. I needed to script and chain processes together as part of a system to create a ground control point database. The rest, as they say, is history.

As the years go by, and as things become more packaged, “easier to use”, and “frameworks” popping up everywhere, I have found time and time again that the trusty UNIX command line and tools have saved the day when the pressure was on (this includes stuff like perl and [recently] python scripting), without the overhead of setting up grandiose configurations and such.

How did you get started with UNIX? Stories involving hacks and / or someone over your shoulder looking in amazement are welcome 🙂

My first major code contribution to MapServer was a server side implementation of OWS Common 1.0.0. For those who are not aware, OWS Common is a specification which unifies common XML constructs, etc. used by OGC specifications, such as Capabilities metadata, exception reports and bounding box encodings. The benefit of OWS Common is that specifications, and subsequently server implementations, can focus on their core specific functionality while leveraging the common bits.

A good example here is MapServer’s SOS server support, which leverages mapowscommon.c, making mapogcsos.c alot lighter as a result. Client implementations can additionally write OWS Common parsers as reusable functionality which they can then use when writing their, say, WFS and SOS clients. I really believe that this is a benefit for those developing SDI components (i.e. why should contact information be encoded differently for WMS and WFS, really?); just imagine the reduction in code as a result of OWS Common!

Sean recently pointed me to OWSlib, a python lib for working with OGC Web Services. I initially thought it would be great to write an OWS Common client / parser for OWSlib, so that when WFS 1.1.0 and SOS 1.0.0 clients are developed, they can use an OWS Common class between them.

But then I thought why not just implement this in MapServer, and have the functionality exposed via mapscript?

My previous post declared my plugging my nose and jumping into Python GIS stuff. Advice and info from Sean have been valuable in getting familiar with things, though I’ll be the first to admit I’m still green (writing Python like a Perl / C hack 🙂 ).

So here’s one of my first attempts at solving a real world problem: reprojecting a bunch of points from a CSV file:

#!/usr/bin/python

import sys
import mapscript

if (len(sys.argv) == 1):

	print sys.argv[0] + " <csvfile>"
	sys.exit(1)
else:
	projInObj  = mapscript.projectionObj("init=epsg:32619")
	projOutObj = mapscript.projectionObj("init=epsg:4326")

	print "id,geom,zone"
	f = open(sys.argv[1], 'r')
	for line in f:
		s = line.strip()
		k = s.split(",")
		wkt = "POINT(" + k[1] + " " + k[2] + ")"
		shape = mapscript.shapeObj.fromWKT(wkt) # man, I love WKT!!
		shape.project(projInObj, projOutObj)
		print k[0] + "," + shape.toWKT() + "," + k[3]
	f.close()

And that’s it! So now you can take the CSV output with the geometry encoded as WKT and hook it up to MapServer with some simple OGR OVF syntax in your mapfile:

CONNECTIONTYPE OGR
CONNECTION "<OGRVRTDataSource>
<OGRVRTLayer name='nb'>
<SrcDataSource>./nb.csv</SrcDataSource>
<GeometryType>wkbPoint</GeometryType>
<GeometryField encoding='WKT' field='geom'/>
</OGRVRTLayer>
</OGRVRTDataSource>"

Mind you, you might question why to reproject when MapServer can do this for you, but I digress. There’s probably more eloquent ways to do this than what’s been done above via mapscript, eh?

Well, I’ve finally decided to give Python an honest try. A born and bred Perl hack, then dipping into PHP, I’ve seen much Python applause in the geospatial software community through many of Sean’s posts. Though I know Sean thinks mapscript is a disaster, it was via mapscript that I started to look more into Python.

So here’s what I typically start off with when doing development:

• HTTP / URLs / Forms: for Perl, I use CGI; for PHP, I use the native, built in functions
• XML Processing: for Perl, I use XML::Simple; for PHP, I use SimpleXMLElement
• Database: for Perl, I use DBI; for PHP, I use the compiled in mysql support, for example

So far I’ve found urllib2 and ElementTree to be useful. Other than that, I’m finding the command line pretty awesome for one offs and testing.

If anyone’s got any other suggestions, that would be great!!

UPDATE (20 June 2007): Manual trackback and great post / response from Sean

Check out these articles from IBM on tuning LAMP applications. Being a longtime LAMP guy (the “P” being long time Perl hack, but slowly moving towards PHP), I found this useful for tweaks and optimizing my applications.

Part 1

Part 2

Looking forward to Part 3, which covers MySQL.

Update (10 June 2007): here’s Part 3.

Just read Cameron’s pointer to this course. All I have to say is wow. This is probably the most user-friendly and comprehensive online course / outreach material I’ve seen in a long time. And the student demo pages are really impressive.

Kudos to Ian and students!!

I’ve been working with MapServer since 2000. In October 2006, I was added as a committer to the codebase. Since then, I’ve been working on mainly OGC support and Perl mapscript type issues. After a few months, here are some things which I would like to see at some point in MapServer from a developer’s point of view.

1./ libxml2 support: it would be of benefit to MapServer to have XML support taken care of by libxml2. Currently, MapServer uses msIO_printf, which, while functional, is prone to XML errors in particular when outputting XML. libxml2 takes care of closing tags, quoting attributes, namespaces, schemas, and a slew of other functionality. Also, coding MapServer XML in libxml2 allows for extensibility (e.g. applying a stylesheet via mapfile configuration).

2./ Better organized codebase: something like:

• mapserver/
  • etc/
  • core/
  • formats/
  • mapscript/
  • ogc/
  • tests/
  • util/

3./ subversion: It’s nice to see MapServer move towards a shared infrastructure like OSGeo‘s and leverage trac for issue tracking. Already an improvement over bugzilla, trac also plays nice with subversion. So hopefully it will just be a matter of time before the codebase gets ported from CVS.

I go to my share of conferences, seminars and meetings. I think it’s part of keeping abreast of the community, technology, seeing colleagues, and making new contacts.

I always say, if there’s one conference that I make my best effort to attend, it’s the FOSS4G conference. Although the name has gone through a few iterations, I think the theme has generally been consistent: open source software for geospatial (I prefer the term geospatial, as oppossed to geo-informatics, GIS, geoscience, etc.).

Some of the things I enjoy are:

• Meeting the actual developers. Many of the developers attend this conference, and it’s a great chance to meet them in person, find out about future developments, or even ask them that burning question about line 642 in file gfoo.c
• The workshops: hands-on workshops are provided which are very useful. For example, I quickly learned PostGIS here and applied it the week after. The developers frequently give or hover these workshops, especially when it pertains to their software, so you can be sure that questions, no matter how complex, will be addressed
• The applications: it’s valuable to see presentations of projects which use FOSS4G approaches. Regardless of how simple you think your application/project/portal is, the fact that you have integrated these approaches is a great indication of your abilities, integration, and the tools used
• The community: last, but not least. The dynamics, the personalities, and the commitment and passion for this cause is exhilarating to see and be a part of

If you come from the open source software world, or have an interest in geospatial, or are a geospatial professional who’s curious about open source tools, technologies, the projects that use them, you’ll want to think about this conference.