tommy's scratchpad

I’m in Boulder, Colorado for the OGC meetings. I had a chance to check out Denver earlier today (16th Street Mall, as well stumbling upon a Latin-American festival), which was very nice. The mountains are absolutely breathtaking!

I’m going to try to hook up with Sean sometime this week, as he is close by. In the meantime, check out some photos from today — I’ll be updating these as the week goes on.

Saw this via digg earlier today, and couldn’t help but appreciate the “magic” of UNIX for the nth time.

My first foray into the UNIX world (years ago) was the result of a colleague (Mike Adair, of mapbuilder fame) who advised me to take UNIX training. I needed to script and chain processes together as part of a system to create a ground control point database. The rest, as they say, is history.

As the years go by, and as things become more packaged, “easier to use”, and “frameworks” popping up everywhere, I have found time and time again that the trusty UNIX command line and tools have saved the day when the pressure was on (this includes stuff like perl and [recently] python scripting), without the overhead of setting up grandiose configurations and such.

How did you get started with UNIX? Stories involving hacks and / or someone over your shoulder looking in amazement are welcome 🙂

My first major code contribution to MapServer was a server side implementation of OWS Common 1.0.0. For those who are not aware, OWS Common is a specification which unifies common XML constructs, etc. used by OGC specifications, such as Capabilities metadata, exception reports and bounding box encodings. The benefit of OWS Common is that specifications, and subsequently server implementations, can focus on their core specific functionality while leveraging the common bits.

A good example here is MapServer’s SOS server support, which leverages mapowscommon.c, making mapogcsos.c alot lighter as a result. Client implementations can additionally write OWS Common parsers as reusable functionality which they can then use when writing their, say, WFS and SOS clients. I really believe that this is a benefit for those developing SDI components (i.e. why should contact information be encoded differently for WMS and WFS, really?); just imagine the reduction in code as a result of OWS Common!

Sean recently pointed me to OWSlib, a python lib for working with OGC Web Services. I initially thought it would be great to write an OWS Common client / parser for OWSlib, so that when WFS 1.1.0 and SOS 1.0.0 clients are developed, they can use an OWS Common class between them.

But then I thought why not just implement this in MapServer, and have the functionality exposed via mapscript?

My previous post declared my plugging my nose and jumping into Python GIS stuff. Advice and info from Sean have been valuable in getting familiar with things, though I’ll be the first to admit I’m still green (writing Python like a Perl / C hack 🙂 ).

So here’s one of my first attempts at solving a real world problem: reprojecting a bunch of points from a CSV file:

#!/usr/bin/python

import sys
import mapscript

if (len(sys.argv) == 1):

	print sys.argv[0] + " <csvfile>"
	sys.exit(1)
else:
	projInObj  = mapscript.projectionObj("init=epsg:32619")
	projOutObj = mapscript.projectionObj("init=epsg:4326")

	print "id,geom,zone"
	f = open(sys.argv[1], 'r')
	for line in f:
		s = line.strip()
		k = s.split(",")
		wkt = "POINT(" + k[1] + " " + k[2] + ")"
		shape = mapscript.shapeObj.fromWKT(wkt) # man, I love WKT!!
		shape.project(projInObj, projOutObj)
		print k[0] + "," + shape.toWKT() + "," + k[3]
	f.close()

And that’s it! So now you can take the CSV output with the geometry encoded as WKT and hook it up to MapServer with some simple OGR OVF syntax in your mapfile:

CONNECTIONTYPE OGR
CONNECTION "<OGRVRTDataSource>
<OGRVRTLayer name='nb'>
<SrcDataSource>./nb.csv</SrcDataSource>
<GeometryType>wkbPoint</GeometryType>
<GeometryField encoding='WKT' field='geom'/>
</OGRVRTLayer>
</OGRVRTDataSource>"

Mind you, you might question why to reproject when MapServer can do this for you, but I digress. There’s probably more eloquent ways to do this than what’s been done above via mapscript, eh?

Well, I’ve finally decided to give Python an honest try. A born and bred Perl hack, then dipping into PHP, I’ve seen much Python applause in the geospatial software community through many of Sean’s posts. Though I know Sean thinks mapscript is a disaster, it was via mapscript that I started to look more into Python.

So here’s what I typically start off with when doing development:

• HTTP / URLs / Forms: for Perl, I use CGI; for PHP, I use the native, built in functions
• XML Processing: for Perl, I use XML::Simple; for PHP, I use SimpleXMLElement
• Database: for Perl, I use DBI; for PHP, I use the compiled in mysql support, for example

So far I’ve found urllib2 and ElementTree to be useful. Other than that, I’m finding the command line pretty awesome for one offs and testing.

If anyone’s got any other suggestions, that would be great!!

UPDATE (20 June 2007): Manual trackback and great post / response from Sean

Check out these articles from IBM on tuning LAMP applications. Being a longtime LAMP guy (the “P” being long time Perl hack, but slowly moving towards PHP), I found this useful for tweaks and optimizing my applications.

Part 1

Part 2

Looking forward to Part 3, which covers MySQL.

Update (10 June 2007): here’s Part 3.

Just read Cameron’s pointer to this course. All I have to say is wow. This is probably the most user-friendly and comprehensive online course / outreach material I’ve seen in a long time. And the student demo pages are really impressive.

Kudos to Ian and students!!

I’ve been working with MapServer since 2000. In October 2006, I was added as a committer to the codebase. Since then, I’ve been working on mainly OGC support and Perl mapscript type issues. After a few months, here are some things which I would like to see at some point in MapServer from a developer’s point of view.

1./ libxml2 support: it would be of benefit to MapServer to have XML support taken care of by libxml2. Currently, MapServer uses msIO_printf, which, while functional, is prone to XML errors in particular when outputting XML. libxml2 takes care of closing tags, quoting attributes, namespaces, schemas, and a slew of other functionality. Also, coding MapServer XML in libxml2 allows for extensibility (e.g. applying a stylesheet via mapfile configuration).

2./ Better organized codebase: something like:

• mapserver/
  • etc/
  • core/
  • formats/
  • mapscript/
  • ogc/
  • tests/
  • util/

3./ subversion: It’s nice to see MapServer move towards a shared infrastructure like OSGeo‘s and leverage trac for issue tracking. Already an improvement over bugzilla, trac also plays nice with subversion. So hopefully it will just be a matter of time before the codebase gets ported from CVS.

I go to my share of conferences, seminars and meetings. I think it’s part of keeping abreast of the community, technology, seeing colleagues, and making new contacts.

I always say, if there’s one conference that I make my best effort to attend, it’s the FOSS4G conference. Although the name has gone through a few iterations, I think the theme has generally been consistent: open source software for geospatial (I prefer the term geospatial, as oppossed to geo-informatics, GIS, geoscience, etc.).

Some of the things I enjoy are:

• Meeting the actual developers. Many of the developers attend this conference, and it’s a great chance to meet them in person, find out about future developments, or even ask them that burning question about line 642 in file gfoo.c
• The workshops: hands-on workshops are provided which are very useful. For example, I quickly learned PostGIS here and applied it the week after. The developers frequently give or hover these workshops, especially when it pertains to their software, so you can be sure that questions, no matter how complex, will be addressed
• The applications: it’s valuable to see presentations of projects which use FOSS4G approaches. Regardless of how simple you think your application/project/portal is, the fact that you have integrated these approaches is a great indication of your abilities, integration, and the tools used
• The community: last, but not least. The dynamics, the personalities, and the commitment and passion for this cause is exhilarating to see and be a part of

If you come from the open source software world, or have an interest in geospatial, or are a geospatial professional who’s curious about open source tools, technologies, the projects that use them, you’ll want to think about this conference.

I thought I’d put in my $0.02 CAD after reading a few similar posts out there. So here are some from my point of view (in no particular order):

Geospatial

• GeoRSS is here: GeoRSS made v1 this year at foss4g2006 and has proven to be a simple, yet very effective way to tag feeds. I’ve used and integrated in all my projects (I use the GML flavour) which produce RSS content by way of outputting an overview map based on the position. It gives the developers endless possibilities, and folks just love to see ‘where’ a post is
• Getting to Know PHP: Earlier this year, ignorant to PHP, I decided to take the plunge and see what all the fuss was about. Boy was I glad I did. So easy, even I could figure it out. So easy that I’ll be porting bits of my website to PHP in the coming year
• MapServer SOS support: It’s great to see OGC SensorWeb support in MapServer. Given that MapServer can already do stuff like spatial, temporal and aspatial queries, I think SOS was a relatively easy initial implementation effort. The more complex effort is the data bindings (sensor data is quite complex and multidimensional, and getting in and out of MapServer, in a generic framework is quite the challenge
• ResEau portal launch: I was very fortunate to work with a great bunch of people to have finally released the ResEau water portal after two years of planning, design and development. I think this portal is a great example of the benefits of using standards-based approaches all the while providing useful information to a vast audience
• MapServer commiter access: I was very honoured to be nominated this year for commit access to the codebase. My main focus was and is to implement the OGC OWS Common Specification for use by other OGC implementations
• Web 2.0, Time article and information overload: Such a good article. And so true. As a result, we are overloaded with information! At one point, I refactored all my bookmarks to be pure RSS feeds just to handle it all. Whoever said computers would lessen the workload!?
• foss4g2006: This was a great conference. The lighthing talks, BOF sessions, and demo fest, as well the gathering of the OSGeo crowd further solidified OSGeo’s existence. Can’t wait until next year’s event!
• Catalogue woes continue: the Cat2.0 ebRIM vs. ISO implementation debate continued. Add to this the lack of (especially open source) implementations, and stuff like owscat, and other catalogue-ish projects continued to exist. The OGC has recently endorsed ebRIM as the future base for Cat2.0 implementations, so hopefully we should start seeing some interoperability between catalogues
• Atlas of Canada 100th Anniversary: Congratulations on the 100th anniversary of this valuable Canadian resource. Kudos to the Atlas!

Other:

• Cool travel: Charlottetown, Alabama, Washington D.C, Lausanne, Las Vegas, San Diego, Moncton, Winnipeg were among some of the neat places I visited this year
• Home renovation: I did my biggest renovation job ever on my house this year, which lasted almost 6 months. I’m really glad the way things turned out, and I’m looking forward to doing some more stuff in 2007
• Condo putters along: construction continued on my condo, and is now ahead of schedule, ready this coming June. Looking good!
• Website changes: This year, I finally succumbed to using software packages to manage my website content. This website originated in 1998, as an HTML learning experience, so I was inclined to do *everything* by hand. While very useful, as time goes on, I find myself with less and less time (go figure!), and there’s so many solid tools out there to make things easy, and standards based (i.e XHTML, CSS, etc.). So why not use these great tools and concentrate more on the stuff you want to do, right?

Looking forward to 2007:

• Further OWS Common support in MapServer: mapowscommon.c/h is almost complete to spec (missing some operations support). The next step will be to begin integrating into the OGC specs as they migrate to OWS Common for stuff like GetCapabilities XML, etc.
• OWSContext: look for further formal development on this specification. OWSContext has been used frequently in the OGC testbeds, and interest is increasing in seeing the spec push forward.
• Further development on discovery and cataloguing: I’m hoping to see some development / experiments on this track. It will be interesting to see how packages like deegree2 progress given the recent motion passed at OGC
• Publications: I’ll be published in two Springer London books this year (The Geospatial Web, as well as Open Source Approaches to Spatial Data Handling)
• MapServer Brazil: I was honoured to be invited to this event, and look forward to speaking on OGC and open standards. Looking forward to this!

Did I miss anything? What else was geospatial-worthy in 2006?

A Beautiful Wedding Photography Season

Compared to 2020, 2021 was a more hopeful and even in some ways “better” year. We had both good and bad moments, but with the bad ones, we need to remember the good ones forever and that’s what Fame Park Studios is specialized in, to save and preserve those beautiful moments, either with your family, with your partner or even by yourself.

At any rate, this closes out 2006 for me in the blogosphere. I wish everyone a great Holiday season, and all the best for 2007!