Jay Varner

My Golden Hammer - or - A Software Engineer Walks into a Design Class

Wed, 16 Aug 2017 09:00:00 -0400

The notion of a golden hammer, “a familiar technology or concept applied obsessively to many software problems”, was introduced into information technology literature in 1998 as an anti-pattern: a programming practice to be avoided.¹

For the past few weeks, I have been building a prototype of an app for my interaction design class. I have been laying out the screens in Sketch. This is not new to me. I often use Sketch to, well, sketch out ideas before writing code. I hit a frustrating snag when I went to convert the sketches into an interactive prototype using Proto.io. For the record, Proto.io was not the problem. It is a very compelling and powerful tool. I hope to learn a better workflow.

I abandoned Proto.io for my three headed golden hammer – JavaScript, HTML, and CSS. I’m not going to say it saved me time, but it did make the exercise more comfortable. Using my day-to-day workflow, I was able to take my Sketch work and quickly build a functional version of the app.

I learned an important thing. I should never prototype with real code. It was kind of fun to not care and violate basic best practices in the interest of time. I cranked it out with the attitude of “this is a prototype…it’s not like this is ever going into production.”

In the real world, it would be so easy to say…

Hey, I spent a lot of time building this prototype. It does “work.” Let’s just get started with this. We’ll fix it before it goes to production.”

We all know it will not get fixed before going to production.

https://en.wikipedia.org/wiki/Law_of_the_instrument#Computer_programming ↩

An Overview of Overviews

Mon, 13 Jun 2016 09:30:00 -0400

In our previous post, Bringing Your Maps into Focus, we covered a two-step process using GDAL. There is a lot going on in the second step, internal tiling and overviews. These processes are hard to research and understand what’s really going on.

We’re going to walk through what is happening using two command line tools; exiftool to inspect the GeoTIFF’s internal metadata and GDAL’s gdalinfo. Installing both is not really a pain. For Mac there are Homebrew formulas for each. Most Linux distros have packages, and there are downloads for Windows .

Let’s look at the exiftool output for a non-georeferenced TIFF. It’s really not that interesting. We’re providing here as a starting point.

$ exiftool Atlanta_1928_Sheet45.tif
ExifTool Version Number         : 10.02
File Name                       : Atlanta_1928_Sheet45.tif
Directory                       : .
File Size                       : 90 MB
File Modification Date/Time     : 2016:05:24 20:59:18-04:00
File Access Date/Time           : 2016:05:24 20:58:53-04:00
File Inode Change Date/Time     : 2016:05:24 20:59:40-04:00
File Permissions                : rw-r--r--
File Type                       : TIFF
File Type Extension             : tif
MIME Type                       : image/tiff
Exif Byte Order                 : Little-endian (Intel, II)
Subfile Type                    : Full-resolution Image
Image Width                     : 5347
Image Height                    : 6348
Bits Per Sample                 : 8 8 8
Compression                     : LZW
Photometric Interpretation      : RGB
Image Description               :
Strip Offsets                   : (Binary data 3521 bytes, use -b option to extract)
Orientation                     : Horizontal (normal)
Samples Per Pixel               : 3
Rows Per Strip                  : 16
Strip Byte Counts               : (Binary data 2778 bytes, use -b option to extract)
X Resolution                    : 300
Y Resolution                    : 300
Planar Configuration            : Chunky
Resolution Unit                 : inches
Software                        : Adobe Photoshop CS5.1 Windows
Modify Date                     : 2012:02:08 17:22:10
Predictor                       : Horizontal differencing
XMP Toolkit                     : Adobe XMP Core 5.0-c061 64.140949, 2010/12/07-10:57:01
Create Date                     : 2010:06:15 15:11:38-04:00
Metadata Date                   : 2012:02:08 17:22:10-05:00
Creator Tool                    : Adobe Photoshop CS3 Windows
Format                          : image/tiff
Description                     :
Color Mode                      : RGB
Instance ID                     : xmp.iid:DDF74EAAA152E111A8D59B79B1F6E115
Document ID                     : uuid:32FACA12C678DF11BE5CBF86910F7516
Original Document ID            : uuid:32FACA12C678DF11BE5CBF86910F7516
History Action                  : saved
History Instance ID             : xmp.iid:DDF74EAAA152E111A8D59B79B1F6E115
History When                    : 2012:02:08 17:22:10-05:00
History Software Agent          : Adobe Photoshop CS5.1 Windows
History Changed                 : /
Marked                          : False
Current IPTC Digest             : 54de4a51baba1e48176bd21acf562b86
Coded Character Set             : UTF8
Application Record Version      : 0
Caption-Abstract                :
IPTC Digest                     : 54de4a51baba1e48176bd21acf562b86
Displayed Units X               : inches
Displayed Units Y               : inches
Global Angle                    : 30
Global Altitude                 : 30
Copyright Flag                  : False
Photoshop Thumbnail             : (Binary data 5790 bytes, use -b option to extract)
Color Space                     : Uncalibrated
Exif Image Width                : 5347
Exif Image Height               : 6348
Image Size                      : 5347x6348
Megapixels                      : 33.9

If we run it on a GeoTIFF, we see some extra stuff:

$ exiftool Atlanta_1928_Sheet45.tif
…
Rows Per Strip                  : 64
…
Geo Tiff Version                : 1.1.0
GT Model Type                   : Projected
GT Raster Type                  : Pixel Is Area
GT Citation                     : NAD83 Georgia State Planes, West Zone, US Foot
Geographic Type                 : NAD83
Geog Citation                   : NAD83
Geog Geodetic Datum             : North American Datum 1983
Geog Angular Units              : Angular Degree
Geog Ellipsoid                  : GRS 1980
Geog Semi Major Axis            : 6378137
Geog Inv Flattening             : 298.257222100911
Projected CS Type               : Unknown (2240)
Projection                      : User Defined
Proj Coord Trans                : Transverse Mercator
Proj Linear Units               : Linear Foot US Survey
Proj Nat Origin Long            : -84.1666666666667
Proj Nat Origin Lat             : 30
Proj False Easting              : 699999.999999999
Proj False Northing             : 0
Proj Scale At Nat Origin        : 0.9999
Image Size                      : 5364x6328
Megapixels                      : 33.9

And let’s look at what gdalinfo has to say:

$ gdalinfo Atlanta_1928_Sheet45.tif
Driver: GTiff/GeoTIFF
Files: Atlanta_1928_Sheet45.tif
Size is 5364, 6328
Coordinate System is:
PROJCS["NAD83 / Georgia West (ftUS)",
    GEOGCS["NAD83",
        DATUM["North_American_Datum_1983",
            SPHEROID["GRS 1980",6378137,298.2572221009113,
                AUTHORITY["EPSG","7019"]],
            AUTHORITY["EPSG","6269"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4269"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",30],
    PARAMETER["central_meridian",-84.1666666666667],
    PARAMETER["scale_factor",0.9999],
    PARAMETER["false_easting",699999.9999999991],
    PARAMETER["false_northing",0],
    UNIT["US survey foot",0.3048006096012192,
        AUTHORITY["EPSG","9003"]],
    AUTHORITY["EPSG","2240"]]
Origin = (2232544.649310299195349,1362031.125677362782881)
Pixel Size = (0.668734999999993,-0.668735000000005)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  ( 2232544.649, 1362031.126) ( 79d 8' 0.51"W, 33d38'30.07"N)
Lower Left  ( 2232544.649, 1357799.371) ( 79d 8' 2.94"W, 33d37'48.36"N)
Upper Right ( 2236131.744, 1362031.126) ( 79d 7'18.24"W, 33d38'28.34"N)
Lower Right ( 2236131.744, 1357799.371) ( 79d 7'20.68"W, 33d37'46.64"N)
Center      ( 2234338.197, 1359915.248) ( 79d 7'40.59"W, 33d38' 8.35"N)
Band 1 Block=5364x64 Type=Byte, ColorInterp=Red
Band 2 Block=5364x64 Type=Byte, ColorInterp=Green
Band 3 Block=5364x64 Type=Byte, ColorInterp=Blue

Strips

TIFF image data can be organized into strips for faster random access and efficient I/O buffering.¹

Looking at the “Rows Per Strip” tag in the exiftool output, we can see that, prior to georeferencing, our TIFF had strips that were 16 rows/pixels tall. Our GeoTIFF’s strips are 64 rows/pixels tall. gdalinfo confirms that:

Band 1 Block=5364x64 Type=Byte, ColorInterp=Red

This will help the speed that our maps are loaded. Rather than loading all 33,943,392 pixels at once, the image will be rendered in 343,296 pixel strips. But what if the only part of the map that is requested is one corner? The client will get the whole strip and a lot of unwanted data. The main point of using WMS and slippy maps is only serving the tile(s) that are requested.

Tiles

For low-resolution to medium-resolution images, the standard TIFF method of breaking the image into strips is adequate. However high-resolution images can be accessed more efficiently—and compression tends to work better—if the image is broken into roughly square tiles instead of horizontally wide, but vertically narrow, strips.²

Above, we see with gdalinfo that each band’s block spans the whole image and in the exiftool output, our GeoTIFF has no internal tiling. The standard tile size used by OpenStreetMap, Google Maps, etc. is 256 x 256. In the previous post, we ran gdalwarp on our GeoTIFF we generated tiles that are 256 x 256 -co 'TILED=YES' -co 'BLOCKXSIZE=256'³

The full command from our previous post:

$ gdalwarp -s_srs <source ESPG> -t_srs <target EPSG> -r average -co 'TILED=YES/NO' -co 'BLOCKXSIZE=XXX' -co 'BLOCKYSIZE=XXX' -co 'COMPRESS=JPEG' </path/to/source/geo.tif> </path/to/new/geo.tif>

Running exiftool on the warped GeoTIFF shows:

Tile Width    : 256
Tile Length   : 256

And gdalinfo shows:

Band 1 Block=256x256 Type=Byte, ColorInterp=Red
Band 2 Block=256x256 Type=Byte, ColorInterp=Green
Band 3 Block=256x256 Type=Byte, ColorInterp=Blue

Overviews aka Subfiles

A TIFF file can contain multiple subfiles⁴. Using gdaladdo we can create multiple subfiles of our map at different resolutions. GDAL calls subfiles overviews. Let’s look back at our example:

gdaladdo --config GDAL_TIFF_OVR_BLOCKSIZE 256 -r average processed/atlanta_1928_sheet45.tif 2 4 8 16 32

gdaladdo tiles the map again and defaults to 128. You can set GDAL_TIFF_OVR_BLOCKSIZE as an environment variable, but here we’re just passing it in as an option. The -r is choosing the resampling algorithm for the overviews. We played around with a few and landed on average. You might want to try others.

Finally, that list of numbers at the end is the amount the image will be reduced: 2 will reduce by half, 4 will reduce by a quarter, and so on.

Let’s look at what happened by looking at the gdalinfo output:

Band 1 Block=256x256 Type=Byte, ColorInterp=Red
  Overviews: 2673x3186, 1337x1593, 669x797, 335x399, 168x200
Band 2 Block=256x256 Type=Byte, ColorInterp=Green
  Overviews: 2673x3186, 1337x1593, 669x797, 335x399, 168x200
Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
  Overviews: 2673x3186, 1337x1593, 669x797, 335x399, 168x200

Just for fun and to prove those low resolution versions really do exist, let’s use a bunch of exiftool options to inspect them (thanks Kyle Fenton!) The output is rather verbose, so we’ll really just show the important stuff.

$ exiftool -s -a -e -G5 atlanta_1928_sheet45.tif.tif
[ExifTool]      ExifToolVersion                 : 10.02
[System]        FileName                        : atlanta_1928_sheet45.tif
…
[TIFF-IFD0]     ImageWidth                      : 5345
[TIFF-IFD0]     ImageHeight                     : 6371
…
[TIFF-IFD2]     ImageWidth                      : 1337
[TIFF-IFD2]     ImageHeight                     : 1593
…
[TIFF-IFD4]     ImageWidth                      : 335
[TIFF-IFD4]     ImageHeight                     : 399
…
[TIFF-IFD5]     SubfileType                     : Reduced-resolution image
[TIFF-IFD5]     ImageWidth                      : 168
[TIFF-IFD5]     ImageHeight                     : 200
[TIFF-IFD5]     BitsPerSample                   : 8 8 8
[TIFF-IFD5]     Compression                     : Uncompressed
[TIFF-IFD5]     PhotometricInterpretation       : RGB
[TIFF-IFD5]     SamplesPerPixel                 : 3
[TIFF-IFD5]     PlanarConfiguration             : Chunky
[TIFF-IFD5]     TileWidth                       : 256
[TIFF-IFD5]     TileLength                      : 256
[TIFF-IFD5]     TileOffsets                     : 142744474
[TIFF-IFD5]     TileByteCounts                  : 196608
[TIFF-IFD5]     SampleFormat                    : Unsigned; Unsigned; Unsigned

Our process and understanding of these concepts have developed and evolved though collaboration, research, and a lot of trial and error. We would love any feedback and suggestions on how the process could be improved.

http://www.awaresystems.be/imaging/tiff/tifftags/rowsperstrip.html ↩
See page 66 of the TIFF spec document ↩
The -co stand for “creation option” and these are specific to the file format. All the options for GeoTIFFs, and all sorts of good info, can be found at http://www.gdal.org/frmt_gtiff.html ↩
See page 16 of the TIFF spec document. ↩

Bringing Your Maps into Focus

Mon, 13 Jun 2016 09:00:00 -0400

Here at Emory, we have some pretty cool old maps of Atlanta that we wanted to share in a more interactive way. But when we tried, the maps looked like this:

See the Pen eExoyO by Jay Varner (@jayvarner) on CodePen.

After a good bit of research, lots of trial and error, and working with our colleagues Eric Willoughby at Georgia State University and Amanda Henley at the University of North Carolina, we came up with a process that gave us this:

See the Pen brzJLx by Jay Varner (@jayvarner) on CodePen.

What We Did

We georeferenced our map in the appropriate coordinate system for the map’s area. In our case EPSG:2240. We’ll refer to specific coordinate systems with their EPSG code.

Next we took the resulting GeoTIFF and uploaded it to our GeoServer so the map could be shared via WMS.

Finally, like the examples above, we used Leaflet to display the maps in the browser overlaid on a modern street map.

What We Did Wrong

Our first problem was that the GeoTIFF is a high resolution image. In the first example on this page, we were squeezing a 5,931 x 5,779 pixel image (the full resolution) into a 326 x 276 pixel box (for the zoom level shown above). There’s just too much data to display clearly. Notice that if you zoom all the way in to our original example, the map looks great.

See the Pen OjdGvE by Jay Varner (@jayvarner) on CodePen.

Our other problem was that we used a coordinate system that is inappropriate for web display. Tiled base maps that are provided by organizations like OpenSteetMap are based on EPSG:3857, aka WGS 84 Web Mercator.

GDAL Can Solve those Problems

“GDAL is a translator library for raster and vector geospatial data formats…” This is our three step process.

gdalwarp

We use gdalwarp to reproject, resample, add internal tiling, and compress¹

Reproject

When you georeference a scanned map, you select a projection. For most GIS needs, you will want to use the projection appropriate for the chunk of Earth your map covers. However, for displaying maps on the web, you want to use EPSG:3857².

Resample

gdalwarp offers various algorithms³ for resampling. We suggest you try a few and see what works best. We’ve had good results using average, but many folks will suggest near.

Tiling

GeoTIFFs are organized in 1 pixel strips by default. OpenStreetMap tiles are 256x256. We match OpenStreetMap by adding internal tiles. We’ll go deeper into tiling in our follow up post.

Compress

GeoTIFFs can be huge and it’s not very nice to make users load large files when they don’t have to. GDAL can also compress our GeoTIFFs. There are many compression algorithms, but we’re pretty happy with the results we’re getting with JPEG.

The Command

Here is an example showing how to set all the options. Note the last argument is a path to a file the command will create. Your original GeoTIFF will be preserved.

Syntax

$ gdalwarp -s_srs <source ESPG> -t_srs <target EPSG> -r average -co 'TILED=YES/NO' -co 'BLOCKXSIZE=XXX' -co 'BLOCKYSIZE=XXX' -co 'COMPRESS=ABC' </path/to/source/geo.tif> </path/to/new/geo.tif>

Example

$ gdalwarp -s_srs EPSG:2240 -t_srs EPSG:3857 -r average -co 'TILED=YES' -co 'BLOCKXSIZE=256' -co 'BLOCKYSIZE=256' -co 'COMPRESS=JPEG' atlanta_1928_sheet45.tif processed/atlanta_1928_sheet45.tif

NOTE: We are creating a new copy of the GeoTIFF in a directory called “processed”. You can save it where ever you like.

Now you have a new GeoTIFF projected in Web Mercator and preserved your original.

Add Overviews

We discuss overviews in greater depth in our follow-up post. Remember when we said the main reason the map looked so bad was because there was just too much data? Well, overviews fix that. What happens here is we create internal versions of our image that are smaller and lower resolution.

We also have to tell it to keep our blocks, or internal tiles, at 256, the default is 128. You can set GDAL_TIFF_OVR_BLOCKSIZE as an environment variable, or just override the default with the --config option.

Syntax

$ gdaladdo --config GDAL_TIFF_OVR_BLOCKSIZE XXX -r average </path/to/new.tif> levels (list of numbers)

Example

$ gdaladdo --config GDAL_TIFF_OVR_BLOCKSIZE 256 -r average processed/atlanta_1928_sheet45.tif 2 4 8 16 32

Automation

Obviously if you have more than two maps to process, running these commands on each will get real old real fast. Fortunately, since GDAL is a command line tool, we can automate the whole process. There are wrappers for GDAL in many languages: Python, PHP, .Net, Ruby (though the Ruby gem is no longer maintained and not updated for GDAL 2), etc.

We have automated our process. It’s nice to just run a script and go enjoy some coffee while the computer does all the work.

Conclusion

This has been an evolving process. We found ways to improve the process every time we prepared for a workshop, conference talk, or writing this post. We would love to hear any feedback or suggestions.

In previous talks and workshops, we presented a three-step process where we did the tiling and compressed the image using gdal_translate. While writing this post we realized a way to combine these processes with gdalwarp. ↩
There is confusion between EPSG:3857 and EPSG:4326. For displaying raster data using something like Leaflet, OpenLayers, etc. 3857 will result in a much clearer image. Here are two links that helped us understand. http://gis.stackexchange.com/a/48952 and http://www.faqoverflow.com/gis/48949.html ↩
You can see a list of available algorithms by running man gdadalwarp and reading the explanation of the -r option. man is short for manual. Use the space bar to scroll through the text and type q to quit the man page viewer. Or you can read it here. ↩