Mar
17
2011
17
Theodolite HD Arrives for iPad 2
I am thrilled to announce that my Theodolite augmented reality app is finally available for a tablet device with the arrival of iPad 2. Customers have been clamoring for this capability, and it's great to see everything come together. Theodolite HD has been an adventure to develop from start to finish.
I began working on Theodolite HD about 10 months ago, long before a camera-equipped iPad was even a rumor. I could have released this app for the original iPad, but it's kind of hard to pitch an augmented reality app without the "reality" provided by cameras. That's more like augmented nothing. I really didn't want to develop a kludgey universal app that was crippled on the iPad, like some have done.
So, while waiting for a future iPad to come along with a camera, I focused on graphics and mapping functions, which could be developed and tested on the original iPad. All the apps in the Theodolite series use OpenGL for graphics, since it's the most reliable and efficient way to get fast rendering. OpenGL lets me knit calculations and graphics together on a very basic code level for optimum performance.
When the iPhone 4 came along, I overhauled the graphics engine in Theodolite so that the app could generate its own resolution-independent artwork and textures internally, on startup (using Core Graphics). This gives me a lot of flexibility to accomodate future devices without having to sit in front of a graphics app on my computer and re-render artwork for different scale factors. My initial plan was to use scale factor 1.0 graphics in the iPad version of the app, since the iPad screen has a scale factor of 1.0 (see note #1 below). However, this just did not look right. The same graphics that were sized perfectly for an iPhone-sized device looked too small and ingrown on the iPad, despite possessing good touch target size, usability, and readability. In short, it didn't really take advantage of the iPad's larger screen like I wanted.
So after thinking about it for a while, I had the idea to trick the app into thinking the iPad actually had a scale factor 2.0 screen, thus letting it take advantage of all the graphics capability I had orginally developed for the iPhone 4. I did this by setting up my OpenGL working space parameters so that there were 1024x768 pixels allocated over a 512x384 point drawing space. From there, Theodolite's graphics engine took care of the rest, creating all OpenGL textures for a scale factor of 2.0. The end result is a crisp interface that scales nicely to take advantage of the iPad's larger real estate:
The size and layout of the interface will be great for all the applications I have in mind: usage in vehicle cockpits, jobsite surveys, sports, field research, architectural planning, engineering measurements, and educational demonstrations (to name a few). This puts the app on par with a typical "clipboard" scale that people are used to. Though the iPad device is not really ergonomic for photography, I think it will be outstanding for augmented reality applications. Both the background camera preview and the content overlay are larger and more life sized, and this gives Theodolite more of a life-like feel than you get on a smaller iPhone screen.
When the iPad 2 was announced in early March, I dusted off my code and prepared to tidy up any loose ends once I had an actual iPad 2 in my hands. Among other things, I needed to determine how to identify the device at app launch, and I also needed to measure the camera field of view so that I could properly calibrate Theodolite's optical rangefinder. It's unfortunate that Apple doesn't give developers early access to new products, and it's even more unfortunate that nobody could pre-order the iPad 2 this time around. Thus, the wait began.
As I counted down the days to March 11, I planned my strategy to score an iPad on launch day. Within a 50 mile radius of my home in tidewater Virginia, I have two Apple stores, several Best Buys, numerous Targets and Wal-Marts, and 4-5 Verizon and AT&T stores. Knowing that the Apple stores always have huge lines, I took those off the list since I have a day job and couldn't get in line at 5am (nor would I want to). So I planned a big loop through all the remaining stores starting at around 3:30pm.
First stop was at Target #1. No iPads had been delivered yet, and it wasn't looking good there. So I proceeded to the closer of two Best Buy stores in my area. I was shocked to see a line of about 50 people there, composed of moms and dads, kids, blue collar workers in their work uniforms, grandparents, seniors, you name it. Not a nerdy gadget crowd at all, just a bunch of really excited regular people. That's when it hit me that this launch was huge.
After about 15 minutes in line, a Best Buy manager announced that they only had 30 WiFi models in stock. Since the line was way longer than 30 people and I really wanted a 3G model (mainly for access to GPS hardware), I split and headed for the next stop in line, an AT&T store. When I arrived, the store was empty except for two employees. There weren't even any crickets chirping, that's how quiet it was. As I bolted through the door in a rush, both employees gave me a look of resignation, as though I was another in a long line of iPad seekers soon to be disappointed. And I was -- they had no iPads in stock, and none on the horizon.
OK, next stop Verizon. There was no line outside the store, but inside it was a madhouse with about 20-30 people milling around in a pretty small space. They had the iPad 2 in stock, and were taking names and giving out tickets. By now it was 4:50 and the air was so thick with excitement I could taste it (kind of like a Cool Mint Cliff Bar actually). It looked like I was marginal for a 64GB model (that was all that was left for grabs by the time I got there). The sales lady got a laugh when she asked me which model/color I was looking for and I responded "3G" to every question. So I waited.
At about 4:55, my wife, my wonderful sweet wife, who has an iPhone and iPad but doesn't care about nerdy gadget stuff, app development, or product launches, called me from Target #2 where she was shopping for household items on her way home from work. They had an unclaimed white 32GB AT&T 3G model. Did I want it? HELL YES! So I skipped over to the Verizon sales lady, told her to take my name off the list, gave her a big kiss (OK, not really), and then windmilled my arms as I ran back to my car. Just then, my wife texted me this photo:
I am more of an even-number person myself, always have been, but that number 3 looked pretty awesome. By the time I got back to the electronics counter at Target, 5:00 had come and gone, and my wife was wrapping up at the checkout. This particular Target received five iPads, and they were all sold by 5:03. As it turns out, this Target store is less than a mile from my house, and was to be the last stop on my 35 mile iPad 2 shopping tour between work and home. If it wasn't for my wife, I would have probably missed the boat. As I gave her a big kiss (really), she said "what -- it's shopping at Target, I was made for this". What a gal.
I got home around 5:15, cracked open a beer, threw a sip over my shoulder for good luck, and setup the new iPad 2 with iTunes. By 5:30 I was ready to compile in Xcode, after about 5 minutes of dicking around with certificates and device provisioning (this always strikes me as a very un-Apple like process even when it goes smoothly). I only had one surprise, and that's the way the iPad 2 is identified in software. The previous iPad showed up as "iPad1,1" regardless of the flavor (which was annoying, as I previously had a need to tell WiFi and 3G models apart for another app project). I was expecting the new iPad 2 to show up as "iPad2,1" following a pattern of past iPhone and iPod devices. Well, my white AT&T model showed up as "iPad2,2". Presumably, Apple now separates out the WiFi, GSM 3G, and CDMA 3G models using this scheme (I suspect it's iPad2,1, iPad2,2, and iPad2,3 respectively).
Next I moved on to measuring the camera's field of view. It would be great if Apple released these specs, or made it a parameter we could query from the SDK (I think I'll submit a feature request for that). In the absence of hard info, I usually take a look at available camera hardware from suppliers, and back it up with measurements. Based on some research, I suspect the iPad 2 uses the OmniVision OV9740 for its back-facing camera, or something very similar. This unit has a 2.296x1.281 mm image sensor array with 1.75 micron elements (for 1312x732 dimensions). It is intended to be a 720p HD sensor (1280x720 pixels) for video recording. For stills, Apple uses the center 4:3 portion of the sensor array to take 960x720 pixel images.
For proper field of view, I always verify specs with some simple tests. One such test is shown in the image below, where I used the iPad 2 to take some screenshots in Theodolite while aiming at a simple right-angle ruler jig I setup on my kitchen counter:
Using some basic trigonometry, this allowed me to determine that 4:3 stills taken with the iPad 2 back camera have an approximate 34.1 degree vertical field of view and an approximate 44.5 degree horizontal field of view. This equates roughly to a hypothetical 43mm focal length lens on a 35mm camera. With the specs calculated, I was able to button up Theodolite HD and upload it for submission to the iTunes App Store around 6:30. Not bad. The app was approved and went on sale four days later on March 15.
As for the camera itself, well, it's pretty lousy and is definiteley the iPad 2's weak point. As I mentioned before, a tablet form factor isn't really ideal for photography (borderline ridiculous in fact) so Apple likely concluded that a high resolution camera wasn't real important. I was expecting the camera to be OK for augmented reality, but it's just barely OK. You'd expect the main issue to be the fact that the camera's 720 pixel vertical resolution is less than the 768 pixel screen height in landscape, but that turns out to not be a big deal. Rather, the camera just doesn't capture great imagery unless the ambient lighting is good and the range of colors, darks, and highlights is narrow. I think Apple really needs to improve the camera next time around, not for the sake of photography, but for the sake of other uses like augmented reality.
Most of my other comments about the iPad 2 echo what you've probably read elsewhere. It's thinner, lighter, faster, better looking, and overall nicer to use than the first iPad. I wouldn't have picked white as a first choice, but now that I have a white unit, I really like the way it looks -- it's classy, and I do not find the bezel to be distracting as some have suggested. While I do like the new iPad's shape and find it more comfortable to hold overall (due to smooth edges), there is one area where the new iPad is less comfortable to me, and that's the radius of the four corners of the device. The iPad 2 has about half the corner radius of the first generation iPad, and I find that those points dig into my palms in certain situations.
And Theodolite HD? Well, it's just plain cool on the iPad 2. I think customers are going to love it. In addition to the larger layout, the graphics performance is outstanding. The GPU on the iPad 2 renders all of Theodolite's complex graphics with ease, well in excess of my desired 50 FPS frame rate. Photo capture and rendering is blazingly fast. The app is just plain great on this new device. If you're interested, check out the Theodolite HD website, or view the app's iTunes listing. Theodolite HD is off to a good start, hovering between #3 and #4 in the iPad paid Navigation category in the US App Store as I write this.
Notes:
[1] Scale factor expresses the relationship between screen pixels and points in the drawing space; the iPad and iPhone 3GS are scale factor of 1.0 devices since pixels and points map 1:1. The iPhone 4 is scale factor 2.0 since there are two pixels on the 960x640 pixel Retina Display for every point dimension in the 480x320 point drawing space. [Return to article]
I began working on Theodolite HD about 10 months ago, long before a camera-equipped iPad was even a rumor. I could have released this app for the original iPad, but it's kind of hard to pitch an augmented reality app without the "reality" provided by cameras. That's more like augmented nothing. I really didn't want to develop a kludgey universal app that was crippled on the iPad, like some have done.
So, while waiting for a future iPad to come along with a camera, I focused on graphics and mapping functions, which could be developed and tested on the original iPad. All the apps in the Theodolite series use OpenGL for graphics, since it's the most reliable and efficient way to get fast rendering. OpenGL lets me knit calculations and graphics together on a very basic code level for optimum performance.
When the iPhone 4 came along, I overhauled the graphics engine in Theodolite so that the app could generate its own resolution-independent artwork and textures internally, on startup (using Core Graphics). This gives me a lot of flexibility to accomodate future devices without having to sit in front of a graphics app on my computer and re-render artwork for different scale factors. My initial plan was to use scale factor 1.0 graphics in the iPad version of the app, since the iPad screen has a scale factor of 1.0 (see note #1 below). However, this just did not look right. The same graphics that were sized perfectly for an iPhone-sized device looked too small and ingrown on the iPad, despite possessing good touch target size, usability, and readability. In short, it didn't really take advantage of the iPad's larger screen like I wanted.
So after thinking about it for a while, I had the idea to trick the app into thinking the iPad actually had a scale factor 2.0 screen, thus letting it take advantage of all the graphics capability I had orginally developed for the iPhone 4. I did this by setting up my OpenGL working space parameters so that there were 1024x768 pixels allocated over a 512x384 point drawing space. From there, Theodolite's graphics engine took care of the rest, creating all OpenGL textures for a scale factor of 2.0. The end result is a crisp interface that scales nicely to take advantage of the iPad's larger real estate:
The size and layout of the interface will be great for all the applications I have in mind: usage in vehicle cockpits, jobsite surveys, sports, field research, architectural planning, engineering measurements, and educational demonstrations (to name a few). This puts the app on par with a typical "clipboard" scale that people are used to. Though the iPad device is not really ergonomic for photography, I think it will be outstanding for augmented reality applications. Both the background camera preview and the content overlay are larger and more life sized, and this gives Theodolite more of a life-like feel than you get on a smaller iPhone screen.
When the iPad 2 was announced in early March, I dusted off my code and prepared to tidy up any loose ends once I had an actual iPad 2 in my hands. Among other things, I needed to determine how to identify the device at app launch, and I also needed to measure the camera field of view so that I could properly calibrate Theodolite's optical rangefinder. It's unfortunate that Apple doesn't give developers early access to new products, and it's even more unfortunate that nobody could pre-order the iPad 2 this time around. Thus, the wait began.
As I counted down the days to March 11, I planned my strategy to score an iPad on launch day. Within a 50 mile radius of my home in tidewater Virginia, I have two Apple stores, several Best Buys, numerous Targets and Wal-Marts, and 4-5 Verizon and AT&T stores. Knowing that the Apple stores always have huge lines, I took those off the list since I have a day job and couldn't get in line at 5am (nor would I want to). So I planned a big loop through all the remaining stores starting at around 3:30pm.
First stop was at Target #1. No iPads had been delivered yet, and it wasn't looking good there. So I proceeded to the closer of two Best Buy stores in my area. I was shocked to see a line of about 50 people there, composed of moms and dads, kids, blue collar workers in their work uniforms, grandparents, seniors, you name it. Not a nerdy gadget crowd at all, just a bunch of really excited regular people. That's when it hit me that this launch was huge.
After about 15 minutes in line, a Best Buy manager announced that they only had 30 WiFi models in stock. Since the line was way longer than 30 people and I really wanted a 3G model (mainly for access to GPS hardware), I split and headed for the next stop in line, an AT&T store. When I arrived, the store was empty except for two employees. There weren't even any crickets chirping, that's how quiet it was. As I bolted through the door in a rush, both employees gave me a look of resignation, as though I was another in a long line of iPad seekers soon to be disappointed. And I was -- they had no iPads in stock, and none on the horizon.
OK, next stop Verizon. There was no line outside the store, but inside it was a madhouse with about 20-30 people milling around in a pretty small space. They had the iPad 2 in stock, and were taking names and giving out tickets. By now it was 4:50 and the air was so thick with excitement I could taste it (kind of like a Cool Mint Cliff Bar actually). It looked like I was marginal for a 64GB model (that was all that was left for grabs by the time I got there). The sales lady got a laugh when she asked me which model/color I was looking for and I responded "3G" to every question. So I waited.
At about 4:55, my wife, my wonderful sweet wife, who has an iPhone and iPad but doesn't care about nerdy gadget stuff, app development, or product launches, called me from Target #2 where she was shopping for household items on her way home from work. They had an unclaimed white 32GB AT&T 3G model. Did I want it? HELL YES! So I skipped over to the Verizon sales lady, told her to take my name off the list, gave her a big kiss (OK, not really), and then windmilled my arms as I ran back to my car. Just then, my wife texted me this photo:
I am more of an even-number person myself, always have been, but that number 3 looked pretty awesome. By the time I got back to the electronics counter at Target, 5:00 had come and gone, and my wife was wrapping up at the checkout. This particular Target received five iPads, and they were all sold by 5:03. As it turns out, this Target store is less than a mile from my house, and was to be the last stop on my 35 mile iPad 2 shopping tour between work and home. If it wasn't for my wife, I would have probably missed the boat. As I gave her a big kiss (really), she said "what -- it's shopping at Target, I was made for this". What a gal.
I got home around 5:15, cracked open a beer, threw a sip over my shoulder for good luck, and setup the new iPad 2 with iTunes. By 5:30 I was ready to compile in Xcode, after about 5 minutes of dicking around with certificates and device provisioning (this always strikes me as a very un-Apple like process even when it goes smoothly). I only had one surprise, and that's the way the iPad 2 is identified in software. The previous iPad showed up as "iPad1,1" regardless of the flavor (which was annoying, as I previously had a need to tell WiFi and 3G models apart for another app project). I was expecting the new iPad 2 to show up as "iPad2,1" following a pattern of past iPhone and iPod devices. Well, my white AT&T model showed up as "iPad2,2". Presumably, Apple now separates out the WiFi, GSM 3G, and CDMA 3G models using this scheme (I suspect it's iPad2,1, iPad2,2, and iPad2,3 respectively).
Next I moved on to measuring the camera's field of view. It would be great if Apple released these specs, or made it a parameter we could query from the SDK (I think I'll submit a feature request for that). In the absence of hard info, I usually take a look at available camera hardware from suppliers, and back it up with measurements. Based on some research, I suspect the iPad 2 uses the OmniVision OV9740 for its back-facing camera, or something very similar. This unit has a 2.296x1.281 mm image sensor array with 1.75 micron elements (for 1312x732 dimensions). It is intended to be a 720p HD sensor (1280x720 pixels) for video recording. For stills, Apple uses the center 4:3 portion of the sensor array to take 960x720 pixel images.
For proper field of view, I always verify specs with some simple tests. One such test is shown in the image below, where I used the iPad 2 to take some screenshots in Theodolite while aiming at a simple right-angle ruler jig I setup on my kitchen counter:
Using some basic trigonometry, this allowed me to determine that 4:3 stills taken with the iPad 2 back camera have an approximate 34.1 degree vertical field of view and an approximate 44.5 degree horizontal field of view. This equates roughly to a hypothetical 43mm focal length lens on a 35mm camera. With the specs calculated, I was able to button up Theodolite HD and upload it for submission to the iTunes App Store around 6:30. Not bad. The app was approved and went on sale four days later on March 15.
As for the camera itself, well, it's pretty lousy and is definiteley the iPad 2's weak point. As I mentioned before, a tablet form factor isn't really ideal for photography (borderline ridiculous in fact) so Apple likely concluded that a high resolution camera wasn't real important. I was expecting the camera to be OK for augmented reality, but it's just barely OK. You'd expect the main issue to be the fact that the camera's 720 pixel vertical resolution is less than the 768 pixel screen height in landscape, but that turns out to not be a big deal. Rather, the camera just doesn't capture great imagery unless the ambient lighting is good and the range of colors, darks, and highlights is narrow. I think Apple really needs to improve the camera next time around, not for the sake of photography, but for the sake of other uses like augmented reality.
Most of my other comments about the iPad 2 echo what you've probably read elsewhere. It's thinner, lighter, faster, better looking, and overall nicer to use than the first iPad. I wouldn't have picked white as a first choice, but now that I have a white unit, I really like the way it looks -- it's classy, and I do not find the bezel to be distracting as some have suggested. While I do like the new iPad's shape and find it more comfortable to hold overall (due to smooth edges), there is one area where the new iPad is less comfortable to me, and that's the radius of the four corners of the device. The iPad 2 has about half the corner radius of the first generation iPad, and I find that those points dig into my palms in certain situations.
And Theodolite HD? Well, it's just plain cool on the iPad 2. I think customers are going to love it. In addition to the larger layout, the graphics performance is outstanding. The GPU on the iPad 2 renders all of Theodolite's complex graphics with ease, well in excess of my desired 50 FPS frame rate. Photo capture and rendering is blazingly fast. The app is just plain great on this new device. If you're interested, check out the Theodolite HD website, or view the app's iTunes listing. Theodolite HD is off to a good start, hovering between #3 and #4 in the iPad paid Navigation category in the US App Store as I write this.
Notes:
[1] Scale factor expresses the relationship between screen pixels and points in the drawing space; the iPad and iPhone 3GS are scale factor of 1.0 devices since pixels and points map 1:1. The iPhone 4 is scale factor 2.0 since there are two pixels on the 960x640 pixel Retina Display for every point dimension in the 480x320 point drawing space. [Return to article]