KABT BioBlog

This morning, I saw a twitter post on an interesting new free phone app.  Here is a brief video on the app posted at YouTube:

and here is what the developers say on their website:

Leafsnap is the first in a series of electronic field guides being developed by researchers from Columbia University, the University of Maryland, and the Smithsonian Institution.  These researchers are working on visual recognition software to help identify species from photographs. Leafsnap is the first in a series of electronic field guides being developed to demonstrate this new technology.  This free mobile app helps identify tree species from photographs of their leaves and contains beautiful high-resolution images of their flowers, fruit, petiole, seeds, and bark. Leafsnap currently includes the trees of New York City and Washington, D.C., and will soon grow to cover the trees of the entire continental United States.

Leafsnap turns users into citizen scientists, automatically sharing images, species identifications, and geo-coded stamps of species locations with a community of scientists who will use the stream of data to map and monitor the ebb and flow of flora nationwide.

The genesis of Leafsnap was the realization that many techniques used for face recognition developed by Professor Peter Belhumeur and Professor David Jacobs, of the Computer Science departments of Columbia University and the University of Maryland, respectively, could be applied to automatic species identification.Professors Jacobs and Belhumeur approached Dr. John Kress, Chief Botanist at the Smithsonian, to start a collaborative effort for designing and building such a system for plant species. Columbia and the University of Maryland designed and implemented the visual recognition system used for automatic identification. In addition, Columbia University designed and wrote the iPhone, iPad, and Android apps, the leafsnap.com website, and wrote the code that powers the recognition servers. The Smithsonian was instrumental in collecting the datasets of leaf species and supervising the curation efforts throughout the course of the project. As part of this effort, the Smithsonian contracted the not-for-profit nature photography group Finding Species, which collected and photographed the high-quality photos available in the apps and the website.

Once downloaded and opened, one can challenge oneself to one of a number of identification “Games”, “Browse” the current field guide of 184 plants, set particular “Options”, and take a picture using the phone’s built in camera that should automatically upload to the leafsnap database via “Snap It!”.  Once you have uploaded images, I assume you can view your own “Collection”.

Once you are browsing, you can view the species contained in the app using their common or scientific name, in a manner familiar to iPhone users and also used by National Geographic’s bird identification app.  By clicking on the image icon in the upper left-hand corner of this screen, the app toggles between displays of a representative leaf, flower, or fruit associated with each tree specimen, a nice feature that could help to confirm one’s hypothesis of identification.

Once you select a particular species, you can view a collection of detailed images of the trees leaves, flowers, fruits, seeds, and bark, or use the image icon in the upper right-hand corner to toggle to a dispaly basic information on the species being viewed.

The games section includes timed quizzes that test one’s ability to identify tree specimens by leaf, flower, or fruit in the “Leaf Challenge”.  The second image below is a screen shot of this game.  The name of the tree leaf to identify with a finger tap is noted, while the number correct and time left are noted in the upperleft and right-hand corners of the screen, respectively. 

In “Green Sweep” you are given four leaves to identify that are floating around the screen.  You identify each by moving them into their appropriately labeled square, with a touch and a drag.  If correct, it challenges you with another four leaves to identify, and continues in this manner until your time is up.  Each of the game categorizes your exepertise based on your score at the end of the game.

In using ”Snap It!” the app uses the phone’s built in camera to take a picture which is then uploaded for identification by the recognition software described previously. 

Sadly, I have not been able to get this function of the app to work completely.   I went out a tried to take a picture of a redbud leave only to realize that your are supposed to take the picture of the leaf on a white background (I hadn’t read the top of the screen).  The app realized this though and informed me which was demonstrates its sophistication. 

After correcting this mistake, I had issues with the app crashing when I took a picture within the app.  So, I exited the app and took the picture within phone’s camera app, and selected to chose that image from within leafsnap.  This time, I experienced crashes while the app attempted to upload the image for identification.  I even changed the image resolution to small in the options setting to no avail.

With the number of people working on this app, I imagine that such problems will be solved in a short time.  I only wish they had a way to report bugs via the app or on their website.  If and when I get this aspect of the app working, I will edit the post and make a comment to report changes.

I look forward to using this app with my students!

Addendum:

Since the time of the orignal post, I e-mailed leafsnap, noticed that an update was posted on twitter, downloaded the update, and tried to upload an image of a Redbud from my back yard again.  This time it worked!

After uploading the image, it loads results to your phone.  The results are ordered, I assume by some probability of it being a match with the outline of the uploaded leaf image (you can see the top three results in the middle image above).  If you click on the “map” button at the top of the page it shows where the leaf was collected (dang, now you know where I live).

To verify a particular result, one finger swipes across the chosen selection to bring up the red label button (first image above), and selects the “label” button by touching it (second image above).  Once this is done, the labelled image now appears in your “Collection.”

Pretty cool! 

Now, I challenge you to be the second to add an image from the Kansas landscape!