Citizen Science Can Improve Visitor Experiences and Research Outcomes in Museums and Cultural Institutions

  • Alexis Garretson

Applying citizen science protocols to imaged museum specimens can help lessen the digitization burden on research staff members and address research gaps in existing citizen science datasets while providing meaningful visitor engagement opportunities. Citizen science is still a relatively young field, but because museum collections are often substantially older, pairing the two can allow us to ask questions on a much broader time scale. Applying consistent protocols to citizen science observations and museum specimens enables high-quality derived data that can be quickly combined without uncertainty in definitions or differences in approaches. Citizen science protocols can be well suited to use with image data because they are often developed for visual-only information (e.g., presence of flowers on a plant, the number of seeds on a page, the number of clouds in the sky). Because many collection objects are imaged as part of the digitization process, the visual information is often already available and searchable as an input.

Utilizing pre-existing citizen science protocols also allows museums to tap into a broader community of existing participants who are already familiar with the protocol. For example, the National Phenology Network has more than 15,000 citizen science observers who are already trained in applying their protocols to living specimens. Citizen scientists from active collection projects can easily transition to the digital interface and vice versa, allowing for mutually beneficial collaborations. Also, during times when citizen scientists cannot make traditional onsite or in-person observations (e.g., global pandemic, short-term disability, long-term disability), digitization and digital engagement opportunities allow for continued engagement with the observer community through citizen science opportunities. It also enables museums to continually digitally interact with users who may not view collection items in person or engage traditionally with museum infrastructure. Finally, citizen science projects and protocols often emphasize education and engagement, which means there are often pre-existing tools that can be modified for use in formal educational settings such as schools and universities and informal educational settings such as museums, libraries, and botanical gardens.

Challenges and Opportunities of Collections-Based Research

Museums and other cultural institutions are rich storehouses of historical information, mainly through objects, documents, and physical specimens. Access to museum collections enables research in diverse fields, including anthropology, paleontology, art history, and women’s studies. Collections-based research is of high and increasing importance to researchers. In particular, natural history specimens, which include preserved insects and pressed plants, are of growing research interest within ecology and environmental sciences because they provide a historical snapshot of biodiversity, species interactions, and even genetics. Yet, researchers often need physical access to specimens or objects, which is a barrier to making collections-research more inclusive to international researchers, disabled researchers, and researchers at smaller institutions. Natural disasters, including the coronavirus pandemic, have also limited and complicated research access to museum collections and have left many collection items vulnerable to threats, including pests and environmentally driven degradation. One way collections are working to make collections more accessible is through digitization, but digitization can be time-consuming and expensive and does not directly equate to accessibility, particularly if digitized collections are not findable, accessible, interoperable, and reusable.

Presently, the best practices for digitization include the database of information about specimens and the imaging and georeferencing of collection items. Databasing involves the standardization of text-based metadata about an item like what it is (e.g., the taxonomic identification or categorization of the item), the date of item collection or creation, the identities of the collector or creator, where it was created, and other details like the preservation methods. There are multiple standardized metadata profiles in the museum setting, including Dublin Core, Darwin Core, Ecological Metadata Language, and the Federal Agencies Digital Guidelines Initiative. Often, the databasing step of digitization includes the assignment of a unique identifier, which may be a collection-specific barcode or a standardized persistent identifier for the object controlled by a domain-specific authority. While databases may provide information about the specimen’s location, georeferencing provides a specific mapped GPS point. Georeferencing is the process of converting a text-based description of a locality to a geospatial coordinate, which allows for rapid, computer-based analyses of geospatial patterns. Finally, imaging typically includes capturing, processing, and archiving a high-quality 2-D image of the specimen, often with a color scale or ruler. As the technological tools available to museums continue to grow, best practices in digitization across fields are expanding to include 3-D scans, microscope images, x-rays, genetic information, or other experimentally derived quantities.

As the scope of digitization expands, fully digitizing museum collections becomes an increasingly challenging process, both in terms of the physical costs of the technological tools and in the hours required to preserve details of their collections for future research use. Digitization can be a significant challenge, particularly at smaller institutions, and can lead to the absence of critical collections in research datasets.

Citizen Science as a Tool for Research and Education

Citizen science is a research method that fosters collaboration between professional and informal scientists. Citizen scientists can include students, museum visitors, civic educators, retired scientists or educators, volunteers, or any individuals interested in scientific work. As research becomes increasingly high-dimensional and data-driven, citizen science provides research datasets that can reach previously unachievable sample sizes, geographic extent, and temporal coverage. Simultaneously, citizen science enables meaningful engagement in the scientific process to individuals often excluded from academic research. Citizen science is quickly proliferating in the physical and natural sciences, and a growing number of projects address research questions in the social and cultural sciences. In the natural sciences, citizen science datasets have enabled broad research investigations into wide-ranging topics such as the spatial differences in dragonfly colors, water quality in local watersheds, bird population trends, and plant responses to climate change.

Beyond the research benefits, citizen science in natural science fields can encourage political actions such as environmental and conservation activities, while improving scientific literacy, public understanding of science, and scientific engagement amongst citizen scientists. Most citizen scientists are motivated to participate by a desire to contribute to scientific discovery. Participation in citizen science projects enables individuals to join a community of other citizen scientists involved in their project, learn more about the science they are involved with, and meaningfully contribute to scientific inquiry.

Many citizen science projects within ecology and environmental sciences are observation-based, with participants observing natural features (e.g., cloud cover, species presence, water conditions) in a specific area. For example, iNaturalist crowdsources photos of organisms and identifications as documentation of species presence and ecology; the OPAL air quality survey used counts and identifications of lichen on tree bark and counts of fungal infections on sycamore to better understand air quality in Great Britain; and Nature’s Notebook, a project of the USA National Phenology Network, leverages more than 15,000 volunteers to document the seasonal phenology (the timing of life stages) in plant and animal species. These projects continue to develop and proliferate across a variety of fields and research topics and provide new opportunities for research and educational partnerships.

Although citizen science projects can provide high-quality data at a regional, continental, or even global scale, many projects are limited in their temporal scope. While biological and environmental studies, particularly those of birds, have greater longevity relative to other disciplines, most citizen science projects are fewer than 15 years old. New projects are frequently under development as the availability of technology and platforms continues to advance. One way to improve this temporal coverage is to encourage participation in citizen science projects into the future, but another opportunity is to supplement citizen science data with historically collected data, including data stored in museums and libraries.

Case Study One: Collections-Based Citizen Science in Undergraduate Research

Digitized herbarium specimens are increasingly used as a source of information on the timing of plant and animal life stages such as flowering, reproduction, and the fall color change—a collection of phenomena known as plant phenology. However, classifying herbarium specimens for phenology is hugely time-consuming, leading to low sample sizes across most studies, especially compared to the enormous volume of available data. Citizen science is well suited to process large volumes of visual data that would otherwise be time-consuming to classify. There are many current phenology citizen science projects, but Nature’s Notebook is the largest, and its protocols provide species-specific, standardized definitions of the life stages of more than 1,000 species. Data contributed to Nature’s Notebook is freely available and can be visualized, aggregated, and queried from their data portal.

Over the past few years, we applied the Nature’s Notebook protocols to more than 3,000 specimens of red and sugar maples housed in more than 50 museums and herbaria across the Eastern United States. This project involved a small number of individuals, with three undergraduate interns and a graduate student leading the process. In the initial project, we did not use a public platform to conduct the classification, though we envision expanding the project to include broad digital participation in the future. Because the specimens used in this project were all digitized and imaged, we could easily integrate data from museum collections we could not have visited during the project, allowing for virtual student engagement with collections across the country.

Our work generated an integrated phenology dataset spanning more than 120 years, with details on various phenological states, including flowering times, fall coloration, and new leaf growth. This dataset allowed us to investigate the impacts of anthropogenic environmental changes such as climate change and land-use change on these important tree species. Comparatively, just using Nature’s Notebook data provided fewer than 10 years of robust data. While significant, these data became much more impactful when paired with the herbarium data. Because of these paired data, we can investigate previously impossible questions about the changes in maple tree phenology, particularly the fall color change, over the past 120 years and into the present.

We discovered changes in maple leaf production, maple reproduction, and maple interactions with other species, including insects and pathogens. Pairing these historical data with ongoing citizen science observations by Nature’s Notebook allows us to continue building our understanding of how climate change impacts the natural world and better understand our ecological and environmental past. Because these data involve protocols developed for citizen scientists and already digitized herbarium resources, this project could be operationalized on a larger scale, across the billions of herbarium specimens worldwide and those that have yet to be collected.

Digital collections offer an unparalleled opportunity for educational and classroom engagement. By supporting students in developing digital collections research projects, students build digital literacy skills while allowing them to engage in integrative and active topical exploration. Freely available lesson plans for integrating natural history collections data into classroom and independent research activities are becoming increasingly common, lowering the barrier to including these modules in digital and in-person educational experiences. Projects such as AIM-UP! (Advancing Integration of Museums into Undergraduate Programs) continue to support the development and implementation of these types of materials, but museums can directly partner with organizations to improve these resources. Developing classroom curriculum and independent research experiences for collections-driven investigations can harness existing museum and citizen science educational materials to enhance student exposure to citizen science and museum research opportunities.

Case Study Two: Digital Citizen Science and Digitization at the Mohonk Preserve

The Mohonk Preserve is a nature preserve and land trust located in the Hudson Valley region of New York. In addition to their land protection activities, the Mohonk Preserve maintains an archive, library, and physical object collection that includes more than 60,000 physical items, 14,000 notecards with natural history observations, and 9,000 photographs. These data have been used to document plant and animal phenology changes, assess impacts of acid rain on fish communities, and predict potential climate change effects in the region. While the data are valuable and expansive, the preserve has a relatively small number of conservation staff members, and few are devoted exclusively to the digital preservation of the preserve’s collections.

In addition to the archives, the Mohonk Preserve supports a robust suite of citizen science programs, including tracking the phenology of local species using iNaturalist, ongoing weather monitoring and tracking, volunteer-driven testing of stream water quality, and a variety of other projects. These data are collected by local individuals, requiring onsite and in-person activities, but provide significantly more data than staff members alone could provide. Volunteers in these programs receive dedicated instruction in using specialized tools, expanded access to preserve lands and archives, and are credited collectively and (as applicable, individually) for their contributions to the resulting datasets.

Beyond the ongoing environmental data collection, citizen scientists and volunteers have played a critical role in the digitization process of the preserve’s many physical objects and card files. For example, at the Mohonk Preserve, volunteers have been instrumental in scanning notecards with wildlife observations, notebooks of weather observations, and written ecological and geological reports. In this case, the scanned cards are the source material for an ongoing Notes from Nature project on Zooniverse.org, where volunteers transcribe the data from the notecards. As of December 2020, more than 1,700 volunteers have provided more than 30,000 classifications of the data preserved on the notecards, allowing staff to provide the associated data more quickly to interested researchers.

Digitization is ongoing in museums, but research staff members are often involved in many other aspects of collection management, in addition to the expectation to rapidly transcribe, describe, and categorize objects resulting from the digitization process. Citizen scientists can often perform these tasks, even without subject matter expertise, while learning and engaging with the collection. Citizen science is not passing work off to the public; it is a collaborative process that allows the public to do genuine work within the museum while giving users unprecedented access to collections and a whole new form of engagement.

A growing number of platforms provide institutions to host and build these types of projects. The largest is Zooniverse.org, the platform we used for Notes from Nature, a collection of online citizen science projects that have enabled over two million online volunteers to contribute to over 250 research projects spanning disciplines from astronomy to zoology. Using these tools for digital engagement in tandem with volunteer- or staff-driven digitization can allow for significant opportunities for continued research, education, and visitor engagement.

Discussion and Conclusions

Mohonk Preserve Citizen Science activities primarily use citizen science as a tool to engage with visitors and improve overall institutional research outcomes (e.g., digitize collections and collect data on preserve property). In contrast, in the herbaria, citizen science protocols were applied to collections in an educational setting to address a particular research question. In both cases, the citizen science monitoring and observing protocols improved the research use of museum collections and holdings, enhanced user engagement with museum collections, and supported crucial environmental research.

Cultural institutions focusing on natural history often have much more extensive and diverse collections than universities and government institutions. These trends may also be present in other types of research collections such as ethnographic and art collections. Using emerging tools like citizen science, particularly on platforms like Zooniverse.org, can improve digital public access to museum collections and their associated data. Particularly in the unprecedented times of the coronavirus pandemic of 2020, cultural institutions are increasingly dependent on robust digital engagement and education opportunities. These types of opportunities can enable meaningful visitor engagement with cultural institutions and their collections while supporting research goals, even when in-person work is not possible.

Citizen science is a very flexible research tool. While I have focused on applications of citizen science in the museum setting to environmental research, there are growing opportunities to apply citizen science, particularly digital citizen science, to other natural, physical, and social science fields. Digital tools are becoming increasingly available to citizen scientists, and more field-specific tools are allowing for more variety in citizen science projects. As these tools continue to evolve and proliferate, the opportunities for engagement, research, and education are rapidly growing. As museums continue to expand their digital presence, citizen science is a tool that can improve and expand both institutional research and visitor educational opportunities.

About the Author

  • Alexis Garretson

    Alexis Garretson (she/her) is a National Science Foundation Graduate Research Fellow and a current PhD student at the Tufts University and Jackson Laboratory collaborative genetics training program. Her research focuses on improving the stewardship of community science and museum data while improving methods to incorporate these data into ongoing investigations into climate change, human health, and global change. Alexis is a Research Associate with the Mohonk Preserve Daniel Smiley Research Center, where she supports the digitization, stewardship, and curation of their collections, which include more than 60,000 physical items and a research library. Alexis received her BS in Environmental and Conservation Biology and her MS in Evolutionary Biology from George Mason University, where her thesis work focused on documenting changes in the phenology of maple trees over the past 200 years using community monitoring data and museum collections. Outside of research work, she enjoys hiking and submitting and identifying observations on iNaturalist.

Notes

  1. Kermish-Allen, Peterman, and Bevc. “The Utility of Citizen Science Projects in K–5 Schools.” Shah and Martinez. “Current Approaches in Implementing Citizen Science in the Classroom.” Edelson and Kirn. “A Report of the Designing Citizen Science for Science and Education Project at BSCS Science Learning.”
  2. Frey. “Superstar Museums.” Ariño. “Approaches to Estimating the Universe of Natural History Collections Data.”
  3. Fowler and Fowler. “Anthropology of the Numa.” Pálfi. “Juvenile Cases of Skeletal Tuberculosis from the Terry Anatomical Collection (Smithsonian Institution, Washington, D.C., USA).”
  4. MacFadden and Guralnick. “Horses in the Cloud.”
  5. Schildkrout and Keim. The Scramble for Art in Central Africa.
  6. Suarez and Tsutsui. “The Value of Museum Collections for Research and Society.”
  7. Schilthuizen et al. “Specimens as Primary Data.” Wandeler, Hoeck, and Keller. “Back to the Future.” Marshall et al. “Quantifying the Dark Data in Museum Fossil Collections as Palaeontology Undergoes a Second Digital Revolution.”
  8. Michalko and OCLC Research. Making Archival and Special Collections More Accessible. Kharouba et al. “Using Insect Natural History Collections to Study Global Change Impacts.”
  9. Baker. “Biodiversity Collections, Data, and COVID.” Spafford-Ricci and Graham. “The Fire at the Royal Saskatchewan Museum, Part 1.” Kelly and Phear. “After the Fire.” Mayernik et al. “Risk Assessment for Scientific Data.”
  10. Drew, Moreau, and Stiassny. “Digitization of Museum Collections Holds the Potential to Enhance Researcher Diversity.” Beaman and Cellinese. “Mass Digitization of Scientific Collections.” Soltis. “Digitization of Herbaria Enables Novel Research.” Wilkinson et al. “The FAIR Guiding Principles for Scientific Data Management and Stewardship.” Kahn and Wilensky. “A Framework for Distributed Digital Object Services.”
  11. Page et al. “Digitization of Biodiversity Collections Reveals Biggest Data on Biodiversity.”
  12. Iwanycki. “Guidelines for Collecting Herbarium Specimens.” Heidorn and Wei. “Automatic Metadata Extraction from Museum Specimen Labels.”
  13. Weibel et al. “Dublin Core Metadata for Resource Discovery.”
  14. Wieczorek et al. “Darwin Core.”
  15. Fegraus et al. “Maximizing the Value of Ecological Data with Structured Metadata.”
  16. Federal Agencies Digital Guidelines Initiative. “Technical Guidelines for Digitizing Cultural Heritage Materials.”
  17. Hobern, Hahn, and Robertson. “Options to Apply the IGSN Model to Biodiversity Data.”
  18. Murphey et al. “Georeferencing of Museum Collections: A Review of Problems and Automated Tools, and the Methodology Developed by the Mountain and Plains Spatio-Temporal Database- Informatics Initiative (Mapstedi).”
  19. Ibid.
  20. Nelson et al. “Digitization Workflows for Flat Sheets and Packets of Plants, Algae, and Fungi.” Giraud et al. “Best Practice Guidelines for Imaging of Herbarium Specimens.”
  21. Schneider et al. “Improved Non-Destructive 2D and 3D X-Ray Imaging of Leaf Venation.” Allan et al. “A Novel Automated Mass Digitisation Workflow for Natural History Microscope Slides.” Carranza-Rojas et al. “Going Deeper in the Automated Identification of Herbarium Specimens.” Leavitt et al. “Shotgun Sequencing Decades-Old Lichen Specimens to Resolve Phylogenomic Placement of Type Material.” Taylor and Swann. “DNA from Herbarium Specimens.”
  22. Mayernik et al. “Risk Assessment for Scientific Data.”
  23. Colombo, Kaehler, and Calvente. “An Inventory of the Bignoniaceae from the Brazilian State of Rio Grande Do Norte Highlights the Importance of Small Herbaria to Biodiversity Studies.”
  24. Eitzel et al. “Citizen Science Terminology Matters.”
  25. Dickinson, Zuckerberg, and Bonter. “Citizen Science as an Ecological Research Tool.” Hampton et al. “Big Data and the Future of Ecology.” Devictor and Bensaude-Vincent. “From Ecological Records to Big Data.”
  26. Moore et al. “Temperature Shapes the Costs, Benefits and Geographic Diversification of Sexual Coloration in a Dragonfly.”
  27. Mohonk Preserve et al. “Mohonk Preserve Stream Water Quality Invasive Species and Macroinvertebrate Sampling in from 2017–Present.”
  28. Taylor and Guralnick. “Opportunistically Collected Photographs Can Be Used to Estimate Large-Scale Phenological Trends.” Howard. “Asclepias Syriaca (Common Milkweed) Flowering Date Shift in Response to Climate Change.” Piao et al. “Plant Phenology and Global Climate Change.” Prevéy et al. “Climate Change Shifts in Habitat Suitability and Phenology of Huckleberry (Vaccinium Membranaceum).”
  29. Brossard, Lewenstein, and Bonney. “Scientific Knowledge and Attitude Change.” Bonney et al. “Citizen Science.” Evans. “Great Expectations—Ethics, Avian Flu and the Value of Progress.” Toomey and Domroese. “Can Citizen Science Lead to Positive Conservation Attitudes and Behaviors?” Crall et al. “The Impacts of an Invasive Species Citizen Science Training Program on Participant Attitudes, Behavior, and Science Literacy.” Aristeidou and Herodotou. “Online Citizen Science.”
  30. Gay et al. “Behaviors and Motivations Observed in the Zoonivers.” Tyson. “NOLS and Nutcrackers.”
  31. Dickinson, Zuckerberg, and Bonter. “Citizen Science as an Ecological Research Tool.” Tyson “NOLS and Nutcrackers.” Gay et al. “Behaviors and Motivations Observed in the Zooniverse.” Bonney et al. “Can Citizen Science Enhance Public Understanding of Science?”
  32. Ueda. “INaturalist Research-Grade Observations.” Unger et al. “INaturalist as an Engaging Tool for Identifying Organisms in Outdoor Activities.” Nugent. “INaturalist.”
  33. Seed et al. “Modelling Relationships Between Lichen Bioindicators, Air Quality and Climate on a National Scale.”
  34. Posthumus and Crimmins. “Nature’s Notebook.” Denny et al. “Standardized Phenology Monitoring Methods to Track Plant and Animal Activity for Science and Resource Management Applications.” USA-National Phenology Network. “USA-National Phenology Network 2019 Annual Report.”
  35. Devictor and Bensaude-Vincent, “From Ecological Records to Big Data.” Hochachka et al. “Data-Intensive Science Applied to Broad-Scale Citizen Science.” Tulloch et al. “Realising the Full Potential of Citizen Science Monitoring Programs.”
  36. Lepczyk. “Integrating Published Data and Citizen Science to Describe Bird Diversity across a Landscape.” Bonney et al. “Citizen Science.”
  37. Theobald et al. “Global Change and Local Solutions.” Cunha et al. “Citizen Science Participation in Research in the Environmental Sciences.”
  38. Spellman and Mulder. “Validating Herbarium-Based Phenology Models Using Citizen-Science Data.” Heberling and Isaac. “INaturalist as a Tool to Expand the Research Value of Museum Specimens.”
  39. Jones and Daehler. “Herbarium Specimens Can Reveal Impacts of Climate Change on Plant Phenology; a Review of Methods and Applications.” Park and Schwartz. “Long-Term Herbarium Records Reveal Temperature-Dependent Changes in Flowering Phenology in the Southeastern USA.” Everill et al. “Determining Past Leaf-out Times of New England’s Deciduous Forests from Herbarium Specimens.”
  40. Jones and Daehler. “Herbarium Specimens Can Reveal Impacts of Climate Change on Plant Phenology; a Review of Methods and Applications.” Page et al. “Digitization of Biodiversity Collections Reveals Biggest Data on Biodiversity.” Lane and Edwards. “The Global Biodiversity Information Facility (GBIF).”
  41. Denny et al. “Standardized Phenology Monitoring Methods to Track Plant and Animal Activity for Science and Resource Management Applications.”
  42. Garretson. “Identifying and Projecting Novel and Long-Term Phenological Trends: Integrating Heterogeneous Data Sources.”
  43. Ibid.
  44. Cook et al. “Natural History Collections as Emerging Resources for Innovative Education.”
  45. Cook et al. Walsh, Giffen, and Thompson, “Teaching Biodiversity with Museum Specimens in an Inquiry-Based Lab.” Hiller et al. “Mutualism in Museums.” Linton et al. “An Investigation of Island Biogeography Using Data from Online Natural History Collections.”
  46. Cook. “AIM-UP! (Advancing Integration of Museums into Undergraduate Programs).” Garroutte, Metzgar, and Mori. “AIM-UP! Coal Balls Lesson.” Rask, “AIM-UP! Reading Scientific Literature Module.”
  47. Mayernik et al. “Risk Assessment for Scientific Data”; Garretson et al. “Vernal Pool Amphibian Breeding Ecology Monitoring from 1931 to Present.”
  48. Charifson et al. “History of Fish Presence and Absence Following Lake Acidification and Recovery in Lake Minnewaska, Shawangunk Ridge, NY.”
  49. Cook et al. “A Homogeneous Record (1896–2006) of Daily Weather and Climate at Mohonk Lake, New York.” Cook and Cole. “On Predicting the Response of Forests in Eastern North America to Future Climatic Change.”
  50. Garretson et al. “Vernal Pool Amphibian Breeding Ecology Monitoring from 1931 to Present.” Mohonk Preserve et al. “Mohonk Preserve Stream Water Quality Invasive Species and Macroinvertebrate Sampling in from 2017–Present.” Preserve et al. “History of Acid Precipitation on the Shawangunk Ridge.” McNamara. “Citizen Science and Digitization.”
  51. Mohonk Preserve et al. “Mohonk Preserve Stream Water Quality Invasive Species and Macroinvertebrate Sampling in from 2017–Present.” Garretson et al. “Vernal Pool Amphibian Breeding Ecology Monitoring from 1931 to Present.” McNamara. “Citizen Science and Digitization.”
  52. Mayernik et al. “Risk Assessment for Scientific Data.” McNamara. “Citizen Science and Digitization.”
  53. Garretson et al. “Vernal Pool Amphibian Breeding Ecology Monitoring from 1931 to Present.” Mayernik et al. “Risk Assessment for Scientific Data.” McNamara. “Citizen Science and Digitization.”
  54. Mohonk Preserve. “A Century of Natural History Observations.”
  55. Borden et al. “Educational Exploration of the Zooniverse.” Cox et al. “Defining and Measuring Success in Online Citizen Science.”
  56. Frey. “Superstar Museums.”

Bibliography

Allan, E. L., Livermore, L., Price, B. W., Shchedrina, O., and Smith, V. S. 2019. “A Novel Automated Mass Digitisation Workflow for Natural History Microscope Slides.” Biodiversity Data Journal 7 (March 1, 2019). https://doi.org/10.3897/BDJ.7.e32342

Ariño, A. H. 2010. “Approaches to Estimating the Universe of Natural History Collections Data.” Biodiversity Informatics 7(2).

Aristeidou, M. and Herodotou, C. 2020. “Online Citizen Science: A Systematic Review of Effects on Learning and Scientific Literacy.” Citizen Science: Theory and Practice 5(1): 1–12.

Baker, B. 2020. “Biodiversity Collections, Data, and COVID.” BioScience 70(10) (October 16, 2020): 841–47. https://doi.org/10/ghg8vp

Beaman, R. S. and Cellinese, N. 2012. “Mass Digitization of Scientific Collections: New Opportunities to Transform the Use of Biological Specimens and Underwrite Biodiversity Science.” ZooKeys (209) (July 20, 2012): 7–17. https://doi.org/10.3897/zookeys.209.3313

Bonney, R. et al. 2009. “Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy.” BioScience 59(11) (December 1, 2009): 977–84. https://doi.org/10.1525/bio.2009.59.11.9

Bonney, R., Phillips, T. B., Ballard, H. L., and Enck, J. W. 2016. “Can Citizen Science Enhance Public Understanding of Science?” Public Understanding of Science 25(1) (January 1, 2016): 2–16. https://doi.org/10.1177/0963662515607406

Borden, K. A., Kapadia, A., Smith, A., and Whyte, L. 2013. “Educational Exploration of the Zooniverse: Tools for Formal and Informal Audience Engagement” 473 (April 1, 2013): 101.

Brossard, D., Lewenstein, B., and Bonney, R. 2005. “Scientific Knowledge and Attitude Change: The Impact of a Citizen Science Project.” International Journal of Science Education 27(9) (January 1, 2005): 1099–1121. https://doi.org/10.1080/09500690500069483

Carranza-Rojas, J., Goeau, H., Bonnet, P., Mata-Montero, E., and Joly, A. 2017.. “Going Deeper in the Automated Identification of Herbarium Specimens.” BMC Evolutionary Biology 17 (August 11, 2017). https://doi.org/10.1186/s12862-017-1014-z

Charifson, D. M. et al. 2015. “History of Fish Presence and Absence Following Lake Acidification and Recovery in Lake Minnewaska, Shawangunk Ridge, NY.” Northeastern Naturalist 22(4) (December 2015): 762–81. https://doi.org/10.1656/045.022.0411

Colombo, B., Kaehler, M., and Calvente, A. 2016. “An Inventory of the Bignoniaceae from the Brazilian State of Rio Grande Do Norte Highlights the Importance of Small Herbaria to Biodiversity Studies.” Phytotaxa 278(1) (October 5, 2016): 19. https://doi.org/10.11646/phytotaxa.278.1.2

Cook, B. I.,et al. 2009. “A Homogeneous Record (1896–2006) of Daily Weather and Climate at Mohonk Lake, New York.” Journal of Applied Meteorology and Climatology 49(3) (October 22, 2009): 544–55. https://doi.org/10.1175/2009JAMC2221.1

Cook, B. I. et al. 2008. “A Cross‐Taxa Phenological Dataset from Mohonk Lake, NY and Its Relationship to Climate.” International Journal of Climatology 28(10) (August 2008): 1369–83. https://doi.org/10.1002/joc.1629

Cook, E. R. and Cole, J. 1991. “On Predicting the Response of Forests in Eastern North America to Future Climatic Change.” Climatic Change 19(3) (October 1, 1991): 271–82. https://doi.org/10.1007/BF00140166

Cox, J. et al. 2015. “Defining and Measuring Success in Online Citizen Science: A Case Study of Zooniverse Projects.” Computing in Science Engineering 17(4) (July 2015): 28–41. https://doi.org/10.1109/MCSE.2015.65

Cook, J. A. et al. 2014. “Natural History Collections as Emerging Resources for Innovative Education.” BioScience 64(8) (August 1, 2014): 725–34. https://doi.org/10.1093/biosci/biu096

Cook, J. A., Edwards, S., Ickert-Bond, S., and Lacy, E. 2015. “AIM-UP! (Advancing Integration of Museums into Undergraduate Programs).” https://qubeshub.org/resources/288

Crall, A. W. 2013. “The Impacts of an Invasive Species Citizen Science Training Program on Participant Attitudes, Behavior, and Science Literacy.” Public Understanding of Science 22(6) (August 1, 2013): 745–64. https://doi.org/10.1177/0963662511434894

Cunha, D. G. et al. 2017. “Citizen Science Participation in Research in the Environmental Sciences: Key Factors Related to Projects’ Success and Longevity.” Anais Da Academia Brasileira de Ciências 89(3): 2229–45. https://doi.org/10.1590/0001-3765201720160548

Denny, E. G. et al. 2014. “Standardized Phenology Monitoring Methods to Track Plant and Animal Activity for Science and Resource Management Applications.” International Journal of Biometeorology 58(4) (May 1, 2014): 591–601. https://doi.org/10.1007/s00484-014-0789-5

Devictor, V. and Bensaude-Vincent, B. 2016. “From Ecological Records to Big Data: The Invention of Global Biodiversity.” History and Philosophy of the Life Sciences 38(4) (December 2016): UNSP 13. https://doi.org/10.1007/s40656-016-0113-2

Dickinson, J. L., Zuckerberg, B., and Bonter, D. N. 2010. “Citizen Science as an Ecological Research Tool: Challenges and Benefits.” Annual Review of Ecology, Evolution, and Systematics 41(1) (November 2, 2010): 149–72. https://doi.org/10.1146/annurev-ecolsys-102209-144636

Drew, J. A., Moreau, C. S., and LJ Stiassny, M. 2017. “Digitization of Museum Collections Holds the Potential to Enhance Researcher Diversity.” Nature Ecology & Evolution 1(12): 1789

Edelson, D. C, and Kirn, S. L. 2018. “A Report of the Designing Citizen Science for Science and Education Project at BSCS Science Learning,” 39.

Eitzel, M. V. et al. “Citizen Science Terminology Matters: Exploring Key Terms.” Citizen Science: Theory and Practice 2(1) (June 5, 2017): 1. https://doi.org/10.5334/cstp.96

Evans, N. G. 2013. “Great Expectations—Ethics, Avian Flu and the Value of Progress.” Journal of Medical Ethics 39(4): 209–213.

Everill, P. H., Primack, R. B., Ellwood, E. R., and Melaas, E. K. 2014. “Determining Past Leaf-out Times of New England’s Deciduous Forests from Herbarium Specimens.” American Journal of Botany 101(8) (August 1, 2014): 1293–1300. https://doi.org/10.3732/ajb.1400045

Federal Agencies Digital Guidelines Initiative. “Technical Guidelines for Digitizing Cultural Heritage Materials.” Accessed June 27, 2018. http://www. Digitizationguidelines. Gov/Guidelines/FADGI_Still_Image_Tech_Guidelines_2016. Pdf

Fegraus, E. H., Andelman, S., Jones, and Schildhauer, M. 2005. “Maximizing the Value of Ecological Data with Structured Metadata: An Introduction to Ecological Metadata Language (EML) and Principles for Metadata Creation.” The Bulletin of the Ecological Society of America 86(3): 158–68. https://www.zotero.org/google-docs/?4u7NpY

Fowler, D. D. and Fowler, C. S. 1971.“Anthropology of the Numa: John Wesley Powell’s Manuscripts on the Numic Peoples of Western North America, 1868–1880.” http://repository.si.edu/xmlui/handle/10088/1329

Frey, Bruno S. 1998. “Superstar Museums: An Economic Analysis.” Journal of Cultural Economics 22(2) (June 1, 1998): 113–25. https://doi.org/10.1023/A:1007501918099

Garretson, A. 2020. “Identifying and Projecting Novel and Long-Term Phenological Trends: Integrating Heterogeneous Data Sources.” Master of Science, George Mason University.

Garretson, A., Napoli, M., Feldsine, N., Adler-Colvin, P., and Long, E. 2020. “Vernal Pool Amphibian Breeding Ecology Monitoring from 1931 to Present: A Harmonised Historical and Ongoing Observational Ecology Dataset.” Biodiversity Data Journal 8 (April 14, 2020): e50121. https://doi.org/10.3897/BDJ.8.e50121

Garroutte, M., Metzgar, J., and Mori, H. 2018. “AIM-UP! Coal Balls Lesson.” https://doi.org/10/ghhfvh

Gay, P. L., Brown, S., Huang, A. D., Lehan, C., and Moon Zoo Team. “Behaviors and Motivations Observed in the Zooniverse.” AGU Fall Meeting Abstracts 11 (December 1, 2010): ED11A-0581.

Giraud, M. et al. “Best Practice Guidelines for Imaging of Herbarium Specimens.” Innovation and Consolidation for Large Scale Digitization of Natural Heritage (January 1, 2018): 41.

Hampton, S. E. et al. 2013 “Big Data and the Future of Ecology.” Frontiers in Ecology and the Environment 11(3): 156–62. https://doi.org/10.1890/120103

Heberling, J. M. and Isaac, B. L. 2019. “INaturalist as a Tool to Expand the Research Value of Museum Specimens.” The Plant Journal (May 9, 2019): e01193. https://doi.org/10.1002/aps3.1193@10.1111/(ISSN)1365-313X.plants_day_collection

Heidorn, P. B. and Wei, Q. 2008. “Automatic Metadata Extraction from Museum Specimen Labels.” International Conference on Dublin Core and Metadata Applications: 57–68.

Hiller, A. E. et al. 2017. “Mutualism in Museums: A Model for Engaging Undergraduates in Biodiversity Science.” PLOS Biology 15(11) (November 21, 2017): e2003318. https://doi.org/10/gckn5k

Hobern, D., Hahn, A., and Robertson, T. 2018. “Options to Apply the IGSN Model to Biodiversity Data.” Biodiversity Information Science and Standards 2 (May 31, 2018): e27087. https://doi.org/10.3897/biss.2.27087

Hochachka, W. M. et al. 2012. “Data-Intensive Science Applied to Broad-Scale Citizen Science.” Trends in Ecology & Evolution, Ecological and Evolutionary Informatics 27(2) (February 1, 2012): 130–37. https://doi.org/10.1016/j.tree.2011.11.006

Howard, A. F. 2018. “Asclepias Syriaca (Common Milkweed) Flowering Date Shift in Response to Climate Change.” Scientific Reports 8(1) (December 13, 2018): 17802. https://doi.org/10.1038/s41598-018-36152-2

Iwanycki, N. 2009. “Guidelines for Collecting Herbarium Specimens.” Royal Botanical Gardens (July 2009): 5.

Jones, C. A. and Daehler, C. C. 2018. “Herbarium Specimens Can Reveal Impacts of Climate Change on Plant Phenology; A Review of Methods and Applications.” PeerJ 6 (April 3, 2018): e4576. https://doi.org/10.7717/peerj.4576

Kahn, R. and Wilensky, R. 2006. “A Framework for Distributed Digital Object Services.” International Journal on Digital Libraries 6(2) (April 1, 2006): 115–23. https://doi.org/10/dj5wcr

Kelly, M. C. S. and Phear, S. 2004. “After the Fire: Salvaging the Stores of the Department of Archaeology & Natural History, Australian National University, Canberra,” 11. New York, NY, 2004.

Kermish-Allen, R., Peterman, K., and Bevc, C. 2019. “The Utility of Citizen Science Projects in K–5 Schools: Measures of Community Engagement and Student Impacts.” Cultural Studies of Science Education 14(3) (September 1, 2019): 627–41. https://doi.org/10.1007/s11422-017-9830-4

Kharouba, H. M. et al. 2019. “Using Insect Natural History Collections to Study Global Change Impacts: Challenges and Opportunities.” Philosophical Transactions of the Royal Society B: Biological Sciences 374(1763) (January 7, 2019): 20170405. https://doi.org/10.1098/rstb.2017.0405

Lane, M. A., and Edwards, J. L. 2007. “The Global Biodiversity Information Facility (GBIF).” Systematics Association Special Volume Series 73: 1–4.

Leavitt, S. D., Keuler, R., Newberry, C. C., Rosentreter, R., and St Clair, L. L. 2019. “Shotgun Sequencing Decades-Old Lichen Specimens to Resolve Phylogenomic Placement of Type Material.” Plant and Fungal Systematics 64(2) (December 1, 2019): 237–47. https://doi.org/10.2478/pfs-2019-0020

Lepage, D. and Francis, C. M. 2002. “Do Feeder Counts Reliably Indicate Bird Population Changes? 21 Years of Winter Bird Counts in Ontario, Canada.” The Condor 104(2) (May 1, 2002): 255–70. https://doi.org/10.1093/condor/104.2.255

Lepczyk, C. A. 2005. “Integrating Published Data and Citizen Science to Describe Bird Diversity across a Landscape.” Journal of Applied Ecology 42(4): 672–77. https://doi.org/10.1111/j.1365-2664.2005.01059.x

Linton, D., Monfils, A., Phillips, M., Ellwood, L., and Cook, J. 2019. “An Investigation of Island Biogeography Using Data from Online Natural History Collections.” https://doi.org/10/ghhftg

MacFadden, B. J. and Guralnick, R. P. 2017. “Horses in the Cloud: Big Data Exploration and Mining of Fossil and Extant Equus (Mammalia: Equidae).” Paleobiology 43(1) (Winter 2017): 1–14. https://doi.org/10.1017/pab.2016.42

Marshall, C. R. et al. 2018. “Quantifying the Dark Data in Museum Fossil Collections as Palaeontology Undergoes a Second Digital Revolution.” Biology Letters 14(9) (September 2018). https://doi.org/10.1098/rsbl.2018.0431

Mayernik, M. S. et al. 2020. “Risk Assessment for Scientific Data.” Data Science Journal 19(1) (March 12, 2020): 10. https://doi.org/10.5334/dsj-2020-010

McNamara, C. 2019. “Citizen Science and Digitization: Examining the Process of Digitization in Crowdsourcing,” August 2019. http://dspace-prod.mse.jhu.edu:8080/handle/1774.2/62119

Michalko, J. and OCLC Research 2015. Making Archival and Special Collections More Accessible. Dublin, Ohio: OCLC Research. http://www.oclc.org/content/dam/research/publications/2015/oclcresearch-making-special-collections-accessible-2015-a4.pdf

Mohonk Preserve. “A Century of Natural History Observations.” Accessed October 29, 2020. https://www.zooniverse.org/projects/mohonkpreservedsrc/a-century-of-natural-history-observations

Mohonk Preserve et al. 2020. “Mohonk Preserve Stream Water Quality Invasive Species and Macroinvertebrate Sampling in from 2017–Present.” Environmental Data Initiative, 2020. https://doi.org/10.6073/PASTA/A00FF0A44F3305F17C99D87E6A0F29FD

Moore, M. P., Lis, C. Gherghel, I., and Martin, R. A. 2019. “Temperature Shapes the Costs, Benefits and Geographic Diversification of Sexual Coloration in a Dragonfly.” Ecology Letters 22(3): 437–46. https://doi.org/10.1111/ele.13200

Murphey, P. C, Guralnick, R. P., Neufeld, D., and Ryan, J. A. 2004. “Georeferencing of Museum Collections: A Review of Problems and Automated Tools, and the Methodology Developed by the Mountain and Plains Spatio-Temporal Database- Informatics Initiative (Mapstedi),” 29.

Nelson, G. et al. 2015. “Digitization Workflows for Flat Sheets and Packets of Plants, Algae, and Fungi.” Applications in Plant Sciences 3(9): 1500065. https://doi.org/10.3732/apps.1500065

Nugent, J. 2018. “INaturalist: Citizen Science for 21st-Century Naturalists.” Science Scope 41(7): 12.

Page, L. M., MacFadden, B. J., Fortes, J. A., Soltis, P. S., and Riccardi, G. 2015. “Digitization of Biodiversity Collections Reveals Biggest Data on Biodiversity.” BioScience 65(9) (September 1, 2015): 841–42. https://doi.org/10.1093/biosci/biv104

Pálfi, G. 2012. “Juvenile Cases of Skeletal Tuberculosis from the Terry Anatomical Collection (Smithsonian Institution, Washington, D.C., USA).” Acta Biologica Szegediensis 56(1) (January 1, 2012): 1–12.

Park, I.W. and Schwartz, M. D. 2015. “Long-Term Herbarium Records Reveal Temperature-Dependent Changes in Flowering Phenology in the Southeastern USA.” International Journal of Biometeorology 59(3) (March 1, 2015): 347–55. https://doi.org/10.1007/s00484-014-0846-0

Piao, S. et al. 2019. “Plant Phenology and Global Climate Change: Current Progresses and Challenges.” Global Change Biology 25(6): 1922–40.

Posthumus, E. and Crimmins, T. 2011. “Nature’s Notebook: A Tool for Education and Research.” The Bulletin of the Ecological Society of America 92(2) (April 1, 2011): 185–87. https://doi.org/10.1890/0012-9623-92.2.185

Preserve, Mohonk et al. 2018. “History of Acid Precipitation on the Shawangunk Ridge: Mohonk Preserve Precipitation Depths and PH, 1976 to Present.” Environmental Data Initiative, 2018. https://doi.org/10.6073/PASTA/734EA90749E78613452EACEC489F419C

Prevéy, J. S., Parker, L. E., Harrington, C. A., Lamb, C. T., and Proctor, M. F. 2020. “Climate Change Shifts in Habitat Suitability and Phenology of Huckleberry (Vaccinium Membranaceum).” Agricultural and Forest Meteorology 280 (January 15, 2020): 107803. https://doi.org/10.1016/j.agrformet.2019.107803

Rask, G. 2015..“AIM-UP! Reading Scientific Literature Module.” https://qubeshub.org/resources/329

Schildkrout, E. and Keim, C. A. 1998. The Scramble for Art in Central Africa. Cambridge University Press.

Schilthuizen, M., Vairappan, C. S., Slade, E. M., Mann, D. J., and Miller, J. A. 2015. “Specimens as Primary Data: Museums and ‘Open Science.’” Trends in Ecology & Evolution 30(5) (May 2015): 237–38. https://doi.org/10.1016/j.tree.2015.03.002

Schneider, J. V. et al. 2018. “Improved Non-Destructive 2D and 3D X-Ray Imaging of Leaf Venation.” Plant Methods 14 (January 19, 2018). https://doi.org/10.1186/s13007-018-0274-y

Seed, L., Wolseley, P., Gosling, L., Davies, L., and Power, S. A. 2013. “Modelling Relationships Between Lichen Bioindicators, Air Quality and Climate on a National Scale: Results from the UK OPAL Air Survey.” Environmental Pollution 182 (November 1, 2013): 437–47. https://doi.org/10.1016/j.envpol.2013.07.045

Shah, H. R. and Martinez, L. R. 2016. “Current Approaches in Implementing Citizen Science in the Classroom.” Journal of Microbiology & Biology Education 17(1) (March 1, 2016): 17–22. https://doi.org/10.1128/jmbe.v17i1.1032

Soltis, P. S. 2017. “Digitization of Herbaria Enables Novel Research.” American Journal of Botany 104(9): 1281–1284.

Spafford-Ricci, S. and Graham, F. 2000. “The Fire at the Royal Saskatchewan Museum, Part 1: Salvage, Initial Response, and the Implications for Disaster Planning.” Journal of the American Institute for Conservation 39(1) (January 1, 2000): 15–36. https://doi.org/10/ghg8v7

Spellman, K. V. and Mulder, C. P. H. 2016. “Validating Herbarium-Based Phenology Models Using Citizen-Science Data.” BioScience 66(10) (October 1, 2016): 897–906. https://doi.org/10.1093/biosci/biw116

Stewart, E. 2016. “A Matter of Rights: The Representation of Women in the Museum of Australian Democracy.” Journal of Australian Studies 40(2) (April 2, 2016): 215–21. https://doi.org/10.1080/14443058.2016.1175493

Suarez, A. V. and Tsutsui, N. D. 2004. “The Value of Museum Collections for Research and Society.” BioScience 54(1) (January 1, 2004): 66–74. https://doi.org/10.1641/0006-3568(2004)054[0066:TVOMCF]2.0.CO;2

Taylor, J. W. and Swann, E. C. 1994. “DNA from Herbarium Specimens.” Ancient DNA: 166–181. Springer, NY, New York.

Taylor, S. D. and Guralnick, R. P. 2019. “Opportunistically Collected Photographs Can Be Used to Estimate Large-Scale Phenological Trends.” BioRxiv (October 7, 2019): 794396. https://doi.org/10.1101/794396

Theobald, E. J. et al. 2015. “Global Change and Local Solutions: Tapping the Unrealized Potential of Citizen Science for Biodiversity Research.” Biological Conservation 181(Supplement C) (January 1, 2015): 236–44. https://doi.org/10.1016/j.biocon.2014.10.021

Toomey, A. H., and Domroese, M. C. 2013. “Can Citizen Science Lead to Positive Conservation Attitudes and Behaviors?” Human Ecology Review 20(1) 50–62.

Tulloch, A. I. T., Possingham, H. P., Joseph, L. N., Szabo, J., and Martin, T. G. 2013. “Realising the Full Potential of Citizen Science Monitoring Programs.” Biological Conservation 165 (September 1, 2013): 128–38. https://doi.org/10.1016/j.biocon.2013.05.025

Tyson, A. 2019. “NOLS and Nutcrackers: The Motivations, Barriers, and Benefits Experienced by Outdoor Adventure Educators in the Context of a Citizen Science Project.” Citizen Science: Theory and Practice 4(1) (June 13, 2019): 20. https://doi.org/10.5334/cstp.127

Ueda, K. 2020. “INaturalist Research-Grade Observations.” https://doi.org/10.15468/ab3s5x

Unger, S., Rollins, M., Tietz, A., and Dumais, H. 2020. “INaturalist as an Engaging Tool for Identifying Organisms in Outdoor Activities.” Journal of Biological Education. (March 15, 2020): 1–11. https://doi.org/10.1080/00219266.2020.1739114

USA-National Phenology Network. 2019. “USA-National Phenology Network 2019 Annual Report.” (December 2019). https://www.usanpn.org/files/npn/reports/USA-NPN_AnnualReport2019.pdf

Walsh, L. L., Giffen, C. J., and Thompson, C. W. 2019. “Teaching Biodiversity with Museum Specimens in an Inquiry-Based Lab.” CourseSource 6. https://doi.org/10/ghhftd

Wandeler, P., Hoeck, P. E. A., and Keller, L. F. 2007. “Back to the Future: Museum Specimens in Population Genetics.” Trends in Ecology & Evolution 22(12): 634–42.

Weibel, S., Kunze, J., Lagoze, C., and Wolf, M. 1998. “Dublin Core Metadata for Resource Discovery.” Internet Engineering Task Force RFC 2413(222): 132.

Wieczorek, J. et al. 2012. “Darwin Core: An Evolving Community-Developed Biodiversity Data Standard.” PloS One 7(1): e29715. https://doi.org/10/fzrpwq

Wilkinson, M. D. et al. 2016. “The FAIR Guiding Principles for Scientific Data Management and Stewardship.” Scientific Data 3(1) (March 15, 2016): 160018. https://doi.org/10/bdd4

Zankowicz, K. 2014.. “In Her Hands: Women’s Educational Work at the Royal Ontario Museum, the Canadian National Exhibition and the Art Gallery of Toronto, 1900s–1950s.” https://tspace.library.utoronto.ca/handle/1807/68428