AI-Ready Open Data

Artificial Intelligence (AI) is rapidly changing society– from education and politics to art and healthcare. How does the AI revolution impact open data, such as the hundreds of free datasets that Cambridge provides on its Open Data Portal? Conversely, how does open data impact AI? And how can you, as a user of open data, use AI tools to enhance your analysis and research?

Cambridge’s Open Data Program is actively engaged with these questions, and we’d like to share a bit of what we’ve learned.

Modern generative AI systems are trained on massive amounts of data: textual data, image data, and numerical data. Open data is already impacting AI as a key part of these systems' "diets." But quality is just as important as quantity. AI systems have their own specific "nutritional" needs, which is why data scientists at the US Department of Commerce, UNESCO, and other organizations are developing guidelines for “AI-ready” open data. This is data published with comprehensive contextual metadata, ensuring that it's not just machine-readable but machine-understandable, and that it has been carefully vetted to ensure that it represents diverse and inclusive populations, so as not to perpetuate biases.

The goal? New standards for open data repositories that will produce AI systems that will be better able to interpret data correctly and produce more reliable results.

Just as open data can enhance AI, AI can improve open data. Current AI systems can identify anomalies in open datasets, flag potential errors, and significantly enhance QA processes. This results in cleaner, more accurate data for everyone. 

Perhaps most immediately relevant to users of open data, AI introduces sophisticated data analysis and exploration capabilities to anyone with a web browser. You can upload datasets and ask tools like ChatGPT to produce specific graphs or calculations. You can request AI to perform a preliminary analysis and highlight interesting patterns. Furthermore, you can ask and AI to clean up a messy dataset for further processing. Data analysis no longer requires vast expertise with complex tools; anyone with an understanding of the underlying concepts can simply articulate their requirements conversationally.

Are you using AI to analyze or explore Cambridge open data? Would you be interested in learning more about how to use modern AI tools with the Open Data Portal? Please let us know at opendata@cambridgema.gov!