GeoAI tools can make geographic and environmental information easier to explore through everyday questions. However, a quick and clearly written answer is not necessarily a complete or reliable one. When users ask about local conditions, land use or weather, the answer must also be relevant to the selected area, supported by suitable data and clear about its limitations.
J-VERSE—JackDaw Validation in Real Spatial Environments—was developed by Catalink through the PoliRuralPlus Enhance Open Call to evaluate JackDaw, a GeoAI tool that allows users to explore territorial information through an interactive map and natural-language questions.
The project examined JackDaw from two complementary perspectives. First, users with different backgrounds were invited to interact with the platform and share their experiences. Second, selected environmental information was compared with an independent weather-data source and measurements from locally deployed IoT devices in Nicosia and Limassol. In this way, J-VERSE considered both how people experienced JackDaw and how well its information reflected the selected external references and local conditions. The aim was not simply to check whether JackDaw could return an answer, but whether that answer was useful, spatially appropriate and consistent with the available evidence.
Testing JackDaw with different users
The user-centred validation involved 35 participants from seven stakeholder categories, including citizens, students, researchers, agricultural participants, NGOs and professionals from the public and private sectors. Participants also had different levels of familiarity with digital and geospatial tools. This diversity was important for examining whether JackDaw could be used not only by specialists, but also by people with limited experience of exploring geographic information.
Following a short introduction to the platform, participants selected an area, tested questions relating to different scenarios, explored topics of their own interest, and interpreted the responses provided by JackDaw. The activity allowed them to experience the main interaction process directly and provide feedback on its ease of use, response time and the usefulness of the information received.
Overall, participants found the platform approachable. Ease of use received an average rating of 4.16 out of 5, while 84% reported receiving an answer in under one minute. These results indicate that most participants could interact with JackDaw without substantial difficulty, even when they did not have specialist GIS knowledge. However, ease of interaction did not always translate into perceived usefulness. Usefulness received a more moderate average rating of 2.75 out of 5, particularly when information was missing, too general or insufficiently connected to the selected location.
Selected results from the J-VERSE user-centred validation
At the same time, the sessions showed that ease and speed alone do not determine whether an answer is considered useful. Participants’ willingness to use or recommend JackDaw also appeared to depend on whether its responses were relevant, clear and sufficiently informative for the question asked.
The feedback therefore identified opportunities to improve user guidance, map interaction and support for questions involving several locations or more complex requests. It also highlighted the importance of clearly informing users when requested information is unavailable, so that they can understand the limits of a response and decide whether further information is needed.
Comparing environmental information
To complement the user feedback, J-VERSE compared JackDaw’s temperature and humidity information with Open-Meteo, an external weather-data reference. The comparisons covered selected locations and periods in Cyprus. A knowledge graph was used to align information from the different sources and support consistent comparisons.
In the controlled comparisons, JackDaw and Open-Meteo often followed similar broad temperature trends across the periods examined. However, the evaluation also identified differences in some aggregated results and in answers returned through the conversational interface.
The evaluation also identified some limitations. Within the requests examined, historical data were not always available for every selected location, date or variable. In certain cases, the conversational responses did not communicate the available information as clearly or consistently as expected. This highlighted an important distinction between the structured JackDaw data used in the controlled comparisons and the answers produced through open-ended conversation.
In one exploratory test, the conversational answer differed from the results obtained by analysing the structured JackDaw values and the Open-Meteo reference. Although the test did not establish the internal cause of the difference, it reinforced the importance of considering the relevant observations and spatial context when interpreting conversational outputs.
What local measurements added
J-VERSE also used locally deployed IoT devices in Nicosia and Limassol to record temperature and humidity in specific settings. These measurements provided a more localised perspective than broader-area weather information.
The local measurements generally followed the same broad temperature trends as the other sources. At the same time, they captured differences associated with their immediate surroundings, including solar exposure, vegetation, airflow and the characteristics of the installation location.
Temperature patterns during the Nicosia pilot (“MeteoAPI” refers to Open-Meteo)
The observed differences reflect the different spatial scales represented by each source. JackDaw and Open-Meteo provide broader-area weather information, whereas locally deployed sensors reflect conditions in their immediate surroundings. The sources are therefore complementary but not interchangeable: broader-area data offer an overall view, while local measurements add microclimate-level detail for a specific site.
This microclimate-level detail may be particularly useful when decisions depend on conditions at a specific site. Combining wider territorial information with appropriately deployed local sensors could therefore strengthen the value of GeoAI tools for users who require more location-specific insights.
What did J-VERSE learn?
Taken together, the findings show that JackDaw can make territorial information easier and faster to explore, including for users without specialist GIS knowledge. At the same time, ease of use did not always mean that the resulting answer was considered useful. Data coverage, spatial-context handling and consistency between structured data and conversational responses remain important areas for improvement.
The main opportunities for improvement concern data coverage, support for complex and multi-location questions, and consistency between the information available to the platform and the answers communicated through its conversational interface. Clearer acknowledgement of unavailable or uncertain information would also help users understand when further verification may be needed.
J-VERSE demonstrates the value of combining user feedback, independent data and local observations when evaluating GeoAI tools. With appropriate adaptation to local data, geography, language and stakeholder needs, this approach could also support similar validation activities in other regions.
Learn more
Readers can explore the stakeholder findings, historical comparisons and pilot measurements through the J-VERSE validation dashboard.
Further information is available in the related PoliRuralPlus articles:
● J-VERSE: Using Knowledge Graphs to Validate JackDaw Against Weather APIs and IoT Data
J-VERSE (JackDaw Validation in Real Spatial Environments) has received funding from the PoliRuralPlus project, which is funded by the European Union’s Horizon Europe research and innovation programme under grant agreement No 101136910. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or European Research Executive Agency (REA). Neither the European Union nor the granting authority can be held responsible for them.
Existující komentáře