Date: TBA
Time: TBA
Place: TBA

Recent advances in Artificial Intelligence, particularly Large Language Models (LLMs), are transforming how researchers extract structured information from textual data. These developments create new opportunities for measuring complex phenomena from unstructured text, while also raising important methodological challenges related to consistency, validity, and reproducibility.

This tutorial presents a structured framework for constructing quantitative indicators from textual data using LLMs. The approach formalizes the full measurement pipeline, transforming raw text into calibrated, reproducible metrics through a sequence of well-defined stages. The framework integrates preprocessing and text segmentation, prompt design based on structured rubrics, LLM-based scoring, and an evaluation layer addressing reliability and validity. Human expert assessments are incorporated to benchmark model outputs, followed by a calibration stage that translates raw scores into standardized indicators.

The tutorial is structured into three parts. The first part introduces the foundations of LLM-based measurement, including data collection, preprocessing, and prompt design. The second part focuses on transforming qualitative text into quantitative indicators through scoring, evaluation, and iterative refinement. The final part presents an end-to-end example and discusses calibration, limitations, and reproducibility.

By participating in this tutorial, attendees will learn how to design robust and transparent pipelines for extracting quantitative indicators from textual data, and how to apply these methods in research contexts where structured data is limited.

Outline

1. Foundations of LLM-Based Measurement from Text
   • Textual data as a source for measurement
   • Challenges in transforming text into quantitative indicators
   • Overview of the measurement pipeline
   • Data collection and preprocessing
     • Text extraction and structuring from different sources
     • Text cleaning and segmentation (chunking strategies)
   • Rubric design and prompt construction
     • Translating abstract constructs into measurable dimensions
     • Designing structured, rubric-based prompts
     • Ensuring consistency and interpretability of outputs
2. From Text to Indicators – Scoring, Evaluation, and Refinement
   • LLM-based scoring of textual content
   • Evaluation framework
     • Reliability assessment
     • Validity assessment
   • Human expert benchmarking
   • Iterative refinement loops
     • Improving preprocessing, rubrics, prompts, and model behavior
   • Common pitfalls
     • Prompt sensitivity
     • Bias and hallucinations
3. Calibration, Applications, and Practical Walkthrough
   • Calibration of model outputs
   • End-to-end example
     • Application of rubric-based scoring to real text
   • Applications across research domains
   • Limitations and reproducibility considerations
   • Open discussion and Q&A

Target Audience

This tutorial is open for any conference participant who wants to gain knowledge about recent methodological developments in text-based measurement using Large Language Models. Although there are no formal prerequisites or required knowledge, we suggest participants have a basic understanding of data analysis and familiarity with general computing environments.

Participants with experience in Python or R may benefit more from the practical components, although this is not required. Attendees are encouraged to bring their laptops if they wish to follow the examples and workflow demonstrations during the session.

Presenters

Xavier Martínez-Barberois a researcher at Universitat Politècnica de València and affiliated with the eSMART Research Center. He holds a PhD in Business Administration and has over 5 years of experience in data and AI-related research, including international experience in the private sector. His work focuses on measuring corporate technological and sustainability adoption using textual data. His research combines text analysis, machine learning, and empirical methods, with applications in innovation, energy transition, and sustainable finance.