Measuring and Evaluating Outcomes of Program Sukatech with a Lean Application of GQM+Strategies Principles and Models

April 8, 2026

Project title: Lean Application of Principles and Models Based on GQM+Strategies for the Enhancement of Measurement and Decision-Making Processes in Program Sukatech

Project type: Undergraduate capstone project, i.e. Technical Residency in Information Systems

Organization: State Agency of Science, Technology and Innovation of Goiás (SECTI-GO)

Contributing team: Brenda Andreia Lima Pinheiro, Fabio Tiago Vieira Soares Marques, Yatherson Lucas Teodoro Souza

Supervisor: Juliano Lopes de Oliveira (ORCID: 0000-0002-8405-2152)

Execution period: March - July 2025

DOI: https://doi.org/10.5281/zenodo.19476767

Public repository: https://github.com/contraexemplo/RTSI-2025

License: CC BY-SA 4.0

Notes: Published under my legal name (ORCID: 0009-0000-2892-1838)

In July 2025, I closed one of the most important chapters of my Bachelor’s degree: I finished my Technical Residency in Information Systems. I led a team of technical residents working with the State Agency of Science, Technology and Innovation of Goiás (SECTI-GO). Our focus was a program called Sukatech, a circular economy initiative that encompasses recycling and refurbishing electronics, empowering and training our population and raising awareness about proper electronic waste disposal.

Our challenge: help a powerhouse of a program to understand more about its operations and impact through measurement and evaluation. What followed was a significant effort to map the very rich system that Sukatech is, and it ended with (1) the development of a data governance and information management framework to help Sukatech understand where it wants to go (2) the creation of a roadmap of optimal applications and solutions to lead Sukatech to what it wants to be.

Sukatech is a program structured in four axes:

Axis 1 - Production Chain;
Axis 2 - Training and Entrepreneurship;
Axis 3 - Environmental Education;
Transverse Axis - Evaluation.

A mind map shows the four axes of the Sukatech program and their respective goals.

Axis 1 - Production Chain structures and operationalizes the reverse logistics of electronic equipment in Computer Refurbishment Centers (CRCs), carrying out the collection, sorting, refurbishment, donation of refurbished equipment, and the environmentally appropriate disposal of unusable materials.

Axis 2 - Training and Entrepreneurship promotes the training of the population in strategic technological areas, certifying students in courses of the circular economy and reverse logistics hub, serving the population from 12 years of age, with criteria for gender equity (60% for women), inclusion of Afro-descendant people and people with disabilities, and prioritization of people in social vulnerability.

Axis 3 - Environmental Education concentrates all the civic and environmental training activities of the Sukatech program. The general population is educated about the importance of the correct and responsible disposal of electronic waste. Asset managers in public administration are educated on the procedural pathways for state assets that have reached the end of their useful life, explaining how these assets can be processed by the Sukatech program through disposal or redirection actions.

The Transverse Axis - Evaluation monitors the execution of all other action axes of the Sukatech program, building intuitive processes and tools for continuous evaluation.

Analyzing the Sukatech program as is from a data perspective, we note that the processes of all operational axes follow the same sequence of key activities with shared execution responsibilities between Programando o Futuro, SECTI, and its key partners: all key axis processes are initiated through the collection of fundamental data. This data is then stored in parallel in various ways and managed—transformed or manipulated—by key activities. This data is then consolidated by axis coordinators to meet accountability requirements. Finally, it is shared with SECTI and analyzed according to what is stipulated by measurement processes parameterized by the general and specific objectives of each axis and by monitoring criteria established by the Transversal Axis. The conclusions of this data analysis activity support the program’s decision-making processes.

We also found that the Sukatech program’s operations are facing problems in two different areas:

Domain 1: Start Point Issues (data collection, storage, and management);
Domain 2: End Point Issues (consolidation, transit, and data analysis).

A BPMN diagram shows a simplified version of Sukatech's data collection, storage, management, consolidation, transit and data analysis flow.

Domain 1, also known as Start Point Issues, encompasses the information systems problems during the execution of operational activities of the Sukatech program. In this domain, we identified two major problems:

Problem 1.1: Unavailability of modules in the software system used by Programming the Future for the administration of CRCs that include specific requirements of all axes of the Sukatech program;
Problem 1.2: Dependence on manual file and folder organization systems for axis data management (i.e., extensive use of Google Drive to manage disposal documents in Axis 1 - Production Chain, all of Axis 2 - Training and Entrepreneurship, and campaigns of Axis 3 - Environmental Education).

We also found that these two problems are interconnected: Problem 1.1 causes Problem 1.2. We also note that the operations of the three axes of the Sukatech program have varying degrees of manualness (that is, how manual the collection, storage and management of data is):

Axis 1 - Production Chain is the axis with the highest degree of technological structuring: Programando o Futuro operates from its own software system responsible for collecting, storing and managing information on donations, disposals, dismantling and recycling of all CRCs that the organization controls;
Axis 2 - Training and Entrepreneurship is the axis with the highest degree of manual work, since it is managed solely from various folders, spreadsheets, and documents hosted on Google Drive;
Axis 3 - Environmental Education operates at a medium level of manual work in the coordination of its campaigns, but some of its activities (such as, for example, campaign data collection) are covered by the software system used by activities of Axis 1.

Domain 2, also known as End Point Issues, encompasses the information systems problems of evaluation and accountability activities of the Sukatech program. We found six main problems:

Problem 2.1: Extensive use of spreadsheets created, consolidated, and managed manually for accountability processes;
Problem 2.2: Slow data transit between systems managed by Programando o Futuro and information portals shared with SECTI;
Problem 2.3: Accidental handling of inconsistent and/or distorted data;
Problem 2.4: Complex, laborious, and extremely manual processes for transforming received semi-structured data into:
- Metrics for monitoring, measuring success and failure;
- Standard reports for accountability processes;
- Customized reports for accountability processes.
Problem 2.5: Insufficient data for key measurements;
Problem 2.6: Late diagnoses of operational problems.

We found that the six problems listed are also interrelated: Problem 2.1 (manual data consolidation) is a direct generating factor of problems 2.2 (slow data transit), 2.3 (inconsistent and/or distorted data), and 2.5 (insufficient data). Problem 2.3 causes Problem 2.4 (difficulties in data transformation), which influences Problem 2.5. Problem 2.5, in turn, influences Problem 2.6 (late diagnoses).

A diagram reveals the dependencies between all problems in Domain 2.

Various reports of major challenges in consolidating data from the Sukatech program point to a strong interconnection between the two domains: the “technical debt” created by the problems of Domain 1 causes the problems of Domain 2. The problems of Domain 2, consequently, perpetuate the problems of Domain 1.

Through its responsibilities for continuous monitoring and evaluation of the Sukatech program, the Transverse Axis establishes itself as the driving force for the application and propagation of changes in its information systems. The specific goals and objectives listed by the Sukatech program management align with our diagnosis of problems observed from a data perspective; therefore, we believe that the execution of this change package will solve most of the identified pain points.

However, when mapping the goals, indicators, and sources indicated in the Work Plan and observing the monthly accountability cycles, we noticed that qualitative indicators (e.g., completed renovations) and quantitative indicators (e.g., number of certified students) of specific objectives are not explicitly accompanied by well-defined parameters of progress, regression, success, or failure. The definition of aspirational results in their entirety occurs punctually in specific cases (e.g., processing 250 tons of electronic waste per year, totaling 750 tons processed during the project execution period).

We have established that SECTI’s strategic planning uses the Objectives and Key Results (OKR) framework to set objectives and evaluate results for the Sukatech program. The Work Plan brings together structuring elements for the formalization of measurement processes for the Sukatech program, but these elements are presented in a disconnected or implicit way in its wording. Such presentation encourages the creation and use of subjective mental models of goal execution strategies, general and specific objectives by its executors, managers, and supervisors. In a context where key information systems activities are extraordinarily dependent on human actions at all points, we believe that distorted interpretations and the disconnection between organizational objectives and operations are the predominant factors in the propagation of errors throughout the Sukatech program’s activity execution chain.

In order to establish system to be, we presented a technical proposal for the systemic management of the two sets of problems faced by the Transverse Axis - Evaluation. We supplement the application of the Objectives and Key Results (OKR) framework with models and principles based on the GQM+Strategies approach.

We proposed the use of three formal models — Organizational Objective, GQM Objective, and Measurement Model — to create a body of knowledge about general and specific goals and objectives. Using instructional cards to facilitate strategic elicitation, we developed a methodology for documenting and communicating goals and objectives focused on data collection and analysis. Finally, we suggested a process of interpretation and learning through the use of relevant measurement models to support the decision-making processes of the Sukatech program.

Instructional cards: Organizational goal

Seven yellow cards reveal seven questions to ask whenever establishing an organizational goal: What do we want to transform? What characteristic of the object we want to transform? What's the size of that transformation? When will we complete that transformation? Who will execute it? What can prevent that transformation for happening? How does it interact with other objectives?

Instructional cards: GQM goal

Five blue cards reveal five questions to ask whenever establishing a GQM goal: What do we want to analyze? Why? What aspect we want to analyze? From what perspective? In what context?

Instructional cards: Measurement model

Four green cards reveal four questions to ask whenever establishing a measurement model: What do we want to know? What do we already know? What may affect our measurements? How that may affect our measurements?