Engaging Evidence Use in EdTech Purchasing: Building Stronger Product Pilots

Piloting is widely viewed as a best practice in EdTech procurement: 84% of districts agree that products should be piloted before purchase.¹ But for many districts, the pilot phase is the hardest part of the adoption process. Pilots are crucial for collecting relevant contextual data about how well a product works in real classrooms with real users—information you just can’t get from academic studies. Often, it’s the only opportunity to get feedback from students and teachers before purchasing, and it helps district leaders determine whether a product will support or disrupt classroom practices. 

However, the work of running a pilot and interpreting the data is overwhelming even for districts with robust procedures in place, let alone those with operational limitations. Districts have to deal with staffing and time constraints, inconsistent feedback collection processes, and behavioral biases that can subtly push decision-makers toward options that, while perhaps satisfactory, fall short of optimal. 

Too often, these barriers stand in the way of evidence use, pushing districts toward decisions rooted in anecdotal reports, intuitive judgment, or surface-level impressions rather than strong evidence of classroom fit and effectiveness. To strengthen EdTech purchasing decisions, districts need avenues to ensure high-quality instructional materials (HQIM) get a chance in classrooms, teachers have the time and capacity to test these tools properly, and educational leaders have ways to generate actionable feedback based on these pilots. 

In this article, we outline clear solutions to common EdTech piloting problems, calling for action from both product vendors and adopters to prioritize relevant data in every purchase decision.

Reducing Perceived Risk in Early-Stage Pilots

When it comes to piloting products, districts tend to gravitate toward known, familiar options. They can be understandably hesitant to test novel options without experiential reviews, especially as more products introduce emerging AI features. This is a common pain point for vendors who approach conferences as opportunities to generate leads, while district leaders tend to use these interactions to validate solutions already on their radar.² Smaller EdTech vendors can have a challenging time breaking into the market, even if they have high-quality products. Because new products have limited evidence of use in real classrooms, dedicating resources to pilot these products can feel unnecessarily risky. Concerns about data privacy further increase the perceived risk of testing unproven tools. By and large, districts don’t want to be early adopters.¹ 

For districts, this hesitance to be the first to pilot new products can mean overlooking high-quality options that may effectively address identified learning gaps. This pattern can stem from zero-risk bias: our natural tendency to choose options that completely eliminate risk, even if alternatives could lead to better overall outcomes. Zero-risk bias becomes pronounced in today’s crowded EdTech market—tech stacks are growing more expensive and difficult to manage, educators are experiencing technology fatigue, and concerns about student screen time are rising. In this environment, asking districts to take the chance on a new tool is increasingly challenging. In fact, districts are only getting more selective about the tools they use.³ 

This is an especially important insight for EdTech vendors. As districts continue prioritizing solutions that manage risk, buyers are setting the bar higher and higher.⁴ EdTech companies need to be prepared to prove their value and meet increasing standards. To satisfy district demands and position HQIM as trustworthy and low-risk, vendors should work closely with districts to provide pilot support in the form of implementation guidance, professional development, and context-specific demos. In fact, districts report that their relationships with vendors are second only to peer recommendations as indicators of quality.¹ By working to forge strong relationships with districts and validate outcomes with third-party certifications, EdTech vendors without an established market presence can encourage districts to give their high-quality products a chance.

Making Pilots Feasible Under Real-World Constraints

Running pilots is undeniably time-intensive. For districts already grappling with time constraints, staffing limitations, and cognitive overload, conducting thorough EdTech pilots is a significant challenge. It’s not unusual for a major tool to go through a year-long pilot, involving a lengthy process of training teachers, collecting feedback, and engaging students.⁵ Smaller districts without sufficient staff and time for these pilots may end up needing to rush the process or test tools less extensively—for instance, by conducting fewer classroom observations than originally planned.¹ This reduces the potential for pilots to produce relevant information regarding how well a tool aligns with classroom-specific needs. 

When pilots are rushed or scaled back to accommodate capacity limitations, district leaders are left making decisions based on incomplete data. As a result, they may be forced to rely on surface-level impressions, like “students seem to like it” or “teachers think it’s great,” rather than systematically comparing pilot results against district goals.¹ The result is that pilots become more procedural than learning-oriented. 

Effort-intensive pilots can also make decision-makers vulnerable to the sunk cost fallacy, a behavioral phenomenon where people become reluctant to abandon a course of action just because they have invested heavily in it, even if changing course would lead to better outcomes. Many district leaders are aware of the sunk cost problem in EdTech piloting, noting that the piloted product is often the one adopted because of the amount of time invested in the process.¹ However, awareness alone is rarely enough to eliminate the bias. Once resources have been used on a pilot, abandoning the product and starting over can feel like a waste.

Solutions to piloting challenges must work within a district’s existing constraints rather than assume ideal situations. When a complete pilot program is just not realistic, districts have other options. For example, they can work with vendors to ensure product demos are consistent across tools. This means they should demonstrate the same use case for the same audience, so tools can be compared fairly without conducting lengthy pilots.⁶ Instead of treating pilots as long, intensive trials, this allows districts to create shorter learning sprints with a tight and controlled scope. Going into pilots with a small number of high-value questions to answer also ensures pilots are focused on key signals that align with established learning gaps and needs.

Translating Pilot Feedback into Actionable Evidence

Data collection presents another area of friction during pilots. While engaging stakeholders is essential for making meaningful adoption decisions and facilitating buy-in for implementation, challenges with feedback collection can limit the beneficial impact of the pilot. Without structured feedback systems to follow, feedback often takes the form of anecdotal reports rather than robust, measurable data on student learning and engagement. 

Anecdotal feedback is difficult to analyze or compare across products, leaving districts unable to properly quantify a new tool’s potential impact. For instance, it becomes challenging to compare and contrast different product parameters, such as impact on student achievement and ease of use for teachers. Without structured evidence, districts are left relying on general impressions to inform product selection, increasing susceptibility to cognitive bias. ¹ 

This challenge underscores the importance of strong data collection processes that surface stakeholder perspectives and generate meaningful data that support careful interpretation of pilot results. For example, to generate feedback, districts can pair structured surveys and interviews with standardized classroom observation techniques.⁶ This approach generates a mix of qualitative and quantitative data to give decision-makers a more comprehensive picture of program efficacy, classroom fit, and stakeholder sentiment. Many EdTech vendors, third-party evaluators, and nonprofits provide rubrics to evaluate pilots, which can help districts create data collection measures and subsequently assess pilot success against various pre-established goals or metrics. Promoting the use of these tools can help districts standardize the pilot process and produce more actionable results.

Strengthening Pilots Through Communication and Coordinated Action

The single biggest challenge with the pilot phase in EdTech purchasing is that pilots often fail to function as true evidence-creation tools and instead become justification exercises that confirm decisions already leaning toward adoption. Hesitance to test new products, capacity constraints, and informal feedback collection procedures weaken districts’ ability to perform meaningful pilots, increasing vulnerability to biases that sway adoption decisions. 

Districts, vendors, and third parties each play a role in strengthening EdTech pilots. Vendors can work to forge strong relationships with districts to reduce uncertainty, encouraging districts to become early adopters of innovative HQIM. Districts with limited capacity for pilots can focus on shorter, focused demos, empowered by the resources and capacity-building support from vendors and education partners. Finally, districts can adopt established processes for conducting pilots to extract more meaningful data about classroom performance and fit. Overall, this coordinated effort can help ensure pilots are centered on student learning to support EdTech adoptions that have a real and meaningful impact on classrooms.

Sources

1. EdSignals Studio, Smarter Demand: Dimensions of Quality in Purchasing Decisions, 2022
2. EdSignals Studio. (2025). 2025 Annual Report. https://edsignals.org/edsignals-studio-2025-annual-report/ 
3. Ullmen, E. (May 16, 2025). Trimming the Edtech Fat: How Districts Are Streamlining Their Digital Ecosystems. EdSurge. https://www.edsurge.com/news/2025-05-16-trimming-the-edtech-fat-how-districts-are-streamlining-their-digital-ecosystems
4. Ng, A. (June 14, 2024). A New, Tougher ‘Bar of Entry’ for Ed-Tech Companies. EdWeek Market Brief. https://marketbrief.edweek.org/meeting-district-needs/a-new-tougher-bar-of-entry-for-ed-tech-companies/2024/06 
5. Ullmen, E. (December 1, 2025). Cleaning House: How Districts Are Rethinking Edtech. ASCD. https://www.ascd.org/el/articles/cleaning-house-how-districts-are-rethinking-edtech
6. McLemore, C. & Rae, J. (June 24, 2024). How District Leaders Make Edtech Purchasing Decisions. EdSurge. https://www.edsurge.com/news/2024-06-24-how-district-leaders-make-edtech-purchasing-decisions