The vision AI partner you choose determines two things. How many defects you catch today. And whether you can scale that capability across more lines, more sites, and more use cases over the next five years.
Most manufacturing evaluations focus on the first. The demo runs. Detection accuracy looks good. The price fits the budget. The decision is made.
Six months into production, the problems appear. The partner cannot integrate with the plant's MES without a custom build that costs more than the original contract. The model that worked for Body Shop inspection needs a full rebuild for Paint Shop. The governance documentation required for the quality audit does not exist in the format the auditor needs. The deployment that looked straightforward becomes a multi-year project.
77% of AI vision implementations in manufacturing never advance beyond the pilot phase. Most do not fail because the technology stopped working. They fail because the evaluation did not test for what production actually requires.
Here is what to evaluate before you commit.
Why Standard Software Evaluation Criteria Are Not Enough
Vision AI for manufacturing is not enterprise software in the traditional sense. It is a combination of hardware, software, AI models, data pipelines, and integration architecture that needs to operate reliably in an industrial environment.
The standard software evaluation criteria, features, pricing, vendor stability, support response times, do matter. But they do not tell you the things that determine whether a vision AI deployment will reach production and stay there.
What matters in manufacturing vision AI evaluation is different. It is about production conditions, not demo conditions. Integration depth, not surface connectivity. Model management at scale, not single-application performance. Governance documentation, not sales collateral. And production track record, not reference calls with early adopters who are still in pilot.
The Eight Questions That Separate Strong Partners From Weak Ones
1. Does the partner have documented production deployments at comparable scale in manufacturing environments like yours?
This is the first and most important question. Not whether the technology works in principle. Whether this partner has delivered it in production, at comparable scale, in a comparable manufacturing environment.
Ask for two or three specific reference deployments. Not case studies on the website. Named facilities with named contacts you can speak to directly. Ask those contacts specifically whether the deployment is in full production, what the detection accuracy is on the production line rather than in testing, and what problems they encountered during deployment that were not anticipated in the pilot.
A partner who hesitates on this question or offers only anonymised case studies has not delivered what they are proposing to deliver for you. That is important information.
2. How does the platform handle model management across multiple manufacturing zones?
A single vision AI application for one inspection station is manageable with almost any platform. A deployment spanning Body Shop, Paint Shop, Assembly, and Press Shop across multiple production lines is a model management challenge that most platforms handle poorly.
Ask specifically how the partner manages model versioning, retraining, and deployment across multiple zones simultaneously. What happens when the Body Shop model needs updating because a new vehicle variant has been introduced? Does that update affect the Paint Shop model? How long does a model update take to deploy to all edge devices in a zone? Who controls that process and what is the approval workflow?
Partners who have solved this problem have specific, detailed answers. Partners who have not will describe what the platform is theoretically capable of rather than how they have done it in practice.
3. What are the integration points with MES, ERP, and quality management systems, and how have you implemented them in production?
The value of vision AI in manufacturing is realised when inspection results connect to production decisions. A defect detection system that generates alerts on a separate screen that an operator monitors is not closing the loop. A system that sends a hold signal to the MES before the defective unit progresses is.
Ask the partner to describe specifically how their integration with your MES, ERP, and quality management systems works. Not what APIs they support. How they have implemented it in a production environment with a system similar to yours.
Native support for industrial protocols including EtherNet/IP, PROFINET, Modbus, and OPC UA is a baseline requirement, not a differentiator. The question is not whether the integration exists. It is how the partner has delivered it and what problems they encountered.
4. How does the platform handle edge inferencing and what are the connectivity requirements?
Manufacturing environments are not always well-connected. Some zones have reliable high-bandwidth connectivity. Others have intermittent connectivity, legacy network infrastructure, or physical environments that make reliable connectivity difficult.
Ask the partner what happens when the connection between the edge device and the central management layer is lost. Does the inspection system continue to operate? Does it degrade gracefully? Does it stop entirely? The answer tells you how the platform was designed, and whether it was designed for manufacturing environments or for data centre environments.
Also ask what the latency requirements are. A vision inspection system that needs to stop a production line when it detects a defect needs to return a result in milliseconds. A system that processes results in the cloud and sends them back cannot meet this requirement regardless of how good the connection is.
5. What governance and compliance documentation does the platform generate automatically?
IATF 16949, the international quality management standard for automotive production, requires traceability of inspection results. The EU AI Act classifies many industrial vision AI systems as high-risk. Automotive customer-specific requirements from OEMs add further documentation obligations.
Ask the partner what compliance documentation the platform generates automatically. Not what it can generate with additional configuration. What it produces as a standard output of the inspection process.
Specifically ask about inspection record completeness, model version documentation, defect classification audit trails, and the format in which these records are stored and retrievable. A partner who has never been through an IATF 16949 audit with their platform in scope will not have thought carefully about these requirements. A partner who has will have specific answers about what the auditor asked for and how the platform provided it.
6. What is the total cost of ownership over three years, not just the initial contract?
Initial deployment costs for vision AI in manufacturing range from $50,000 to $250,000 per application. That is the number that appears in the original contract. It is not the number that appears in the three-year total cost of ownership.
Infrastructure costs surpass development costs within 12 to 18 months in most manufacturing AI deployments. Model retraining when new product variants are introduced costs money and engineering time. Integration maintenance costs money when connected systems are updated. Hardware refresh cycles cost money when edge devices reach end of life.
Ask the partner to model the three-year total cost of ownership explicitly. Include compute infrastructure, model retraining, integration maintenance, hardware refresh, and the governance and compliance overhead. A partner who can only quote the initial deployment cost has not thought carefully about what it costs to operate their platform in production.
7. What does the deployment timeline look like from contract to production inspection?
The gap between signing a contract with a vision AI partner and having a system in production catching defects is where most manufacturing AI timelines exceed their original estimates.
Ask the partner to describe the specific steps between contract signature and full production operation. How long does site assessment take? How long does camera and lighting installation take? How long does model training take for your specific defect types? How long does MES integration take? What are the acceptance testing criteria and how long does that process take?
Partners with production deployment experience have specific timelines backed by actual projects. Partners without it will give you the theoretical minimum without the buffer that real deployments require.
8. What is your support model once the system is in production?
Vision AI systems in production encounter problems that were not anticipated during deployment. A new product variant changes the defect profile. Lighting conditions change seasonally. A camera accumulates contamination. A MES update breaks an integration. New production shift patterns generate edge cases the model has not seen before.
Ask the partner specifically how they handle these problems once the system is live. What is the support response time for a production inspection system that has stopped working? Who is responsible for model retraining when the defect profile changes? Is there an engineering team available to work inside your facility when problems require physical access?
Vision systems are long-term investments. The partner you need is the one who can support your applications as they evolve, not the one who delivered the initial deployment and moved on.
What to Look For in a Reference Call
The reference call is the most valuable step in any vision AI partner evaluation. Here is what to ask.
Ask whether the system is in full production or still in pilot. Full production means it is the primary quality gate for that inspection station, not running alongside manual inspection as a comparison.
Ask what the false positive rate is on the production line. A high false positive rate, where the system flags good parts as defective, means the line is being stopped unnecessarily. Operators lose trust in the system and start overriding it. This is a failure mode that does not appear in demo accuracy numbers.
Ask what went wrong during deployment that was not anticipated in the pilot. Every production deployment encounters problems. A reference contact who says the deployment was seamless has not been asked this question directly before, or is not being candid. A reference who describes specific problems and how they were resolved gives you valuable information about the partner's problem-solving capability.
Ask whether the MES integration is working as intended. This is the integration that closes the loop between detection and production decision. If it is working, the reference will describe specific examples of defects being caught and production decisions being made automatically. If it is not, the inspection results are still reaching an operator who then manually decides what to do.
The Deployment Model That Matters as Much as the Platform
The platform is one part of what determines whether a vision AI deployment reaches production. The deployment model is the other.
A vision AI platform implemented by engineers who have never worked inside a manufacturing facility will encounter problems that are only visible from inside the production environment. Lighting conditions that change across shifts. Network infrastructure that behaves differently under production load. Production schedules that cannot accommodate installation downtime. Quality management workflows that were not documented in the RFI response.
The deployment model that consistently produces production outcomes is one where the implementation engineers work inside the client's actual environment. They see the production conditions directly. They design the integration against the actual MES configuration, not the documented API. They train the model on images captured under actual production lighting conditions, not in a controlled setup.
This is the forward deployed engineering principle applied to vision AI. The complexity of manufacturing environments is only fully visible from the inside. The engineering team that works inside the environment discovers it before it becomes a production problem.
Vishleshan AI's forward deployed engineering (FDE) approach to vision AI deployment in manufacturing works inside client environments from day one. We design integration against actual MES and ERP configurations. We train models on images captured under production conditions. We build governance documentation into the deployment architecture from the start. Our manufacturing AI experience spans automotive and industrial production environments where the evaluation criteria above are not theoretical. They are what we have been tested against in production. Book a Consultation