Contact

Contact HaxiTAG for enterprise services, consulting, and product trials.

Showing posts with label enterprise AI applications. Show all posts
Showing posts with label enterprise AI applications. Show all posts

Friday, July 3, 2026

When AI Agents Enter the Office: The Hidden Management War Behind the ServiceNow Case

 

An Unexpected Overtime in the Age of AI

On an autumn night in 2025, only one light remained on in the building housing the IT service desk of the City of Raleigh, North Carolina.

Under that light, the service desk supervisor stared at a stream of conversation logs on his screen, brow furrowed. A month earlier, he had been thrilled about the city’s adoption of an AI agent—the ServiceNow-deployed IT Helpdesk Agent, which promised to automatically handle high-frequency issues such as password resets and software installation guidance. He had naively believed this would give his team some breathing room.

But now, his workload had increased rather than decreased.

During the day, he managed his five human employees; at night, he had to “train” the AI agent—correcting its mistakes, checking for missed information, and monitoring every interaction it had with citizens. What he had thought would be a digital colleague now felt more like an intern requiring round-the-clock supervision.

He couldn’t help but recall a remark from Jacqui Canney, ServiceNow’s Chief People and AI Enablement Officer:

“Managers had a hard job before. Now, they have a harder job.”

This is merely the tip of the iceberg. A quiet battle over the “hidden management cost” of AI agents is now unfolding within organizations across the globe.


ServiceNow’s Shift: From Copilot to Agent

1. Early 2025

The enterprise software giant ServiceNow made a strategic decision: to fully embrace AI agents.

Unlike the “copilot” model popular in 2023–2024—where humans write prompts and AI assists with fragmented tasks—ServiceNow’s AI agents were given a brand‑new identity: autonomous executors.

These intelligent agents, called “AI Specialists,” can complete entire workflows end‑to‑end in three domains: IT service management, customer relationship management, and security and risk. They no longer require human confirmation at every step; instead, they receive a goal, break down tasks, call tools, and deliver results—much like a real employee.

At the same time, ServiceNow launched its “AI Control Tower”—a command center designed for management. What can this control tower do?

  • Observe agent behavior: replay every decision path like a dashboard camera.
  • Track ROI: precisely calculate how much money each agent call costs and how much value it creates.
  • Govern and circuit‑break: allow managers to intervene with one click when agent behavior goes off track or costs become abnormal.

ServiceNow’s hope was that companies would buy AI agents like software and manage them effortlessly through the control tower.

But the real story is far more complex than the product brochures suggest.

2. A Contradictory Signal: Management Isn’t Getting Easier

In an interview with The Deep View, Jacqui Canney revealed a telling nuance:

“I actually hope our managers aren’t thinking ‘I’ve got five agents on my team.’ Instead, they should see agents as embedded parts of new workflows.”

The subtext: ServiceNow has discovered that if managers continue to treat agents as “colleagues,” they will fall into a huge management trap.

What trap? Blurred responsibility.

When traditional software fails, it’s a bug—call IT to fix it. When a human employee fails, it’s a performance issue—the manager has a conversation. But when an AI agent fails?

  • Is it insufficient model capability?
  • Is it bias in the training data?
  • Is it the manager failing to set up the right prompts?
  • Or is it the user’s unclear question?

No one is naturally responsible for it. Yet the person who ends up cleaning up the mess is still that supervisor sitting in front of the service desk.


Raleigh’s Front‑Line Report: A Manager’s Nightmare Week

Scene: The First Month of a Municipal IT Service Desk

Let’s return to Raleigh.

Chief Information Officer Mark Wittenburg described the case to the media in detail:

After deploying the AI agent, the service desk supervisor’s first week was a shock.

Monday: The agent went live. The supervisor spent half a day manually importing the city’s internal knowledge base—3,000 frequently‑asked questions, system permission guides, emergency procedures—into the agent’s training set. But import was not a one‑time task; because the agent kept encountering new questions not covered in the knowledge base, the supervisor had to supplement it constantly.

Wednesday: The agent began answering independently. The supervisor found that for clear instructions like “reset my password,” the agent performed well. But for ambiguous descriptions such as “I can’t log in—maybe my account is locked, or maybe it’s a browser issue,” the agent started giving incomplete or wrong answers. The supervisor had to label every suspicious session: “Correct,” “Partially Correct,” or “Wrong.”

Friday: The agent had processed 147 requests with an accuracy rate of about 82%. That meant the supervisor had to manually verify the 18% of erroneous cases, apologize to users, and make corrections. Worse, he discovered that in one session the agent had inadvertently leaked an internal server IP address—fortunately not causing a security incident, but it sent a chill down his spine.

Week summary: The supervisor originally spent 40% of his time on team management and process optimization. Now that 40% was entirely consumed by agent supervision, plus an extra two hours in the evening. His team did feel some relief (the agent handled repetitive tasks), but he himself was trapped in an unprecedented, high‑intensity “human‑machine sandwich” state.

Wittenburg admitted:

“That’s been a transition for the supervisor.”

The Core of the Conflict: The “Invisible Transfer” of Management Duties

Companies often calculate the ROI of AI agents using a financial model: software license fees + API call fees + implementation costs, compared to labor hours saved. But in Raleigh’s case, one hidden cost item was completely overlooked: the manager’s attention cost.

The manager’s role shifted from “managing people” to “managing people + managing agents + managing human‑agent interaction quality.” These three activities require entirely different skill sets:

  • Managing people: needs empathy, motivation, performance feedback.
  • Managing agents: needs technical understanding, log analysis, model debugging.
  • Managing human‑agent interaction quality: needs process design, exception handling, rapid decision‑making.

Few managers possess all three capabilities. Even fewer companies provide additional training or compensation for this.


Financial and Security Time Bombs

If Raleigh’s story brought “management burden” to the surface, a warning from Jayney Howson, ServiceNow’s Chief Learning Officer, shone a spotlight on another layer of hidden cost: token economics.

1. Token: The “New Oil” of the AI Era

In the world of generative AI, a “token” is the billing unit, roughly equivalent to 0.75 English words. On the surface, the cost of a single API call is pitifully low—a few dollars per million tokens. But when AI agents run at high frequency across an organization, the bill expands rapidly.

Howson points to a troubling trend: the combined use of employees and agents is quietly driving up costs.

For example: A marketing specialist uses an AI agent to draft a customer email (agent calls GPT‑4, consumes 500 tokens), then manually revises it and asks the agent to polish it again (another 300 tokens). Next, the agent calls an internal database to pull last week’s sales data and generates an analysis paragraph (1,500 tokens). A single simple email task can involve hundreds of token calls—while the manager remains unaware, until month‑end when a bill for tens of thousands of dollars arrives.

Even more alarming is the risk of data leakage. When agents are granted access to sensitive internal systems (e.g., HR systems, customer databases), every call may transmit data fragments to external models. Without strict “data permission boundaries” and “output auditing,” a small mistake can become a major compliance disaster.

Howson’s original words:

“If managers aren’t prepared, they will be left cleaning up the mess.”

2. A Fictional Yet Highly Realistic Scenario

Consider a typical mid‑sized company:

  • Deployed 5 AI agents (IT support, HR Q&A, sales assistant, finance reconciliation, compliance review).
  • 200 employees frequently interact with the agents every day.
  • Agents call each other to complete complex tasks (e.g., “generate a contract for a new client and check compliance”), forming agent chains.

After one month, the manager faces three “surprises”:

  1. Billing surprise: The expected monthly AI cost of $5,000 becomes $35,000. The cause: unsupervised circular calls between agents (A calls B, B calls A, infinite recursion).
  2. Security surprise: While processing a contract, the compliance review agent sends the client’s non‑public financial data as context to a third‑party model, whose logs are externally accessible.
  3. Labor surprise: The IT manager spends an entire week manually tracing the abnormal call chain, writing new guardrails, and explaining and remediating the situation with affected clients.

In this mess‑cleaning process, no one rewards the manager. Instead, executives only ask: “Why did AI governance get out of control?”


AI Agents Are Not Employees—They Are a Runtime System

The ServiceNow case is a landmark because it forces the entire industry to confront a fundamental question:

What exactly are we managing?

The traditional answer: people and tools. The new answer: a runtime system that exhibits autonomous behavior, continuously evolves, and blurs accountability.

1. From “Software Lifecycle” to “AI Runtime Governance”

In the past, enterprise software deployment followed the classic “requirements‑development‑testing‑launch‑maintenance” model. After launch, software behavior was deterministic—Excel doesn’t make calculation errors because it’s in a bad mood.

But an AI agent is entirely different:

  • It has no finite state machine; its behavior is based on probabilistic models.
  • Its output shifts with model version updates, prompt tuning, and context changes.
  • Its errors are emergent—even if each step is correct, the combination can be absurd.

This means enterprises can no longer manage AI agents like they manage software. They must establish a completely new governance paradigm: runtime governance.

Runtime governance demands:

  • Real‑time monitoring: not a weekly review, but tracking the agent’s decision path every second.
  • Dynamic guardrails: not predefined rules, but real‑time adjustments of permissions and boundaries based on agent behavior.
  • Accountability tracing: every error must be attributable to a specific model, prompt, data, or management action.

ServiceNow’s AI Control Tower is essentially an attempt to implement this runtime governance. But as the Raleigh case shows, tools alone are far from enough—managers need new skills, new organizational support, and new incentives.

2. The Future of Managers: From “Running the Business” to “Running AI Operations”

The most powerful sentence in the case comes from Jacqui Canney:

“I actually hope our managers aren’t thinking ‘I’ve got five agents on my team.’ Instead, they should see agents as embedded parts of new workflows.”

This is not just a change in wording; it is a fundamental shift in worldview.

“Having agents on the team” means the manager still sees themselves as a manager of people, with agents as extra “digital subordinates.” This mindset leads the manager to micro‑manage every agent, ultimately sinking into the quagmire of micromanagement.

“Workflows are re‑embedded by AI” means the manager’s core task becomes designing, maintaining, and optimizing human‑machine hybrid workflows. Under this view:

  • AI agents are not subordinates; they are autonomous nodes in the process.
  • The manager’s value is no longer “controlling every step” but “ensuring the entire process converges on cost, quality, and risk.”

This requires managers to possess three new core capabilities:

  1. Process engineering: ability to map business flows and identify which steps are suitable for agents and which must remain human.
  2. AI economics: ability to calculate token ROI for each step and optimize calling strategies.
  3. Exception design: ability to pre‑set automatic fallbacks, human backup, and post‑incident recovery mechanisms when agents fail.

Unfortunately, the vast majority of mid‑level managers today do not have these capabilities. Even more unfortunately, no business school systematically teaches “AI process governance.”


The True Watershed for Enterprise AI Adoption

The ServiceNow case leaves us not with an easy answer, but with a heavy exam.

Question No. 1: Are You Willing to Acknowledge the “Hidden Costs”?

Many companies still calculate AI agent ROI using Excel models and are delighted to find “payback in less than six months.” But they overlook:

  • How much is the manager’s extra time worth, converted into salary?
  • How much customer churn cost is caused by agent errors?
  • How much additional insurance and audit expense is incurred due to data leakage risks?
  • How much “decision‑delay cost” arises from unpredictable agent behavior?

Acknowledging these hidden costs is the first step to maturity.

Question No. 2: Are You Willing to Restructure Management Capability Models?

ServiceNow has already begun internal action: they have formally incorporated “AI enablement” into all managers’ job descriptions and established a mandatory “AI governance certification” course. The curriculum includes:

  • How to read an agent’s trace log?
  • How to set prompt guardrails?
  • How to calculate token ROI for each process?
  • How to design human‑machine breakpoints?

This is no longer a “nice‑to‑have” skill; it is a fundamental capability for future managers.

Question No. 3: Are You Willing to Invest in “AI Observability Infrastructure”?

Without measurement, there is no management. ServiceNow’s AI Control Tower provides a template, but it may not fit every enterprise. The key is that enterprises need to build their own:

  • Agent behavior logging system: record every input, output, and intermediate reasoning step.
  • Cost attribution system: trace token consumption to specific departments, processes, managers, and agent instances.
  • Anomaly circuit‑breaking system: automatically pause and notify the manager when a single agent’s call cost exceeds a threshold or when sensitive data is attempted to be transmitted.

The construction cost of these infrastructures is not trivial, but they are the only guarantee against “uncontrolled chaos.”


The Real Winners in the Age of AI Agents

Today, in 2025, every tech company is talking about AI agents. But the ServiceNow case reveals a sobering truth:

The first enterprises to deploy AI agents are not necessarily the winners. The real winners are those that can govern AI agents with the lowest management cost and the highest reliability.

What will become of that service desk supervisor in Raleigh? If he receives adequate training and tools, he may slowly transform from an “agent babysitter” into a “workflow architect”—no longer checking every agent answer line by line, but designing an automated quality sampling and feedback loop. His team will no longer be bogged down by simple repetitive labor, but will focus on complex requests that truly require human empathy and judgment.

If he does not receive support? He will burn out, resign, and become another silent casualty on the road to enterprise AI transformation.

And the ultimate message from ServiceNow is:

Don’t ask “What can AI agents do?” Ask “Is our organization ready to manage AI agents?”

This quiet war over “hidden management costs” has just begun. The enterprises that win this war will define the organizational form of the next decade. And those that fixate solely on technical ROI while ignoring governance systems will eventually discover—

The most expensive cost is always the one that never appears on an invoice, hidden in the tired eyes of managers and the low‑value redundancy that employees are forced to perform.

Related topic:

Wednesday, May 6, 2026

CyberAgent's Enterprise-Level AI Agent Deployment: Unpacking the 93% Active User Rate Through Voluntary Adoption Strategy

Case Overview and Core Themes

Company Background and AI Strategic Framework

CyberAgent, a leading Japanese internet company with diversified business operations spanning advertising, media and IP, as well as gaming sectors, stands as a representative enterprise in the Asia-Pacific technology, media, and entertainment industries. The company's journey into artificial intelligence began as early as 2016, when it established a dedicated AI laboratory (AI Lab) focused on AI research and development related to digital marketing. This early strategic investment laid a solid technical foundation and cultivated an organizational culture that would later prove instrumental in the successful deployment of enterprise-level AI agents.

In 2020, CyberAgent launched the "Kiwami Prediction AI" system, specifically designed for intelligent optimization of advertising creative production. By 2023, the company further established the "AI Operations Office" to oversee the construction of an enterprise-level AI application framework and governance system at the organizational level. This clearly delineated developmental trajectory demonstrates CyberAgent's strategic positioning of AI as a core organizational asset rather than merely a technological tool.

Core Deployed Products and Tool Ecosystem

In terms of specific product deployment, CyberAgent adopted a dual-core tool strategy. ChatGPT Enterprise serves as a general-purpose AI assistant, primarily addressing daily office scenarios including research analysis, content creation, and information organization. Codex functions as a professional-grade programming assistant, covering specialized development workflows such as code review, design discussions, documentation, and development planning. This clearly differentiated tool configuration strategy not only satisfies the diverse business needs of the enterprise but also ensures deep application value in specialized scenarios.

Central Theme: Voluntary Adoption and Culture-Driven AI Integration

The most remarkable aspect of the CyberAgent case lies in its distinctive approach characterized by a "non-mandatory, voluntary adoption" strategy. Without implementing any compulsory usage policies, ChatGPT Enterprise achieved a remarkable 93% monthly active user rate, with usage spanning virtually all departments and over 100 employees participating in more than ten training sessions. This achievement subverts the conventional wisdom that "mandatory enforcement is necessary to ensure adoption rates" in traditional enterprise software deployment, revealing instead the possibilities that emerge when AI achieves deep organizational penetration through cultural construction and knowledge sharing.

In-Depth Analysis of Application Scenarios and Effectiveness Assessment

Multi-Scenario Application Practices of ChatGPT Enterprise

Within daily office operations, the application of ChatGPT Enterprise exhibits remarkable breadth and depth. Research analysts leverage it for rapid market intelligence consolidation and competitive analysis. Content operations teams utilize it for copywriting and creative brainstorming. Product managers employ it for structured documentation of requirements and efficient meeting minutes generation. Crucially, ChatGPT Enterprise does not simply replace human work; instead, it assumes the role of a "thinking partner," helping employees gain multi-dimensional reference information in complex decision-making scenarios.

In terms of information security, CyberAgent fully leveraged the enterprise-grade security capabilities of ChatGPT Enterprise, including account management, usage visibility, and access control. The company established a comprehensive internal guideline system that clearly delineates acceptable information types for AI tool input while implementing strict protection for confidential data. This security governance framework achieves an effective balance between AI application scalability and data protection.

Deep Integration of Codex in Development Workflows

The introduction of Codex brought significant transformation to CyberAgent's development workflow. In design review processes, Codex can comprehensively evaluate and stress-test design proposals from multiple perspectives, helping teams achieve more thorough consensus before implementation and significantly reducing rework caused by design flaws. Developer Hidekazu Hora remarked: "Codex functions like a reliable partner, supporting the entire process from discussing implementation approaches to execution, effectively enhancing development speed."

In the code review dimension, Codex not only generates improvement suggestions but also assists teams in selecting optimal options among multiple alternatives. Notably, Codex's value extends beyond mere coding speed improvement to systematic enhancement of development quality. As Sou Yoshihara, a senior Codex power user from the AI Business Division, evaluated: "Compared with other programming models, Codex gives the impression of producing higher-quality proposals. It is not merely a tool but rather a methodology for optimizing the overall development process."

Signature Project Cases: Kiwami Prediction AI and WormEscape

The Kiwami Prediction AI project deeply applied Codex's MCP (Model Context Protocol) capabilities during its design and implementation planning phases, achieving high-integration AI capability with the professional development environment through the Cursor editor. This case demonstrates how AI Agent capabilities can be seamlessly embedded within existing professional development toolchains.

The development cycle for the WormEscape game was completed for a soft launch in approximately one month, with Codex playing a pivotal role. This case powerfully validates AI Agent's practical value in accelerating product development cycles while demonstrating that AI can effectively help developers rapidly overcome knowledge barriers even in areas where they lack prior experience.

Utility Analysis and Value Assessment

Dual-Dimensional Examination of Quantitative Metrics and Qualitative Benefits

From a quantitative perspective, the 93% monthly active user rate, participation exceeding 100 employees per training session across more than ten sessions, and usage coverage spanning virtually all departments—these metrics fully validate the high penetration and acceptance of AI tools within CyberAgent. However, what deserves greater attention are the driving factors and sustainability mechanisms behind this success.

From a qualitative dimension, CyberAgent's AI application achieves multi-layered value: enhanced decision quality—through multi-perspective analysis supporting more comprehensive judgment; improved collaboration efficiency—the application of Codex in design reviews significantly reduced internal communication costs and rework frequency; strengthened knowledge transfer—AI tools emerged as effective supplementary means for newcomers to rapidly familiarize themselves with business and technology; unleashed innovation capacity—employees liberated from repetitive tasks channeled more energy into creative endeavors.

The Success Logic Behind the Non-Mandatory Strategy

CyberAgent's choice to forgo mandatory adoption policies achieved high penetration rates through the following mechanisms:

Knowledge sharing mechanisms constitute the core driving force. Internal promotion of effective prompts and successful application cases created a virtuous knowledge dissemination network. Rather than being compelled to use AI, employees proactively learned and experimented after witnessing high-value applications by colleagues. This bottom-up diffusion model possesses stronger sustainability and deeper penetration than top-down administrative mandates.

Visibility-based incentives likewise played a significant role. The company established an internal usage ranking system; while data was not used for performance evaluation, it provided employees with benchmarks for self-reference and target pursuit. This transparent feedback mechanism satisfied employees' cognitive needs for self-improvement while avoiding resistance stemming from coercion.

Automated follow-ups ensured implementation continuity. For employees who had not used the tools for extended periods, the system automatically sent reminders via Slack, though these follow-ups represented gentle guidance rather than mandatory requirements. This design respected employees' learning pace while ensuring sustained tool promotion.

Tiered training systems addressed differentiated needs. Training courses spanning from beginner to advanced levels covered employees of varying roles and skill levels, ensuring everyone could find a suitable learning path.

The Art of Balancing Security and Scalability

In advancing AI applications, CyberAgent fully recognized the prerequisite importance of security governance. Through establishing clear internal guidelines, strict account management systems, and usage visibility functions, the company effectively controlled information security risks while expanding AI application scope. As Ken Takao, Manager of the Data Technology Department, summarized: "With enterprise features such as account management and visibility into usage, ChatGPT Enterprise made it possible to support business use of a wide range of information, excluding confidential data. As a result, the scope of AI use across the company has expanded, and many employees now integrate AI into their daily work."

Inspirational Significance and the Elevation of AI Intelligence Applications

Universal Lessons for the Industry

CyberAgent's practices provide invaluable reference frameworks for enterprise-level AI Agent deployment. First and foremost, the priority of cultural construction should proceed in parallel with technology deployment. The achievement of a 93% active user rate reflects, on the surface, the success of tools, but at a deeper level, represents a triumph of organizational culture. When employees perceive AI as a partner enhancing their capabilities rather than a surveillance mechanism or replacement threat, voluntary adoption becomes the natural outcome.

Secondly, gradual expansion outperforms radical replacement. CyberAgent did not attempt to replace all work with AI in a single stride; instead, it progressively expanded AI application boundaries through continuous scenario discovery and successful case sharing. This strategy reduced transformation resistance, cultivated employees' AI literacy, and created conditions for subsequently deeper integration.

Thirdly, the value positioning of tools determines the depth of application. Positioning AI as a "quality judgment improvement tool" rather than a mere "speed enhancement tool" elevated Codex's application value beyond simple efficiency calculations, extending into higher dimensions such as decision quality, workflow optimization, and professional capability enhancement.

Industry Trend Insights on AI Agent Development

The CyberAgent case reflects several significant trends in the AI Agent field. From the technology integration dimension, AI agents are evolving from independent tools toward deeply embedded workflow components. The integration of Codex with Cursor through the MCP protocol demonstrates how AI capability can be seamlessly connected with professional development environments to unlock greater value.

From the role positioning dimension, AI agents are transitioning from "executors" to "collaborative partners." Employee feedback consistently emphasized AI's auxiliary value in discussion, review, and decision-making processes requiring human judgment, rather than merely replacement functions at the execution level.

From the governance model dimension, enterprise AI applications are forming a三位一体 (three-in-one) advancement paradigm of "security first, value-driven, culture-supported." Pure technology deployment cannot guarantee success; radical promotion lacking security frameworks carries substantial risks; and strategies lacking cultural support struggle to sustain.

Prospects for Intelligent Applications Toward the Future

CyberAgent regards AI as a pivotal technology that may become part of the next-generation internet industry standard. This judgment carries profound strategic insight. When AI capabilities become part of the work infrastructure, enterprise competitive advantages will no longer derive merely from "whether AI is used," but rather from "how AI is deeply integrated to unlock unique value."

For enterprises planning AI Agent deployment, the CyberAgent case provides a clear success pathway: establish a forward-looking AI strategic framework (such as the creation of an AI Operations Office); construct a comprehensive security governance system (application of enterprise-grade security features and establishment of internal guidelines); design culture-driven promotion mechanisms (knowledge sharing, voluntary adoption, tiered training); pursue deep integration rather than superficial application (embed AI into core workflows to enhance decision quality and development quality).

Conclusion

The CyberAgent AI Agent enterprise-level deployment case serves as a profound textbook on successfully transforming cutting-edge AI technology into organizational productivity. Behind its 93% monthly active user rate lies the power of culture rather than the pressure of coercion. The quality improvements brought by Codex reflect deep practice of human-machine collaboration philosophy rather than simple tool replacement logic.

The core value of this case lies in revealing the success equation for enterprise AI Agent deployment: advanced technological tools + comprehensive security governance + voluntarily-driven cultural mechanisms = sustainable deep application. As AI Agent technology continues to evolve, CyberAgent's experience reminds us that the decisive factor in technological success often lies not in the technology itself but in the depth of integration between technology, organization, and culture.

Related topic:

Thursday, April 23, 2026

Enterprise AI Inference Security Architecture: A Deep Dive into On-Premise Deployment vs. Public Cloud Services

When enterprises introduce AI capabilities, they face a fundamental security decision: Should they deploy models and inference services on their own infrastructure (on-premise/private deployment), or leverage public cloud AI inference services? This choice not only affects costs and performance but also profoundly determines the enterprise's data security posture, compliance capabilities, and risk exposure surface. Recently, Omdia's report "Rethinking Critical AI Infrastructure" shared significant research findings. Drawing from the report's key data insights and conclusions, along with fundamental security architecture principles, this article conducts a systematic analysis across four dimensions—threat models, compliance constraints, supply chain risks, and practical validation methodologies—to provide enterprise decision-makers with a clear security assessment framework and actionable verification pathways.


The Essence of LLM Inference Security: Where the Data Goes, the Risk Follows

The core security proposition of AI inference services is: To what extent does the enterprise's proprietary data (queries, context, feedback, internal information, knowledge, know-how, and core business data) leave its own control boundary?

Standard public cloud inference service workflow:

Enterprise Application → Send Prompt (with sensitive data) → Cloud Provider API → Model Processing → Return Results

In this process, both the enterprise's input data and output results pass through the cloud provider's infrastructure. Even though cloud vendors promise "not used for training," data remains exposed to risks across transmission channels, server-side logs, memory dumps, and operator access points.

On-premise/private deployment (including on-premises servers, enterprise-controlled private clouds, and local inference on endpoint devices) differs fundamentally:

Enterprise Application → Local Model → Return Results

Data physically remains within the enterprise boundary, fundamentally eliminating risks of transmission and third-party access.

Omdia's survey validates this understanding: 76% of enterprises worry about data breaches caused by cloud services, while 99% of enterprises use proprietary data in AI workflows. The tension between these two figures is the core driving force behind the security value of on-premise deployment.


Comparative Analysis from a Security Perspective: On-Premise vs. Public Cloud

Threat Model Comparison

Risk DimensionPublic Cloud Inference ServiceOn-Premise Deployment
Data Breach in TransitExists (TLS encrypts, but endpoints and keys managed by cloud provider)None (data doesn't leave internal network or device)
Server-side Data ResidueExists (logs, cache, debug dumps may retain user data)Controllable (enterprise configures log policies independently)
Cloud Provider Internal Personnel AccessExists (requires trust in cloud provider's employee behavior controls)None (or reduced to enterprise internal IAM controls)
Multi-tenant Side-channel AttacksTheoretically exists (GPU sharing, memory isolation risks)None (exclusive resource allocation)
Compliance Data Cross-borderHigh risk (user data may route to overseas regions)Avoidable (enterprise controls physical data location)
Model Supply Chain SecurityBlack box (enterprise cannot verify if model contains backdoors or bias)Transparent (can use open-source or self-developed models, fully auditable)
API Key Leakage RiskExists (key management becomes new attack surface)Not applicable

Special Considerations for Compliance Constraints

For regulated industries (finance, healthcare, government, legal), compliance requirements often directly exclude public cloud inference:

  • Data Residency Regulations: EU GDPR, China's Data Security Law, and US HIPAA all require that specific data not leave the country. While cloud providers can meet regional requirements, their global operational systems may still expose data to overseas support personnel.
  • Audit Traceability: On-premise deployment can provide complete internal audit logs (who, when, and what data was queried), while cloud service logs are controlled by the cloud provider, making it difficult for enterprises to obtain comprehensive audit trails.
  • Third-party Data Processing: Many enterprises' customer contracts explicitly prohibit providing data to third parties (including cloud providers as "data processors"). On-premise deployment can avoid triggering this clause.

Omdia's report notes that only 9% of enterprises believe their strategic AI partners fully meet their requirements, with security and compliance being the primary gaps.

Underestimated Risk: Model Supply Chain Security

Public cloud inference services typically offer "closed models" (e.g., GPT-5, Claude 4.6). Enterprises cannot:

  • Audit whether the model's training data contains infringement or bias
  • Verify whether the model contains backdoors or data poisoning attacks
  • Ensure the model's inference behavior complies with enterprise security policies

With on-premise deployment using open-source models (e.g., Kimi 1.5, MiniMax 2.5, Qwen 3.5), enterprises can:

  • Review model cards and training data sources
  • Run security scanning tools to detect backdoors
  • Perform additional security alignment fine-tuning on the model

This represents a new extension of supply chain security in the AI era—models are software, and closed-source models have zero supply chain transparency.


How to Make the Right Decision for Your Enterprise

Security decisions should not be based on intuition or vendor marketing. Below is a four-step validation framework to help enterprises quantitatively assess the security suitability of on-premise versus public cloud solutions.

Step 1: Data Classification and Risk Mapping

Operation: Classify all data that might enter the AI system into three levels:

LevelDefinitionExamplesRecommended Deployment Mode
L3 - Extremely SensitiveDisclosure would cause significant legal/financial/reputational damagePatient health information, personal identity information, unpublished financial reports, source codeMandatory on-premise (on-prem or edge)
L2 - Moderately SensitiveDisclosure has some impact but is manageableInternal meeting minutes, non-confidential product documentsOn-premise preferred, or strict DPA with cloud provider
L1 - Low SensitivityPublicly available informationPublic market data, published product descriptionsPublic cloud acceptable

Step 2: Threat Modeling and Attack Path Analysis

For the selected public cloud inference service, map out complete attack paths:

[Employee Endpoint] → (API Key Leakage) → [Cloud API Gateway] → (Man-in-the-Middle Attack) → [Inference Server] → (Memory Dump) → [Log System]

Evaluate each path for:

  • Attack feasibility (technical门槛)
  • Potential impact (data exposure volume)
  • Existing control measures (guarantees provided by cloud provider)

If unacceptable risk paths exist (e.g., "cloud provider operations personnel can directly read user prompts"), on-premise deployment becomes a necessary condition.

Step 3: On-Premise Deployment Feasibility Validation (Pilot)

Select 1-2 typical AI use cases at L2/L3 level for on-premise deployment pilot:

Pilot Option A - Edge Inference:

  • Hardware: Employee existing endpoints (e.g., 16GB RAM laptops) or uniformly procured high-memory devices
  • Models: Open-source models with <10 billion parameters (e.g., Qwen-7B, Llama 3 8B), using 4-bit quantization
  • Tools: Ollama, llama.cpp, MLX
  • Validation metrics: Inference latency, zero data exfiltration (confirmed via network packet capture), user experience

Pilot Option B - Private Cloud Inference:

  • Hardware: Enterprise internal GPU servers (e.g., 2x A10)
  • Models: vLLM or TGI deployment framework
  • Comparison: Latency, throughput, and operational costs versus public cloud APIs

Step 4: Residual Risk Acceptance Decision

After validation, form a risk matrix:

Deployment ModeMajor Residual RisksAcceptability Judgment
Public CloudCloud provider internal access, compliance violations, opaque supply chainL1 data only
On-PremiseHardware failure, malicious internal employees, model capability ceilingMitigated through access control and monitoring

Key Decision Principle: Security is not "no risk," but "risk is controllable." For L3 data, the residual risk of on-premise deployment (internal personnel) is far lower than public cloud (external + internal), and should be mandatory.


Practical Case Analysis: Real-World Paths for Enterprise Security Validation

Based on Omdia's report and industry practices, here are security validation results from two typical industries:

Case 1: Multinational Financial Institution (Fortune 500)

  • Scenario: Using AI to analyze suspicious patterns in transaction flows
  • Data Sensitivity: L3 (customer account information, transaction amounts)
  • Initial Plan: Using a certain public cloud AI API for prototype testing
  • Issues Discovered:
  • Compliance team found cloud API logs retained account information in prompts, violating internal data retention policies
  • Security audit showed API calls might route through overseas data centers, violating data residency requirements
  • Validation Action: Deployed Llama 3 70B (post-quantization) on internal GPU clusters; inference latency increased by 15%, but fully compliant
  • Final Decision: All inference involving real transaction data migrated to on-premise; cloud APIs retained only for public data testing

Case 2: Medical AI Startup

  • Scenario: Extracting structured diagnostic information from physician notes
  • Data Sensitivity: L3 (Protected Health Information/PHI)
  • Initial Plan: Planning to use publicly hosted open-source model services
  • Issues Discovered:
  • HIPAA requirements mandate signing business cooperation agreements with cloud providers, but the startup couldn't afford audit costs
  • Some patient data carries "de-identification" risk; any transmission constitutes a violation
  • Validation Action: Running Mistral 7B model locally on MacBook Pro (64GB RAM); data never leaves the laptop
  • Final Decision: All PHI processing completed on-device; cloud services only handle anonymized statistical information

Security Is Not Black and White, But Structured Decision-Making Is Possible

Core Conclusions

  1. Security boundaries are determined by physical data location. No matter how public cloud inference services encrypt or authenticate, they cannot change the fact that "data leaves the enterprise's control domain." For extremely sensitive data, on-premise deployment is the only choice that aligns with zero-trust architecture.

  2. The security advantages of on-premise deployment extend beyond breach prevention to include auditability, controllability, and isolation. Enterprises can independently decide log retention, access permissions, and model versions, unaffected by cloud provider policy changes.

  3. Model supply chain security is an emerging high-priority risk. Using closed-source cloud models means fully delegating inference logic security to third parties; enterprises cannot verify whether models contain backdoors, bias, or poisoning. On-premise deployment combined with open-source models provides full-stack transparency.

  4. "Hybrid security architecture" is a pragmatic path. Not all data requires equal protection. Enterprises should establish data classification systems: L3 data mandates on-premise deployment, L2 data prioritizes on-premise but can accept strict DPAs, and L1 data can safely use public cloud services.

Omdia Report's Core Contributions on Security Issues

The report debunked two myths with empirical data:

  • Myth One: "Only super-large models have value, and super-large models must be cloud-based." The report indicates 57% of enterprise models have fewer than 10 billion parameters, and unified memory architecture can run hundred-billion-parameter models locally. The technical feasibility of on-premise deployment has been validated.
  • Myth Two: "Cloud provider security certifications are sufficiently reliable." The report shows only 9% of enterprises are completely satisfied with their partners, while 76% worry about data breaches. Security is not just about certifications; it's about trust and architectural choices.

Limiting Conditions (Honest Boundaries)

On-premise deployment is not without security challenges:

  • Internal Threats: After data localization, malicious or negligent internal personnel may directly access models and raw data. This requires strict IAM, audit, and DLP measures.
  • Device Physical Security: Loss or theft of endpoint devices (laptops, workstations) becomes a new risk surface. Full-disk encryption and remote wipe capabilities must be enabled.
  • Model Leakage Risk: Model files deployed in private environments are intellectual property themselves and require protection against unauthorized copying and exfiltration.
  • Update and Patch Management: Models and inference frameworks in on-premise deployment require continuous security updates, increasing operational burden.

Final Recommendations

For Enterprise Decision-Makers:

  1. Immediately initiate data classification and AI use case risk assessment, clarifying "which data will never go to the cloud."
  2. For L3 data, mandate on-premise inference pilots to verify technical feasibility and costs.
  3. Don't default to cloud APIs as the first choice; instead, treat them as "low-sensitivity data exclusive channels."
  4. Incorporate model supply chain security into procurement evaluation systems, prioritizing open-source models that can be deployed locally.

For Security Teams:

  1. Include AI inference in data loss prevention monitoring to detect whether sensitive data is being sent to cloud AI APIs.
  2. Establish security baselines for on-premise inference: encryption, access control, log auditing, and model integrity verification.
  3. Conduct regular penetration tests of public cloud AI services (within authorized scope) to verify data isolation commitments.

One-Sentence Summary: The choice of security architecture is essentially the design of trust boundaries. Privatizing AI inference means contracting the trust boundary to within the enterprise's controllable scope—this is the most straightforward, yet most effective, security principle.


This analytical framework is based on Omdia's "Rethinking Critical AI Infrastructure" (January 2026) research data, supplemented by the NIST AI Risk Management Framework, OWASP LLM Security Cheat Sheet, and other publicly available standards.

Related topic: