Generative AI (GenAI) produces new content (text, code, structured data) by learning statistical patterns from vast training datasets. Traditional ML in ServiceNow predicts values from patterns in your own historical data. The distinction matters because they solve different problems and are configured and maintained differently.

• Trained on your instance's historical data (resolved incidents, categorisation patterns)
• Predicts discrete field values: which category, which assignment group, which priority
• Output is always a prediction with a confidence score, never generated text
• Gets better as your instance accumulates more resolved, correctly-labelled records
• Fully on-platform: no external model, no external API call

• Uses pre-trained large language models (LLMs) trained on vast datasets of human language
• Generates new content: summaries, draft replies, code, flow logic, knowledge articles
• Output is generated text or structured content, not a predicted value
• Quality depends on the LLM provider and the prompt, not on your instance data alone
• May involve an external API call (to Azure OpenAI, Claude, Gemini) unless using Now LLM

Now LLM is ServiceNow's proprietary large language model, currently on version 2.0 (released October 2025). ServiceNow built its own model to address three gaps that external LLMs cannot fully close.

When using Now LLM, customer data never leaves ServiceNow infrastructure. No external API call is made to any third party. This is the only option for organisations with absolute data residency or sovereignty requirements — zero-retention agreements with external providers still technically cross the platform boundary. Now LLM does not.

Now LLM is fine-tuned on ServiceNow-specific content: GlideRecord API patterns, table names, workflow structures, ITSM terminology, and platform-specific conventions. It generates more accurate ServiceNow-specific code and content than a general-purpose LLM working from a generic prompt.

Using an on-platform model removes dependency on third-party pricing changes, API rate limits, service disruptions, or model version updates from an external vendor that could alter production behaviour.

Now LLM v2.0 improvements over v1: longer context window, faster response latency, and training data updated through 2025.

The Generative AI Controller is the platform-level infrastructure layer that manages all connections between ServiceNow and LLM providers. It is the abstraction layer between the Now Assist skills that agents and developers use and the actual language models processing their requests.

• Stores and manages LLM connection credentials, API keys, endpoints, and deployment names for each configured provider, using ServiceNow Credential records (not hardcoded)
• Routes prompt requests from Now Assist and NASK skills to the correct LLM provider based on each skill's configuration
• Handles token limit management, retry logic, and rate limiting to maintain stable AI responses at scale
• Maintains audit logs of all LLM requests, response times, and error states for compliance and debugging

Because the Generative AI Controller decouples skills from their LLM provider, the same Now Assist skill can use Now LLM in a production environment and Azure OpenAI in a UAT environment without any change to the skill code. Only the provider connection record in the Controller changes. This also means switching LLM providers requires changing the Controller configuration, not modifying every skill.

ServiceNow supports multiple external LLM providers through the Generative AI Controller, allowing organisations to choose the provider that best fits their enterprise agreements, model performance requirements, or cost preferences.

• Azure OpenAI (Microsoft) – GPT-4o, GPT-4 Turbo via Azure endpoint. Most commonly deployed in large enterprise environments with existing Microsoft agreements.
• Anthropic Claude (via AWS Bedrock) – Claude 3 / 3.5 model family. Popular for its instruction-following accuracy and long-context handling.
• Google Gemini (via Vertex AI) – Gemini Pro and advanced models. Preferred in organisations with Google Cloud agreements.
• BYOLLM – Connect any compatible model via a custom endpoint. For proprietary or self-hosted models.
• Now LLM – ServiceNow's on-platform model. No external call, data residency guaranteed.

Provider selection can be set at multiple levels: globally for all Now Assist skills, for a specific department's skills, or per individual skill. A common enterprise pattern: use Azure OpenAI for ITSM summarisation (best model quality, existing agreement) and Now LLM for HR cases (data sensitivity requires on-platform processing).

Both are enterprise control options that address the two most common GenAI security concerns in regulated organisations: which model processes the data, and how the data is encrypted. They are separate controls and can be used independently.

Allows an organisation to connect a custom or self-hosted language model to the Generative AI Controller instead of using Now LLM or a listed partner provider. BYOLLM is the answer when an organisation has strict data residency requirements that prohibit any external provider (even with zero-retention), already owns enterprise licenses for a fine-tuned proprietary model, or needs a model specialised in a non-English language.

Configuration: set up the custom model endpoint as a Connection record in the Generative AI Controller, specifying the API format (OpenAI-compatible, custom schema) and authentication method.

Allows an organisation to provide their own encryption keys to control how GenAI data is encrypted at rest within the ServiceNow platform. BYOK adds an additional encryption layer without changing which LLM processes the data. It is a platform-level control managed through the ServiceNow security configuration, not the Generative AI Controller.

ServiceNow provides three distinct data protection options for organisations to choose from based on their compliance requirements and risk tolerance.

Data is processed entirely within ServiceNow infrastructure. No API call is made to any external service. This satisfies “data must never leave the platform” requirements, including government data sovereignty mandates, defence sector restrictions, and the strictest healthcare data residency requirements.

When using Azure OpenAI, Anthropic Claude, or Google Gemini, ServiceNow negotiates zero data retention agreements: the external provider processes the prompt but does not store, log, or train on the content. The data is transient, used to generate the response and immediately discarded. Suitable for most enterprise deployments where data residency within the platform is not a hard requirement.

Fields configured as sensitive (names, phone numbers, email addresses, national IDs, account numbers) are anonymised in the prompt before any external LLM call. The model processes “[CALLER_NAME]” instead of the actual name. The response is de-anonymised before displaying to the agent. Appropriate for organisations that want additional protection layered on top of zero-retention agreements.

No data from one ServiceNow instance is ever used as context or training data for another instance. Tenant isolation is enforced at the infrastructure level.

ServiceNow completed the acquisition of Moveworks in December 2025 for approximately $2.85 billion. Moveworks was an enterprise AI assistant platform deployed at hundreds of large organisations with deep expertise in conversational AI and cross-system enterprise search.

• Conversational AI front-end – A more polished, natural-language-first interface for employees to resolve IT and HR requests via Microsoft Teams, Slack, or email, with more nuanced multi-turn conversation handling than standard Virtual Agent.
• Cross-system enterprise search – Ability to answer queries by searching simultaneously across SharePoint, Confluence, Jira, ServiceNow, and other enterprise systems in a single query. Addresses the fragmented knowledge problem at enterprises with content in multiple systems.
• Autonomous task completion – End-to-end handling of password resets, access requests, and HR enquiries without human agent involvement, building on ServiceNow's workflow execution engine.
• AI research talent – Hundreds of ML engineers and AI researchers joined ServiceNow's product team, accelerating the AI roadmap across Now Assist, AI Agents, and the AI Platform.

Moveworks' capabilities are being absorbed into the ServiceNow AI Platform under the Autonomous Workforce initiative. Moveworks brings the front-end conversational experience; ServiceNow provides the back-end workflow engine, governance platform, and enterprise ITSM depth.

Retrieval-Augmented Generation (RAG) is a technique that grounds an LLM's output in specific documents or data sources, rather than relying solely on knowledge baked into the model during training. ServiceNow uses RAG as the foundation of AI Search and NASK Retrievers.

A general-purpose LLM knows what it was trained on, but it does not know your organisation's specific policies, runbooks, or procedures. Without RAG, asking Now Assist “what is our password reset policy?” produces a generic answer. With RAG, the system retrieves the actual policy document from your knowledge base and uses it as context for the response.

1. The user query (or record context) is converted into a semantic embedding vector
2. The vector is compared against a pre-indexed content corpus (knowledge base articles, NASK-configured tables, external sources) to find the most semantically relevant chunks
3. The top-N most relevant chunks are injected into the LLM prompt as context
4. The LLM generates its response grounded in that specific retrieved content, not generic training knowledge
5. The response cites or is traceable to the source content

• AI Search: user query is matched against the knowledge index; AI generates a direct answer from the most relevant articles
• NASK Retrievers: the Retriever component implements RAG for custom skills, pulling from custom tables or external APIs
• Now Assist knowledge article generation: the LLM grounds the article in the resolved incident notes and referenced knowledge content

Prompt engineering is the practice of designing the text instructions sent to an LLM to produce consistent, high-quality output. In Now Assist and NASK, the system prompt is the developer's primary quality control tool.

• Role definition – Tell the LLM what persona to adopt. “You are an ITSM specialist who writes clear, concise technical summaries for IT support agents.”
• Task specification – Be explicit about exactly what to produce. “Summarise the incident history in 4–6 bullet points.”
• Output format – Specify structure. “Return your response as a numbered list. Each item: action verb + specific detail. Do not include background context or general advice.”
• Length constraints – “Maximum 150 words.” Without this, LLMs trend towards verbosity.
• Scope boundaries – Tell the LLM what NOT to include. “Do not include information not present in the provided context. If the context is insufficient, say so.” This prevents hallucination.

Poor: “Summarise this incident.”
Better: “You are an ITSM specialist. Summarise the following incident history in 4–6 bullet points, focusing on: the reported problem, steps taken, current status, and outstanding actions. Maximum 150 words. Use only information provided. Do not speculate.”

LLMs can only process a finite amount of text in a single call, the context window. When an incident has accumulated hundreds of work notes over days or weeks, the full text often exceeds what can be sent to the LLM in one prompt.

If the full incident history is passed to the LLM without processing, one of two things happens: the LLM provider rejects the call with a token limit error, or (for models with very long context windows) the LLM degrades in accuracy on content from the middle of a long document, the “lost in the middle” problem where attention weakens for content far from the start and end of the context.

• Chunking and prioritisation – The Now Assist summarisation skill does not send all work notes. It selects the most recent N work notes (typically the last 20–30, or those from the last 48 hours) as the most contextually relevant for a summary.
• Now LLM v2.0 long-context improvement – Now LLM v2.0 was specifically improved for longer context reasoning compared to v1, handling large incident histories more reliably without accuracy degradation.
• NASK Script pre-processing – For NASK custom skills, a pre-processing Script can summarise or filter retrieved content before it is injected into the prompt, ensuring the context passed to the LLM is within token budget.

Both approaches improve LLM output quality for a specific domain or organisation, but they work at different layers and have very different implementation costs.

Continues training the LLM on domain-specific data to update the model's weights. The model literally learns organisation-specific knowledge by adjusting its internal parameters. Fine-tuning produces a model that has baked-in domain knowledge but requires significant compute resources, large high-quality datasets, and deep ML expertise to execute. The resulting model also needs retraining whenever the domain knowledge changes.

Retrieves relevant content at inference time and provides it as context in the prompt. The model weights do not change, instead, relevant information is given to the model to reason over each time it generates a response. RAG requires no compute-intensive training, is much easier to update (just update the indexed content), and allows the same base model to serve multiple domains by switching the retrieval source. This is the approach ServiceNow uses.

ServiceNow's customers have diverse, frequently changing knowledge bases. A fine-tuned model for each customer's specific knowledge would be impractical to maintain. RAG allows Now Assist to ground responses in each customer's current knowledge base without retraining. Now LLM is fine-tuned for the ServiceNow platform domain (GlideRecord, workflow concepts), while RAG handles the organisation-specific layer at runtime.

Hallucination is when an LLM generates plausible-sounding but factually incorrect content with apparent confidence. It is a fundamental characteristic of how LLMs work, they predict probable next tokens, which can produce fluent but wrong text. In a ServiceNow context, hallucinated resolution notes or knowledge article content that agents trust and act on represents a real operational risk.

• Insufficient context: the LLM has no relevant knowledge to draw from and “fills in” a plausible-sounding answer
• Ambiguous prompts: the skill prompt does not specify that the LLM should only use provided content
• Long context: the LLM loses track of specific details in very long incident histories

• RAG grounding – Providing explicit retrieved context reduces the LLM's need to guess. When told “answer based only on the following documents,” hallucination rates drop significantly compared to open-domain generation.
• Prompt constraints – “Do not include information not present in the provided context. If you cannot find the answer in the context, say so.” Explicit scope instructions reduce off-topic generation.
• Human review gates – AI-generated knowledge articles go through a mandatory knowledge manager review before publishing. Resolution notes are agent-reviewed before persisting. No auto-write of generated content for high-stakes fields.
• Source citation – AI Search shows which articles the generated answer was sourced from, allowing agents to verify the source if needed.

Vector embeddings are numerical representations of text that capture semantic meaning rather than just the words used. They are what enable AI Search to match a user's intent rather than just their exact words.

Searches for the literal words in the query. “Can't access email” would return articles containing those specific words. Articles about “Outlook connectivity issues” might not appear even if they answer the question, because the words do not overlap sufficiently.

Converts the query and all indexed content into embedding vectors. Measures the cosine similarity between the query vector and content vectors. Articles about “Outlook connectivity issues,” “email client setup,” and “Microsoft 365 authentication problems” all score highly for the query “Can't access email” because their semantic vectors are close, even though the words differ.

Semantic search correctly handles informal employee language (“computer is slow”, “can't get in”, “printer won't work”) and maps it to the formal IT documentation that answers the question. This is why AI Search deflects tickets that keyword search failed to deflect, the user found their answer even with imperfect search terms.

Use Now LLM, ServiceNow's on-platform model. It is the only option where data never leaves ServiceNow infrastructure. No external API call is made and no data passes to any third party, even temporarily.

Configure this in the Generative AI Controller by setting Now LLM as the provider for all relevant Now Assist skills. Audit each skill's configuration to confirm there are no individual overrides pointing to an external provider that might have been set during earlier testing.

Zero data retention agreements with Azure OpenAI, Claude, or Gemini mean the provider does not store the data after processing, but the data still leaves the ServiceNow platform via an external API call. For organisations with absolute data residency requirements (certain government agencies, defence contractors, some healthcare organisations subject to national data localisation laws), “data never crosses the platform boundary” is the hard requirement. A zero-retention call still crosses that boundary. Now LLM does not.

The choice depends on four factors: data residency requirements, model quality needs, existing enterprise agreements, and cost. Walk through each.

If the organisation has an absolute requirement that data never leaves ServiceNow infrastructure (government, defence, certain healthcare) the decision is made: Now LLM. No further evaluation needed.

Compare on model quality vs. cost. Azure OpenAI GPT-4o generally outperforms Now LLM on complex reasoning tasks (multi-paragraph summarisation of complex incidents, nuanced email drafting). Now LLM v2.0 performs comparably on standard ITSM summarisation and response suggestion tasks. For organisations where model quality is critical and data controls are satisfied by zero-retention agreement, Azure OpenAI often wins on quality.

Large enterprises frequently already have Azure Enterprise Agreements with OpenAI credits. Using Azure OpenAI may have negligible marginal cost, making it the pragmatic choice even if Now LLM quality is acceptable.

Recommend starting with Now LLM for HR and Finance skills (highest data sensitivity) and Azure OpenAI for ITSM summarisation (highest volume, where quality gain is most impactful). The Generative AI Controller supports per-skill provider configuration, so this is a viable production architecture.

The configuration has two parts: the provider connection (Generative AI Controller) and the skill assignment (Now Assist Admin Center).

1. Navigate to Generative AI Controller (System AI → Generative AI Controller)
2. Create a new LLM Provider Connection record and select Azure OpenAI as the provider type
3. Enter the Azure OpenAI endpoint URL, API key, and deployment name (the specific model deployment, e.g., gpt-4o-production)
4. Run the built-in connection test to verify the credentials and connectivity before saving
5. Navigate to the Now Assist Admin Center (All → Now Assist → Admin Center)
6. Open the Incident Summarisation skill configuration
7. Change the LLM Provider field from Now LLM to the Azure OpenAI connection just created
8. Before going live: confirm the client has a zero data retention agreement with Microsoft for their specific Azure OpenAI deployment (this is not automatic with an Azure subscription; it requires an enterprise data processing agreement)
9. Test the skill with 5–10 representative incidents and compare output quality against the Now LLM baseline before activating for all agents

Verbosity and tone problems are prompt problems, not model problems. They are fixed by adding explicit constraints to the system prompt.

Access the skill configuration in the Now Assist Admin Center and review the current system prompt. Check for: missing length constraint, missing tone instruction, missing format specification. Most default or developer-written prompts that produce verbose output have all three gaps.

Add explicit word or bullet point limits: “Maximum 100 words” or “Maximum 5 bullet points.” Without this instruction, LLMs default to comprehensive answers that cover edge cases the user did not ask about.

Add a tone instruction: “Write in plain, direct language. No jargon. Write as if briefing a colleague, not writing a report.” Or for technical agents: “Technical and concise. Avoid explanatory language, agents are experienced professionals.” The right tone depends on the agent audience.

If agents want bullet points, specify: “Return a bulleted list. Each bullet: action verb + specific detail. No introductory sentence. No concluding sentence.” Removing sentence scaffolding (intro + body + conclusion) dramatically reduces output length.

Run the revised prompt against 10 test incidents. Measure average word count and share with pilot agents for tone feedback before pushing to production.

LLM provider timeouts at peak hours are a capacity and configuration issue, not a ServiceNow platform issue. Work through the diagnosis from the external provider inward.

Check the Generative AI Controller audit logs for error types. Distinguish between: timeout errors (request sent but no response in time), rate limit errors (too many requests per minute), and connection errors (cannot reach the endpoint). Each has a different fix.

Azure OpenAI deployments have Tokens Per Minute (TPM) and Requests Per Minute (RPM) quotas set at the Azure deployment level. Peak usage may be hitting these quotas. Check Azure OpenAI Studio → Deployments → model deployment metrics for quota utilisation at the times timeouts occur.

If quota is the constraint, request a TPM increase from Azure for the deployment. Standard Azure subscriptions have lower defaults; enterprise agreements can get higher quotas approved within 24–48 hours.

In the Generative AI Controller, confirm retry logic is enabled for transient failures. Consider configuring a fallback provider (e.g., Now LLM) for the affected skill so that if the primary provider times out, the skill attempts Now LLM as a backup rather than returning an error to the agent.

If peak load cannot be handled by a single Azure deployment, create multiple Azure OpenAI deployments and configure the Generative AI Controller to distribute requests across them (if this is supported in the current release), or stagger agent rollout to flatten peak demand.