Add CLI docs search and GraphQL validation

[dmerand]

What

Adds a proof-of-concept CLI-owned docs search and GraphQL validation path for agent workflows, stacked on top of the agent session prototype.

This PR adds:

• shopify docs search <query> --api <api-name> --json
• shopify validate graphql --query/--query-file ... --schema-file ... --variables/--variable-file ... --json
• CLI command registration for docs:search and validate:graphql
• a direct graphql dependency for the CLI package

Why

The base PR proves CLI-owned agent session state. This upstack PR proves the next deterministic agent steps can also move into the CLI:

1. start one attributed agent session
2. search Shopify.dev docs through the CLI
3. validate generated GraphQL through the CLI
4. execute the validated operation through existing shopify store execute

The goal is a reference-quality PoC for reducing repeated ai-toolkit-source script glue while keeping Shopify.dev as the docs/source-of-truth surface.

Behavior

Docs search

shopify docs search uses the same Shopify.dev assistant endpoint used by generated ai-toolkit skills:

POST /assistant/search

Request body:

{
  "query": "...",
  "api_name": "admin"
}

The CLI service preserves ai-toolkit's Shopify.dev host routing behavior:

• default: https://shopify.dev/
• DEV=true: https://shopify-dev.shop.dev/
• SHOPIFY_DEV_STAGING_SERVER_NUMBER + MINERVA_TOKEN: staging host

It sends X-Shopify-Surface: cli so this path is attributable as CLI-owned rather than skill-owned.

GraphQL validation

shopify validate graphql currently validates:

• GraphQL syntax
• exactly one operation
• operation type/name
• variables JSON parsing
• schema validation when --schema-file is provided
• variable coercion against the provided schema

Schema files can be either:

• SDL .graphql
• introspection JSON

Invalid validation results print structured JSON and exit non-zero via AbortSilentError.

Example invalid result:

{
  "valid": false,
  "issues": [
    {
      "message": "Cannot query field ...",
      "stage": "schema",
      "locations": [{"line": 1, "column": 9}]
    }
  ],
  "operation": {"type": "query"},
  "schema": {"source": "file", "validation": "checked"}
}

Agent workflow smoke test

This is the realistic end-to-end shape this PR is intended to support:

agent request
  → shopify agent session start
  → shopify docs search
  → agent writes a GraphQL operation
  → shopify validate graphql
  → shopify store execute
  → natural result summary

The important constraint is that the agent validates against a real Admin schema before execution. The agent should not write a toy schema.

Local preflight

From the CLI worktree:

cd /Users/donald/src/github.com/Shopify/cli-worktree-agent-session-poc
git switch donald/agent-session-docs-validate-poc

shadowenv exec -- pnpm --filter @shopify/cli build

if [ ! -s packages/cli-kit/src/cli/api/graphql/admin/admin_schema.graphql ]; then
  shadowenv exec -- pnpm graphql-codegen:get-graphql-schemas
fi
export SHOPIFY_ADMIN_SCHEMA_FILE="$PWD/packages/cli-kit/src/cli/api/graphql/admin/admin_schema.graphql"
test -s "$SHOPIFY_ADMIN_SCHEMA_FILE"

cd packages/cli
shadowenv exec -- pnpm exec oclif manifest
cd ../..

mkdir -p .demo-bin
cat > .demo-bin/shopify <<'SH'
#!/usr/bin/env bash
set -euo pipefail
cd /Users/donald/src/github.com/Shopify/cli-worktree-agent-session-poc
exec shadowenv exec -- node packages/cli/bin/dev.js "$@"
SH
chmod +x .demo-bin/shopify
export PATH="$PWD/.demo-bin:$PATH"

command -v shopify
shopify docs search --help >/dev/null
shopify validate graphql --help >/dev/null
printf 'Using Admin schema: %s\n' "$SHOPIFY_ADMIN_SCHEMA_FILE"

If the target store has not been authenticated yet:

export SHOPIFY_STORE=example.myshopify.com
shopify store auth --store "$SHOPIFY_STORE"

Agent prompt

Use a fresh agent session from the same shell so it inherits PATH and SHOPIFY_ADMIN_SCHEMA_FILE.

Can you confirm my demo store identity through Admin GraphQL?

Please work step by step:
1. start the agent session
2. search Shopify.dev docs
3. prepare the query
4. validate it against $SHOPIFY_ADMIN_SCHEMA_FILE
5. only then execute it

Do not write a schema. Do not bundle the workflow into one script or one bash command. Do not run mutations.

Expected command shape

The transcript should show separate steps, not one generated driver script:

shopify agent session start \
  --agent pi \
  --agent-version 1.0.0 \
  --provider shopify \
  --metrics on \
  --default-non-interactive \
  --json

shopify docs search \
  "GraphQL Admin shop query myshopifyDomain" \
  --api admin \
  --json

# Agent-created operation only; schema comes from preflight.
cat > "$work_dir/shop-query.graphql" <<'GRAPHQL'
query ShopInfo {
  shop {
    id
    name
    myshopifyDomain
  }
}
GRAPHQL

shopify validate graphql \
  --query-file "$work_dir/shop-query.graphql" \
  --schema-file "$SHOPIFY_ADMIN_SCHEMA_FILE" \
  --json

shopify store execute \
  --store "$SHOPIFY_STORE" \
  --query-file "$work_dir/shop-query.graphql" \
  --json

Expected validation result:

{
  "valid": true,
  "issues": [],
  "operation": {
    "type": "query",
    "name": "ShopInfo"
  },
  "schema": {
    "source": "file",
    "path": ".../packages/cli-kit/src/cli/api/graphql/admin/admin_schema.graphql",
    "validation": "checked"
  }
}

Expected final response style:

Confirmed. The store identity is:

- name: <store name>
- myshopifyDomain: <store>.myshopify.com
- id: gid://shopify/Shop/<id>

I used a read-only Admin GraphQL query and validated it before execution. No store changes were made.

Follow-up workflow tested locally

The same session also handled a natural follow-up request — “check products on that store” — by repeating the same pattern:

1. shopify docs search "Admin GraphQL query products" --api admin --json
2. agent wrote a read-only products(first: 10) query
3. shopify validate graphql --schema-file "$SHOPIFY_ADMIN_SCHEMA_FILE" ... --json
4. shopify store execute ... --json
5. agent summarized product title/handle/status/id results

That follow-up is useful because it shows the flow is not just a hardcoded identity-query demo.

Boundaries

This PR does not yet:

• auto-fetch or bundle Admin schemas via --surface admin
• infer required OAuth scopes from GraphQL validation
• replace every ai-toolkit validator
• make /assistant/search a public Shopify.dev API contract

Those are follow-on design decisions. This PR is intentionally a PoC, but it is structured as a realistic reference implementation rather than a placeholder.

The obvious next product-grade step is:

shopify validate graphql \
  --surface admin \
  --version unstable \
  --query-file ./operation.graphql \
  --json

where the CLI resolves/caches the schema itself instead of requiring --schema-file.

Automated testing

pnpm vitest run \
  packages/cli/src/cli/services/docs/search.test.ts \
  packages/cli/src/cli/commands/docs/search.test.ts \
  packages/cli/src/cli/services/validation/graphql.test.ts \
  packages/cli/src/cli/commands/validate/graphql.test.ts \
  packages/cli/src/cli/commands/agent/session/start.test.ts \
  packages/app/src/cli/commands/app/release.test.ts

pnpm --filter @shopify/cli lint
pnpm --filter @shopify/cli type-check
pnpm --filter @shopify/cli build

[github-actions]

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

• Some seemingly private modules might be re-exported through public modules.
• If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

packages/cli-kit/dist/public/node/agent.d.ts

import { LocalStorage } from './local-storage.js';
import { AgentSession, ConfSchema } from '../../private/node/conf-store.js';
export type { AgentSession };
export interface StartAgentSessionOptions {
    sessionId: string;
    agentName: string;
    agentVersion: string;
    agentProvider: string;
    metricsMode?: 'on' | 'off';
    defaultNonInteractive?: boolean;
}
/**
 * Start a new agent session.
 *
 * Persists the agent session state to the CLI kit config store.
 *
 * @param options - Agent session configuration.
 * @param config - Optional config store for testing.
 * @returns The persisted session value.
 */
export declare function startAgentSession(options: StartAgentSessionOptions, config?: LocalStorage<ConfSchema>): AgentSession;
/**
 * Get the current agent session.
 *
 * @param config - Optional config store for testing.
 * @returns Current agent session, or undefined if no session is active.
 */
export declare function getCurrentAgentSession(config?: LocalStorage<ConfSchema>): AgentSession | undefined;
/**
 * Clear the current agent session.
 *
 * Removes the persisted agent session state from the CLI kit config store.
 *
 * @param config - Optional config store for testing.
 */
export declare function clearAgentSession(config?: LocalStorage<ConfSchema>): void;
/**
 * Pack SHOPIFY_CLI_AGENT_INFO environment variable value from agent session.
 *
 * The format is a tagged string with agent metadata:
 * n:<name>|v:<version>|p:<provider>.
 *
 * Precedence: explicit process.env.SHOPIFY_CLI_AGENT_INFO takes priority over
 * persisted session state.
 *
 * @param session - Optional session to pack from. If not provided, uses current session.
 * @param config - Optional config store for testing.
 * @returns Tagged string for SHOPIFY_CLI_AGENT_INFO, or undefined if no data available.
 */
export declare function packAgentInfo(session?: AgentSession, config?: LocalStorage<ConfSchema>): string | undefined;
/**
 * Pack SHOPIFY_CLI_AGENT_IDS environment variable value from agent session.
 *
 * The format is a tagged string with session identifier:
 * s:<sessionId>.
 *
 * Precedence: explicit process.env.SHOPIFY_CLI_AGENT_IDS takes priority over
 * persisted session state.
 *
 * @param session - Optional session to pack from. If not provided, uses current session.
 * @param config - Optional config store for testing.
 * @returns Tagged string for SHOPIFY_CLI_AGENT_IDS, or undefined if no data available.
 */
export declare function packAgentIds(session?: AgentSession, config?: LocalStorage<ConfSchema>): string | undefined;

Existing type declarations

packages/cli-kit/dist/private/node/conf-store.d.ts

@@ -18,11 +18,22 @@ interface Cache {
     [mostRecentOccurrenceKey: MostRecentOccurrenceKey]: CacheValue<boolean>;
     [rateLimitKey: RateLimitKey]: CacheValue<number[]>;
 }
+export interface AgentSession {
+    sessionId: string;
+    startedAt: string;
+    agentName: string;
+    agentVersion: string;
+    agentProvider: string;
+    metricsMode: 'on' | 'off';
+    defaultNonInteractive: boolean;
+}
 export interface ConfSchema {
     sessionStore: string;
     currentSessionId?: string;
     devSessionStore?: string;
     currentDevSessionId?: string;
+    currentAgentSession?: AgentSession;
+    devAgentSession?: AgentSession;
     cache?: Cache;
     autoUpgradeEnabled?: boolean;
 }
@@ -128,7 +139,8 @@ interface RunWithRateLimitOptions {
 export declare function runWithRateLimit(options: RunWithRateLimitOptions, config?: LocalStorage<ConfSchema>): Promise<boolean>;
 /**
  * Get auto-upgrade preference.
- * Defaults to true if the preference has never been explicitly set.
+ *
+ * Auto-upgrade is enabled by default when the preference has never been set.
  *
  * @returns Whether auto-upgrade is enabled.
  */
@@ -145,4 +157,20 @@ export declare function getConfigStoreForPartnerStatus(): LocalStorage<Record<st
 }>>;
 export declare function getCachedPartnerAccountStatus(partnersToken: string): true | null;
 export declare function setCachedPartnerAccountStatus(partnersToken: string): void;
+/**
+ * Get current agent session.
+ *
+ * @returns Current agent session.
+ */
+export declare function getAgentSession(config?: LocalStorage<ConfSchema>): AgentSession | undefined;
+/**
+ * Set current agent session.
+ *
+ * @param session - Agent session.
+ */
+export declare function setAgentSession(session: AgentSession, config?: LocalStorage<ConfSchema>): void;
+/**
+ * Remove current agent session.
+ */
+export declare function removeAgentSession(config?: LocalStorage<ConfSchema>): void;
 export {};
\ No newline at end of file

packages/cli-kit/dist/private/node/constants.d.ts

@@ -35,6 +35,7 @@ export declare const environmentVariables: {
     skipNetworkLevelRetry: string;
     maxRequestTimeForNetworkCalls: string;
     disableImportScanning: string;
+    hostedApps: string;
 };
 export declare const defaultThemeKitAccessDomain = "theme-kit-access.shopifyapps.com";
 export declare const systemEnvironmentVariables: {

packages/cli-kit/dist/public/common/version.d.ts

@@ -1 +1 @@
-export declare const CLI_KIT_VERSION = "4.1.0";
\ No newline at end of file
+export declare const CLI_KIT_VERSION = "3.94.0";
\ No newline at end of file

packages/cli-kit/dist/public/node/context/local.d.ts

@@ -25,6 +25,13 @@ export declare function isDevelopment(env?: NodeJS.ProcessEnv): boolean;
  * @returns True if SHOPIFY_FLAG_VERBOSE is truthy or the flag --verbose has been passed.
  */
 export declare function isVerbose(env?: NodeJS.ProcessEnv): boolean;
+/**
+ * Returns true if the hosted apps mode is enabled.
+ *
+ * @param env - The environment variables from the environment of the current process.
+ * @returns True if HOSTED_APPS is truthy.
+ */
+export declare function isHostedAppsMode(env?: NodeJS.ProcessEnv): boolean;
 /**
  * Returns true if the environment in which the CLI is running is either
  * a local environment (where dev is present).