Compare commits

...

6 Commits

Author SHA1 Message Date
diegosouzapw
997e83f685 refactor(dashboard): move the webhook forms' URL check and field messages into helpers (KEA-9386)
The inline validation added to the Custom, Slack and Discord forms pushed
their components over the complexity ratchet (new-code mode): function
length, cyclomatic and cognitive complexity.

No behaviour change. The debounced server check the three forms each
carried a copy of becomes the `useWebhookUrlCheck` hook, and what a field
shows under itself is derived by pure helpers in webhookFieldValidation.ts
(`webhookUrlFieldFeedback`, `webhookSecretFieldError`).
2026-11-20 20:48:41 -03:00
diegosouzapw
5454b20a1c docs(changelog): add the fragment for #20834 2026-11-20 19:29:29 -03:00
diegosouzapw
7f6351956e fix(i18n): serve the formValidation namespace to the forms that use it
The access-token, webhook and eval-suite forms read their inline messages
from the new formValidation namespace, but the per-section namespace map was
not regenerated, so the dashboard would not have sent those messages to the
client. Regenerated with `npm run gen:i18n-routes`; the drift test
tests/unit/i18n-route-namespaces.test.ts fails without it.
2026-11-20 18:43:14 -03:00
diegosouzapw
edd7ecd731 fix(dashboard): make the eval case model optional and flag suite errors inline (KEA-9388, KEA-9389)
Quality > New suite refused to save a case whose "Model (optional)" was
empty, and reported every problem only as a toast ("This case has
validation errors") with nothing highlighted.

- The model is optional, as labelled: the save schema already takes a
  case without one and the runner resolves it (the run's model or combo
  target, else the case's model, else the default). The dialog no longer
  requires it and omits `model` from the request when empty, since the
  schema refuses an empty string. A hint under the field says what an
  empty model runs on; the default is now one constant shared by the
  runner and the hint.
- A refused save flags each missing field inline (suite name, case name,
  prompt, expected value), marks the required ones, and moves focus -
  and the dialog's scroll position - to the first of them. The messages
  clear as the fields are filled.
2026-11-20 18:11:09 -03:00
diegosouzapw
05359effd5 fix(dashboard): validate the webhook wizard's fields inline (KEA-9386)
On the configure step, Next was silently disabled while the destination
was empty; plain text such as "not-a-url" was reported 600 ms later as
"a blocked private address"; a URL or secret over the route's limits was
sent and the refusal reached the banner as "[object Object]".

- Next stays enabled: a refused click flags the fields inline and focuses
  the first one (all four integration kinds).
- A value that is not an http(s) URL, or is over the limit, is said at
  once next to the field, without a round trip. The limits (URL 2000,
  secret 500) are the ones the routes' Zod schemas already enforced, now
  shared through src/shared/constants/formLimits.ts. Nothing is cut: the
  secret field has no maxLength, and a 400-character key reaches the API
  whole.
- POST /api/webhooks/validate-url answers `invalid_url` for text that is
  not a URL instead of `blocked_private`.
- A validation refusal from the API is shown by its message.
2026-11-20 18:08:13 -03:00
diegosouzapw
26cb8f9014 fix(dashboard): explain the access-token Name field inline (KEA-9387)
Create was silently disabled while Name was empty, and a name over the
route's 100-character limit was sent and came back as a generic
"Invalid request".

Create now stays enabled: a refused click flags Name inline (required,
or over the limit with the current count) and focuses it; an over-long
name is flagged as it is typed and is never cut or sent. The limit lives
in src/shared/constants/formLimits.ts and is the one the route's Zod
schema enforces. Validation messages come from a new shared
`formValidation` namespace (en.json only).
2026-11-20 18:04:53 -03:00
29 changed files with 1335 additions and 198 deletions

View File

@@ -0,0 +1 @@
- **fix(dashboard):** the access-token, webhook and eval-suite forms say which field is wrong, where it is wrong — the submit button stays enabled and a refused click marks each invalid field inline and focuses the first one; the eval case model is optional as labelled; `not-a-url` in the webhook wizard is no longer reported as "a blocked private address" (`POST /api/webhooks/validate-url` answers `invalid_url`), and the form limits are the constants the route schemas use ([#20834](https://github.com/cheaper-inference/OmniRoute/pull/20834))

View File

@@ -23,6 +23,7 @@ import {
EmptyState,
Modal,
} from "@/shared/components";
import { ACCESS_TOKEN_NAME_MAX_LENGTH } from "@/shared/constants/formLimits";
interface AccessTokenRow {
id: string;
@@ -51,6 +52,7 @@ async function fetchTokens(): Promise<AccessTokenRow[]> {
export default function AccessTokensTab() {
const t = useTranslations("settings");
const tc = useTranslations("common");
const tv = useTranslations("formValidation");
const nameRef = useRef<HTMLInputElement>(null);
const format = useFormatter();
const scopeLabels = {
@@ -66,6 +68,8 @@ export default function AccessTokensTab() {
const [scope, setScope] = useState("read");
const [expires, setExpires] = useState("");
const [creating, setCreating] = useState(false);
// Set by a refused Create: an untouched form is not flagged for being empty.
const [showNameRequired, setShowNameRequired] = useState(false);
const [newSecret, setNewSecret] = useState<string | null>(null);
// Name + scope of the token just created, shown as the secret preview's header.
const [newSecretMeta, setNewSecretMeta] = useState({ name: "", scope: "read" });
@@ -101,12 +105,31 @@ export default function AccessTokensTab() {
};
}, [t]);
// The request carries the trimmed name, so that is what the server's limit is measured on.
const trimmedName = name.trim();
const nameTooLong = trimmedName.length > ACCESS_TOKEN_NAME_MAX_LENGTH;
const nameField = t("accessTokensColName");
const nameError = nameTooLong
? tv("maxLength", {
field: nameField,
max: ACCESS_TOKEN_NAME_MAX_LENGTH,
count: trimmedName.length,
})
: showNameRequired && !trimmedName
? tv("required", { field: nameField })
: undefined;
const createToken = async () => {
if (!name.trim()) return;
if (!trimmedName || nameTooLong) {
// Create stays enabled: the click is what tells the user which field needs fixing.
setShowNameRequired(true);
nameRef.current?.focus();
return;
}
setCreating(true);
setError("");
try {
const body: Record<string, unknown> = { name: name.trim(), scope };
const body: Record<string, unknown> = { name: trimmedName, scope };
const days = Number(expires);
if (expires && Number.isFinite(days) && days > 0) body.expiresInDays = days;
const res = await fetch("/api/cli/tokens", {
@@ -123,9 +146,10 @@ export default function AccessTokensTab() {
);
return;
}
setNewSecretMeta({ name: name.trim(), scope });
setNewSecretMeta({ name: trimmedName, scope });
setNewSecret(data.token);
setName("");
setShowNameRequired(false);
setExpires("");
setScope("read");
await load();
@@ -159,8 +183,8 @@ export default function AccessTokensTab() {
: format.dateTime(date, { dateStyle: "medium", timeStyle: "short" });
};
// The dialog's opener ("Create") is disabled again once the name is cleared, so the Modal's
// own focus return lands on <body>; send focus back to the name field instead.
// After the secret dialog closes, the next thing to do is name another token: send focus to
// the name field rather than leaving it on the dialog's opener ("Create").
const dismissSecret = () => {
setNewSecret(null);
requestAnimationFrame(() => nameRef.current?.focus());
@@ -176,12 +200,16 @@ export default function AccessTokensTab() {
{/* Create */}
<Card title={t("accessTokensCreateHeading")}>
<div className="grid gap-3 sm:grid-cols-4 sm:items-end">
{/* Top-aligned so the message under Name does not push its input out of the row; the
button drops by the height of a label (sm:mt-7) to sit level with the inputs. */}
<div className="grid gap-3 sm:grid-cols-4 sm:items-start">
<Input
ref={nameRef}
label={t("accessTokensColName")}
label={nameField}
placeholder={t("accessTokensNamePlaceholder")}
value={name}
required
error={nameError}
onChange={(e) => setName(e.target.value)}
/>
<Select
@@ -203,7 +231,7 @@ export default function AccessTokensTab() {
value={expires}
onChange={(e) => setExpires(e.target.value)}
/>
<Button onClick={createToken} disabled={creating || !name.trim()}>
<Button className="sm:mt-7" onClick={createToken} disabled={creating}>
{creating ? t("accessTokensCreating") : t("accessTokensCreate")}
</Button>
</div>

View File

@@ -1,6 +1,6 @@
"use client";
import { memo, useEffect, useState } from "react";
import { memo, useEffect, useRef, useState } from "react";
import { useTranslations } from "next-intl";
import { buttonVariants } from "@omniroute/ui-kit/button";
import { Field } from "@omniroute/ui-kit/field";
@@ -20,6 +20,9 @@ import Icon from "@/shared/components/Icon";
import { useNotificationStore } from "@/store/notificationStore";
import { useConfirmDialog } from "@/shared/hooks/useConfirmDialog";
import { matchesSearch } from "@/shared/utils/turkishText";
import { focusFirstInvalidField } from "@/shared/utils/focusFirstInvalidField";
import { EVAL_DEFAULT_CASE_MODEL } from "@/shared/constants/evals";
import { hasSuiteDraftErrors, validateSuiteDraft, type SuiteDraftErrors } from "./evalSuiteDraft";
type EvalTargetType = "suite-default" | "model" | "combo";
@@ -458,6 +461,10 @@ export default function EvalsTab() {
const [isBuilderOpen, setIsBuilderOpen] = useState(false);
const [suiteDraft, setSuiteDraft] = useState<EvalSuiteDraft>(createEmptySuiteDraft());
const [savingSuite, setSavingSuite] = useState(false);
// Set by a refused save: the builder then flags the fields inline (an untouched form is not
// flagged for being empty) and `refusedSaves` moves focus to the first of them.
const [showSuiteErrors, setShowSuiteErrors] = useState(false);
const [refusedSaves, setRefusedSaves] = useState(0);
const [deletingSuiteId, setDeletingSuiteId] = useState<string | null>(null);
useEffect(() => {
@@ -661,29 +668,17 @@ export default function EvalsTab() {
async function handleSaveSuite() {
const suiteName = suiteDraft.name.trim();
if (!suiteName) {
notify.warning(t("suiteBuilderNameRequired"));
return;
}
if (suiteDraft.cases.length === 0) {
notify.warning(t("suiteBuilderCasesRequired"));
return;
}
const invalidCaseIndex = suiteDraft.cases.findIndex((draftCase) => {
const hasMessage =
draftCase.systemPrompt.trim().length > 0 || draftCase.userPrompt.trim().length > 0;
return (
!draftCase.name.trim() ||
!draftCase.expectedValue.trim() ||
!hasMessage ||
!draftCase.model.trim()
);
});
if (invalidCaseIndex >= 0) {
notify.warning(t("suiteBuilderCaseInvalid", { index: invalidCaseIndex + 1 }));
// Each missing field is flagged next to itself by the builder; a toast would only repeat
// it away from the field. A case's model is optional (see evalSuiteDraft.ts).
if (hasSuiteDraftErrors(validateSuiteDraft(suiteDraft))) {
setShowSuiteErrors(true);
setRefusedSaves((count) => count + 1);
return;
}
@@ -704,7 +699,8 @@ export default function EvalsTab() {
return {
...(draftCase.id.startsWith("draft-") ? {} : { id: draftCase.id }),
name: draftCase.name.trim(),
model: draftCase.model.trim(),
// Omitted when empty: the save schema takes a missing model, not an empty one.
...(draftCase.model.trim() ? { model: draftCase.model.trim() } : {}),
input: {
messages,
},
@@ -739,6 +735,7 @@ export default function EvalsTab() {
await refreshDashboard();
setExpanded(result?.suite?.id || suiteDraft.id || null);
setIsBuilderOpen(false);
setShowSuiteErrors(false);
setSuiteDraft(createEmptySuiteDraft());
notify.success(
isEditing ? t("suiteBuilderUpdated") : t("suiteBuilderCreated"),
@@ -1858,10 +1855,13 @@ export default function EvalsTab() {
onClose={() => {
if (savingSuite) return;
setIsBuilderOpen(false);
setShowSuiteErrors(false);
setSuiteDraft(createEmptySuiteDraft());
}}
onSave={() => void handleSaveSuite()}
saving={savingSuite}
errors={showSuiteErrors ? validateSuiteDraft(suiteDraft) : null}
focusInvalidRequest={refusedSaves}
t={t}
/>
{confirmDialog}
@@ -1925,6 +1925,8 @@ function SuiteBuilderModal({
onClose,
onSave,
saving,
errors,
focusInvalidRequest,
t,
}: {
draft: EvalSuiteDraft;
@@ -1933,9 +1935,19 @@ function SuiteBuilderModal({
onClose: () => void;
onSave: () => void;
saving: boolean;
/** The draft's problems once a save was refused; `null` before that. */
errors: SuiteDraftErrors | null;
/** Bumped on each refused save: focus (and scroll) goes to the first flagged field. */
focusInvalidRequest: number;
t: (key: string, values?: Record<string, unknown>) => string;
}) {
const tv = useTranslations("formValidation");
const editableStrategies = STRATEGIES.filter((strategy) => strategy.name !== "custom");
const bodyRef = useRef<HTMLDivElement>(null);
useEffect(() => {
if (focusInvalidRequest > 0) focusFirstInvalidField(bodyRef.current);
}, [focusInvalidRequest]);
function updateCase(caseId: string, patch: Partial<EvalCaseDraft>) {
onChange({
@@ -2001,13 +2013,15 @@ function SuiteBuilderModal({
title={draft.id ? t("suiteBuilderEditTitle") : t("suiteBuilderCreateTitle")}
onClose={onClose}
>
<div className="flex max-h-[75vh] flex-col gap-4 overflow-y-auto pr-1">
<div ref={bodyRef} className="flex max-h-[75vh] flex-col gap-4 overflow-y-auto pr-1">
<div className="grid grid-cols-1 gap-4 md:grid-cols-2">
<Input
label={t("suiteBuilderNameLabel")}
value={draft.name}
onChange={(event) => onChange({ ...draft, name: event.target.value })}
placeholder={t("suiteBuilderNamePlaceholder")}
required
error={errors?.name ? t("suiteBuilderNameRequired") : undefined}
/>
<Input
label={t("suiteBuilderDescriptionLabel")}
@@ -2033,6 +2047,8 @@ function SuiteBuilderModal({
const selectedStrategy = editableStrategies.find(
(strategy) => strategy.name === draftCase.strategy
);
const caseErrors = errors?.cases[draftCase.id];
const promptErrorId = `suite-case-${draftCase.id}-prompt-error`;
return (
<Card key={draftCase.id} className="p-4">
@@ -2071,12 +2087,22 @@ function SuiteBuilderModal({
value={draftCase.name}
onChange={(event) => updateCase(draftCase.id, { name: event.target.value })}
placeholder={t("suiteBuilderCaseNamePlaceholder")}
required
error={
caseErrors?.name
? tv("required", { field: t("suiteBuilderCaseNameLabel") })
: undefined
}
/>
<Input
label={t("suiteBuilderCaseModelLabel")}
value={draftCase.model}
onChange={(event) => updateCase(draftCase.id, { model: event.target.value })}
placeholder={t("suiteBuilderCaseModelPlaceholder")}
hint={t("suiteBuilderCaseModelHint", {
target: t("targetSuiteDefaults"),
fallback: EVAL_DEFAULT_CASE_MODEL,
})}
/>
<Input
label={t("suiteBuilderCaseTagsLabel")}
@@ -2130,9 +2156,18 @@ function SuiteBuilderModal({
<Field
label={t("suiteBuilderCaseUserPromptLabel")}
htmlFor={`suite-case-${draftCase.id}-userPrompt`}
error={
caseErrors?.prompt ? (
<span id={promptErrorId} role="alert">
{t("suiteBuilderCasePromptRequired")}
</span>
) : undefined
}
>
<Textarea
id={`suite-case-${draftCase.id}-userPrompt`}
error={Boolean(caseErrors?.prompt)}
aria-describedby={caseErrors?.prompt ? promptErrorId : undefined}
value={draftCase.userPrompt}
onChange={(event) =>
updateCase(draftCase.id, { userPrompt: event.target.value })
@@ -2149,6 +2184,12 @@ function SuiteBuilderModal({
}
placeholder={getExpectedPlaceholder(draftCase.strategy)}
hint={getExpectedHint(draftCase.strategy)}
required
error={
caseErrors?.expectedValue
? tv("required", { field: t("suiteBuilderCaseExpectedLabel") })
: undefined
}
/>
</div>
</Card>

View File

@@ -0,0 +1,48 @@
/**
* Validation of the Quality › suite builder draft, kept apart from the dialog so the rules can
* be read (and tested) in one place. A case needs a name, at least one prompt and an expected
* value. Its model is optional: the runner sends the case to the run's target and, on "Suite
* defaults" with no model on the case, to its built-in default (src/lib/evals/runtime.ts).
*/
export interface SuiteCaseDraftFields {
id: string;
name: string;
systemPrompt: string;
userPrompt: string;
expectedValue: string;
}
export interface SuiteDraftFields {
name: string;
cases: SuiteCaseDraftFields[];
}
export interface SuiteCaseErrors {
name: boolean;
prompt: boolean;
expectedValue: boolean;
}
export interface SuiteDraftErrors {
name: boolean;
/** Keyed by case id; only cases with at least one problem are present. */
cases: Record<string, SuiteCaseErrors>;
}
export function validateSuiteDraft(draft: SuiteDraftFields): SuiteDraftErrors {
const cases: Record<string, SuiteCaseErrors> = {};
for (const draftCase of draft.cases) {
const errors: SuiteCaseErrors = {
name: !draftCase.name.trim(),
prompt: !draftCase.systemPrompt.trim() && !draftCase.userPrompt.trim(),
expectedValue: !draftCase.expectedValue.trim(),
};
if (errors.name || errors.prompt || errors.expectedValue) cases[draftCase.id] = errors;
}
return { name: !draft.name.trim(), cases };
}
export function hasSuiteDraftErrors(errors: SuiteDraftErrors): boolean {
return errors.name || Object.keys(errors.cases).length > 0;
}

View File

@@ -1,10 +1,11 @@
"use client";
import { useEffect, useState } from "react";
import { useEffect, useRef, useState } from "react";
import { useTranslations } from "next-intl";
import { Alert } from "@omniroute/ui-kit/alert";
import Button from "@/shared/components/Button";
import Modal from "@/shared/components/Modal";
import { focusFirstInvalidField } from "@/shared/utils/focusFirstInvalidField";
import { Step1ChooseIntegration } from "./steps/Step1ChooseIntegration";
import {
Step2ConfigureIntegration,
@@ -17,6 +18,11 @@ import { Step3EventsAndTest } from "./steps/Step3EventsAndTest";
import { HowItWorksSidebar } from "./HowItWorksSidebar";
import type { WebhookItem } from "./WebhookCard";
import type { WebhookKind } from "./shared/IntegrationCard";
import {
webhookErrorText,
webhookSecretTooLong,
webhookUrlProblem,
} from "../webhookFieldValidation";
interface AddWebhookWizardProps {
isOpen: boolean;
@@ -67,15 +73,20 @@ function stateFromWebhook(webhook: WebhookItem | null | undefined): WizardState
function step2Valid(state: WizardState, isEditing: boolean): boolean {
const { kind } = state;
if (kind === "slack") return state.slack.webhookUrl.trim().length > 0;
if (kind === "slack") return webhookUrlProblem(state.slack.webhookUrl) === null;
if (kind === "telegram") {
return (
state.telegram.chatId.trim().length > 0 &&
(isEditing || state.telegram.botToken.trim().length > 0)
);
}
if (kind === "discord") return state.discord.webhookUrl.trim().length > 0;
if (kind === "custom") return state.custom.endpointUrl.trim().length > 0;
if (kind === "discord") return webhookUrlProblem(state.discord.webhookUrl) === null;
if (kind === "custom") {
return (
webhookUrlProblem(state.custom.endpointUrl) === null &&
!webhookSecretTooLong(state.custom.secretKey)
);
}
return false;
}
@@ -93,6 +104,15 @@ export function AddWebhookWizard({
const tc = useTranslations("common");
const [error, setError] = useState<string | null>(null);
const [createdId, setCreatedId] = useState<string | null>(null);
// Next stays enabled on step 2: a refused click is what flags the fields that need fixing
// (inline, by the forms) and moves focus to the first of them.
const [showStep2Errors, setShowStep2Errors] = useState(false);
const [refusedNext, setRefusedNext] = useState(0);
const bodyRef = useRef<HTMLDivElement>(null);
useEffect(() => {
if (refusedNext > 0) focusFirstInvalidField(bodyRef.current);
}, [refusedNext]);
useEffect(() => {
if (!isOpen) return;
@@ -100,6 +120,7 @@ export function AddWebhookWizard({
await Promise.resolve();
setStep(1);
setState(stateFromWebhook(editingWebhook));
setShowStep2Errors(false);
setError(null);
setCreatedId(editingWebhook?.id ?? null);
})();
@@ -109,6 +130,7 @@ export function AddWebhookWizard({
if (saving) return;
setStep(1);
setState(INITIAL);
setShowStep2Errors(false);
setError(null);
setCreatedId(null);
onClose();
@@ -134,6 +156,11 @@ export function AddWebhookWizard({
// Called when Next is clicked on step 2: create (or update) the webhook so the
// test button is available at step 3 before the user clicks Finish.
const handleNextFromStep2 = async () => {
if (!step2Valid(state, isEditing)) {
setShowStep2Errors(true);
setRefusedNext((count) => count + 1);
return;
}
setSaving(true);
setError(null);
try {
@@ -144,7 +171,7 @@ export function AddWebhookWizard({
body: JSON.stringify(buildConfigPayload()),
});
const data = await res.json().catch(() => ({}));
if (!res.ok) throw new Error(data.error || t("saveFailed"));
if (!res.ok) throw new Error(webhookErrorText(data.error, t("saveFailed")));
} else {
const res = await fetch("/api/webhooks", {
method: "POST",
@@ -152,7 +179,7 @@ export function AddWebhookWizard({
body: JSON.stringify(buildConfigPayload()),
});
const data = await res.json().catch(() => ({}));
if (!res.ok) throw new Error(data.error || t("saveFailed"));
if (!res.ok) throw new Error(webhookErrorText(data.error, t("saveFailed")));
setCreatedId(data.webhook?.id ?? null);
}
setStep(3);
@@ -180,7 +207,7 @@ export function AddWebhookWizard({
}),
});
const data = await res.json().catch(() => ({}));
if (!res.ok) throw new Error(data.error || t("saveFailed"));
if (!res.ok) throw new Error(webhookErrorText(data.error, t("saveFailed")));
onCreated();
handleClose();
} catch (err) {
@@ -190,8 +217,6 @@ export function AddWebhookWizard({
}
};
const canGoNext = step === 1 ? true : step === 2 ? step2Valid(state, isEditing) : true;
const stepTitle =
step === 1
? t("wizard.step1Title")
@@ -230,7 +255,7 @@ export function AddWebhookWizard({
<Button
variant="primary"
onClick={() => (step === 2 ? void handleNextFromStep2() : setStep((s) => s + 1))}
disabled={!canGoNext || saving}
disabled={saving}
loading={saving && step === 2}
>
{t("wizard.next")}
@@ -249,7 +274,7 @@ export function AddWebhookWizard({
</div>
}
>
<div className="flex gap-6">
<div ref={bodyRef} className="flex gap-6">
<div className="min-w-0 flex-1">
{error && (
<Alert
@@ -263,7 +288,10 @@ export function AddWebhookWizard({
{step === 1 && (
<Step1ChooseIntegration
selected={state.kind}
onSelect={(kind) => setState((s) => ({ ...s, kind }))}
onSelect={(kind) => {
setState((s) => ({ ...s, kind }));
setShowStep2Errors(false);
}}
t={t}
/>
)}
@@ -280,6 +308,7 @@ export function AddWebhookWizard({
onChangeCustom={(v) => setState((s) => ({ ...s, custom: v }))}
t={t}
isEditing={isEditing}
showRequired={showStep2Errors}
/>
)}
{step === 3 && (

View File

@@ -18,6 +18,8 @@ interface Step2ConfigureIntegrationProps {
onChangeCustom: (v: CustomConfig) => void;
t: (key: string, opts?: Record<string, unknown>) => string;
isEditing?: boolean;
/** Set by a refused Next: the forms then flag their empty required fields. */
showRequired?: boolean;
}
export function Step2ConfigureIntegration({
@@ -32,8 +34,12 @@ export function Step2ConfigureIntegration({
onChangeCustom,
t,
isEditing,
showRequired,
}: Step2ConfigureIntegrationProps) {
if (kind === "slack") return <SlackConfigForm value={slack} onChange={onChangeSlack} t={t} />;
if (kind === "slack")
return (
<SlackConfigForm value={slack} onChange={onChangeSlack} t={t} showRequired={showRequired} />
);
if (kind === "telegram")
return (
<TelegramConfigForm
@@ -41,11 +47,27 @@ export function Step2ConfigureIntegration({
onChange={onChangeTelegram}
t={t}
isEditing={isEditing}
showRequired={showRequired}
/>
);
if (kind === "discord")
return <DiscordConfigForm value={discord} onChange={onChangeDiscord} t={t} />;
return <CustomConfigForm value={custom} onChange={onChangeCustom} t={t} isEditing={isEditing} />;
return (
<DiscordConfigForm
value={discord}
onChange={onChangeDiscord}
t={t}
showRequired={showRequired}
/>
);
return (
<CustomConfigForm
value={custom}
onChange={onChangeCustom}
t={t}
isEditing={isEditing}
showRequired={showRequired}
/>
);
}
export type { SlackConfig, TelegramConfig, DiscordConfig, CustomConfig };

View File

@@ -1,8 +1,10 @@
"use client";
import { useEffect, useState } from "react";
import { useTranslations } from "next-intl";
import Input from "@/shared/components/Input";
import { HmacRecipeBlock } from "../../shared/HmacRecipeBlock";
import { webhookSecretFieldError, webhookUrlFieldFeedback } from "../../../webhookFieldValidation";
import { useWebhookUrlCheck } from "./useWebhookUrlCheck";
export interface CustomConfig {
endpointUrl: string;
@@ -14,52 +16,29 @@ interface CustomConfigFormProps {
onChange: (v: CustomConfig) => void;
t: (key: string) => string;
isEditing?: boolean;
/** Set by a refused Next: an untouched form is not flagged for being empty. */
showRequired?: boolean;
}
type UrlState = "idle" | "checking" | "ok" | "blocked" | "invalid";
export function CustomConfigForm({ value, onChange, t, isEditing }: CustomConfigFormProps) {
const [urlState, setUrlState] = useState<UrlState>("idle");
useEffect(() => {
const url = value.endpointUrl.trim();
const controller = new AbortController();
const delay = url ? 600 : 0;
const timer = setTimeout(async () => {
if (!url) {
setUrlState("idle");
return;
}
setUrlState("checking");
try {
const res = await fetch("/api/webhooks/validate-url", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ url }),
signal: controller.signal,
});
const data = await res.json().catch(() => ({}));
setUrlState(data.valid ? "ok" : data.reason === "blocked_private" ? "blocked" : "invalid");
} catch {
if (!controller.signal.aborted) setUrlState("invalid");
}
}, delay);
return () => {
clearTimeout(timer);
controller.abort();
};
}, [value.endpointUrl]);
const urlHint =
urlState === "checking"
? t("validateUrl.checking")
: urlState === "ok"
? t("validateUrl.ok")
: urlState === "blocked"
? t("validateUrl.blockedPrivate")
: urlState === "invalid" && value.endpointUrl.trim()
? t("validateUrl.invalidUrl")
: "";
export function CustomConfigForm({
value,
onChange,
t,
isEditing,
showRequired,
}: CustomConfigFormProps) {
const tv = useTranslations("formValidation");
const urlState = useWebhookUrlCheck(value.endpointUrl);
const url = webhookUrlFieldFeedback(value.endpointUrl, urlState, {
label: t("custom.endpointUrl"),
showRequired,
t,
tv,
});
const secretError = webhookSecretFieldError(value.secretKey, {
label: t("custom.secretKey"),
tv,
});
return (
<div className="space-y-4">
@@ -68,11 +47,14 @@ export function CustomConfigForm({ value, onChange, t, isEditing }: CustomConfig
value={value.endpointUrl}
onChange={(e) => onChange({ ...value, endpointUrl: e.target.value })}
placeholder={t("custom.endpointUrlPlaceholder")}
error={urlHint && urlState !== "ok" && urlState !== "checking" ? urlHint : undefined}
required
error={url.error}
hint={
<>
{urlHint && (urlState === "ok" || urlState === "checking") && (
<span className={`block ${urlState === "ok" ? "text-success" : ""}`}>{urlHint}</span>
{url.status && (
<span className={`block ${urlState === "ok" ? "text-success" : ""}`}>
{url.status}
</span>
)}
</>
}
@@ -84,6 +66,7 @@ export function CustomConfigForm({ value, onChange, t, isEditing }: CustomConfig
onChange={(e) => onChange({ ...value, secretKey: e.target.value })}
placeholder={isEditing ? t("secretEditPlaceholder") : t("custom.secretKeyPlaceholder")}
autoComplete="new-password"
error={secretError}
hint={t("custom.secretKeyHint")}
/>
<HmacRecipeBlock

View File

@@ -1,8 +1,10 @@
"use client";
import { Disclosure, DisclosureContent, DisclosureTrigger } from "@omniroute/ui-kit/disclosure";
import { useEffect, useState } from "react";
import { useTranslations } from "next-intl";
import Input from "@/shared/components/Input";
import { webhookUrlFieldFeedback } from "../../../webhookFieldValidation";
import { useWebhookUrlCheck } from "./useWebhookUrlCheck";
export interface DiscordConfig {
webhookUrl: string;
@@ -12,52 +14,19 @@ interface DiscordConfigFormProps {
value: DiscordConfig;
onChange: (v: DiscordConfig) => void;
t: (key: string) => string;
/** Set by a refused Next: an untouched form is not flagged for being empty. */
showRequired?: boolean;
}
type UrlState = "idle" | "checking" | "ok" | "blocked" | "invalid";
export function DiscordConfigForm({ value, onChange, t }: DiscordConfigFormProps) {
const [urlState, setUrlState] = useState<UrlState>("idle");
useEffect(() => {
const url = value.webhookUrl.trim();
const controller = new AbortController();
const delay = url ? 600 : 0;
const timer = setTimeout(async () => {
if (!url) {
setUrlState("idle");
return;
}
setUrlState("checking");
try {
const res = await fetch("/api/webhooks/validate-url", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ url }),
signal: controller.signal,
});
const data = await res.json().catch(() => ({}));
setUrlState(data.valid ? "ok" : data.reason === "blocked_private" ? "blocked" : "invalid");
} catch {
if (!controller.signal.aborted) setUrlState("invalid");
}
}, delay);
return () => {
clearTimeout(timer);
controller.abort();
};
}, [value.webhookUrl]);
const urlHint =
urlState === "checking"
? t("validateUrl.checking")
: urlState === "ok"
? t("validateUrl.ok")
: urlState === "blocked"
? t("validateUrl.blockedPrivate")
: urlState === "invalid" && value.webhookUrl.trim()
? t("validateUrl.invalidUrl")
: "";
export function DiscordConfigForm({ value, onChange, t, showRequired }: DiscordConfigFormProps) {
const tv = useTranslations("formValidation");
const urlState = useWebhookUrlCheck(value.webhookUrl);
const url = webhookUrlFieldFeedback(value.webhookUrl, urlState, {
label: t("discord.webhookUrl"),
showRequired,
t,
tv,
});
return (
<div className="space-y-4">
@@ -66,11 +35,14 @@ export function DiscordConfigForm({ value, onChange, t }: DiscordConfigFormProps
value={value.webhookUrl}
onChange={(e) => onChange({ webhookUrl: e.target.value })}
placeholder={t("discord.webhookUrlPlaceholder")}
error={urlHint && urlState !== "ok" && urlState !== "checking" ? urlHint : undefined}
required
error={url.error}
hint={
<>
{urlHint && (urlState === "ok" || urlState === "checking") && (
<span className={`block ${urlState === "ok" ? "text-success" : ""}`}>{urlHint}</span>
{url.status && (
<span className={`block ${urlState === "ok" ? "text-success" : ""}`}>
{url.status}
</span>
)}
<span className="block">{t("discord.webhookUrlHint")}</span>
</>

View File

@@ -1,8 +1,10 @@
"use client";
import { Disclosure, DisclosureContent, DisclosureTrigger } from "@omniroute/ui-kit/disclosure";
import { useEffect, useState } from "react";
import { useTranslations } from "next-intl";
import Input from "@/shared/components/Input";
import { webhookUrlFieldFeedback } from "../../../webhookFieldValidation";
import { useWebhookUrlCheck } from "./useWebhookUrlCheck";
export interface SlackConfig {
webhookUrl: string;
@@ -12,52 +14,19 @@ interface SlackConfigFormProps {
value: SlackConfig;
onChange: (v: SlackConfig) => void;
t: (key: string) => string;
/** Set by a refused Next: an untouched form is not flagged for being empty. */
showRequired?: boolean;
}
type UrlState = "idle" | "checking" | "ok" | "blocked" | "invalid";
export function SlackConfigForm({ value, onChange, t }: SlackConfigFormProps) {
const [urlState, setUrlState] = useState<UrlState>("idle");
useEffect(() => {
const url = value.webhookUrl.trim();
const controller = new AbortController();
const delay = url ? 600 : 0;
const timer = setTimeout(async () => {
if (!url) {
setUrlState("idle");
return;
}
setUrlState("checking");
try {
const res = await fetch("/api/webhooks/validate-url", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ url }),
signal: controller.signal,
});
const data = await res.json().catch(() => ({}));
setUrlState(data.valid ? "ok" : data.reason === "blocked_private" ? "blocked" : "invalid");
} catch {
if (!controller.signal.aborted) setUrlState("invalid");
}
}, delay);
return () => {
clearTimeout(timer);
controller.abort();
};
}, [value.webhookUrl]);
const urlHint =
urlState === "checking"
? t("validateUrl.checking")
: urlState === "ok"
? t("validateUrl.ok")
: urlState === "blocked"
? t("validateUrl.blockedPrivate")
: urlState === "invalid" && value.webhookUrl.trim()
? t("validateUrl.invalidUrl")
: "";
export function SlackConfigForm({ value, onChange, t, showRequired }: SlackConfigFormProps) {
const tv = useTranslations("formValidation");
const urlState = useWebhookUrlCheck(value.webhookUrl);
const url = webhookUrlFieldFeedback(value.webhookUrl, urlState, {
label: t("slack.webhookUrl"),
showRequired,
t,
tv,
});
return (
<div className="space-y-4">
@@ -66,11 +35,14 @@ export function SlackConfigForm({ value, onChange, t }: SlackConfigFormProps) {
value={value.webhookUrl}
onChange={(e) => onChange({ webhookUrl: e.target.value })}
placeholder={t("slack.webhookUrlPlaceholder")}
error={urlHint && urlState !== "ok" && urlState !== "checking" ? urlHint : undefined}
required
error={url.error}
hint={
<>
{urlHint && (urlState === "ok" || urlState === "checking") && (
<span className={`block ${urlState === "ok" ? "text-success" : ""}`}>{urlHint}</span>
{url.status && (
<span className={`block ${urlState === "ok" ? "text-success" : ""}`}>
{url.status}
</span>
)}
<span className="block">{t("slack.webhookUrlHint")}</span>
</>

View File

@@ -1,6 +1,7 @@
"use client";
import { Disclosure, DisclosureContent, DisclosureTrigger } from "@omniroute/ui-kit/disclosure";
import { useTranslations } from "next-intl";
import Input from "@/shared/components/Input";
export interface TelegramConfig {
@@ -13,9 +14,20 @@ interface TelegramConfigFormProps {
onChange: (v: TelegramConfig) => void;
t: (key: string) => string;
isEditing?: boolean;
/** Set by a refused Next: an untouched form is not flagged for being empty. */
showRequired?: boolean;
}
export function TelegramConfigForm({ value, onChange, t, isEditing }: TelegramConfigFormProps) {
export function TelegramConfigForm({
value,
onChange,
t,
isEditing,
showRequired,
}: TelegramConfigFormProps) {
const tv = useTranslations("formValidation");
// When editing, a blank token keeps the stored one.
const tokenRequired = !isEditing;
return (
<div className="space-y-4">
<Input
@@ -25,6 +37,12 @@ export function TelegramConfigForm({ value, onChange, t, isEditing }: TelegramCo
onChange={(e) => onChange({ ...value, botToken: e.target.value })}
placeholder={isEditing ? t("secretEditPlaceholder") : t("telegram.botTokenPlaceholder")}
autoComplete="new-password"
required={tokenRequired}
error={
showRequired && tokenRequired && !value.botToken.trim()
? tv("required", { field: t("telegram.botToken") })
: undefined
}
hint={t("telegram.botTokenHint")}
/>
<Input
@@ -32,6 +50,12 @@ export function TelegramConfigForm({ value, onChange, t, isEditing }: TelegramCo
value={value.chatId}
onChange={(e) => onChange({ ...value, chatId: e.target.value })}
placeholder={t("telegram.chatIdPlaceholder")}
required
error={
showRequired && !value.chatId.trim()
? tv("required", { field: t("telegram.chatId") })
: undefined
}
hint={t("telegram.chatIdHint")}
/>
<Disclosure className="rounded-lg border border-border bg-bg-subtle">

View File

@@ -0,0 +1,45 @@
"use client";
import { useEffect, useState } from "react";
import { webhookUrlProblem, type WebhookUrlCheckState } from "../../../webhookFieldValidation";
/**
* Asks the server whether a destination is allowed (it refuses private addresses), 600 ms after
* the last keystroke. A value that is empty, over the limit or not an http(s) URL is never
* sent: the field says so itself.
*/
export function useWebhookUrlCheck(value: string): WebhookUrlCheckState {
const [urlState, setUrlState] = useState<WebhookUrlCheckState>("idle");
useEffect(() => {
const url = value.trim();
const controller = new AbortController();
const askServer = webhookUrlProblem(value) === null;
const delay = askServer ? 600 : 0;
const timer = setTimeout(async () => {
if (!askServer) {
setUrlState("idle");
return;
}
setUrlState("checking");
try {
const res = await fetch("/api/webhooks/validate-url", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ url }),
signal: controller.signal,
});
const data = await res.json().catch(() => ({}));
setUrlState(data.valid ? "ok" : data.reason === "blocked_private" ? "blocked" : "invalid");
} catch {
if (!controller.signal.aborted) setUrlState("invalid");
}
}, delay);
return () => {
clearTimeout(timer);
controller.abort();
};
}, [value]);
return urlState;
}

View File

@@ -0,0 +1,131 @@
import { WEBHOOK_SECRET_MAX_LENGTH, WEBHOOK_URL_MAX_LENGTH } from "@/shared/constants/formLimits";
export type WebhookUrlProblem = "required" | "tooLong" | "invalid";
/** Where the server's check of a destination stands (see `useWebhookUrlCheck`). */
export type WebhookUrlCheckState = "idle" | "checking" | "ok" | "blocked" | "invalid";
/**
* What is wrong with a webhook destination before the server is asked: empty, over the route's
* length limit, or not an http(s) URL (the first thing the outbound guard refuses). `null` means
* the value is worth sending — the private-address check stays on the server.
*/
export function webhookUrlProblem(value: string): WebhookUrlProblem | null {
const url = value.trim();
if (!url) return "required";
// The request carries the value as typed, so the limit is measured on it too.
if (value.length > WEBHOOK_URL_MAX_LENGTH) return "tooLong";
try {
const parsed = new URL(url);
const isHttp = parsed.protocol === "http:" || parsed.protocol === "https:";
return isHttp && parsed.hostname !== "" ? null : "invalid";
} catch {
return "invalid";
}
}
type Translate = (key: string, values?: Record<string, string | number>) => string;
interface WebhookUrlFieldOptions {
/** The field's visible label, named in the message. */
label: string;
showRequired?: boolean;
/** The `webhooks` translator the wizard is given. */
t: Translate;
/** The shared `formValidation` translator. */
tv: Translate;
}
/**
* The inline message for a destination field, or `undefined` when the value can go to the
* server check. "Required" is only said once the user has tried to continue (`showRequired`);
* a malformed or over-long value is said as it is typed.
*/
function webhookUrlFieldError(value: string, options: WebhookUrlFieldOptions): string | undefined {
const problem = webhookUrlProblem(value);
if (problem === "tooLong") {
return options.tv("maxLength", {
field: options.label,
max: WEBHOOK_URL_MAX_LENGTH,
count: value.length,
});
}
if (problem === "invalid") return options.t("validateUrl.invalidUrl");
if (problem === "required" && options.showRequired) {
return options.tv("required", { field: options.label });
}
return undefined;
}
/** The server check's verdict in words; empty while there is nothing to say. */
function webhookUrlCheckText(state: WebhookUrlCheckState, value: string, t: Translate): string {
if (state === "checking") return t("validateUrl.checking");
if (state === "ok") return t("validateUrl.ok");
if (state === "blocked") return t("validateUrl.blockedPrivate");
if (state === "invalid" && value.trim()) return t("validateUrl.invalidUrl");
return "";
}
/**
* What a destination field shows under itself. `error` is the field's own verdict (empty, over
* the limit, not an http(s) URL — said at once, without a round trip) or, for a value that
* passes it, the server's refusal. `status` is the server check while it runs or once it
* accepted the value; empty otherwise.
*/
export function webhookUrlFieldFeedback(
value: string,
state: WebhookUrlCheckState,
options: WebhookUrlFieldOptions
): { error: string | undefined; status: string } {
const text = webhookUrlCheckText(state, value, options.t);
const pendingOrAccepted = state === "ok" || state === "checking";
return {
error: webhookUrlFieldError(value, options) ?? (text && !pendingOrAccepted ? text : undefined),
status: pendingOrAccepted ? text : "",
};
}
/** The signing secret is sent trimmed; over the route's limit it is refused, never cut. */
export function webhookSecretTooLong(value: string): boolean {
return value.trim().length > WEBHOOK_SECRET_MAX_LENGTH;
}
/**
* The inline message for a signing secret over the route's limit. The key is flagged, never
* cut: a truncated signing key would sign payloads the receiver cannot verify.
*/
export function webhookSecretFieldError(
value: string,
options: { label: string; tv: Translate }
): string | undefined {
if (!webhookSecretTooLong(value)) return undefined;
return options.tv("maxLength", {
field: options.label,
max: WEBHOOK_SECRET_MAX_LENGTH,
count: value.trim().length,
});
}
/**
* Text of the `error` field of a webhooks API reply. A validation failure arrives as an object
* (`{ message, details: [{ field, message }] }`), which must not reach the UI as
* "[object Object]".
*/
export function webhookErrorText(error: unknown, fallback: string): string {
if (typeof error === "string" && error.trim() !== "") return error;
if (error !== null && typeof error === "object") {
const bag = error as { message?: unknown; details?: unknown };
const details = Array.isArray(bag.details)
? bag.details
.map((detail) =>
detail !== null && typeof detail === "object"
? String((detail as { message?: unknown }).message ?? "")
: ""
)
.filter((text) => text !== "")
: [];
if (details.length > 0) return details.join(". ");
if (typeof bag.message === "string" && bag.message.trim() !== "") return bag.message;
}
return fallback;
}

View File

@@ -6,6 +6,7 @@ import { resolveOperatorCredential } from "@/lib/api/operatorCredential";
import { createAccessToken, listAccessTokens } from "@/lib/db/accessTokens";
import { ACCESS_SCOPES, scopeSatisfies, type AccessScope } from "@/lib/accessTokens/scopes";
import { isValidationFailure, validateBody } from "@/shared/validation/helpers";
import { ACCESS_TOKEN_NAME_MAX_LENGTH } from "@/shared/constants/formLimits";
/**
* /api/cli/tokens — manage scoped CLI access tokens. Admin-only: the path is in
@@ -29,7 +30,7 @@ export async function GET(request: Request) {
}
const createSchema = z.object({
name: z.string().min(1).max(100),
name: z.string().min(1).max(ACCESS_TOKEN_NAME_MAX_LENGTH),
scope: z.enum(ACCESS_SCOPES).optional(),
expiresInDays: z.number().int().positive().max(3650).optional(),
});

View File

@@ -16,15 +16,16 @@ import { isEncryptionEnabled } from "@/lib/db/encryption";
import { parseAndValidateWebhookUrl } from "@/shared/network/outboundUrlGuardPolicy";
import { WEBHOOK_EVENT_VALUES } from "@/lib/webhooks/eventDescriptions";
import { WEBHOOK_SECRET_MAX_LENGTH, WEBHOOK_URL_MAX_LENGTH } from "@/shared/constants/formLimits";
const WEBHOOK_KINDS = ["slack", "telegram", "discord", "custom"] as const;
const WEBHOOK_EVENT_VALUES_WITH_WILDCARD = ["*", ...WEBHOOK_EVENT_VALUES] as const;
const updateWebhookSchema = z
.object({
url: z.string().min(1).max(2000).optional(),
url: z.string().min(1).max(WEBHOOK_URL_MAX_LENGTH).optional(),
events: z.array(z.enum(WEBHOOK_EVENT_VALUES_WITH_WILDCARD)).optional(),
secret: z.string().max(500).optional(),
secret: z.string().max(WEBHOOK_SECRET_MAX_LENGTH).optional(),
description: z.string().max(1000).optional(),
enabled: z.boolean().optional(),
kind: z.enum(WEBHOOK_KINDS).optional(),

View File

@@ -16,15 +16,16 @@ import { isEncryptionEnabled } from "@/lib/db/encryption";
import { parseAndValidateWebhookUrl } from "@/shared/network/outboundUrlGuardPolicy";
import { WEBHOOK_EVENT_VALUES } from "@/lib/webhooks/eventDescriptions";
import { WEBHOOK_SECRET_MAX_LENGTH, WEBHOOK_URL_MAX_LENGTH } from "@/shared/constants/formLimits";
const WEBHOOK_KINDS = ["slack", "telegram", "discord", "custom"] as const;
const WEBHOOK_EVENT_VALUES_WITH_WILDCARD = ["*", ...WEBHOOK_EVENT_VALUES] as const;
const createWebhookSchema = z
.object({
url: z.string().min(1).max(2000),
url: z.string().min(1).max(WEBHOOK_URL_MAX_LENGTH),
events: z.array(z.enum(WEBHOOK_EVENT_VALUES_WITH_WILDCARD)).optional().default(["*"]),
secret: z.string().max(500).optional(),
secret: z.string().max(WEBHOOK_SECRET_MAX_LENGTH).optional(),
description: z.string().max(1000).optional().default(""),
kind: z.enum(WEBHOOK_KINDS).optional().default("custom"),
metadata: z.record(z.string(), z.string()).optional(),

View File

@@ -9,9 +9,10 @@ import { requireManagementAuth } from "@/lib/api/requireManagementAuth";
import { OutboundUrlGuardError } from "@/shared/network/outboundUrlGuard";
import { parseAndValidateWebhookUrl } from "@/shared/network/outboundUrlGuardPolicy";
import { validateBody, isValidationFailure } from "@/shared/validation/helpers";
import { WEBHOOK_URL_MAX_LENGTH } from "@/shared/constants/formLimits";
const validateUrlSchema = z.object({
url: z.string().min(1).max(2000),
url: z.string().min(1).max(WEBHOOK_URL_MAX_LENGTH),
});
export async function POST(request: Request) {
@@ -36,7 +37,9 @@ export async function POST(request: Request) {
parseAndValidateWebhookUrl(url);
return NextResponse.json({ valid: true });
} catch (err) {
if (err instanceof OutboundUrlGuardError) {
// The guard also throws for text that is not an http(s) URL at all (OUTBOUND_URL_INVALID):
// that is "invalid", not "a blocked private address".
if (err instanceof OutboundUrlGuardError && err.code !== "OUTBOUND_URL_INVALID") {
return NextResponse.json({ valid: false, reason: "blocked_private" });
}
return NextResponse.json({ valid: false, reason: "invalid_url" });

View File

@@ -1607,7 +1607,7 @@
"checking": "Checking URL...",
"ok": "URL is valid",
"blockedPrivate": "URL is a blocked private address",
"invalidUrl": "Invalid URL format"
"invalidUrl": "Enter a valid URL that starts with https:// or http://"
},
"custom": {
"endpointUrl": "Endpoint URL",
@@ -11054,6 +11054,8 @@
"suiteLatestRunsHint": "Most recent evaluation runs for this suite.",
"saving": "Saving",
"suiteBuilderCaseModelLabel": "Model (optional)",
"suiteBuilderCaseModelHint": "Used when a run targets \"{target}\". If empty, the case runs on {fallback}.",
"suiteBuilderCasePromptRequired": "Add a system prompt or a user prompt",
"evalControlsTitle": "Evaluation Controls",
"suiteBuilderEditTitle": "Edit Eval Suite",
"suiteBuilderCaseStrategyLabel": "Validation Strategy",
@@ -16687,5 +16689,9 @@
"recoveryHelp": "Recovery reuses the original payload and key. After reload, restore the exact endpoint, connection, transport and input. New generation starts a separate billable request. Up to 20 receipts are kept in this tab.",
"streamIncomplete": "The stream ended without a complete final event. Upstream completion is unknown.",
"streamError": "The upstream stream reported an error. Received files remain available."
},
"formValidation": {
"required": "{field} is required",
"maxLength": "{field} can have at most {max, number} characters (currently {count, number})"
}
}

View File

@@ -205,6 +205,7 @@
"common",
"costs",
"cursorAuthModal",
"formValidation",
"header",
"kiroAuthModal",
"kiroSocialOAuthModal",
@@ -1637,6 +1638,7 @@
"cursorAuthModal",
"endpoint",
"featureFlags",
"formValidation",
"header",
"health",
"kiroAuthModal",
@@ -1796,6 +1798,7 @@
"connectionTest",
"costs",
"cursorAuthModal",
"formValidation",
"freeBudget",
"header",
"kiroAuthModal",
@@ -1828,6 +1831,7 @@
"common",
"costs",
"cursorAuthModal",
"formValidation",
"header",
"kiroAuthModal",
"kiroSocialOAuthModal",

View File

@@ -4,6 +4,7 @@ import { saveEvalRun } from "@/lib/db/evals";
import { resolveSelfLoopBearer } from "@/shared/middleware/chatAdmissionIdentity";
import { getApiKeyById } from "@/lib/db/apiKeys";
import { getCombos } from "@/lib/db/combos";
import { EVAL_DEFAULT_CASE_MODEL } from "@/shared/constants/evals";
import { getSuite, listSuites, runSuite } from "./evalRunner";
export interface EvalTargetInput {
@@ -178,10 +179,10 @@ function resolveCaseModel(evalCase: Record<string, unknown>, target: EvalTargetI
typeof evalCase.model === "string" && evalCase.model.trim().length > 0 ? evalCase.model : null;
if (target.type === "model" || target.type === "combo") {
return targetId || caseModel || "gpt-4o";
return targetId || caseModel || EVAL_DEFAULT_CASE_MODEL;
}
return caseModel || "gpt-4o";
return caseModel || EVAL_DEFAULT_CASE_MODEL;
}
/**

View File

@@ -0,0 +1,6 @@
/**
* Model an eval case is sent to when it names none and the run targets "Suite defaults"
* (src/lib/evals/runtime.ts). The suite builder names it in the hint of its optional Model
* field, so the two read the same value.
*/
export const EVAL_DEFAULT_CASE_MODEL = "gpt-4o";

View File

@@ -0,0 +1,14 @@
/**
* Field limits shared by a dashboard form and the Zod schema of the route that receives it.
* The form shows an inline message at the limit and the route refuses beyond it; importing the
* same constant on both sides keeps the two from drifting apart.
*/
/** `name` of POST /api/cli/tokens (Settings › Access tokens). */
export const ACCESS_TOKEN_NAME_MAX_LENGTH = 100;
/** `url` of POST/PUT /api/webhooks and POST /api/webhooks/validate-url. */
export const WEBHOOK_URL_MAX_LENGTH = 2000;
/** `secret` of POST/PUT /api/webhooks (the HMAC signing key of a custom webhook). */
export const WEBHOOK_SECRET_MAX_LENGTH = 500;

View File

@@ -0,0 +1,11 @@
/**
* Moves focus — and with it the scroll position of the enclosing dialog — to the first control
* flagged `aria-invalid="true"` inside `container`. Forms call it after a refused submit, once
* the inline messages have rendered, so the user lands on the field that needs fixing.
*/
export function focusFirstInvalidField(container: ParentNode | null | undefined): boolean {
const field = container?.querySelector<HTMLElement>('[aria-invalid="true"]');
if (!field) return false;
field.focus();
return true;
}

View File

@@ -0,0 +1,149 @@
/**
* KEA-9386: the limits and the wording the Custom webhook dialog shows inline are the ones
* the API enforces.
*
* - POST /api/webhooks and PUT /api/webhooks/[id] refuse a `url` over WEBHOOK_URL_MAX_LENGTH
* and a `secret` over WEBHOOK_SECRET_MAX_LENGTH, naming the field; values at the limit and
* the 340 / 400 character values of the ticket are accepted and stored whole.
* - POST /api/webhooks/validate-url tells "not a URL" apart from "blocked private address":
* it used to answer `blocked_private` for plain text, so the form told the user that
* `not-a-url` was a private address.
*/
import { describe, it, after } from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-webhook-limits-"));
process.env.DATA_DIR = TEST_DATA_DIR;
process.env.NODE_ENV = "test";
process.env.DISABLE_SQLITE_AUTO_BACKUP = "true";
const core = await import("../../../../src/lib/db/core.ts");
const { updateSettings } = await import("../../../../src/lib/db/settings.ts");
await updateSettings({ requireLogin: false });
const webhooksRoute = await import("../../../../src/app/api/webhooks/route.ts");
const webhookByIdRoute = await import("../../../../src/app/api/webhooks/[id]/route.ts");
const validateUrlRoute = await import("../../../../src/app/api/webhooks/validate-url/route.ts");
const { getWebhook } = await import("../../../../src/lib/db/webhooks.ts");
const { WEBHOOK_SECRET_MAX_LENGTH, WEBHOOK_URL_MAX_LENGTH } =
await import("../../../../src/shared/constants/formLimits.ts");
after(() => {
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
});
function jsonRequest(url: string, method: string, body: unknown): Request {
return new Request(url, {
method,
headers: { "Content-Type": "application/json" },
body: JSON.stringify(body),
});
}
const urlOfLength = (length: number) => {
const prefix = "https://example.com/hook/";
return prefix + "a".repeat(length - prefix.length);
};
const create = (body: unknown) =>
webhooksRoute.POST(jsonRequest("http://localhost/api/webhooks", "POST", body));
const update = (id: string, body: unknown) =>
webhookByIdRoute.PUT(jsonRequest(`http://localhost/api/webhooks/${id}`, "PUT", body), {
params: Promise.resolve({ id }),
});
const validateUrl = async (url: string) => {
const res = await validateUrlRoute.POST(
jsonRequest("http://localhost/api/webhooks/validate-url", "POST", { url })
);
return { status: res.status, body: await res.json() };
};
describe("webhook field limits shared with the dashboard form", () => {
it("exposes the limits the routes enforce", () => {
assert.equal(WEBHOOK_URL_MAX_LENGTH, 2000);
assert.equal(WEBHOOK_SECRET_MAX_LENGTH, 500);
});
it("POST stores the ticket's 340-character URL and 400-character secret whole", async () => {
const url = urlOfLength(340);
const secret = "s".repeat(400);
const res = await create({ kind: "custom", url, secret });
assert.equal(res.status, 201);
const { webhook } = await res.json();
const stored = getWebhook(webhook.id);
assert.equal(stored?.url, url);
assert.equal(stored?.secret?.length, 400, "the secret is not truncated on the way in");
});
it("POST accepts values exactly at the limits", async () => {
const res = await create({
kind: "custom",
url: urlOfLength(WEBHOOK_URL_MAX_LENGTH),
secret: "s".repeat(WEBHOOK_SECRET_MAX_LENGTH),
});
assert.equal(res.status, 201);
});
it("POST refuses a URL over the limit, naming the field", async () => {
const res = await create({ kind: "custom", url: urlOfLength(WEBHOOK_URL_MAX_LENGTH + 1) });
assert.equal(res.status, 400);
const body = await res.json();
assert.ok(body.error.details.some((detail: { field: string }) => detail.field === "url"));
});
it("POST refuses a secret over the limit, naming the field", async () => {
const res = await create({
kind: "custom",
url: "https://example.com/hook",
secret: "s".repeat(WEBHOOK_SECRET_MAX_LENGTH + 1),
});
assert.equal(res.status, 400);
const body = await res.json();
assert.deepEqual(
body.error.details.map((detail: { field: string }) => detail.field),
["secret"]
);
});
it("PUT enforces the same limits", async () => {
const created = await create({ kind: "custom", url: "https://example.com/hook" });
const { webhook } = await created.json();
const longUrl = await update(webhook.id, { url: urlOfLength(WEBHOOK_URL_MAX_LENGTH + 1) });
assert.equal(longUrl.status, 400);
const longSecret = await update(webhook.id, {
secret: "s".repeat(WEBHOOK_SECRET_MAX_LENGTH + 1),
});
assert.equal(longSecret.status, 400);
const atLimit = await update(webhook.id, {
url: urlOfLength(WEBHOOK_URL_MAX_LENGTH),
secret: "s".repeat(WEBHOOK_SECRET_MAX_LENGTH),
});
assert.equal(atLimit.status, 200);
});
});
describe("POST /api/webhooks/validate-url — reason", () => {
it("answers invalid_url for text that is not a URL", async () => {
for (const url of ["not-a-url", "<script>x</script> %% 😀", "ftp://example.com/hook"]) {
const { status, body } = await validateUrl(url);
assert.equal(status, 200);
assert.deepEqual(body, { valid: false, reason: "invalid_url" }, url);
}
});
it("answers blocked_private for a private address", async () => {
const { body } = await validateUrl("http://127.0.0.1:20128/hook");
assert.deepEqual(body, { valid: false, reason: "blocked_private" });
});
it("accepts a public https URL and refuses one over the limit", async () => {
assert.deepEqual((await validateUrl("https://example.com/hook")).body, { valid: true });
assert.equal((await validateUrl(urlOfLength(WEBHOOK_URL_MAX_LENGTH + 1))).status, 400);
});
});

View File

@@ -0,0 +1,80 @@
/**
* KEA-9387: POST /api/cli/tokens enforces the same Name limit the dashboard form shows
* (ACCESS_TOKEN_NAME_MAX_LENGTH), so the client message and the server refusal cannot drift.
*/
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-cli-tokens-name-"));
const ORIGINAL_ENV = {
DATA_DIR: process.env.DATA_DIR,
API_KEY_SECRET: process.env.API_KEY_SECRET,
INITIAL_PASSWORD: process.env.INITIAL_PASSWORD,
JWT_SECRET: process.env.JWT_SECRET,
DISABLE_SQLITE_AUTO_BACKUP: process.env.DISABLE_SQLITE_AUTO_BACKUP,
};
process.env.DATA_DIR = TEST_DATA_DIR;
process.env.DISABLE_SQLITE_AUTO_BACKUP = "true";
process.env.API_KEY_SECRET = "cli-tokens-name-api-key-secret";
process.env.JWT_SECRET = "cli-tokens-name-jwt-secret";
process.env.INITIAL_PASSWORD = "cli-tokens-name-password";
const core = await import("../../src/lib/db/core.ts");
const apiKeysDb = await import("../../src/lib/db/apiKeys.ts");
const accessTokensDb = await import("../../src/lib/db/accessTokens.ts");
const { updateSettings } = await import("../../src/lib/db/settings.ts");
const tokensRoute = await import("../../src/app/api/cli/tokens/route.ts");
const { ACCESS_TOKEN_NAME_MAX_LENGTH } = await import("../../src/shared/constants/formLimits.ts");
await updateSettings({ requireLogin: true });
const manageKey = await apiKeysDb.createApiKey("manage-automation", "machine-cli-name", ["manage"]);
test.after(() => {
try {
apiKeysDb.resetApiKeyState?.();
} catch {}
try {
core.resetDbInstance();
} catch {}
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
for (const [name, value] of Object.entries(ORIGINAL_ENV)) {
if (value === undefined) delete process.env[name];
else process.env[name] = value;
}
});
function mint(name: string) {
return tokensRoute.POST(
new Request("http://localhost:20128/api/cli/tokens", {
method: "POST",
headers: { "content-type": "application/json", authorization: `Bearer ${manageKey.key}` },
body: JSON.stringify({ name, scope: "read" }),
})
);
}
test("the limit the form shows is the one the route enforces", () => {
assert.equal(ACCESS_TOKEN_NAME_MAX_LENGTH, 100);
});
test("a name one character over the limit is refused with a field-level detail", async () => {
const name = "n".repeat(ACCESS_TOKEN_NAME_MAX_LENGTH + 1);
const res = await mint(name);
assert.equal(res.status, 400);
const body = await res.json();
assert.equal(body.token, undefined);
assert.equal(body.error.details[0].field, "name");
assert.equal(accessTokensDb.listAccessTokens().filter((t) => t.name === name).length, 0);
});
test("a name exactly at the limit is accepted", async () => {
const name = "n".repeat(ACCESS_TOKEN_NAME_MAX_LENGTH);
const res = await mint(name);
assert.equal(res.status, 200);
const body = await res.json();
assert.equal(body.name, name);
});

View File

@@ -0,0 +1,60 @@
/**
* KEA-9388: a suite-builder case is valid without a model — the field is labelled optional,
* the save schema marks it optional, and the runner has a fallback. The other required fields
* are reported per field so the dialog can flag each one inline (KEA-9389).
*/
import test from "node:test";
import assert from "node:assert/strict";
import {
hasSuiteDraftErrors,
validateSuiteDraft,
} from "../../src/app/(dashboard)/dashboard/usage/components/evalSuiteDraft.ts";
import { evalSuiteSaveSchema } from "../../src/shared/validation/schemas/evals.ts";
const filledCase = {
id: "draft-1",
name: "greets",
model: "",
systemPrompt: "",
userPrompt: "Say hello",
expectedValue: "hello",
};
test("a case with an empty model has no errors", () => {
const errors = validateSuiteDraft({ name: "QA-test-suite", cases: [filledCase] });
assert.deepEqual(errors, { name: false, cases: {} });
assert.equal(hasSuiteDraftErrors(errors), false);
});
test("the save schema accepts a case without `model` and refuses an empty one", () => {
const body = (model?: string) => ({
name: "QA-test-suite",
cases: [
{
name: "greets",
...(model === undefined ? {} : { model }),
input: { messages: [{ role: "user", content: "Say hello" }] },
expected: { strategy: "contains", value: "hello" },
},
],
});
assert.equal(evalSuiteSaveSchema.safeParse(body()).success, true);
// Why the dialog omits the key instead of sending "": the schema is `min(1)` when present.
assert.equal(evalSuiteSaveSchema.safeParse(body("")).success, false);
});
test("each missing required field is reported on its own, per case", () => {
const errors = validateSuiteDraft({
name: " ",
cases: [
filledCase,
{ ...filledCase, id: "draft-2", name: " ", userPrompt: "", expectedValue: "" },
{ ...filledCase, id: "draft-3", userPrompt: "", systemPrompt: "Be brief" },
],
});
assert.equal(errors.name, true);
assert.deepEqual(errors.cases, {
"draft-2": { name: true, prompt: true, expectedValue: true },
});
assert.equal(hasSuiteDraftErrors(errors), true);
});

View File

@@ -0,0 +1,44 @@
/**
* KEA-9388: why the suite builder's "Model (optional)" can really be optional — the runner
* resolves a model for a case that names none. A run aimed at a model or combo uses that
* target; a run on "Suite defaults" uses the case's model and, without one, the default the
* builder names in the field's hint (EVAL_DEFAULT_CASE_MODEL).
*/
import { test } from "node:test";
import assert from "node:assert/strict";
import { buildEvalCaseRequest } from "../../src/lib/evals/runtime.ts";
import { EVAL_DEFAULT_CASE_MODEL } from "../../src/shared/constants/evals.ts";
const input = { messages: [{ role: "user", content: "Say hello" }] };
async function modelSentFor(
evalCase: Record<string, unknown>,
target: { type: "suite-default" | "model" | "combo"; id?: string | null }
) {
const body = await buildEvalCaseRequest(evalCase, target, null).json();
return body.model;
}
test("a case without a model runs on the default when the run targets Suite defaults", async () => {
assert.equal(EVAL_DEFAULT_CASE_MODEL, "gpt-4o");
assert.equal(await modelSentFor({ input }, { type: "suite-default" }), EVAL_DEFAULT_CASE_MODEL);
assert.equal(
await modelSentFor({ input, model: " " }, { type: "suite-default" }),
EVAL_DEFAULT_CASE_MODEL
);
});
test("a case without a model runs on the run's model or combo target", async () => {
assert.equal(
await modelSentFor({ input }, { type: "model", id: "claude-sonnet-4-20250514" }),
"claude-sonnet-4-20250514"
);
assert.equal(await modelSentFor({ input }, { type: "combo", id: "my-combo" }), "my-combo");
});
test("a case's own model is used on Suite defaults", async () => {
assert.equal(
await modelSentFor({ input, model: "gpt-4o-mini" }, { type: "suite-default" }),
"gpt-4o-mini"
);
});

View File

@@ -0,0 +1,117 @@
// @vitest-environment jsdom
// KEA-9387: the access-token Name field says what is wrong, where it is wrong. Create stays
// enabled; clicking it with an empty or over-long name flags the field inline and focuses it,
// and nothing is sent to the API.
import React from "react";
import { cleanup, fireEvent, render, screen } from "@testing-library/react";
import { NextIntlClientProvider } from "next-intl";
import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import AccessTokensTab from "@/app/(dashboard)/dashboard/settings/components/AccessTokensTab";
import en from "@/i18n/messages/en.json";
import { ACCESS_TOKEN_NAME_MAX_LENGTH } from "@/shared/constants/formLimits";
// The component formats dates with `useFormatter`, which the global key-only mock lacks:
// exercise the real catalog and provider instead.
vi.unmock("next-intl");
// The kit Table measures itself with ResizeObserver; jsdom lacks it.
beforeAll(() => {
globalThis.ResizeObserver ??= class {
observe() {}
unobserve() {}
disconnect() {}
} as unknown as typeof ResizeObserver;
});
const reply = (data: unknown, status = 200) => new Response(JSON.stringify(data), { status });
const fetchMock = vi.fn<typeof fetch>();
const posts = () => fetchMock.mock.calls.filter(([, init]) => init?.method === "POST");
beforeEach(() => {
fetchMock.mockReset();
fetchMock.mockImplementation(async () => reply({ tokens: [] }));
vi.stubGlobal("fetch", fetchMock);
});
afterEach(() => {
cleanup();
vi.unstubAllGlobals();
});
// Plain DOM lookups: role queries walk the whole accessibility tree on every call.
const createButton = () =>
Array.from(document.querySelectorAll("button")).find(
(node) => node.textContent === "Create"
) as HTMLButtonElement;
const alerts = () =>
Array.from(document.querySelectorAll('[role="alert"]')).map((node) => node.textContent);
async function mount() {
render(
<NextIntlClientProvider
locale="en"
timeZone="UTC"
messages={{
settings: en.settings,
common: en.common,
formValidation: en.formValidation,
}}
onError={(error) => {
throw error;
}}
>
<AccessTokensTab />
</NextIntlClientProvider>
);
// The list request settles first; generous for a loaded CI runner.
await screen.findByText("No access tokens yet.", undefined, { timeout: 10_000 });
return screen.getByPlaceholderText("Name (e.g. laptop)") as HTMLInputElement;
}
describe("access-token Name validation (KEA-9387)", () => {
it("explains an empty name inline when Create is clicked, instead of a silent disabled button", async () => {
const name = await mount();
const create = createButton();
expect(create.disabled).toBe(false);
expect(name.required).toBe(true);
// Nothing is flagged before the user tries to create.
expect(alerts()).toEqual([]);
fireEvent.click(create);
expect(alerts()).toEqual(["Name is required"]);
expect(name.getAttribute("aria-invalid")).toBe("true");
expect(document.activeElement).toBe(name);
expect(posts()).toHaveLength(0);
// Typing a name clears the message.
fireEvent.change(name, { target: { value: "laptop" } });
expect(alerts()).toEqual([]);
});
it("flags a name over the server limit as it is typed and does not send it", async () => {
const name = await mount();
const tooLong = "n".repeat(ACCESS_TOKEN_NAME_MAX_LENGTH + 1);
fireEvent.change(name, { target: { value: tooLong } });
expect(alerts()).toEqual(["Name can have at most 100 characters (currently 101)"]);
expect(name.getAttribute("aria-invalid")).toBe("true");
// Nothing is cut: the user sees what they typed and decides what to remove.
expect(name.value).toHaveLength(ACCESS_TOKEN_NAME_MAX_LENGTH + 1);
fireEvent.click(createButton());
expect(posts()).toHaveLength(0);
expect(document.activeElement).toBe(name);
});
it("accepts a name exactly at the limit", async () => {
const name = await mount();
const atLimit = "n".repeat(ACCESS_TOKEN_NAME_MAX_LENGTH);
fireEvent.change(name, { target: { value: ` ${atLimit} ` } });
expect(alerts()).toEqual([]);
fetchMock.mockResolvedValueOnce(reply({ token: "omni_disposable_test_secret" }));
fireEvent.click(createButton());
await screen.findByText("omni_disposable_test_secret", undefined, { timeout: 10_000 });
expect(JSON.parse(String(posts()[0][1]?.body))).toEqual({ name: atLimit, scope: "read" });
});
});

View File

@@ -0,0 +1,170 @@
// @vitest-environment jsdom
// KEA-9388 + KEA-9389: the Quality › New suite dialog.
// - "Model (optional)" is optional: a case without a model saves (the runner falls back to the
// run target, then to its default), and the request omits `model` rather than sending "".
// - Missing required fields are flagged inline next to the field and the first one is focused;
// the errors are no longer toast-only.
import React from "react";
import { cleanup, fireEvent, render, screen, waitFor, within } from "@testing-library/react";
import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import EvalsTab from "@/app/(dashboard)/dashboard/usage/components/EvalsTab";
import { useNotificationStore } from "@/store/notificationStore";
beforeAll(() => {
globalThis.ResizeObserver ??= class {
observe() {}
unobserve() {}
disconnect() {}
} as unknown as typeof ResizeObserver;
});
const reply = (data: unknown, status = 200) => new Response(JSON.stringify(data), { status });
const fetchMock = vi.fn<typeof fetch>();
const saves = () =>
fetchMock.mock.calls.filter(
([url, init]) => url === "/api/evals/suites" && init?.method === "POST"
);
// The page shows an empty state (and no "New Suite") without at least one suite; real
// installs always carry the built-in ones.
const builtIn = {
id: "golden-set",
name: "Golden Set",
source: "built-in",
cases: [
{
id: "gs-1",
name: "greets",
model: "gpt-4o",
input: { messages: [{ role: "user", content: "Say hello" }] },
expected: { strategy: "contains", value: "hello" },
},
],
};
beforeEach(() => {
useNotificationStore.setState({ notifications: [] });
fetchMock.mockReset();
fetchMock.mockImplementation(async (url, init) =>
url === "/api/evals/suites" && init?.method === "POST"
? reply({ suite: { id: "suite-1" } }, 201)
: reply({ suites: [builtIn], recentRuns: [], scorecard: null, targets: [], apiKeys: [] })
);
vi.stubGlobal("fetch", fetchMock);
});
afterEach(() => {
cleanup();
vi.unstubAllGlobals();
});
// Plain DOM lookups: role queries walk the whole accessibility tree of this large page on
// every call.
const button = (root: ParentNode, text: string) =>
Array.from(root.querySelectorAll("button")).find((node) =>
node.textContent?.includes(text)
) as HTMLButtonElement;
async function openBuilder() {
render(<EvalsTab />);
await screen.findByText("New Suite");
fireEvent.click(button(document, "New Suite"));
await screen.findByText("Create Eval Suite");
return document.querySelector<HTMLElement>('[role="dialog"]')!;
}
const field = (dialog: HTMLElement, label: RegExp) =>
within(dialog).getByLabelText(label) as HTMLInputElement | HTMLTextAreaElement;
const type = (node: HTMLElement, value: string) => fireEvent.change(node, { target: { value } });
const alerts = (dialog: HTMLElement) =>
Array.from(dialog.querySelectorAll('[role="alert"]')).map((node) => node.textContent);
const toasts = () => useNotificationStore.getState().notifications.map((n) => n.message);
function fillRequired(dialog: HTMLElement) {
type(field(dialog, /^Suite Name/), "QA-test-suite");
type(field(dialog, /^Case Name/), "greets");
type(field(dialog, /^User Prompt/), "Say hello");
type(field(dialog, /^Expected Value/), "hello");
}
describe("Quality › New suite (KEA-9388, KEA-9389)", () => {
it("saves a case whose optional Model is empty, omitting `model` from the request", async () => {
const dialog = await openBuilder();
fillRequired(dialog);
expect(field(dialog, /^Model \(optional\)/).value).toBe("");
fireEvent.click(button(dialog, "Create Suite"));
await waitFor(() => expect(saves()).toHaveLength(1));
const body = JSON.parse(String(saves()[0][1]?.body));
expect(body.name).toBe("QA-test-suite");
expect(body.cases).toHaveLength(1);
expect(body.cases[0]).toEqual({
name: "greets",
input: { messages: [{ role: "user", content: "Say hello" }] },
expected: { strategy: "contains", value: "hello" },
tags: [],
});
expect(alerts(dialog)).toEqual([]);
});
it("still sends the model when one is typed", async () => {
const dialog = await openBuilder();
fillRequired(dialog);
type(field(dialog, /^Model \(optional\)/), " gpt-4o-mini ");
fireEvent.click(button(dialog, "Create Suite"));
await waitFor(() => expect(saves()).toHaveLength(1));
expect(JSON.parse(String(saves()[0][1]?.body)).cases[0].model).toBe("gpt-4o-mini");
});
it("says what the empty Model does", async () => {
const dialog = await openBuilder();
const model = field(dialog, /^Model \(optional\)/);
const hint = document.getElementById(model.getAttribute("aria-describedby") ?? "");
expect(hint?.textContent).toBe(
'Used when a run targets "Suite defaults". If empty, the case runs on gpt-4o.'
);
});
it("flags every missing required field inline and focuses the first one", async () => {
const dialog = await openBuilder();
// Nothing is flagged before the user tries to save.
expect(alerts(dialog)).toEqual([]);
fireEvent.click(button(dialog, "Create Suite"));
expect(alerts(dialog)).toEqual([
"Suite name is required",
"Case Name is required",
"Add a system prompt or a user prompt",
"Expected Value is required",
]);
for (const label of [/^Suite Name/, /^Case Name/, /^User Prompt/, /^Expected Value/]) {
expect(field(dialog, label).getAttribute("aria-invalid")).toBe("true");
}
// The optional fields are not flagged.
expect(field(dialog, /^Model \(optional\)/).getAttribute("aria-invalid")).toBeNull();
expect(field(dialog, /^System Prompt/).getAttribute("aria-invalid")).toBeNull();
await waitFor(() => expect(document.activeElement).toBe(field(dialog, /^Suite Name/)));
expect(saves()).toHaveLength(0);
// The field carries the message: no toast repeats it off-screen.
expect(toasts()).toEqual([]);
});
it("clears each message as its field is filled and focuses the next invalid field", async () => {
const dialog = await openBuilder();
fireEvent.click(button(dialog, "Create Suite"));
type(field(dialog, /^Suite Name/), "QA-test-suite");
type(field(dialog, /^Case Name/), "greets");
expect(alerts(dialog)).toEqual([
"Add a system prompt or a user prompt",
"Expected Value is required",
]);
fireEvent.click(button(dialog, "Create Suite"));
await waitFor(() => expect(document.activeElement).toBe(field(dialog, /^User Prompt/)));
// A system prompt alone satisfies "at least one message".
type(field(dialog, /^System Prompt/), "Be brief");
expect(alerts(dialog)).toEqual(["Expected Value is required"]);
});
});

View File

@@ -0,0 +1,173 @@
// @vitest-environment jsdom
// KEA-9386: the Custom webhook step says what is wrong, where it is wrong. Next stays enabled;
// clicking it with an empty, malformed or over-long value flags the field inline and focuses
// it, and nothing is sent to the API. Limits mirror the route's Zod schema.
import React from "react";
import { cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
import { createTranslator } from "use-intl/core";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { AddWebhookWizard } from "@/app/(dashboard)/dashboard/webhooks/components/AddWebhookWizard";
import en from "@/i18n/messages/en.json";
import { WEBHOOK_SECRET_MAX_LENGTH, WEBHOOK_URL_MAX_LENGTH } from "@/shared/constants/formLimits";
// The page hands the wizard its `webhooks` translator: use the real English catalog.
const translate = createTranslator({
locale: "en",
messages: en as never,
namespace: "webhooks",
onError: () => {},
getMessageFallback: ({ key }) => key,
});
const t = (key: string, opts?: Record<string, unknown>) =>
String(translate(key as never, opts as never));
const reply = (data: unknown, status = 200) => new Response(JSON.stringify(data), { status });
const fetchMock = vi.fn<typeof fetch>();
const callsTo = (path: string, method: string) =>
fetchMock.mock.calls.filter(([url, init]) => url === path && init?.method === method);
const creates = () => callsTo("/api/webhooks", "POST");
beforeEach(() => {
fetchMock.mockReset();
fetchMock.mockImplementation(async (url) =>
url === "/api/webhooks/validate-url"
? reply({ valid: true })
: reply({ webhook: { id: "wh-1" } }, 201)
);
vi.stubGlobal("fetch", fetchMock);
});
afterEach(() => {
cleanup();
vi.unstubAllGlobals();
});
// Plain DOM lookups: role queries walk the whole accessibility tree on every call.
const button = (text: string) =>
Array.from(document.querySelectorAll("button")).find((node) =>
node.textContent?.includes(text)
) as HTMLButtonElement | undefined;
const next = () => button("Next") as HTMLButtonElement;
const urlField = () => screen.getByLabelText(/^Endpoint URL/) as HTMLInputElement;
const secretField = () => screen.getByLabelText(/^Secret Key/) as HTMLInputElement;
const type = (field: HTMLInputElement, value: string) =>
fireEvent.change(field, { target: { value } });
const alerts = () =>
Array.from(document.querySelectorAll('[role="alert"]')).map((node) => node.textContent);
const urlOfLength = (length: number) => {
const prefix = "https://example.com/hook/";
return prefix + "a".repeat(length - prefix.length);
};
/** Opens the wizard and walks to the Custom webhook step. */
async function openCustomStep() {
render(<AddWebhookWizard isOpen onClose={vi.fn()} onCreated={vi.fn()} t={t} />);
// The first screen is the integration picker — step 1 — and it has nothing to go back to.
await screen.findByText("Add Webhook — Select Integration");
expect(button("Back")).toBeUndefined();
await waitFor(() => expect(next().disabled).toBe(false));
fireEvent.click(button("Custom Webhook")!);
fireEvent.click(next());
await screen.findByLabelText(/^Endpoint URL/);
}
describe("Custom webhook step validation (KEA-9386)", () => {
it("explains an empty Endpoint URL inline when Next is clicked, instead of a silent disabled button", async () => {
await openCustomStep();
expect(next().disabled).toBe(false);
// Nothing is flagged before the user tries to continue.
expect(alerts()).toEqual([]);
fireEvent.click(next());
expect(alerts()).toEqual(["Endpoint URL is required"]);
expect(urlField().getAttribute("aria-invalid")).toBe("true");
await waitFor(() => expect(document.activeElement).toBe(urlField()));
expect(creates()).toHaveLength(0);
});
it.each(["not-a-url", "<script>x</script> %% 😀", "ftp://example.com/hook"])(
"says at once that %j is not an http(s) URL, without asking the server",
async (value) => {
await openCustomStep();
type(urlField(), value);
expect(alerts()).toEqual(["Enter a valid URL that starts with https:// or http://"]);
expect(urlField().getAttribute("aria-invalid")).toBe("true");
fireEvent.click(next());
await waitFor(() => expect(document.activeElement).toBe(urlField()));
expect(creates()).toHaveLength(0);
expect(callsTo("/api/webhooks/validate-url", "POST")).toHaveLength(0);
}
);
it("accepts the ticket's 340-character URL and flags one over the server limit", async () => {
await openCustomStep();
type(urlField(), urlOfLength(340));
expect(alerts()).toEqual([]);
type(urlField(), urlOfLength(WEBHOOK_URL_MAX_LENGTH));
expect(alerts()).toEqual([]);
type(urlField(), urlOfLength(WEBHOOK_URL_MAX_LENGTH + 1));
expect(alerts()).toEqual(["Endpoint URL can have at most 2,000 characters (currently 2,001)"]);
// Nothing is cut: the user sees what they pasted.
expect(urlField().value).toHaveLength(WEBHOOK_URL_MAX_LENGTH + 1);
fireEvent.click(next());
expect(creates()).toHaveLength(0);
});
it("keeps a 400-character Secret Key whole, from the field to the request", async () => {
await openCustomStep();
const secret = "s".repeat(400);
type(urlField(), "https://example.com/hook");
type(secretField(), secret);
expect(secretField().hasAttribute("maxlength")).toBe(false);
expect(secretField().value).toHaveLength(400);
expect(alerts()).toEqual([]);
fireEvent.click(next());
await waitFor(() => expect(creates()).toHaveLength(1));
expect(JSON.parse(String(creates()[0][1]?.body))).toEqual({
kind: "custom",
url: "https://example.com/hook",
secret,
});
});
it("flags a Secret Key over the server limit instead of cutting or sending it", async () => {
await openCustomStep();
type(urlField(), "https://example.com/hook");
type(secretField(), "s".repeat(WEBHOOK_SECRET_MAX_LENGTH + 1));
expect(alerts()).toEqual(["Secret Key can have at most 500 characters (currently 501)"]);
expect(secretField().value).toHaveLength(WEBHOOK_SECRET_MAX_LENGTH + 1);
expect(secretField().getAttribute("aria-invalid")).toBe("true");
fireEvent.click(next());
await waitFor(() => expect(document.activeElement).toBe(secretField()));
expect(creates()).toHaveLength(0);
});
it("shows the server's reason, not [object Object], when the API refuses the URL", async () => {
fetchMock.mockImplementation(async (url) =>
url === "/api/webhooks/validate-url"
? reply({ valid: true })
: reply(
{
error: {
message: "Invalid request",
details: [{ field: "url", message: "Blocked private or local provider URL" }],
},
},
400
)
);
await openCustomStep();
type(urlField(), "https://internal.example/hook");
fireEvent.click(next());
await screen.findByText("Blocked private or local provider URL");
expect(document.body.textContent).not.toContain("[object Object]");
});
});