Distinction between provider and model fields, dynamic model properties

[krassowski]

I tried to update GPT4All to fix the issues with the recent packages. Some old models are not supported any more, many new models are now available. GPT4All describes the supported local models, with a machine-readable JSON list like:

[
  {
    "order": "a",
    "md5sum": "48de9538c774188eb25a7e9ee024bbd3",
    "name": "Mistral OpenOrca",
    "filename": "mistral-7b-openorca.Q4_0.gguf",
    "filesize": "4108927744",
    "requires": "2.5.0",
    "ramrequired": "8",
    "parameters": "7 billion",
    "quant": "q4_0",
    "type": "Mistral",
    "systemPrompt": " ",
    "description": "<strong>Best overall fast chat model</strong><br><ul><li>Fast responses</li><li>Chat based model</li><li>Trained by Mistral AI<li>Finetuned on OpenOrca dataset curated via <a href=\"https://atlas.nomic.ai/\">Nomic Atlas</a><li>Licensed for commercial use</ul>",
    "url": "https://gpt4all.io/models/gguf/mistral-7b-openorca.Q4_0.gguf"
  },
  {
    "order": "b",
    "md5sum": "97463be739b50525df56d33b26b00852",
    "name": "Mistral Instruct",
    "filename": "mistral-7b-instruct-v0.1.Q4_0.gguf",
    "filesize": "4108916384",
    "requires": "2.5.0",
    "ramrequired": "8",
    "parameters": "7 billion",
    "quant": "q4_0",
    "type": "Mistral",
    "systemPrompt": " ",
    "description": "<strong>Best overall fast instruction following model</strong><br><ul><li>Fast responses</li><li>Trained by Mistral AI<li>Uncensored</li><li>Licensed for commercial use</li></ul>",
    "url": "https://gpt4all.io/models/gguf/mistral-7b-instruct-v0.1.Q4_0.gguf",
    "promptTemplate": "[INST] %1 [/INST]"
  },
  {
    "order": "c",
    "md5sum": "31cb6d527bd3bfb5e73c2e9dfbc75033",
    "name": "GPT4All Falcon",
    "filename": "gpt4all-falcon-q4_0.gguf",
    "filesize": "4210419040",
    "requires": "2.5.0",
    "ramrequired": "8",
    "parameters": "7 billion",
    "quant": "q4_0",
    "type": "Falcon",
    "systemPrompt": " ",
    "description": "<strong>Very fast model with good quality</strong><br><ul><li>Fastest responses</li><li>Instruction based</li><li>Trained by TII<li>Finetuned by Nomic AI<li>Licensed for commercial use</ul>",
    "url": "https://gpt4all.io/models/gguf/gpt4all-falcon-q4_0.gguf",
    "promptTemplate": "### Instruction:\n%1\n### Response:\n"
  },

I hoped to expose:

• filesize, ramrequired and description in the user interface in the model selector, to inform the user about the model requirements
• add a ModelPathField allowing user to either:
  • point to a path on disk (which we could check with md5sum), or
  • click a download button (which would have a nice progress bar) which would send a download request to the backend and then backend would change the model file path to the downloaded location
  • clear the manually set path so that the version on disk downloaded earlier could be used

Problem

The frontend thinks about the current fields as ModelField, defined here:

jupyter-ai/packages/jupyter-ai/src/components/settings/model-fields.tsx

Line 11 in 7c09863

export function ModelField(props: ModelFieldProps): JSX.Element {

but the fields are really defined per provider, not per model as seen in the BaseProvider model:

jupyter-ai/packages/jupyter-ai-magics/jupyter_ai_magics/providers.py

Lines 130 to 132 in 7c09863

	fields: ClassVar[List[Field]] = []
	"""User inputs expected by this provider when initializing it. Each `Field` `f`
	should be passed in the constructor as a keyword argument, keyed by `f.key`."""

and in the handler logic:

jupyter-ai/packages/jupyter-ai/src/handler.ts

Lines 162 to 171 in 7c09863

	export type ListProvidersEntry = {
	id: string;
	name: string;
	model_id_label?: string;
	models: string[];
	help?: string;
	auth_strategy: AuthStrategy;
	registry: boolean;
	fields: Field[];
	};

Further, there is no generic description field suitable for per-model information display, as the help is a class variable with a very narrow scope:

jupyter-ai/packages/jupyter-ai-magics/jupyter_ai_magics/providers.py

Lines 110 to 112 in 7c09863

	help: ClassVar[str] = None
	"""Text to display in lieu of a model list for a registry provider that does
	not provide a list of models."""

Proposed Solution

• Add per-model configuration fields
• Add per-model description fields
• Allow the model configuration to be dynamically updated if state changes on the server side

[krassowski]

Of note, the machine-readable JSON also contains ready-to-use model-specific prompts that were brought up before.