[msluszniak]

Description

• Adds LFM 3 1.7B, LFM 3 VL 2B and LFM 3 Audio 2B (XNNPACK 8da4w, MLX int4/int8), 8K context.
• The multimodal runner accepts an audio_encoder model (LLM_AUDIO_SPEC) in place of vision_encoder.
• nlp app: LFM 3 entries, LFM 3 tool-call parser, hides the draft LFM 3 writes before </think> with thinking off.

Introduces a breaking change?

• Yes
• No

Type of change

• Bug fix (change which fixes an issue)
• New feature (change which adds functionality)
• Documentation update (improves or adds clarity to existing documentation)
• Other (chores, tests, code style improvements etc.)

Tested on

• iOS
• Android

Testing instructions

• nlp app, LLM screen: LFM 3 1.7B (chat, tool calling), LFM 3 VL 2B (image), LFM 3 Audio 2B (attach a WAV URL).

Screenshots

Related issues

Checklist

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have updated the documentation accordingly
• My changes generate no new warnings

Additional notes

• Vulkan variants follow separately; they need an ExecuTorch Vulkan SDPA fix in the libs.

[barhanc]

You can use the combineStats helper here as in gemma example.

[barhanc]

Suggested change

	const chunkSamples = Math.min(numSamples - offset, chunkStride);
	// Right-pad to window + k * hop samples, as the feature extractor does.
	const padded =
	chunkSamples <= window ? window : window + Math.ceil((chunkSamples - window) / hop) * hop;
	const numFrames = 1 + (padded - window) / hop;
	const numAudioTokens = Math.ceil(numFrames / framesPerToken);
	const chunkSamples = Math.min(numSamples - offset, chunkStride);
	// Right-pad to window + k * hop samples, as the feature extractor does.
	const k = Math.ceil((chunkSamples - window) / hop);
	const padded = chunkSamples <= window ? window : window + k * hop;
	const numFrames = 1 + (padded - window) / hop;
	const numAudioTokens = Math.ceil(numFrames / framesPerToken);

[barhanc]

Maybe something like

  const specModalities = {
    vision: ['image'],
    audio: ['audio'],
    omni: ['image', 'audio'],
  }[variant];

  if (
    modalities?.length !== specModalities.length ||
    modalities.some((m) => !specModalities.includes(m))
  ) {
    throw RnExecuTorchError(
      'INVALID_ARGUMENT',
      `Multimodal model requires [${specModalities}] modalities.`
    );
  }

// ...

  let imgShape: readonly [1, 3, number, number];
  let numVisualTokens: number;
  if (modalities.includes('image')) {
    imgShape = [1, 3, ...dims.constant('imgH', 'imgW')];
    numVisualTokens = dim('visualTokens', 'constant');
  }

  let audioSamples: Range;
  let framesPerToken: number;
  if (modalities.includes('audio')) {
    [audioSamples] = dims.range('audioSamples');
    [framesPerToken] = model.execute('get_audio_frames_per_token', [], []) as [number];
  }

with checks changed to

    if (!modalities.includes('image')) {
      throw RnExecuTorchError('INVALID_ARGUMENT', 'prefill: This model has no vision encoder.');
    }

would allow better extensibility also for omni model.

[barhanc]

I tested the new models in example app and they work great. I'm super impressed by the audio model---the prefill and decode speed and accuracy of the transcription are incredible (especially compared to Gemma) 🤯. I don't have iOS device to test the MLX versions so that will have to wait for tomorrow.

[msluszniak]

@kfaracik could you check the MLX and XNNPACK variants of these models on iOS since you have told me you have been adding these models.