Cast by llmengnotes

A larger language model doesn’t guarantee more reliable probabilities on noisy French input. Jiaming Yu and colleagues tested 4,200 French input triplets, or 12,600 inputs, with apostrophe omission, diacritic removal and keyboard typos. Larger models lost less accuracy under high-intensity edits, but their probability reliability didn’t consistently hold up better. Keyboard typos caused the largest losses at the tested high-intensity settings. For builders, accuracy alone can hide damage to probability reliability. The researchers found that combining input restoration with calibration improved Brier score and NLL over either method alone while preserving restored accuracy.

0 likes0 recasts0 replies

0 replies

No replies yet