03 / Liquid Router
Liquid Router Review & Experiments
Review Maxime's TF-IDF router baseline, then test whether inexpensive LFM token embeddings and static sentence vectors improve its quality-cost tradeoff.
Baseline
Maxime's router uses character TF-IDF features with per-model logistic heads and a global-regret policy. It is the strongest lightweight baseline available in the accessible five-model evaluation data.
This is not apples-to-apples with Maxime's published six-model release. These experiments use router-eval-budget-588: 588 prompts, five models, and 8,820 evidence rows.
Experiment
Test whether LFM token embeddings add routing signal beyond character TF-IDF while keeping inference cheap enough for a router.
| Variant | Features | Status |
|---|---|---|
| Baseline | Character TF-IDF + per-model logistic heads | Existing winner |
| Token pool | LFM token embedding mean, max, and IDF-weighted mean | Evaluated |
| TF-IDF + token pool | Sparse TF-IDF plus pooled LFM token embeddings | Evaluated |
| Full feature mix | TF-IDF, pooled embeddings, and length/capability heuristics | Evaluated |
Protocol
Use the existing group-disjoint five-fold router leaderboard. Every row uses the same global-regret policy with a predeclared 0.03 quality gap. The run sweeps C = 0.03, 0.1, 0.3, 0.5, 1.0 on seed 0 with 200 bootstrap samples; C=0.5 matches the baseline.
Token-Pooling Results
| Router | Quality | Cost | Savings @ bar | Finding |
|---|---|---|---|---|
| TF-IDF + LFM token pool, C=0.03 | 0.6635 | 0.010604 | +16% | Best tuned seed-0 result; misses strict bar. |
| TF-IDF global-regret, C=0.5 | 0.6642 | 0.010888 | +14% | Maxime's baseline; misses strict bar. |
| LFM token pool only, C=0.1 | 0.6509 | 0.011623 | +8% | Best tuned token-pool-only result. |
| TF-IDF + LFM token pool, C=0.5 | 0.6595 | 0.012172 | +4% | Fixed-C comparison with baseline. |
| TF-IDF + token pool + heuristics, C=1.0 | 0.6533 | 0.012593 | +1% | Best tuned full-feature result. |
| TF-IDF + token pool + heuristics, C=0.5 | 0.6590 | 0.012856 | -1% | Fixed-C comparison with baseline. |
| LFM token pool only, C=0.5 | 0.6555 | 0.013167 | -4% | Fixed-C comparison with baseline. |
After controlling the routing policy, TF-IDF plus frozen token pooling at C=0.03 slightly improves nominal savings over TF-IDF alone. The gain is small, regularization-sensitive, and does not clear the strict statistical quality bar.
Static Sentence Vectors
Frozen fastText Common Crawl 300D and Sent2Vec Wikipedia-bigram 700D vectors were tested with per-model logistic heads, one shared hidden layer, and cosine-nearest model prototypes.
| Features / head | Quality | Cost | Nominal savings [90% lo] | Result |
|---|---|---|---|---|
| Sent2Vec 700D / nearest prototype | 0.6372 | 0.008023 | +37% [+21%] | Fixed point; misses bar |
| fastText 300D / nearest prototype | 0.6247 | 0.009829 | +23% [+13%] | Fixed point; misses bar |
| Sent2Vec 700D / MLP-128 | 0.6526 | 0.010566 | +16% [+7%] | Sensitivity; misses bar |
| Sent2Vec 700D / MLP-64 | 0.6574 | 0.010907 | +13% [+4%] | Primary; misses bar |
| fastText 300D / linear | 0.6507 | 0.011120 | +12% [+7%] | Primary; misses bar |
| Sent2Vec 700D / linear | 0.6570 | 0.011439 | +10% [-2%] | Primary; misses bar |
| Sent2Vec 700D / MLP-32 | 0.6566 | 0.011460 | +9% [-0%] | Sensitivity; misses bar |
| fastText 300D / MLP-64 | 0.6577 | 0.011605 | +8% [+4%] | Primary; misses bar |
| fastText 300D / MLP-32 | 0.6611 | 0.011915 | +6% [+1%] | Sensitivity; misses bar |
| fastText 300D / MLP-128 | 0.6574 | 0.012165 | +4% [-1%] | Sensitivity; misses bar |
Conclusion
TF-IDF remains the strongest lightweight baseline. Pooled LFM embeddings produce only a small, regularization-sensitive nominal gain. Static sentence vectors and nearest prototypes reduce cost by sacrificing too much quality.
CPU-only cluster timing: warm batched vectorization of all 588 prompts took 3.9 seconds for fastText and 5.1 seconds for Sent2Vec; each five-fold classifier leaderboard took about 15-16 seconds. Asset download and setup dominated total runtime.