Software SushiMultilingual Relevance Reranker

Available on AWS Marketplace · Version 1.0.0

Model and evaluation card

Version 1.0.0 packages original BAAI/bge-reranker-v2-m3, revision 953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e, weights SHA-256 d9e3e081faff1eefb84019509b2f5558fd74c1a05a2c7db22f74174fcedb5286. No Software Sushi adaptation or additional training is included. Original weights load through FP16 into FP32 masters; CUDA scoring uses BF16 autocast and eager attention. Scores are raw passage-relevance logits.

Human-judged public diagnostics

The frozen comparison preserves original human relevance grades and complete labeled candidate pools. Its 1,517 original queries contain 15,845 pairs. The bounded API admits 1,343 queries and 13,699 pairs. Results below are from the original-model Thor diagnostic, not an AWS performance measurement.

Source Admitted queries / full cohort BM25 nDCG@5 Original BGE nDCG@5
MIRACL English dev 713 /799 0.45099 0.82882
MIRACL Spanish dev 606 /648 0.60009 0.82679
Quati Portuguese human supplement 24 /24 0.56618 0.73452
NormasTCU Portuguese legal full documents 0 /46 Not admitted Not admitted

MIRACL uses binary human grades. Quati uses the median of three native 0..3 ratings and also reports each-rater sensitivity. Its 24-query slice is small; it does not establish broad Portuguese or legal-domain performance. NormasTCU's complete documents exceed the bounded API; they are retained as unsupported, with zero contribution to full-cohort results, instead of being shortened or dropped. The 86 excluded English and 42 excluded Spanish pools are likewise reported separately in the full-cohort results.

nDCG@5 uses gain 2^grade-1 with one contribution per original query. It measures ordering within these supplied judged pools, not corpus-wide retrieval recall, generated-answer factuality, answer completeness or customer accuracy. These public upstream sources are not sealed, unseen model-provider holdouts. No target-model tuning or threshold fitting used this native comparison. These native benchmarks cover monolingual supplied-pool relevance; independent native cross-language and customer-workflow performance remain unmeasured.

The original model ranked a nonzero-relevant passage first in 576/713 English, 500/606 Spanish and 23/24 Portuguese admitted queries. Nonzero relevance does not mean complete answer: the maximum native Portuguese grade ranked first in 13/24. Avoid converting these figures into a universal accuracy promise.

Four checkpoints were compared on the same pools. The original was stronger than each tested adaptation in English and Spanish; the small Portuguese slice did not establish a decisive adaptation advantage. Earlier complete-procedure experiments failed at least one unchanged target and remain failed. API 1.0 is a separately scoped original-model relevance integration, not a relabeling of those targets as successes.

Additional generated stress data contains related fictional families and is used for development and demonstrations. It is not independent human customer acceptance. A reserved Luna set remains target-unscored. Native customer-workflow acceptance has not been performed.

Operational boundaries

Results can be wrong or incomplete in any language. The component has no missing-answer detector, safety moderation, access-control decision, freshness check or verified-answer generator. It supports bounded passage workflows; full legal documents and arbitrary long-input retrieval are outside its scope. Use upstream authorization and document version checks, inspect the original source and evaluate the final application on representative authorized data.

Actual AWS real-time qualification

The exact original-model image completed all 1,343 admitted requests and all 7 contract rejection probes on one ml.g5.xlarge/A10G, with network isolation and no data capture. AWS nDCG@5 was 0.82858008 English, 0.82691842 Spanish and 0.73452309 Portuguese. Complete-order identity with Thor was 1,183/1,343; minor backend/numerical reorderings preserved native relevance within the pre-frozen 0.01-per-dataset portability tolerance. Scores are not bit-identical across these frameworks and GPUs.

Sequential client round-trip p50 was 136 ms and p95 was 205 ms from the Mac, after startup. This is not maximum concurrency, throughput, a latency SLA or a customer-workload performance guarantee. Batch qualification is reported below.

Actual AWS batch qualification

AWS completed Marketplace model-package certification for the same immutable image using three fictional EN/ES/PT inputs. A separate, subsequent native run on ml.g6e.xlarge/L40S produced all 1,343 expected outputs. Its nDCG@5 was 0.82836101 English, 0.82764245 Spanish and 0.73437183 Portuguese. Every dataset passed the pre-frozen maximum 0.01 regression tolerance against Thor. The full 1,517-query cohort and all 174 unsupported pools remain in the report.

This run used SingleRecord, one concurrent transform, the explicit AWS driver 535 batch AMI and the documented compatibility environment setting. Actual transform execution lasted about 170 seconds after provisioning; startup and billing include additional time. Certification validates packaging behavior; it is not a public Marketplace listing or independent customer acceptance.