Available on AWS Marketplace · Version 1.0.0
Model and evaluation card
Version 1.0.0 packages original BAAI/bge-reranker-v2-m3, revision
953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e, weights SHA-256
d9e3e081faff1eefb84019509b2f5558fd74c1a05a2c7db22f74174fcedb5286.
No Software Sushi adaptation or additional training is included. Original
weights load through FP16 into FP32 masters; CUDA scoring uses BF16 autocast
and eager attention. Scores are raw passage-relevance logits.
Human-judged public diagnostics
The frozen comparison preserves original human relevance grades and complete
labeled candidate pools. Its 1,517 original queries contain 15,845 pairs. The
bounded API admits 1,343 queries and 13,699 pairs. Results below are from the
original-model Thor diagnostic, not an AWS performance measurement.
| Source |
Admitted queries / full cohort |
BM25 nDCG@5 |
Original BGE nDCG@5 |
| MIRACL English dev |
713 /799 |
0.45099 |
0.82882 |
| MIRACL Spanish dev |
606 /648 |
0.60009 |
0.82679 |
| Quati Portuguese human supplement |
24 /24 |
0.56618 |
0.73452 |
| NormasTCU Portuguese legal full documents |
0 /46 |
Not admitted |
Not admitted |
MIRACL uses binary human grades. Quati uses the median of three native 0..3
ratings and also reports each-rater sensitivity. Its 24-query slice is small;
it does not establish broad Portuguese or legal-domain performance. NormasTCU's
complete documents exceed the bounded API; they are retained as unsupported,
with zero contribution to full-cohort results, instead of being shortened or
dropped. The 86 excluded English and 42 excluded Spanish pools are likewise
reported separately in the full-cohort results.
nDCG@5 uses gain 2^grade-1 with one contribution per original query. It
measures ordering within these supplied judged pools, not corpus-wide retrieval
recall, generated-answer factuality, answer completeness or customer accuracy.
These public upstream sources are not sealed, unseen model-provider holdouts.
No target-model tuning or threshold fitting used this native comparison. These
native benchmarks cover monolingual supplied-pool relevance; independent native
cross-language and customer-workflow performance remain unmeasured.
The original model ranked a nonzero-relevant passage first in 576/713 English,
500/606 Spanish and 23/24 Portuguese admitted queries. Nonzero relevance does
not mean complete answer: the maximum native Portuguese grade ranked first in
13/24. Avoid converting these figures into a universal accuracy promise.
Four checkpoints were compared on the same pools. The original was stronger
than each tested adaptation in English and Spanish; the small Portuguese slice
did not establish a decisive adaptation advantage. Earlier complete-procedure
experiments failed at least one unchanged target and remain failed. API 1.0
is a separately scoped original-model relevance integration, not a relabeling of
those targets as successes.
Additional generated stress data contains related fictional families and is
used for development and demonstrations. It is not independent human customer
acceptance. A reserved Luna set remains target-unscored. Native customer-workflow
acceptance has not been performed.
Operational boundaries
Results can be wrong or incomplete in any language. The component has no
missing-answer detector, safety moderation, access-control decision, freshness
check or verified-answer generator. It supports bounded passage workflows;
full legal documents and arbitrary long-input retrieval are outside its scope.
Use upstream authorization and document version checks, inspect the original
source and evaluate the final application on representative authorized data.
Actual AWS real-time qualification
The exact original-model image completed all 1,343 admitted requests and all 7
contract rejection probes on one ml.g5.xlarge/A10G, with network isolation and
no data capture. AWS nDCG@5 was 0.82858008 English, 0.82691842 Spanish and
0.73452309 Portuguese. Complete-order identity with Thor was 1,183/1,343;
minor backend/numerical reorderings preserved native relevance within the
pre-frozen 0.01-per-dataset portability tolerance. Scores are not bit-identical
across these frameworks and GPUs.
Sequential client round-trip p50 was 136 ms and p95 was 205 ms from the Mac,
after startup. This is not maximum concurrency, throughput, a latency SLA or
a customer-workload performance guarantee. Batch qualification is reported below.
Actual AWS batch qualification
AWS completed Marketplace model-package certification for the same immutable
image using three fictional EN/ES/PT inputs. A separate, subsequent native run
on ml.g6e.xlarge/L40S produced all 1,343 expected outputs. Its nDCG@5 was
0.82836101 English, 0.82764245 Spanish and 0.73437183 Portuguese. Every dataset
passed the pre-frozen maximum 0.01 regression tolerance against Thor. The full
1,517-query cohort and all 174 unsupported pools remain in the report.
This run used SingleRecord, one concurrent transform, the explicit AWS driver
535 batch AMI and the documented compatibility environment setting. Actual
transform execution lasted about 170 seconds after provisioning; startup and
billing include additional time. Certification validates packaging behavior;
it is not a public Marketplace listing or independent customer acceptance.