The assignment was straightforward on paper. Take a CSV of 10,000 e-commerce orders. Train a model to predict whether an order will be delayed. Put it behind an API. Build a portal. Add observability. Submit a live URL.
Eight to ten hours of work, the brief said. That estimate was right, but not for the reasons I expected.
The architecture
Five layers, one direction of data flow.
Azure SQL holds the raw order history. A view called ml.DelayTraining exposes only the nine training columns, nothing else. Training code reads from the view, not the table. This matters: if you read the table, you might accidentally include columns you do not want the model to see, or pull identifiers that have no predictive value.
A Python script exports the view to CSV and registers it as a data asset in Azure ML. The training job runs there against a compute cluster, picks the best model, and saves a joblib artifact.
That artifact gets loaded by an Azure Functions API. The browser never talks to the model directly. It calls /api/predict, which validates the input, calls the model, writes a log document to Cosmos DB, and returns a result. The browser only knows the function endpoint URL. The model scoring key and the Cosmos key live in the function app's application settings, not in code or in the repository.
The front end is a Static Web App: a form, a result card, a health panel, a table of recent requests.
Cosmos DB collects two things: one document per training run (in model_runs) and one document per prediction request (in requests). The requests container has a 30-day TTL. You do not want to pay to store every test prediction indefinitely.
The metric that almost fooled me
The dataset has 18.1% delayed orders. The remaining 81.9% are on time.
The first model I trained without any class weighting scored 81.9% accuracy. It also caught zero delayed orders. The confusion matrix was a single block in the top-left corner: 1,639 true negatives, zero true positives, zero false negatives, 361 false positives. The model had learned to say "on time" for everything.
This is the class imbalance trap. Accuracy rewards the majority class. A model that learns nothing scores higher on accuracy than a model that actually finds the rare event you care about.
The fix was compute_sample_weight(class_weight="balanced"), which increases the penalty for missing a delayed order during training. After balancing, accuracy dropped to 61.8%, AUC rose to 0.589, and recall on the delayed class went from 0% to 46.3%. The model now finds roughly half of actual delays. That is not great, but it is not zero.
AUC is the right metric here. It measures whether the model ranks delayed orders above on-time orders across all possible thresholds. An AUC of 0.589 means the model is better than random, but not dramatically so. Real improvement would require real data with more signal.
What I learned about partition keys
Cosmos DB requires you to choose a partition key when you create a container. The wrong choice costs you money and throughput later. You cannot change it after creation.
For model_runs, I used /modelName. Reads are almost always scoped to one model: "give me all runs for delay-classifier-v1." The data stays in one partition, the query is cheap, and the container will never have enough data for hot partitions to matter.
For requests, I used /userId. Each user's trace lives in one partition. Their recent-requests query reads only their partition, which costs one RU per document instead of a cross-partition fan-out. TTL keeps each partition from growing indefinitely.
Why not /carrier? Four carriers handling millions of requests. Delhivery handles more volume than the others. You end up with one hot partition taking most of the load while three others sit idle. Cosmos scales by partition, so a hot partition is a throughput ceiling.
Why not a constant key like /type = "request"? All data in one partition. Eventually you hit the 20GB per-partition limit or the throughput ceiling. Every query becomes a single-partition scan. This is the second-most-common Cosmos mistake after using the document ID as the partition key.
Observability before shipping
A model nobody can measure is a demo, not a service. The /api/metrics endpoint reads the last 100 request documents from Cosmos and computes p50, p95, slowest latency, and error rate over the last 15 minutes. The portal polls it every 30 seconds.
During the load test (100 requests at concurrency 5), p50 was around 380ms and p95 was around 620ms. The first request after idle took ~1,400ms — that is the Function cold start. After that, the container stays warm and latency is consistent.
The failure test was useful. Breaking the ML endpoint key and sending 10 requests confirmed that the error rate in the health panel rose above zero and the failed requests showed status: error in the requests table. Restoring the key brought the error rate back down. This matters for an FDE: if you cannot show a client that your failure detection works, you cannot claim the system is observable.
What an FDE takes from this
Three things.
First, the gap between "training a model" and "shipping a model" is mostly about the infrastructure around it, not the model itself. The training job was a few hundred lines. The API, the portal, the observability layer, the secret management, the partition key decisions: that is where the hours went.
Second, metrics lie if you let them. AUC of 0.589 sounds unimpressive. But comparing it to the unweighted baseline (AUC 0.559, recall 0%) makes it clear that class balancing produced a genuinely different model, not just a slightly tuned version of the same thing. Showing both models side by side is what honest evaluation looks like.
Third, security is not a layer you add at the end. The decision to never put the ML key in front-end code, to validate inputs on the server even when the browser already validates them, to exclude local.settings.json from git: those decisions are made in the design phase. An FDE working on a client's cloud stack is responsible for not leaving credentials in the repository. That standard applies from the first commit.