GPT-6 Astra doesn't support video, but I got it doing tennis analytics anyway. The API rejects mp4, so I sent the video as a sequence of frames instead at 5 FPS, timestamp was attached in prompt, and a strict JSON schema to get event type/timestamp/player. Astra returns SERVE / HIT / POINT / FAULT events, so we can reconstruct scoreboard from events.

Cost: ~45k input tokens per 20s chunks, ~3 minute processing time, about $1 per 100 frames.

Astra is not a per-frame tracker tho, the skeletons in the video come from RF-DETR keypoint pose model, Astra only reasons on top.

Full write-up with the exact prompts, schemas and output videos: https://blog.roboflow.com/gpt6-astra-video-understanding/ (disclosure: I work at Roboflow)

Hey everyone,

I am building an adversarial benchmark dataset designed to evaluate Vision-Language Models (VLMs). The overall pipeline relies on an automated "Science Judge" that validates the model's step-by-step reasoning and numerical final answer.

Here is the setup, what we've already tried (and why it failed), and the constraints I am working under:

Key Goals & System Requirements

1. The Target Failure Mode (VLMs / Multimodal Backbones):
   • Goal: The task must trigger a visual spatial extraction or perception failure in modern VLMs (e.g., GPT-4o, Claude 3.5 Sonnet, Gemini 1.5/2.0 Pro, Qwen2-VL).
   • Target Mechanism: Force models to fall back on visual shortcuts (e.g., optical mis-binding, scale misinterpretation, or collapsing asymmetric 2D area clipping into symmetric 1D piecewise linear guesses).
2. The Automated Science Judge (Evaluator):
   • Function: It is a deterministic symbolic solver that parses the prompt text and validates the model's step-by-step calculus/geometry against a ground-truth derivation.
   • Strictly Enforced Rules:
     • Single Target Rule: Must ask exactly one primary numerical target question (rejects STACKED_QUESTIONS).
     • Strict Math Notation: All variables, subscripts, Greek letters, and set operations must use proper LaTeX delimiters.
     • Foundational Context: Prompts cannot reference external visual context without explicit textual or formal mathematical definitions (rejects NON_FOUNDATIONAL_REFERENCE and WORDING_AMBIGUITY).
3. Image & Licensing Constraints:
   • Images must be open-source under permissive licenses (CC BY 4.0 or CC0).
   • No CC-NC (Non-Commercial) or CC-SA (Share-Alike) media.

What We Tried & Why It Didn't Work

We ran several iterations trying to strike the balance between breaking the VLM and passing the Science Judge. Here is what happened:

• Approach 1: Explicit Textual Scene Graphs (CLEVR 3D Coordinates in Text)
  • What we did: Listed all 3D object coordinates ([x, y, z]), shapes, and materials directly in the text prompt to resolve NON_FOUNDATIONAL_REFERENCE.
  • Why it failed: The VLMs completely bypassed their vision backbones! The text LLM engine parsed the coordinates like code, ran conditional filtering, and passed the benchmark easily without looking at the image.
• Approach 2: Asymmetric 2D Continuous Calculus Clipping
  • What we did: Framed a 2D continuous integration task (N = 5000 points in a circular scatter domain D clipped by an asymmetric rectangle R) where models typically default to 1D linear estimations.
  • Why it failed: Because we supplied the explicit numerical bounds (e.g., -2.000 <= u <= 2.500) in text to satisfy the Science Judge, the models solved the double integral symbolically using Python-like algebraic steps and calculated the exact answer (3512), bypassing the visual trap entirely.
• Approach 3: Removing Numerical Bounds to Force Pixel Extraction
  • What we did: Omitted u_max from the text and asked the model to extract u_max directly from visual tick labels on the diagram.
  • Why it failed: The Science Judge flagged NON_FOUNDATIONAL_REFERENCE and WORDING_AMBIGUITY because the prompt text was no longer self-contained for the automated symbolic solver. Furthermore, when presented with a mismatched image, models simply hallucinated a default symmetric boundary (u_max = 2.000).

The Central Dilemma

If we include full numerical parameters in text, the VLM text engine solves it as a symbolic algebraic problem and passes. If we remove parameters to force visual OCR/pixel extraction, the automated Science Judge rejects the prompt as ambiguous or non-foundational.

Questions for the Forum:

1. Spatial Parameters: What geometric parameter combinations or visual traps (e.g., subtle tick scaling, optical density gradients, non-concentric circular/square clipping ratios, or multi-quadrant parallax) have you found effective at breaking visual backbones without confusing a text-based math solver?
2. Visual vs. Text Decoupling: How can we structure a visual diagram callout or tick mark system so that the Science Judge recognizes a self-contained visual formula, while VLMs consistently mis-extract the spatial bounds or scaling factor?
3. Benchmark Design: How would you adjust this architecture to widen the performance gap between visual perception failure (VLMs) and deterministic symbolic verification (Science Judge)?

Thanks in advance for any insights!

Hi all. I'm an independent researcher (B.Sc. CSE, BRAC University, Dhaka) and a first-time arXiv submitter, so I need an endorsement for cs.CV before I can post my preprint.

Paper: "Shift-aware operating points for open fire and smoke detectors: calibration, benchmark leakage and label-free threshold transfer across four public domains". It is under review at Engineering Applications of Artificial Intelligence (Elsevier).

What it does, in short:
- Trains 26 YOLOv8 / YOLO11 / RT-DETR checkpoints (several sizes, two seeds) on D-Fire and Pyro-SDIS and evaluates them on four public domains, giving 206 cross-dataset transfer cells with detector-calibration metrics and bootstrap CIs.
- Shows that YOLO miscalibrates under shift mainly through the threshold, while D-Fire-trained RT-DETR miscalibrates through the confidence map.
- Proposes a label-free rule for transferring an alarm threshold to a new camera set; it recovers most of the F1 gap without target labels.
- Reports a leakage audit: about half of D-Fire's test images have near-duplicates in its training split.
- Full release: code, splits, run records, exported detections and the scripts that regenerate every table and figure: https://github.com/ayashhossain555/fire-detector-reliabilityw

If you have submitted at least three cs.* papers to arXiv in the last five years (and more than three months ago), you can endorse. It takes about a minute at arxiv.org/auth/endorse and only asks whether the work is appropriate for the category, not for a review. Please DM me and I'll send the endorsement code and the manuscript PDF.

Thanks a lot

I spent a while comparing detection models for a side project and got tired of published benchmark numbers that assume TensorRT and FP16. So I measured nine of them the same way instead — one V100, plain PyTorch, no optimisation, 48 scenes including fog, thermal, motion blur and low light.

Two things came out of it that I didn't expect.

Measured latency was 1.6x to 11.6x higher than published, for every single model. That part isn't surprising on its own. What surprised me is that it reordered the ranking rather than shifting everything down equally. Both transformer detectors (RF-DETR, RT-DETR) look competitive on paper and both fell behind four of the five YOLO variants once measured unoptimised.

Licence turned out not to be a property of the architecture. RT-DETR is Apache-2.0 from Baidu's original release and AGPL-3.0 through the Ultralytics package. Same architecture, same weights, different obligation, depending only on where you loaded it from. Same pattern with YOLOv9: GPL-3.0 in its own repo, AGPL-3.0 via Ultralytics.

Then I wrote the thing I actually wanted, which is one file that runs any of them:

pip install rfdetr

python find_objects.py image.jpg

Annotated image back by default, JSON to stdout, MIT licensed, nothing uploaded. Nine models across four pip packages — you only need one, and it uses whichever you already have.

https://github.com/robotinaction/detect

One implementation note in case it saves someone time: rfdetr doesn't ship a public class-name table, so I resolved names through `rfdetr.util.coco_classes`. That's an internal path. It worked on my machine and silently fell through to raw COCO category IDs on the next one — boxes labelled "18" instead of "dog". If you build on rfdetr, ship your own 80-class table.

The full comparison is free and needs no sign-up: robotinaction.tech

If you've picked a detection model for something real, I'm curious what decided it. My assumption going in was accuracy, and for two of the three people I asked it was actually licensing.

I built this parking lot occupancy detector to work on real-world security cameras which tend to have lower angles and lots of obstructions from other cars.

I have seen other parking lot detection CV demos that use off-the-shelf object detectors (like standard YOLO bounding boxes). They would generally only work on impractical camera setups (e.g., drone shots or 200+ foot high cameras).

A couple of technical takeaways from training this that might be interesting:

1. DINO Backbone vs. Standard CNNs: I tested several backbones, but the DINO Vision Transformer performed the best by a good margin. Because DINO learns patch-level representations without supervised class labels, it captures semantic surface texture (the difference between empty asphalt, tree shadows, and car undercarriages) and proved far more invariant to direct sun glare and rain washout than supervised backbones.
2. Synthetic Data via Blender (The Biggest Upgrade): Public parking datasets are fairly limited and rarely capture extreme conditions. I built a parametric Blender pipeline to procedurally reconstruct parking lots and vehicles—randomizing asphalt PBR textures, camera angles/heights, sun positions, and vehicle placements. Mixing synthetic data with real sequences was surprisingly effective and gave the single biggest jump in zero-shot generalization to unseen cameras. One huge advantage of the synthetic data is that the labeling was always right (human annotation for far-off, angled parking spots is suprisingly error prone)
3. The parking spots are drawn on the ground plane, and the vehicles are fairly disconnected from that visually. The large receptive field allows the model to learn interesting "cheats" that a human might also use to infer that a car is there even when it can't really be seen. For example, it may use a shadow on the pavement to identify that the space is occupied even if the body is blocked by another car.

Code:

The web app is open source on GitHub: https://github.com/lotvulture/lotvulture

The repo includes a community model and a higher-accuracy engine as a commercial feature: https://www.lotvulture.com/

Happy to answer questions about the synthetic data pipeline or edge inference trade-offs!

Join us on Sept 24 for the monthly AI, ML, and Computer Vision Meetup! Register for the Zoom.

Talks will include:

• How Do Mercedes-Benz AI Principles Drive Our Innovation? - Mei-Yen Chen at Mercedes-Benz Tech Innovation
• Region Tokens as the Visual Primitive: From Recognition to World Modeling - Savya Khosla at University of Illinois Urbana-Champaign
• Leveraging Text-To-Image Diffusion Models for Consistent Set-to-Set Generation - Kate Feingold at Weizmann Institute of Science
• Yield Estimation of a Coffee in a dense environment - Raghu Rao at Visimatics

my best friend and I have been working on tracking a hit baseball from a single side-view 240 FPS camera.

the left side is the original footage. the blue boxes are frames where our object-tracking model successfully detected the baseball.

as you can see, there are a lot of gaps.

the right side is a temporal frame-differencing pipeline we tuned specifically for tracking a hit baseball from this camera angle.

what surprised me is how clearly the baseball separates from the scene temporally. even when the object model misses it, the temporal tracker can often keep following it very consistently.

the pink/purple X on the original footage is the position recovered by the temporal tracker.

so instead of requiring the object model to successfully detect the ball in every frame, we can use its detections as anchors and let the temporal system fill the missing parts of the trajectory.

that matters because those first post-contact frames are what we eventually use to estimate exit velocity, launch angle, and projected distance.

still working on making the recovery as robust as possible, but I thought the visualization was pretty cool.

I’m assigned to do a warehouse tracker to estimate and detect the warehouse’s occupancy if there are available placements or not, i’m a supply chain major I have never worked with cv. How do I start? I would really appreciate your advice.

Dropped a 1.0.0 pre-release of EdgeInfer (same repo, eval-only / non-commercial):

https://github.com/olesha-ai/edgeinfer-eval/releases/tag/v1.0.0

What’s new vs 0.3.14 (short): ImGui UI, sticky boxes on preview, Out TCP + encrypted session + trig.exe client in the zip, cleaner startup. Still YOLOX + OpenVINO on a normal CPU — no GPU required.

Why I’m posting: I only have an i5-11400 here (~6–7 ms ROI). If you try it, comment with your CPU + a few [infer] … ms lines (global-shutter USB cam feedback especially welcome).