NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language

Danial Kamali, Parisa Kordjamshidi

Michigan State University

International Conference on Learning Representations (ICLR), 2026

Best Paper Award, MSLD 2026 Oral, SpaVLE Workshop @ NeurIPS 2025

TL;DR Vision-language models often fail at questions that combine several concepts. NePTune has an LLM translate the question into a short Python program. Inside the program, a declarative first-order-logic formula combines soft concept scores from a vision-language model, while imperative Python handles loops, conditionals and counting. NePTune works zero-shot, and because the logic operators are differentiable, the same programs can also be used to fine-tune the vision-language model.

From question to program to tree

This walkthrough follows the example in Figure 2 of the paper. The program is shown first as Python code and then as a tree. Green nodes are concept scores from the vision-language model (VLM), blue nodes are soft logic, and purple nodes are ordinary Python. The declarative part scores every detected box at once. The imperative part then loops over the boxes, branches on their scores and counts the answers. Press ▶ to play or pause, use the numbered steps to jump, and hover over a line of code or a node to see its counterpart.

Loading the example…

Tunable end to end

Every soft-logic operator in NePTune has a differentiable implementation, so a program also carries gradients. This walkthrough uses a referring expression from Ref-GTA, Expression 1 of Figure 3 in the paper, whose program is a single first-order-logic formula. After the forward pass, a binary cross-entropy loss on the program’s answer flows back through iota and the soft AND into the concept scores, and from there into the VLM. This is how the paper fine-tunes a VLM for Ref-GTA, a video game domain that is new to VLMs trained on natural images.

Loading the example…

Abstract

Modern Vision-Language Models (VLMs) have achieved impressive performance in various tasks, yet they often struggle with compositional reasoning, the ability to decompose and recombine concepts to solve novel problems. While neuro-symbolic approaches offer a promising direction, they are typically constrained by crisp logical execution or predefined predicates, which limit flexibility. In this work, we introduce NePTune, a neuro-symbolic framework that overcomes these limitations through a hybrid execution model that integrates the perception capabilities of foundation vision models with the compositional expressiveness of symbolic reasoning. NePTune dynamically translates natural language queries into executable Python programs that blend imperative control flow with soft logic operators capable of reasoning over VLM-generated uncertainty. Operating in a training-free manner, NePTune, with a modular design, decouples perception from reasoning, yet its differentiable composition operations support fine-tuning. We evaluate NePTune on multiple visual reasoning benchmarks and various domains, utilizing adversarial tests, and demonstrate a significant improvement over base models, as well as its effective compositional generalization and adaptation capabilities in novel environments.

How NePTune works

A query is decomposed into the concepts behind, brown, red and sphere, which are scored and composed into a logical form — Figure 1 of the paper. The query is decomposed into symbolic concepts such as *red* and *sphere*, which are then composed to reason explicitly over objects and their relations.

1Program generationAn LLM (DeepSeek-V3 in the paper) acts as a few-shot parser. In one call, it writes a Python program for the question and lists the object names that the detector should look for.

2Perceptual groundingGrounding DINO proposes boxes for those objects. A VLM then grounds each atomic concept through visual prompting: score draws the box in red and returns p(Yes) from the logits of “Yes” and “No”, and query returns a short text answer.

3Symbolic executorA standard Python interpreter runs the program. Overloaded operators such as & and | compose the score tensors with soft logic, and loops, conditionals and variables give the program the full power of Python.

Soft logic operators

Syntax	Logical form	Description	Differentiable implementation
`α_x.exists()`	∃x α_x	Existential quantification	max(α_x)
`α_x.forall()`	∀x α_x	Universal quantification	min(α_x)
`α_x & α_y`	α_x ∧ α_y	Logical conjunction	min(α_x, α_y)
`α_x & β_xy`	α_x ∧ β_xy	Relational conjunction	∑_y α_x · β_xy
`α_x \| α_y`	α_x ∨ α_y	Logical disjunction	max(α_x, α_y)
`α_x.implies(α_y)`	α_x → α_y	Logical implication	max(1 − α_x, α_y)
`not α_x`	¬α_x	Logical negation	1 − α_x
`α_x.iota(var)`	ι(var, α_x)	Best match	softmax(α_x)
`α_x.count()`	count(α_x)	Counting elements	∑ α_x
`s₁ == s₂`	s₁ = s₂	Scalar equality	σ(τ(γ − \|s₁ − s₂\|) / γ)
`s₁ > s₂`	s₁ > s₂	Scalar inequality	σ(τ(s₁ − s₂ − 1 + γ))

Table 2 of the paper. α is an object-centric or scalar probabilistic score, β is a relation probability score, τ = 0.25 is a temperature and γ = 0.25 is a margin.

Key results

CLEVR

	InternVL2.5	NePTune	ViperGPT	NeSyCoCo	LEFT
Zero-shot: the backbone VLM (end-to-end), NePTune and ViperGPT (neuro-symbolic). Trained: NeSyCoCo and LEFT
Final accuracy	90.25	92.65	36.05	99.68	99.50
Exist	87.10	93.19	48.75	99.28	98.92
Query attribute	98.26	96.81	29.42	100.00	99.86
Compare attribute	98.61	91.94	53.06	99.44	99.72
Count	74.60	87.10	21.37	99.79	98.99
Compare number	90.86	92.57	48.57	100.00	100.00

Table 3 of the paper, showing accuracy (%) by question category. Bold marks the best zero-shot result. NePTune is the strongest zero-shot neuro-symbolic method, and its largest gain over the backbone is on counting (+12.50).

Declarative and imperative together

Ablation setting	CLEVR-Humans
Declarative + trained concepts	56.12
+ VLM concepts	68.48 (+12.36)
+ Imperative reasoning	87.67 (+19.19)

Table 10 of the paper, showing accuracy (%). Starting from a purely declarative reasoner, scoring concepts with a VLM adds 12.36 points, and adding imperative reasoning adds another 19.19.

CLEVR-Humans

Method	Accuracy
Trained
LEFT	56.69
NeSyCoCo	56.12
MDETR	81.73
Zero-shot
Qwen2VL-7B	84.12
InternVL2.5-8B	85.95
Ovis1.6-9B	79.96
ViperGPT	31.05
NePTune	87.67

Table 5 of the paper, showing accuracy (%) on human-written questions.

CLEVR extensions

Method	Ref	Puzzles	RPM
Trained
NeSyCoCo†	100.00	95.00	100.00
NeSyCoCo	94.00	94.00	74.00
LEFT	94.00	85.00	87.00
Zero-shot
Qwen2VL-7B	21.00	43.00	53.00
InternVL2.5-8B	27.00	52.00	47.00
Ovis1.6-9B	4.00	47.00	49.00
ViperGPT	8.00	34.00	4.00
VisProg	35.00	27.00	51.00
NePTune†	99.00	65.00	99.00
NePTune	91.00	60.00	80.00

Table 4 of the paper, showing accuracy (%) on referring expressions (Ref), visual puzzles and Raven’s progressive matrices (RPM) built on CLEVR. † marks methods that use ground-truth programs.

Domain shift and fine-tuning: Ref-GTA

Method	Ref-GTA
GroundingDINO-B	27.90
Florence2-L	58.65
Ovis1.6-9B	2.78
InternVL2.5-8B	6.95
InternVL2.5-1B	1.64
+ Fine-tuning	32.61 ±0.35
ViperGPT	1.40
NAVER†	54.84
NAVER	58.73
NePTune‡	62.73
NePTune	69.69
NePTune (1B)	34.92
+ Fine-tuning	69.90 ±1.16

Table 7 of the paper, showing grounding accuracy (%) on images from a game engine, a domain shift that the VLMs were not trained on. Fine-tuning a 1B VLM through NePTune’s differentiable computations with only 1,000 samples reaches 69.90, while standard fine-tuning of the same VLM reaches 32.61.

Real images: Ref-Adv

Method	Ref-Adv
Grounding DINO-B	60.85
Florence2-L	71.73
Ovis1.6-9B	30.70
InternVL2-8B	72.92
InternVL2.5-8B	76.13
ViperGPT	60.66
NAVER†	36.45
NAVER	65.13
NePTune‡	63.71
+ Verification	75.54
NePTune	71.57
+ Verification	78.08
NePTune (1B)	60.69
+ Fine-tuning	68.06 ±0.56
+ Verification	74.59 ±0.12

Table 6 of the paper, showing grounding accuracy (%) on RefCOCO-Adversarial. NePTune‡ uses the same backbones as NAVER, and NAVER† is NAVER’s execution step on its own.

NePTune was accepted to ICLR 2026 and was presented as an oral at the SpaVLE workshop at NeurIPS 2025.

BibTeX

@inproceedings{kamali2026neptune,
title={Ne{PT}une: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language},
author={Danial Kamali and Parisa Kordjamshidi},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=8H0TkSusWI}
}

Share on

Twitter Facebook LinkedIn