Benchmark Results - ISONGraph | Performance Comparison

Knowledge Graph Benchmark

100 questions across 7 datasets, testing format understanding and graph reasoning.

Rank	Format	Avg Tokens	Token Savings	Accuracy	Efficiency Score
1	ISONGraph	1,698	68.6%	90.0%	53.00
2	ISON	1,976	63.4%	88.0%	44.53
3	JSON Compact	2,893	46.5%	88.0%	30.42
4	TOON	2,934	45.7%	85.0%	28.97
5	Cypher (Neo4j)	3,522	34.9%	89.0%	25.27
6	JSON	5,406	0.0%	87.0%	16.09
7	RDF/Turtle	4,166	22.9%	58.0%	13.92
8	YAML	4,891	9.5%	82.0%	16.77
9	CSV	2,156	60.1%	54.0%	25.05
10	GraphML	9,093	-68.2%	87.0%	9.57

Efficiency Score = (Accuracy / Tokens) * 1000 - measures accuracy per token cost.

Data Traversal Benchmark

50 questions across 4 datasets, testing multi-hop traversal, path finding, and graph analysis.

Rank	Format	Avg Tokens	Accuracy	Efficiency (Acc/1K tokens)
1	ISONGraph	639	92.0%	143.97
2	ISON	685	88.0%	128.47
3	TOON	856	80.0%	93.46
4	JSON Compact	1,072	82.0%	76.49
5	JSON	2,039	84.0%	41.20

Query Type Analysis

Performance breakdown by question type:

Query Type	ISONGraph	JSON	Cypher	GraphML
Direct Lookup	98%	96%	94%	92%
Single-hop Traversal	94%	88%	92%	86%
Multi-hop Traversal	80%	48%	72%	68%
Path Finding	86%	62%	84%	74%
Aggregation	92%	90%	88%	82%

Key Finding: ISONGraph excels at multi-hop traversal (80%) where other formats struggle (40-70%). The tabular format makes relationship chains easier for LLMs to follow.

Token Efficiency Comparison

JSON (5,406 tokens)

{
  "nodes": [
    {"id": 1, "type": "person", "name": "Alice", "age": 30},
    {"id": 2, "type": "person", "name": "Bob", "age": 25},
    {"id": 3, "type": "person", "name": "Carol", "age": 28}
  ],
  "edges": [
    {"source": 1, "target": 2, "relation": "KNOWS", "since": 2020},
    {"source": 2, "target": 3, "relation": "KNOWS", "since": 2021}
  ]
}

                                    ISONGraph (1,698 tokens)
                                    68.6% savings
                                

                                    nodes.person
id  name    age
1   Alice   30
2   Bob     25
3   Carol   28

edges.KNOWS
source      target      since
:person:1   :person:2   2020
:person:2   :person:3   2021
                                

Why ISONGraph Wins

Tabular Format

LLMs are trained on billions of tables (CSV, markdown, logs). The tabular layout is instantly recognizable and parseable.

No Key Repetition

Column headers define field names once. JSON repeats every key for every object, wasting tokens.

Clear Node References

The :type:id syntax makes relationships explicit and easy to follow across tables.

Type Grouping

Nodes and edges are grouped by type, making it easy to understand the graph structure at a glance.

Whitespace Alignment

Aligned columns are visually scannable and help LLMs track relationships across rows.

No Nesting

Flat structure eliminates the cognitive overhead of tracking nested JSON objects and arrays.

Methodology

Datasets

Social Network - Users, follows, posts, likes
E-commerce - Products, orders, reviews, categories
Movie Database - Movies, actors, directors, genres
Academic Citations - Papers, authors, citations, venues
Organization Chart - Employees, departments, reports-to
Transportation - Stations, routes, schedules
Recipe Database - Recipes, ingredients, techniques

Question Types

Direct property lookup
Single-hop neighbor queries
Multi-hop traversal (2-4 hops)
Shortest path finding
Aggregation queries (count, sum, average)
Pattern matching
Cycle detection

Evaluation

Model: Claude 3.5 Sonnet
Token Count: cl100k_base tokenizer
Accuracy: Exact match on expected answer
Runs: 3 runs per question, majority vote

Reproduce the Benchmark

# Clone repository
git clone https://github.com/maheshvaikri-code/ison
cd ison/isongraph/benchmark

# Install dependencies
pip install -r requirements.txt

# Run Knowledge Graph Benchmark
cd KnowledgeGraph_Benchmark
python benchmark_kg_100.py

# Run Data Traversal Benchmark
cd ../DataTraversal_Benchmark
python benchmark_graph.py

Full methodology and raw results available in the benchmark directory.