Benchmarks

Codegiant vs. frontier AI on real COBOL

We ran each platform's own mainframe tooling against the same 78-file AWS CardDemo estate and scored how much of each scorecard it met.

OpenAI's 6.1 Sol
47.4%
Codegiant
100%
Category OpenAI's 6.1 Sol Codegiant
Analysis
60 / 100
100 / 100
Criterion OpenAI's 6.1 Sol OpenAI score Codegiant Codegiant score
Inventory, records, keys Pass, typed by hand 9/10 Parsed, deterministic 10/10
Steps, programs, datasets Pass, copied as text 9/10 Every node in the graph 10/10
DD to dataset bindings 21%, DD names lost 4/10 Every binding with its line 10/10
Dispositions 39% 4/10 Every one resolved 10/10
Program to file links 14% 4/10 Every one, with the open mode 10/10
Copybook use 0% 1/10 Every link, with its impact 10/10
Calls 80% 7/10 Every call site 10/10
Job order through data Partial, contradicts itself 5/10 Every hand-off 10/10
POSTTRAN business logic 94% 9/10 Every rule checked against its statements 10/10
Business logic, other programs 93%, CBTRN03C wrong 8/10 Every program as written 10/10
Dependency map
39 / 80
80 / 80
Criterion OpenAI's 6.1 Sol OpenAI score Codegiant Codegiant score
Map: source lines behind each link Pass, every line checks out 9/10 Every node and edge 10/10
Map: copybook, call, step and dataset links Pass, basic links complete 9/10 Every link, built deterministically 10/10
Map: job order through data 13% certain 2/10 Every hand-off stated 10/10
Map: datasets built by utilities Missed as suppliers 4/10 Every producer linked 10/10
Map: IDCAMS conditions Listed only 2/10 Tied to what they gate 10/10
Map: guessed links 148 uncertain 5/10 Proven links only 10/10
Map: usable view Hairball 5/10 Navigable, described on click 10/10
Map: repeatable One-off file 3/10 Same input, same map 10/10
Real defects found
9 / 10
10 / 10
Criterion OpenAI's 6.1 Sol OpenAI score Codegiant Codegiant score
Real defects found Pass, three found 9/10 Every defect surfaced, with its line 10/10
Java conversion
39 / 70
70 / 70
Criterion OpenAI's 6.1 Sol OpenAI score Codegiant Codegiant score
Posting logic and quirks Pass on the cases run 9/10 Proven on every path 10/10
Invalid digits Fail, garbage posts as 0 3/10 A data exception, as on the mainframe 10/10
Arithmetic Pass, hand-written per statement 9/10 One exact rule everywhere 10/10
Timestamp layout Pass, copied by a formatter 9/10 The view as defined 10/10
FILE STATUS and decode Fail, four fixed codes 3/10 The real status for every outcome 10/10
I/O failure behaviour Partial 4/10 As the COBOL 10/10
Java form Fail, raw byte offsets 2/10 Named types and fields, traced to the COBOL 10/10
PostgreSQL data
15 / 30
30 / 30
Criterion OpenAI's 6.1 Sol OpenAI score Codegiant Codegiant score
Fields as columns 0%, one bytea per row 0/10 Every field typed 10/10
Keys, order, alternate index Pass 8/10 Typed keys, mainframe order 10/10
Load Pass, unchecked 7/10 Every field checked 10/10
POSTTRAN as a job
18 / 20
20 / 20
Criterion OpenAI's 6.1 Sol OpenAI score Codegiant Codegiant score
POSTTRAN step, DDs, return codes Pass, hand-written 9/10 Generated from the JCL 10/10
Reject generation Pass 9/10 Pass 10/10
Proof
24 / 40
40 / 40
Criterion OpenAI's 6.1 Sol OpenAI score Codegiant Codegiant score
Proof method Pass, local compiler only 8/10 Plus production replay 10/10
Decision coverage 65% 5/10 Every outcome 10/10
Bad data and faults 0% 2/10 Every boundary and fault 10/10
README, NOTES, rerunnable proof Pass 9/10 Complete, with a certificate per program and job 10/10
Platform beyond the prompt
0 / 80
80 / 80
Criterion OpenAI's 6.1 Sol OpenAI score Codegiant Codegiant score
One-click deployment Not delivered in this run 0/10 One-click deployment to AWS, Google Cloud, Azure, Oracle Cloud, DigitalOcean, Cloudflare or Fly.io, on-premises, or Kubernetes 10/10
Managed database deployment Schema and load supplied; managed database deployment not delivered 0/10 Tables go to the client's own managed database, deployed with the app 10/10
Included DevSecOps Integrated toolchain not delivered 0/10 Git, CI/CD, Terraform, code review, docs, observability, error and performance monitoring, uptime checks and status pages included 10/10
After go-live / AI SRE Ongoing operations service not delivered 0/10 24/7 AI SRE monitors the system, repairs incidents, tunes cost and scaling, catches security drift and maintains runbooks 10/10
Integrations and automation Integrated automation platform not delivered 0/10 Kronos Automate connects to 3,000+ services, with a dedicated automation tenant per modernized app 10/10
Build on the modernized system App, agent and workflow builders not delivered 0/10 App builder, workflow builder and automation engine included for new apps, agents and workflows on the transformed code 10/10
Platform governance Platform access controls, audit trails and policy enforcement not delivered 0/10 Role- and attribute-based access, audit trails and policy enforcement 10/10
Purpose-built models and private hosting Purpose-built, self-hosted modernization and automation models not delivered 0/10 Kronos models purpose-built for mainframe modernization and automation, self-hostable inside the client's perimeter 10/10

At a glance

The comparison by area.

Area OpenAI's Sol Codegiant
Analysis Counts and basic facts pass; DD bindings reach 21%, program-to-file links 14% and copybook use 0% in the analysis. The parser resolves inventory, layouts, DD bindings, copybook use and calls to source lines.
Dependency map Basic links and cited lines pass; 148 edges are uncertain, utility suppliers are missed and only 13% of job hand-offs are certain. The deterministic graph includes producers, consumers and conditions, with source lines behind the relationships.
Java conversion Posting and quirks pass on tested cases, but invalid digits post as zero, status decoding is absent and Java uses raw byte offsets. Named types and fields preserve number, status and error behaviour, with blocks traced to COBOL.
PostgreSQL data Keys, order and alternate index pass; each record is stored in one bytea column and loaded unchecked. Copybook fields become typed columns, mainframe order is kept and records are checked before cutover.
POSTTRAN Step, DDs, reject generation and return codes pass in hand-written job handling. The job is generated from JCL, including catalogue behaviour and return codes.
Proof Outputs are compared with a local compiler; decision coverage is 65%, with no I/O faults or bad-data cases triggered. Decision outcomes, error paths and field boundaries are checked, alongside captured production replay.
Platform beyond the prompt The eight integrated platform capabilities were not delivered in this run. Deployment, managed database deployment, DevSecOps, AI SRE, integrations, builders, governance and private hosting are included.
Beyond the demo The brief identifies further issues around context, uniformity, site settings, production data, batch windows, operations, failures, proof and scale. The nine detailed sections compare those risks with estate-wide definitions, mainframe behaviour, restart, independent proof and production operations.

Detailed findings

The checks behind each part of the brief.

1. Analysis: Sol against our deterministic facts
Fact family GPT-6.1 Sol Codegiant
Inventory Pass. Counts right, but they come from reading the folder, not from a parse Every program, copybook, job, PROC and data file, counted from the parsed source
Record lengths and keys Pass. Typed by hand from the copybooks into a table Every record and key taken from the copybook and the DEFINE, deterministic
Steps and what each runs Pass. The EXEC lines copied out of the JCL Every step resolved to its program or utility, with PROC steps expanded
Datasets Pass. Names copied as text Every dataset as a node in the graph, with its data and index parts
Which DD binds which dataset 21%. Continuation lines pasted as bare DSN= text with the DD name lost Every reference bound to its DD, step and line
Disposition per DD 39%. The rest left as "see the original JCL" Every disposition resolved per DD, defaults included
Program to file links 14%. Only the six POSTTRAN files; the rest in prose Every SELECT tied through its DD to the dataset, with the open mode
Copybook use 0%. No copybook-to-program link at all Every COPY link, so a layout change shows every program it touches
Calls 80%. The abend routine used by 11 programs is not named Every call site, internal and system
Job order through data Partial. Prose chains, and one statement contradicts its own table Every hand-off between jobs, with the producing and consuming lines
2. The dependency map
Check GPT-6.1 Sol Codegiant
Source lines behind each link Pass. Every cited line checks out Every node and edge, with its line
COPY, CALL, step to program, program to dataset Pass. The basic links are complete, with generations kept Every link built deterministically, and rebuilt the same way on every change
Job order through data 13% certain. Most hand-offs flagged "uncertain" Every hand-off stated, with both lines
Datasets created or loaded by utilities Missed as suppliers. The GDG definitions, backup and reload jobs ahead of POSTTRAN are not linked Every producer, utility or program
IDCAMS conditions Listed only. No link from a condition to the command or return code it gates Every condition tied to what it gates
Guessed links 148 uncertain edges. A program with no job gets datasets matched by DD name from other jobs Proven links only
Usable view Hairball. Fields, paragraphs and spool lines in one graph; the drawn overview covers one job Every job, step and dataset navigable, with a description on click
Repeatable One-off. A model's file; the next run can differ Same input, same map, every time
3. CBTRN02C to Java (asked)
Construct GPT-6.1 Sol Codegiant
Validation and posting Pass. Identical on the cases it ran Identical, proven on every path
Quirks Pass. All kept All kept, because every statement is converted as written
Signed amounts Pass on clean data. Sign read from a hand-written nibble test Every sign rule of the compiler reproduced exactly
Invalid digits Fail. Spaces read as 0, and the bad data posts A data exception, as on the mainframe
Arithmetic Pass. Truncation written by hand at each statement Exact decimal arithmetic from the target picture, one rule everywhere
Timestamp REDEFINES Pass. A Java formatter that copies the layout The view over the same bytes, exactly as defined
FILE STATUS and the status decode Fail. Four fixed codes; the decode branch was never converted The real status for every outcome, decode included
I/O failures Partial. Four codes; one path stops where the COBOL continues The same behaviour as the COBOL on every failure
Java form Fail as modernization. Raw byte offsets, no field names Named record types and fields, every block traced to its COBOL line
4. The slice's datasets in PostgreSQL (asked)
Check GPT-6.1 Sol Codegiant
Fields as columns 0%. Each record dumped into one bytea Every copybook field as a typed column
Keys and order Pass. EBCDIC byte order on a generated key Typed keys, mainframe order kept
Alternate index Pass. An expression index on the bytes Same lookups and the same order as the VSAM path
Load Pass. Bytes copied in unchecked Every record verified before cutover
5. POSTTRAN as a job (asked)
Check GPT-6.1 Sol Codegiant
Step, DDs, program Pass. Written by hand in a shell switch Generated from the JCL
DALYREJS(+1), NEW,CATLG,DELETE Pass Catalogue behaviour as on z/OS
Return codes Pass As on z/OS
6. Proof of the slice (asked)
Check GPT-6.1 Sol Codegiant
Method Pass. Full output compared against a local compiler Full output compared, plus replay of captured production runs
Decision coverage 65%. Happy paths and the four reject reasons only Every decision outcome
Error paths 0%. No open, read, write or close failure was ever triggered Every error path triggered and compared
Bad data 0%. No invalid digits, odd signs or LOW-VALUES Field boundaries, a census of the real data, missing and locked datasets

Beyond what was asked

Where AI modernization breaks at estate scale.

7. The complete scorecard: asked items and platform
Asked-items subtotal 204/350 (58%) 350/350 (100%)
Added platform subtotal 0/80 (0%) 80/80 (100%)
Combined total 204/430 (47.4%) 430/430 (100%)
8.1 The prompt treadmill
Where it breaks What it costs Codegiant
Context runs out Program 400 is converted without what was learned on program 4 Nothing depends on what a model remembers
The same input gives a different output Rerun a program and the Java changes; nothing can be signed off twice Same input, same output, every time
Every session decides again Layouts, names, types and error handling drift from prompt to prompt Decided once for the estate
Nobody can review it all Thousands of hand-shaped programs, each a little different Uniform by construction; nothing hand-shaped to review
The model grades its own work The same model writes the code, the tests and the verdict Proof is independent of the writer
8.2 Uniformity across the estate
Where it breaks What it costs Codegiant
One copybook, many conversions The same record reads differently in different programs, and each passes its own tests One answer for every program that shares it
Numbers, signs and views handled case by case Totals, rounding and REDEFINES behave differently depending on which prompt wrote which method One behaviour across the estate
Schema drift between waves Wave 2 meets wave 1's tables and does not fit; rework mid-program Every wave lands on the same design
8.3 The program outside the source
Where it breaks What it costs Codegiant
Behaviour set outside the code Silent differences in arithmetic, signs, characters and file handling Captured and applied per client
Source that isn't what runs A faithful conversion of the wrong program Caught before conversion
8.4 The data
Where it breaks What it costs Codegiant
Real data breaks the rules Bad values post as zero, or the migration weekend stops at record 3 million Every exception found and reported before cutover
Bytes no program names Lost on the first rewrite, and nobody notices for months Preserved
Order SQL returns rows in a different order than VSAM did, and every control break moves Mainframe order on every read
Generations (0), (+1) and (-1) collapse into one table Every generation side by side
8.5 PostgreSQL and the batch window
Where it breaks What it costs Codegiant
Chatty, unbatched access Fine on a 300-record sample, hours late on production volume Engineered for production volume
A job dies halfway Starts again from zero, or posts the same transactions twice Restarts where it stopped, never posts twice
Two runs at once Two copies of a job update the same rows Prevented, as on the mainframe
Records stored as blobs No index, query or report without decoding bytes Typed, queryable tables
8.6 JCL and operations
Where it breaks What it costs Codegiant
Jobs written by hand Every JCL or parameter change becomes a code change and a release Every job converted into a Java batch job that does everything its JCL did, and keeps working when the JCL changes
Step logic simplified Steps run that should be skipped; files kept, deleted or catalogued at the wrong time Full JCL coverage: step conditions, file dispositions, generations, parameters and in-stream data
Utility and console steps dropped Sol dropped all six online-file close and open steps; the utility steps were hand-coded or partly ignored The utilities run as real steps: file definition and copy, sort, delete and create, conditional return codes
Return and abend codes changed The scheduler reacts to codes it has never seen The scheduler sees what it sees today
8.7 When things fail
Where it breaks What it costs Codegiant
Error paths converted loosely or not at all The first real I/O error in production behaves unlike the last 20 years Same status, same message, same outcome
Invented or "fixed" behaviour Checks, messages and abends the COBOL never had, or quirks quietly corrected Nothing added, nothing removed
8.8 Proof that holds up
Where it breaks What it costs Codegiant
A local compiler and happy-path inputs The paths that hurt in production are never run Proven against production behaviour, on the inputs that hurt
No trace, no reconciliation, no sign-off Auditors get a chat log Every line traceable; a certificate per program and job
AWS Transform for mainframe
6.5%
Codegiant
100%
Category AWS Transform for mainframe Codegiant
The converted code
0 / 7
7 / 7
Requirement AWS Transform for mainframe AWS score Codegiant Codegiant score
Runtime The generated code imports their runtime. Remove it and it won't compile 0/1 No proprietary runtime. It's standard Java and compiles anywhere 1/1
Data fields Your data is stored in AWS's own custom classes, which only their runtime understands 0/1 Your data uses ordinary Java types any developer can read 1/1
Program shape Each COBOL paragraph becomes its own method, so the code still reads like COBOL 0/1 Methods are named for what the business does, like postTransaction() 1/1
AWS Reforge Their fix is an AI rewrite applied afterwards, and anything it can't convert is left for you 0/1 The Java is structured correctly from the start, and every piece is checked before it's kept 1/1
The mainframe inside The app still runs like a mainframe internally, using the mainframe's own EBCDIC character encoding in memory 0/1 Normal Java in memory. The mainframe character encoding is handled only where files come in and go out 1/1
Changing it later Your COBOL is first rewritten into their engine's private language, and only their engine can turn it into Java 0/1 Every step is our own code, with no proprietary binary or engine, so you can easily update or change it later 1/1
Outside the code Settings like number handling and encoding are properties on their runtime, set by hand 0/1 The same COBOL behaves differently at two companies, so we read all of it from the client's own files and build it into plain Java 1/1
Data
0 / 3
3 / 3
Requirement AWS Transform for mainframe AWS score Codegiant Codegiant score
VSAM Each record is stored as raw, unreadable bytes inside the database 0/1 Records become real database tables with real columns and indexes 1/1
Using the data You can't query a balance directly. It's buried in a blob only their runtime reads 0/1 Any reporting tool or new service reads the data directly 1/1
Write safety Updates are batched and written about a second later 0/1 Every write reaches the database immediately 1/1
DB2
0 / 4
4 / 4
Requirement AWS Transform for mainframe AWS score Codegiant Codegiant score
SQL handling Your SQL is pulled out into lookup files that only their runtime can execute 0/1 Your SQL is translated into normal database code inside the generated classes 1/1
One model for files and tables Files and database tables are handled by two separate mechanisms inside their runtime 0/1 Every file call and every database call returns the same kind of result 1/1
One database, one commit VSAM data and DB2 data still live in separate worlds 0/1 Both end up in PostgreSQL, so a file update and a table update can commit together 1/1
Every SQL call tracked No per-call check against the original 0/1 Every SQL call has an identity checked against the Java 1/1
Batch
0 / 3
3 / 3
Requirement AWS Transform for mainframe AWS score Codegiant Codegiant score
Jobs Batch jobs become scripts that only run on their runtime 0/1 Standard Java jobs, with high-throughput job queueing already built into the transformed output, and started by whatever scheduler you already use 1/1
Utility steps Sort and file-management steps are pasted in as text for their runtime to interpret 0/1 Those steps become ordinary batch steps, with the same results as the mainframe 1/1
Restart Restart depends on their runtime 0/1 The checkpoint is saved in the same commit as the data, so a rerun never posts twice 1/1
Running and cost
0 / 5
5 / 5
Requirement AWS Transform for mainframe AWS score Codegiant Codegiant score
Where it runs Documented only on AWS servers 0/1 Any of 7 clouds, on-premises, or Kubernetes 1/1
Runtime fee $0.31 per CPU core per hour, forever, on top of the servers 0/1 Pure Java output. No runtime to license or pay for 1/1
AWS can change the rules AWS has already closed two of its own mainframe runtime products to new customers 0/1 Nothing in your system depends on a vendor's roadmap 1/1
Tools needed You need several separate tools, downloaded and set up individually 0/1 One platform, from analysis through to deployment 1/1
Speed Nothing checks that the converted jobs run as fast as the original 0/1 Every job is timed against the mainframe's batch window and fails if it's slower 1/1
How each one uses AI
0 / 3
3 / 3
Requirement AWS Transform for mainframe AWS score Codegiant Codegiant score
Where AI sits in the pipeline A rules engine that copies COBOL's structure, then optionally an AI rewrite over whole classes, with failures left for you 0/1 Rules do everything with one right answer. AI only writes decisions, inside a locked skeleton 1/1
What the rules decide before AI is involved The rules engine copies COBOL's structure, and the rest is left to the AI rewrite 0/1 Everything: parser facts, technical analysis, data classes, access classes, schema, job plans, function slices, skeletons, names and variable maps, no AI involved 1/1
What every line has to pass No per-line check against the original, and no check that the rewrite invented nothing 0/1 Compilation, real methods, allowed constructs, every field written, conditions the right way round, every number traced to source. Two repair rounds, then it is set aside 1/1
Before conversion
1 / 7
7 / 7
Requirement AWS Transform for mainframe AWS score Codegiant Codegiant score
Analysis without AI Their documentation and business rules are generated by AI 0/1 A full technical document built from the code itself, every fact tied to a source line, no AI involved 1/1
Statement-level facts None 0/1 Every COBOL statement is captured with its real operands 1/1
Business logic grounded in facts None 0/1 The AI narrates only facts already deterministically proven, and a separate judge catches anything it drifts into 1/1
Business functions from outputs None 0/1 Each function is traced backwards from what the program actually produces 1/1
Estate graph with line-level proof AWS Transform gives you graphs and domains 0/1 Click any link and see the exact source line behind it, and the conversion decision it drove 1/1
Migration plan as a calculation Domain splitting and wave planning 1/1 Groups are formed where nothing inside a group is written from outside it 1/1
Dead code checks None 0/1 The source is checked against what actually runs in production. Dead code is flagged rather than converted 1/1
After conversion
2 / 6
6 / 6
Requirement AWS Transform for mainframe AWS score Codegiant Codegiant score
Equivalence Their compare tool diffs files and tables, and you set up and run it yourself 1/1 The Java and the original COBOL run on the same input and are compared automatically, byte for byte 1/1
Test data Generated scripts you run on the mainframe to collect data 1/1 Real production inputs and outputs captured and replayed, plus edge cases built from the copybooks 1/1
Business totals No control-total check 0/1 Record counts and balances must match what the COBOL already computes 1/1
Hallucination Their AI rewrite comes with no check that it invented nothing 0/1 Anything in the Java without a basis in the COBOL is flagged 1/1
Known bug patterns None 0/1 A checker catches the classic conversion mistakes 1/1
Cutover A test environment inside your AWS account 0/1 A parallel run on live traffic, old and new side by side, before anything switches over 1/1
The platform
0 / 8
8 / 8
Requirement AWS Transform for mainframe AWS score Codegiant Codegiant score
One-click deploy Deploys to AWS only 0/1 Deploys in one click to AWS, Google Cloud, Azure, Oracle Cloud, DigitalOcean, Cloudflare or Fly.io, or on-premises, or any Kubernetes infrastructure you run 1/1
Database Your data lives in their proprietary store, tied to their runtime 0/1 Your tables go to your own managed database, deployed with the app 1/1
DevSecOps AWS Transform hands you templates and you build the rest yourself 0/1 The whole toolchain is included, nothing to assemble 1/1
After go-live Their offering stops once the code is delivered 0/1 An AI operations assistant watches the system around the clock, repairs incidents and tunes cost 1/1
Connect None 0/1 Kronos Automate, Codegiant's own LLM, connects to 3,000+ services so you can build automations and workflows on top of your modernized transformation. Each modernized app gets its own dedicated automation tenant, running an enterprise-grade engine that supports hundreds of millions of events per month 1/1
Build None 0/1 New apps, agents and workflows on the transformed code, with the app builder, workflow builder and automation engine included 1/1
Governance None 0/1 Role- and attribute-based access, audit trails and policy enforcement 1/1
Purpose Built Models None 0/1 Kronos models purpose-built for mainframe modernization and automation, which can be self-hosted inside your perimeter for added safety 1/1
Anthropic's COBOL plugin
1.6%
Codegiant
100%
Category Anthropic's COBOL plugin Codegiant
Coverage: how much converts
0 / 2
2 / 2
Requirement Anthropic's COBOL plugin Anthropic score Codegiant Codegiant score
How much gets converted after 222 AI agents, 1 of the 15 programs was converted. 0/1 every program in the system goes through the same pipeline and the same checks in one run. 1/1
Nothing gets lost the Java is rewritten from a description of the program, not from the program, so there’s no way to see what was left out. The program’s error-reporting routine and seven error checks disappeared, and nothing flagged it. 0/1 readable Java, plus a ledger proving every line of COBOL ended up somewhere in the Java. 1/1
The data model: records and tables
0 / 2
2 / 2
Requirement Anthropic's COBOL plugin Anthropic score Codegiant Codegiant score
Reading the records the code that turns mainframe records into Java values is hand-written for this one program. Sharing it with the next program is left as a to-do. 0/1 one compiler produces the Java record, the code that reads and writes it, and the database table, all from the same layout, for every program. They can’t disagree. 1/1
The database design each converted program creates its own version of the tables it uses. 0/1 one table design per file, created once and versioned. 1/1
Discovery: knowing what you have
0 / 5
5 / 5
Requirement Anthropic's COBOL plugin Anthropic score Codegiant Codegiant score
Counting what you have the AI wrote the inventory and got it wrong. It counted 9 VSAM files where there are 11 and 8 generation groups (GDGs) where there are 7, and it predicted 13 rejected transactions where the run had 38. 0/1 the inventory comes from a parser, not an AI, and every fact points to the line of code it came from. Counted, not guessed. 1/1
Checkable facts nothing it says about the program is stored in a form anyone can check. 0/1 every statement, field and path in the program is recorded with its line number. For this one program that’s 339 statements and 128 fields. 1/1
Shared layouts its map of the system leaves out copybooks (the shared record layouts) entirely. 0/1 every copybook and every place it’s used is on the map. Change a layout and you see every program it touches. 1/1
A map you can trust its map of the system disagrees with itself. It says five programs read the transaction file in one place and seven in another. And it says one program updates a file it never touches. 0/1 every link on our map comes from a real line in the JCL or the program, and you can click it to see that line. 1/1
How jobs hand off to each other the map loses which file generation each job reads and writes, and which steps only run if an earlier step succeeded. 0/1 generations and step conditions are kept for every job step. 1/1
Rules and the plan
0 / 4
4 / 4
Requirement Anthropic's COBOL plugin Anthropic score Codegiant Codegiant score
Finding all the rules rule extraction stopped at its built-in limit while it was still finding new rules. 0/1 rules are written from the proven facts of each program, with no limit to run out of. 1/1
Rules that agree with the code two of its rules say the transaction file is wiped at the start of every run. Its own Java does the opposite: it refuses to start unless the file is already empty. 0/1 a separate checker tests every rule against the code it describes. A rule that disagrees doesn’t ship. 1/1
Keeping the quirks it spotted that the COBOL carries on with the previous record when a read fails, wrote that down, then built Java that stops instead. 0/1 business functions are traced backwards from what the program produces, so every path the COBOL takes is kept. 1/1
The plan the migration plan is written by the AI from its own memory of the session. 0/1 the plan is calculated from the stored map of the system, so anyone can regenerate it and get the same answer. 1/1
Data fidelity: numbers, bytes and layouts
0 / 9
9 / 9
Requirement Anthropic's COBOL plugin Anthropic score Codegiant Codegiant score
Records nobody changed stay unchanged when it saves a record, it rewrites every field, including ones the program never touched, and changes how their signs are stored. Its own check proved this. 0/1 only the fields the program actually changed are written back. Everything else stays the same, byte for byte. 1/1
Your mainframe’s number settings one fixed way of storing numbers, whatever your shop’s settings are. 0/1 number handling follows your own compiler settings. 1/1
Special “empty” bytes mainframe LOW-VALUES are turned into a Unicode placeholder character, in the code and in the database. 0/1 the real bytes are kept, and stored as raw bytes only in the columns where they actually appear. 1/1
Characters and languages one hand-built character table for US EBCDIC. 0/1 the character set comes from your settings, including the euro sign and Japanese double-byte text. 1/1
Fields that share space the program reads the same two bytes in two ways to show the VSAM error code. That trick was dropped. 0/1 every shared-space layout (REDEFINES) is kept as two views of the same bytes. 1/1
Unused space in records it assumes unused space starts as blanks. Its laptop test setup makes the same assumption, so the test agreed with itself. 0/1 taken from your run-time settings and checked against your real records. 1/1
Size limits on numbers counters that wrap around on the mainframe never wrap in Java. 0/1 every number field keeps the mainframe’s size limits and arithmetic rules. 1/1
Loading bad data one bad byte in one record stops the whole data load, without saying which record. 0/1 every record is checked first. Bad values are stored safely, nothing is skipped, and every problem is reported by field. 1/1
Sort order the database sorts keys like a dictionary, not like the mainframe. 0/1 keys sort exactly as they did on the mainframe, so programs that read in key order get the same order. 1/1
Batch and JCL
0 / 11
11 / 11
Requirement Anthropic's COBOL plugin Anthropic score Codegiant Codegiant score
Restarting after a failure transactions are written one at a time, with no database transaction around them and no restart. A rerun posts again what was already posted. 0/1 work is saved in batches together with a restart point, so a rerun never posts anything twice. 1/1
File generations the database has no idea of generations. 0/1 every table keeps generations, so today’s and yesterday’s versions live side by side. 1/1
Every kind of VSAM file only one kind (keyed files). 0/1 all kinds: keyed, sequential (ESDS), numbered-slot (RRDS), variable-length, and alternate indexes. 1/1
Which file a program uses the JCL’s file assignments become one database and two file paths. Pointing a run at test or production data is one typo away. 0/1 the JCL is compiled, so every file assignment works the way it did on the mainframe. 1/1
Generations of output the rejects file, which the mainframe numbers automatically and keeps five copies of, becomes a file name the operator types in. 0/1 generations are numbered, kept and rolled off automatically, as on the mainframe. 1/1
Two runs at once nothing stops the same posting job running twice at the same time. Its own reviewer said so. 0/1 files are locked the way the mainframe locks them, and the same job can’t run twice. 1/1
Steps in a job the job becomes one Java application, so rules like “skip this step if the last one failed” are gone. 0/1 every step is kept, its return code decides what runs next, and utility steps (IDCAMS, SORT) run too. 1/1
What happens to files after a step the program renames its own output file. 0/1 files are kept, cataloged or deleted at the end of each step, by the job, as on the mainframe. 1/1
Errors on files when 3 of the files fail to open or 4 fail to close, there’s no handling at all, and one close is never checked. 0/1 every file action returns the same IBM status code the COBOL checks. 1/1
Messages for the night operator 14 of the 26 messages are gone, including the ones that say which error happened and that the program is stopping. 0/1 every message is kept with its exact text. A missing one stops the build. 1/1
How a failure looks to the scheduler the mainframe’s abend code (U0999) becomes exit code 16, which the scheduler’s rules don’t recognise. 0/1 the scheduler sees the same abend codes it always has. 1/1
Hard programs, layouts and truthfulness
0 / 6
6 / 6
Requirement Anthropic's COBOL plugin Anthropic score Codegiant Codegiant score
Made-up behaviour it added a crash the COBOL can’t cause, a check the COBOL never makes, a warning the COBOL never prints, and new message texts. 0/1 anything in the Java that has no source in the COBOL is flagged automatically. Claude doesn’t check for this at all. 1/1
The odd behaviour, kept where the COBOL carries on after certain errors, the Java stops. 0/1 the same behaviour, odd parts included, because that’s what “the same” means. 1/1
Compiler settings it never asks how your programs were compiled. None of its 5 setup questions covers it, and it assumes IBM’s defaults. 0/1 we read your compile listings and run-time settings (16 of them), because they change how the same code behaves. 1/1
Everything outside the code it noticed files that come from outside the system, then didn’t ask about them. 0/1 we account for 125 things that change a program from outside its source: the file catalog, the data, the site setup, the scheduler and mainframe services. 1/1
The hardest program one program changes its own control flow while it runs (ALTER) and reads the mainframe’s job tables directly. Claude said it “can’t be translated” and planned to rewrite it from the rules instead. 0/1 we convert it. The jumps are restructured, and the job information it reads is supplied from the actual run. 1/1
Programs its test can’t run 4 of the 15 programs don’t compile on its laptop COBOL, so they can’t be compared at all. 0/1 every program is handled, with mainframe behaviour taken from IBM’s rules, not from a laptop compiler. 1/1
Proof: what is checked and against what
0 / 7
7 / 7
Requirement Anthropic's COBOL plugin Anthropic score Codegiant Codegiant score
What gets compared the output files and 3 lines of the job log. The full log was saved and never compared. 0/1 every output file, return code, file status and log line. 1/1
What it’s compared against an open-source COBOL on a laptop, with a stand-in for the mainframe’s abend routine. 0/1 real recorded runs from your mainframe. 1/1
The test data sample data that couldn’t show its biggest bug, plus test cases it picked itself. Its bad-data test only hit records that were rejected before the bug could happen. 0/1 test data built from every field’s limits and from your real data, plus missing and locked files. 1/1
Who checks the work its own report says the check ran in the same session that wrote the code. 0/1 the proof is completely separate from the conversion. 1/1
The verdict it first reported “20 of 20 identical”. With new inputs, its own verdict became NOT PROVEN. 0/1 one verdict over every input, and nothing ships unproven. 1/1
Money and test strength it breaks the code on purpose only twice to see whether the tests notice. 0/1 counts and money totals per file and per account, and every condition is tested by deliberately breaking it. 1/1
Who approved the changes changes from the original are marked “approved” with no name. One schema choice was made by the AI against its own plan. 0/1 every change has a named approver, a date and the evidence. 1/1
Scale, determinism and your data
0 / 2
2 / 2
Requirement Anthropic's COBOL plugin Anthropic score Codegiant Codegiant score
A real bank by its own numbers, a large bank would need about 155,000 to 215,000 AI agents across 150+ separate runs, merged by hand. 0/1 no AI until the code is split into small pieces, and no limit on size. 1/1
Same answer twice its own README says running the same command again can give different results. 0/1 every run gives exactly the same result. 1/1
Architecture: memory and handoffs
0 / 5
5 / 5
Requirement Anthropic's COBOL plugin Anthropic score Codegiant Codegiant score
Helpers who each see one piece the analysis was split across 97 helper agents, each reading a slice and reporting back a summary. A link between two slices survives only if a summary happened to mention it. That’s how the final map lost every copybook and miscounted who reads the transaction file. 0/1 links between programs come from the parser and the JCL compiler, across the whole system at once. No summary sits in between. 1/1
Programs stop knowing about each other after conversion each program is converted on its own, with its own code for reading records and its own copy of the tables. What the next program needs to know is written as a to-do note in a text file. 0/1 one shared definition per file, used by every program that touches it. A program can’t read a record differently from the program that wrote it. 1/1
The hand-off between programs is never tested each program is proven against its own COBOL, using test data it made itself. Nobody tests that what the posting program writes is exactly what the interest program reads the next night, and seven programs read the transaction file. 0/1 whole job chains run end to end in the proof (backup, then posting, then interest, then combine), with every hand-off compared. 1/1
Programs that call each other called programs and the data they pass are not converted together. In this run, the statement program and the program it calls weren’t converted at all. 0/1 called programs, the data passed between them and their shared storage are compiled as one unit. 1/1
One brain, with a limited memory one AI session runs the whole migration and holds the plan in its working memory. Even on this small system, the plan was written from that memory without re-reading the analysis files. A bank’s estate is thousands of times bigger, so the session has to summarise itself over and over, and later decisions are made from summaries of summaries. 0/1 nothing important lives in an AI’s memory. Every fact, decision and link is stored, and every stage re-reads it. 1/1
The platform
1 / 8
8 / 8
Requirement Anthropic's COBOL plugin Anthropic score Codegiant Codegiant score
One-click deploy doesn’t deploy one-click anywhere. The conversion stops at Java in a folder. 0/1 deploys in one click to AWS, Google Cloud, Azure, Oracle Cloud, DigitalOcean, Cloudflare or Fly.io, or on-premises, or any Kubernetes infrastructure you run. 1/1
Database your data lives in whatever database the conversion picked, and each converted program carries its own copy of the tables. 0/1 your tables go to your own managed database, deployed with the app. 1/1
DevSecOps the plugin hands you Java in a folder, and you assemble the rest yourself. 0/1 the whole toolchain is included, nothing to assemble. 1/1
After go-live the offering stops once the code is converted. 0/1 an AI operations assistant watches the system around the clock, repairs incidents and tunes cost. 1/1
Connect MCP and connectors bring the tools you already use into the work: Google Drive, Gmail, Slack, Microsoft 365, Notion, Figma, HubSpot and more. 1/1 Kronos Automate, Codegiant’s own LLM, connects to 3,000+ services so you can build automations and workflows on top of your modernized transformation. Each modernized app gets its own dedicated automation tenant. 1/1
Build none. 0/1 new apps, agents and workflows on the transformed code, with the app builder, workflow builder and automation engine included. 1/1
Governance none. 0/1 role- and attribute-based access, audit trails and policy enforcement. 1/1
Purpose Built Models only Claude can run it, and your source goes to the AI provider with usage reporting on by default. 0/1 Kronos models purpose-built for mainframe modernization and automation, which can be self-hosted inside your perimeter for added safety. 1/1

Start building today.

Harness the power of enterprise-grade AI and thousands of connectors to build what’s next.