We ran each platform's own mainframe tooling against the same 78-file AWS CardDemo estate and scored how much of each scorecard it met.
| Criterion | OpenAI's 6.1 Sol | OpenAI score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Inventory, records, keys | Pass, typed by hand | 9/10 | Parsed, deterministic | 10/10 |
| Steps, programs, datasets | Pass, copied as text | 9/10 | Every node in the graph | 10/10 |
| DD to dataset bindings | 21%, DD names lost | 4/10 | Every binding with its line | 10/10 |
| Dispositions | 39% | 4/10 | Every one resolved | 10/10 |
| Program to file links | 14% | 4/10 | Every one, with the open mode | 10/10 |
| Copybook use | 0% | 1/10 | Every link, with its impact | 10/10 |
| Calls | 80% | 7/10 | Every call site | 10/10 |
| Job order through data | Partial, contradicts itself | 5/10 | Every hand-off | 10/10 |
| POSTTRAN business logic | 94% | 9/10 | Every rule checked against its statements | 10/10 |
| Business logic, other programs | 93%, CBTRN03C wrong | 8/10 | Every program as written | 10/10 |
| Criterion | OpenAI's 6.1 Sol | OpenAI score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Map: source lines behind each link | Pass, every line checks out | 9/10 | Every node and edge | 10/10 |
| Map: copybook, call, step and dataset links | Pass, basic links complete | 9/10 | Every link, built deterministically | 10/10 |
| Map: job order through data | 13% certain | 2/10 | Every hand-off stated | 10/10 |
| Map: datasets built by utilities | Missed as suppliers | 4/10 | Every producer linked | 10/10 |
| Map: IDCAMS conditions | Listed only | 2/10 | Tied to what they gate | 10/10 |
| Map: guessed links | 148 uncertain | 5/10 | Proven links only | 10/10 |
| Map: usable view | Hairball | 5/10 | Navigable, described on click | 10/10 |
| Map: repeatable | One-off file | 3/10 | Same input, same map | 10/10 |
| Criterion | OpenAI's 6.1 Sol | OpenAI score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Real defects found | Pass, three found | 9/10 | Every defect surfaced, with its line | 10/10 |
| Criterion | OpenAI's 6.1 Sol | OpenAI score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Posting logic and quirks | Pass on the cases run | 9/10 | Proven on every path | 10/10 |
| Invalid digits | Fail, garbage posts as 0 | 3/10 | A data exception, as on the mainframe | 10/10 |
| Arithmetic | Pass, hand-written per statement | 9/10 | One exact rule everywhere | 10/10 |
| Timestamp layout | Pass, copied by a formatter | 9/10 | The view as defined | 10/10 |
| FILE STATUS and decode | Fail, four fixed codes | 3/10 | The real status for every outcome | 10/10 |
| I/O failure behaviour | Partial | 4/10 | As the COBOL | 10/10 |
| Java form | Fail, raw byte offsets | 2/10 | Named types and fields, traced to the COBOL | 10/10 |
| Criterion | OpenAI's 6.1 Sol | OpenAI score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Fields as columns | 0%, one bytea per row | 0/10 | Every field typed | 10/10 |
| Keys, order, alternate index | Pass | 8/10 | Typed keys, mainframe order | 10/10 |
| Load | Pass, unchecked | 7/10 | Every field checked | 10/10 |
| Criterion | OpenAI's 6.1 Sol | OpenAI score | Codegiant | Codegiant score |
|---|---|---|---|---|
| POSTTRAN step, DDs, return codes | Pass, hand-written | 9/10 | Generated from the JCL | 10/10 |
| Reject generation | Pass | 9/10 | Pass | 10/10 |
| Criterion | OpenAI's 6.1 Sol | OpenAI score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Proof method | Pass, local compiler only | 8/10 | Plus production replay | 10/10 |
| Decision coverage | 65% | 5/10 | Every outcome | 10/10 |
| Bad data and faults | 0% | 2/10 | Every boundary and fault | 10/10 |
| README, NOTES, rerunnable proof | Pass | 9/10 | Complete, with a certificate per program and job | 10/10 |
| Criterion | OpenAI's 6.1 Sol | OpenAI score | Codegiant | Codegiant score |
|---|---|---|---|---|
| One-click deployment | Not delivered in this run | 0/10 | One-click deployment to AWS, Google Cloud, Azure, Oracle Cloud, DigitalOcean, Cloudflare or Fly.io, on-premises, or Kubernetes | 10/10 |
| Managed database deployment | Schema and load supplied; managed database deployment not delivered | 0/10 | Tables go to the client's own managed database, deployed with the app | 10/10 |
| Included DevSecOps | Integrated toolchain not delivered | 0/10 | Git, CI/CD, Terraform, code review, docs, observability, error and performance monitoring, uptime checks and status pages included | 10/10 |
| After go-live / AI SRE | Ongoing operations service not delivered | 0/10 | 24/7 AI SRE monitors the system, repairs incidents, tunes cost and scaling, catches security drift and maintains runbooks | 10/10 |
| Integrations and automation | Integrated automation platform not delivered | 0/10 | Kronos Automate connects to 3,000+ services, with a dedicated automation tenant per modernized app | 10/10 |
| Build on the modernized system | App, agent and workflow builders not delivered | 0/10 | App builder, workflow builder and automation engine included for new apps, agents and workflows on the transformed code | 10/10 |
| Platform governance | Platform access controls, audit trails and policy enforcement not delivered | 0/10 | Role- and attribute-based access, audit trails and policy enforcement | 10/10 |
| Purpose-built models and private hosting | Purpose-built, self-hosted modernization and automation models not delivered | 0/10 | Kronos models purpose-built for mainframe modernization and automation, self-hostable inside the client's perimeter | 10/10 |
The comparison by area.
| Area | OpenAI's Sol | Codegiant |
|---|---|---|
| Analysis | Counts and basic facts pass; DD bindings reach 21%, program-to-file links 14% and copybook use 0% in the analysis. | The parser resolves inventory, layouts, DD bindings, copybook use and calls to source lines. |
| Dependency map | Basic links and cited lines pass; 148 edges are uncertain, utility suppliers are missed and only 13% of job hand-offs are certain. | The deterministic graph includes producers, consumers and conditions, with source lines behind the relationships. |
| Java conversion | Posting and quirks pass on tested cases, but invalid digits post as zero, status decoding is absent and Java uses raw byte offsets. | Named types and fields preserve number, status and error behaviour, with blocks traced to COBOL. |
| PostgreSQL data | Keys, order and alternate index pass; each record is stored in one bytea column and loaded unchecked. | Copybook fields become typed columns, mainframe order is kept and records are checked before cutover. |
| POSTTRAN | Step, DDs, reject generation and return codes pass in hand-written job handling. | The job is generated from JCL, including catalogue behaviour and return codes. |
| Proof | Outputs are compared with a local compiler; decision coverage is 65%, with no I/O faults or bad-data cases triggered. | Decision outcomes, error paths and field boundaries are checked, alongside captured production replay. |
| Platform beyond the prompt | The eight integrated platform capabilities were not delivered in this run. | Deployment, managed database deployment, DevSecOps, AI SRE, integrations, builders, governance and private hosting are included. |
| Beyond the demo | The brief identifies further issues around context, uniformity, site settings, production data, batch windows, operations, failures, proof and scale. | The nine detailed sections compare those risks with estate-wide definitions, mainframe behaviour, restart, independent proof and production operations. |
The checks behind each part of the brief.
| Fact family | GPT-6.1 Sol | Codegiant |
|---|---|---|
| Inventory | Pass. Counts right, but they come from reading the folder, not from a parse | Every program, copybook, job, PROC and data file, counted from the parsed source |
| Record lengths and keys | Pass. Typed by hand from the copybooks into a table | Every record and key taken from the copybook and the DEFINE, deterministic |
| Steps and what each runs | Pass. The EXEC lines copied out of the JCL | Every step resolved to its program or utility, with PROC steps expanded |
| Datasets | Pass. Names copied as text | Every dataset as a node in the graph, with its data and index parts |
| Which DD binds which dataset | 21%. Continuation lines pasted as bare DSN= text with the DD name lost | Every reference bound to its DD, step and line |
| Disposition per DD | 39%. The rest left as "see the original JCL" | Every disposition resolved per DD, defaults included |
| Program to file links | 14%. Only the six POSTTRAN files; the rest in prose | Every SELECT tied through its DD to the dataset, with the open mode |
| Copybook use | 0%. No copybook-to-program link at all | Every COPY link, so a layout change shows every program it touches |
| Calls | 80%. The abend routine used by 11 programs is not named | Every call site, internal and system |
| Job order through data | Partial. Prose chains, and one statement contradicts its own table | Every hand-off between jobs, with the producing and consuming lines |
| Check | GPT-6.1 Sol | Codegiant |
|---|---|---|
| Source lines behind each link | Pass. Every cited line checks out | Every node and edge, with its line |
| COPY, CALL, step to program, program to dataset | Pass. The basic links are complete, with generations kept | Every link built deterministically, and rebuilt the same way on every change |
| Job order through data | 13% certain. Most hand-offs flagged "uncertain" | Every hand-off stated, with both lines |
| Datasets created or loaded by utilities | Missed as suppliers. The GDG definitions, backup and reload jobs ahead of POSTTRAN are not linked | Every producer, utility or program |
| IDCAMS conditions | Listed only. No link from a condition to the command or return code it gates | Every condition tied to what it gates |
| Guessed links | 148 uncertain edges. A program with no job gets datasets matched by DD name from other jobs | Proven links only |
| Usable view | Hairball. Fields, paragraphs and spool lines in one graph; the drawn overview covers one job | Every job, step and dataset navigable, with a description on click |
| Repeatable | One-off. A model's file; the next run can differ | Same input, same map, every time |
| Construct | GPT-6.1 Sol | Codegiant |
|---|---|---|
| Validation and posting | Pass. Identical on the cases it ran | Identical, proven on every path |
| Quirks | Pass. All kept | All kept, because every statement is converted as written |
| Signed amounts | Pass on clean data. Sign read from a hand-written nibble test | Every sign rule of the compiler reproduced exactly |
| Invalid digits | Fail. Spaces read as 0, and the bad data posts | A data exception, as on the mainframe |
| Arithmetic | Pass. Truncation written by hand at each statement | Exact decimal arithmetic from the target picture, one rule everywhere |
| Timestamp REDEFINES | Pass. A Java formatter that copies the layout | The view over the same bytes, exactly as defined |
| FILE STATUS and the status decode | Fail. Four fixed codes; the decode branch was never converted | The real status for every outcome, decode included |
| I/O failures | Partial. Four codes; one path stops where the COBOL continues | The same behaviour as the COBOL on every failure |
| Java form | Fail as modernization. Raw byte offsets, no field names | Named record types and fields, every block traced to its COBOL line |
| Check | GPT-6.1 Sol | Codegiant |
|---|---|---|
| Fields as columns | 0%. Each record dumped into one bytea | Every copybook field as a typed column |
| Keys and order | Pass. EBCDIC byte order on a generated key | Typed keys, mainframe order kept |
| Alternate index | Pass. An expression index on the bytes | Same lookups and the same order as the VSAM path |
| Load | Pass. Bytes copied in unchecked | Every record verified before cutover |
| Check | GPT-6.1 Sol | Codegiant |
|---|---|---|
| Step, DDs, program | Pass. Written by hand in a shell switch | Generated from the JCL |
| DALYREJS(+1), NEW,CATLG,DELETE | Pass | Catalogue behaviour as on z/OS |
| Return codes | Pass | As on z/OS |
| Check | GPT-6.1 Sol | Codegiant |
|---|---|---|
| Method | Pass. Full output compared against a local compiler | Full output compared, plus replay of captured production runs |
| Decision coverage | 65%. Happy paths and the four reject reasons only | Every decision outcome |
| Error paths | 0%. No open, read, write or close failure was ever triggered | Every error path triggered and compared |
| Bad data | 0%. No invalid digits, odd signs or LOW-VALUES | Field boundaries, a census of the real data, missing and locked datasets |
Where AI modernization breaks at estate scale.
| Asked-items subtotal | 204/350 (58%) | 350/350 (100%) | |||
|---|---|---|---|---|---|
| Added platform subtotal | 0/80 (0%) | 80/80 (100%) | |||
| Combined total | 204/430 (47.4%) | 430/430 (100%) |
| Where it breaks | What it costs | Codegiant |
|---|---|---|
| Context runs out | Program 400 is converted without what was learned on program 4 | Nothing depends on what a model remembers |
| The same input gives a different output | Rerun a program and the Java changes; nothing can be signed off twice | Same input, same output, every time |
| Every session decides again | Layouts, names, types and error handling drift from prompt to prompt | Decided once for the estate |
| Nobody can review it all | Thousands of hand-shaped programs, each a little different | Uniform by construction; nothing hand-shaped to review |
| The model grades its own work | The same model writes the code, the tests and the verdict | Proof is independent of the writer |
| Where it breaks | What it costs | Codegiant |
|---|---|---|
| One copybook, many conversions | The same record reads differently in different programs, and each passes its own tests | One answer for every program that shares it |
| Numbers, signs and views handled case by case | Totals, rounding and REDEFINES behave differently depending on which prompt wrote which method | One behaviour across the estate |
| Schema drift between waves | Wave 2 meets wave 1's tables and does not fit; rework mid-program | Every wave lands on the same design |
| Where it breaks | What it costs | Codegiant |
|---|---|---|
| Behaviour set outside the code | Silent differences in arithmetic, signs, characters and file handling | Captured and applied per client |
| Source that isn't what runs | A faithful conversion of the wrong program | Caught before conversion |
| Where it breaks | What it costs | Codegiant |
|---|---|---|
| Real data breaks the rules | Bad values post as zero, or the migration weekend stops at record 3 million | Every exception found and reported before cutover |
| Bytes no program names | Lost on the first rewrite, and nobody notices for months | Preserved |
| Order | SQL returns rows in a different order than VSAM did, and every control break moves | Mainframe order on every read |
| Generations | (0), (+1) and (-1) collapse into one table | Every generation side by side |
| Where it breaks | What it costs | Codegiant |
|---|---|---|
| Chatty, unbatched access | Fine on a 300-record sample, hours late on production volume | Engineered for production volume |
| A job dies halfway | Starts again from zero, or posts the same transactions twice | Restarts where it stopped, never posts twice |
| Two runs at once | Two copies of a job update the same rows | Prevented, as on the mainframe |
| Records stored as blobs | No index, query or report without decoding bytes | Typed, queryable tables |
| Where it breaks | What it costs | Codegiant |
|---|---|---|
| Jobs written by hand | Every JCL or parameter change becomes a code change and a release | Every job converted into a Java batch job that does everything its JCL did, and keeps working when the JCL changes |
| Step logic simplified | Steps run that should be skipped; files kept, deleted or catalogued at the wrong time | Full JCL coverage: step conditions, file dispositions, generations, parameters and in-stream data |
| Utility and console steps dropped | Sol dropped all six online-file close and open steps; the utility steps were hand-coded or partly ignored | The utilities run as real steps: file definition and copy, sort, delete and create, conditional return codes |
| Return and abend codes changed | The scheduler reacts to codes it has never seen | The scheduler sees what it sees today |
| Where it breaks | What it costs | Codegiant |
|---|---|---|
| Error paths converted loosely or not at all | The first real I/O error in production behaves unlike the last 20 years | Same status, same message, same outcome |
| Invented or "fixed" behaviour | Checks, messages and abends the COBOL never had, or quirks quietly corrected | Nothing added, nothing removed |
| Where it breaks | What it costs | Codegiant |
|---|---|---|
| A local compiler and happy-path inputs | The paths that hurt in production are never run | Proven against production behaviour, on the inputs that hurt |
| No trace, no reconciliation, no sign-off | Auditors get a chat log | Every line traceable; a certificate per program and job |
| Requirement | AWS Transform for mainframe | AWS score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Runtime | The generated code imports their runtime. Remove it and it won't compile | 0/1 | No proprietary runtime. It's standard Java and compiles anywhere | 1/1 |
| Data fields | Your data is stored in AWS's own custom classes, which only their runtime understands | 0/1 | Your data uses ordinary Java types any developer can read | 1/1 |
| Program shape | Each COBOL paragraph becomes its own method, so the code still reads like COBOL | 0/1 | Methods are named for what the business does, like postTransaction() | 1/1 |
| AWS Reforge | Their fix is an AI rewrite applied afterwards, and anything it can't convert is left for you | 0/1 | The Java is structured correctly from the start, and every piece is checked before it's kept | 1/1 |
| The mainframe inside | The app still runs like a mainframe internally, using the mainframe's own EBCDIC character encoding in memory | 0/1 | Normal Java in memory. The mainframe character encoding is handled only where files come in and go out | 1/1 |
| Changing it later | Your COBOL is first rewritten into their engine's private language, and only their engine can turn it into Java | 0/1 | Every step is our own code, with no proprietary binary or engine, so you can easily update or change it later | 1/1 |
| Outside the code | Settings like number handling and encoding are properties on their runtime, set by hand | 0/1 | The same COBOL behaves differently at two companies, so we read all of it from the client's own files and build it into plain Java | 1/1 |
| Requirement | AWS Transform for mainframe | AWS score | Codegiant | Codegiant score |
|---|---|---|---|---|
| VSAM | Each record is stored as raw, unreadable bytes inside the database | 0/1 | Records become real database tables with real columns and indexes | 1/1 |
| Using the data | You can't query a balance directly. It's buried in a blob only their runtime reads | 0/1 | Any reporting tool or new service reads the data directly | 1/1 |
| Write safety | Updates are batched and written about a second later | 0/1 | Every write reaches the database immediately | 1/1 |
| Requirement | AWS Transform for mainframe | AWS score | Codegiant | Codegiant score |
|---|---|---|---|---|
| SQL handling | Your SQL is pulled out into lookup files that only their runtime can execute | 0/1 | Your SQL is translated into normal database code inside the generated classes | 1/1 |
| One model for files and tables | Files and database tables are handled by two separate mechanisms inside their runtime | 0/1 | Every file call and every database call returns the same kind of result | 1/1 |
| One database, one commit | VSAM data and DB2 data still live in separate worlds | 0/1 | Both end up in PostgreSQL, so a file update and a table update can commit together | 1/1 |
| Every SQL call tracked | No per-call check against the original | 0/1 | Every SQL call has an identity checked against the Java | 1/1 |
| Requirement | AWS Transform for mainframe | AWS score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Jobs | Batch jobs become scripts that only run on their runtime | 0/1 | Standard Java jobs, with high-throughput job queueing already built into the transformed output, and started by whatever scheduler you already use | 1/1 |
| Utility steps | Sort and file-management steps are pasted in as text for their runtime to interpret | 0/1 | Those steps become ordinary batch steps, with the same results as the mainframe | 1/1 |
| Restart | Restart depends on their runtime | 0/1 | The checkpoint is saved in the same commit as the data, so a rerun never posts twice | 1/1 |
| Requirement | AWS Transform for mainframe | AWS score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Where it runs | Documented only on AWS servers | 0/1 | Any of 7 clouds, on-premises, or Kubernetes | 1/1 |
| Runtime fee | $0.31 per CPU core per hour, forever, on top of the servers | 0/1 | Pure Java output. No runtime to license or pay for | 1/1 |
| AWS can change the rules | AWS has already closed two of its own mainframe runtime products to new customers | 0/1 | Nothing in your system depends on a vendor's roadmap | 1/1 |
| Tools needed | You need several separate tools, downloaded and set up individually | 0/1 | One platform, from analysis through to deployment | 1/1 |
| Speed | Nothing checks that the converted jobs run as fast as the original | 0/1 | Every job is timed against the mainframe's batch window and fails if it's slower | 1/1 |
| Requirement | AWS Transform for mainframe | AWS score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Where AI sits in the pipeline | A rules engine that copies COBOL's structure, then optionally an AI rewrite over whole classes, with failures left for you | 0/1 | Rules do everything with one right answer. AI only writes decisions, inside a locked skeleton | 1/1 |
| What the rules decide before AI is involved | The rules engine copies COBOL's structure, and the rest is left to the AI rewrite | 0/1 | Everything: parser facts, technical analysis, data classes, access classes, schema, job plans, function slices, skeletons, names and variable maps, no AI involved | 1/1 |
| What every line has to pass | No per-line check against the original, and no check that the rewrite invented nothing | 0/1 | Compilation, real methods, allowed constructs, every field written, conditions the right way round, every number traced to source. Two repair rounds, then it is set aside | 1/1 |
| Requirement | AWS Transform for mainframe | AWS score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Analysis without AI | Their documentation and business rules are generated by AI | 0/1 | A full technical document built from the code itself, every fact tied to a source line, no AI involved | 1/1 |
| Statement-level facts | None | 0/1 | Every COBOL statement is captured with its real operands | 1/1 |
| Business logic grounded in facts | None | 0/1 | The AI narrates only facts already deterministically proven, and a separate judge catches anything it drifts into | 1/1 |
| Business functions from outputs | None | 0/1 | Each function is traced backwards from what the program actually produces | 1/1 |
| Estate graph with line-level proof | AWS Transform gives you graphs and domains | 0/1 | Click any link and see the exact source line behind it, and the conversion decision it drove | 1/1 |
| Migration plan as a calculation | Domain splitting and wave planning | 1/1 | Groups are formed where nothing inside a group is written from outside it | 1/1 |
| Dead code checks | None | 0/1 | The source is checked against what actually runs in production. Dead code is flagged rather than converted | 1/1 |
| Requirement | AWS Transform for mainframe | AWS score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Equivalence | Their compare tool diffs files and tables, and you set up and run it yourself | 1/1 | The Java and the original COBOL run on the same input and are compared automatically, byte for byte | 1/1 |
| Test data | Generated scripts you run on the mainframe to collect data | 1/1 | Real production inputs and outputs captured and replayed, plus edge cases built from the copybooks | 1/1 |
| Business totals | No control-total check | 0/1 | Record counts and balances must match what the COBOL already computes | 1/1 |
| Hallucination | Their AI rewrite comes with no check that it invented nothing | 0/1 | Anything in the Java without a basis in the COBOL is flagged | 1/1 |
| Known bug patterns | None | 0/1 | A checker catches the classic conversion mistakes | 1/1 |
| Cutover | A test environment inside your AWS account | 0/1 | A parallel run on live traffic, old and new side by side, before anything switches over | 1/1 |
| Requirement | AWS Transform for mainframe | AWS score | Codegiant | Codegiant score |
|---|---|---|---|---|
| One-click deploy | Deploys to AWS only | 0/1 | Deploys in one click to AWS, Google Cloud, Azure, Oracle Cloud, DigitalOcean, Cloudflare or Fly.io, or on-premises, or any Kubernetes infrastructure you run | 1/1 |
| Database | Your data lives in their proprietary store, tied to their runtime | 0/1 | Your tables go to your own managed database, deployed with the app | 1/1 |
| DevSecOps | AWS Transform hands you templates and you build the rest yourself | 0/1 | The whole toolchain is included, nothing to assemble | 1/1 |
| After go-live | Their offering stops once the code is delivered | 0/1 | An AI operations assistant watches the system around the clock, repairs incidents and tunes cost | 1/1 |
| Connect | None | 0/1 | Kronos Automate, Codegiant's own LLM, connects to 3,000+ services so you can build automations and workflows on top of your modernized transformation. Each modernized app gets its own dedicated automation tenant, running an enterprise-grade engine that supports hundreds of millions of events per month | 1/1 |
| Build | None | 0/1 | New apps, agents and workflows on the transformed code, with the app builder, workflow builder and automation engine included | 1/1 |
| Governance | None | 0/1 | Role- and attribute-based access, audit trails and policy enforcement | 1/1 |
| Purpose Built Models | None | 0/1 | Kronos models purpose-built for mainframe modernization and automation, which can be self-hosted inside your perimeter for added safety | 1/1 |
| Requirement | Anthropic's COBOL plugin | Anthropic score | Codegiant | Codegiant score |
|---|---|---|---|---|
| How much gets converted | after 222 AI agents, 1 of the 15 programs was converted. | 0/1 | every program in the system goes through the same pipeline and the same checks in one run. | 1/1 |
| Nothing gets lost | the Java is rewritten from a description of the program, not from the program, so there’s no way to see what was left out. The program’s error-reporting routine and seven error checks disappeared, and nothing flagged it. | 0/1 | readable Java, plus a ledger proving every line of COBOL ended up somewhere in the Java. | 1/1 |
| Requirement | Anthropic's COBOL plugin | Anthropic score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Reading the records | the code that turns mainframe records into Java values is hand-written for this one program. Sharing it with the next program is left as a to-do. | 0/1 | one compiler produces the Java record, the code that reads and writes it, and the database table, all from the same layout, for every program. They can’t disagree. | 1/1 |
| The database design | each converted program creates its own version of the tables it uses. | 0/1 | one table design per file, created once and versioned. | 1/1 |
| Requirement | Anthropic's COBOL plugin | Anthropic score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Counting what you have | the AI wrote the inventory and got it wrong. It counted 9 VSAM files where there are 11 and 8 generation groups (GDGs) where there are 7, and it predicted 13 rejected transactions where the run had 38. | 0/1 | the inventory comes from a parser, not an AI, and every fact points to the line of code it came from. Counted, not guessed. | 1/1 |
| Checkable facts | nothing it says about the program is stored in a form anyone can check. | 0/1 | every statement, field and path in the program is recorded with its line number. For this one program that’s 339 statements and 128 fields. | 1/1 |
| Shared layouts | its map of the system leaves out copybooks (the shared record layouts) entirely. | 0/1 | every copybook and every place it’s used is on the map. Change a layout and you see every program it touches. | 1/1 |
| A map you can trust | its map of the system disagrees with itself. It says five programs read the transaction file in one place and seven in another. And it says one program updates a file it never touches. | 0/1 | every link on our map comes from a real line in the JCL or the program, and you can click it to see that line. | 1/1 |
| How jobs hand off to each other | the map loses which file generation each job reads and writes, and which steps only run if an earlier step succeeded. | 0/1 | generations and step conditions are kept for every job step. | 1/1 |
| Requirement | Anthropic's COBOL plugin | Anthropic score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Finding all the rules | rule extraction stopped at its built-in limit while it was still finding new rules. | 0/1 | rules are written from the proven facts of each program, with no limit to run out of. | 1/1 |
| Rules that agree with the code | two of its rules say the transaction file is wiped at the start of every run. Its own Java does the opposite: it refuses to start unless the file is already empty. | 0/1 | a separate checker tests every rule against the code it describes. A rule that disagrees doesn’t ship. | 1/1 |
| Keeping the quirks | it spotted that the COBOL carries on with the previous record when a read fails, wrote that down, then built Java that stops instead. | 0/1 | business functions are traced backwards from what the program produces, so every path the COBOL takes is kept. | 1/1 |
| The plan | the migration plan is written by the AI from its own memory of the session. | 0/1 | the plan is calculated from the stored map of the system, so anyone can regenerate it and get the same answer. | 1/1 |
| Requirement | Anthropic's COBOL plugin | Anthropic score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Records nobody changed stay unchanged | when it saves a record, it rewrites every field, including ones the program never touched, and changes how their signs are stored. Its own check proved this. | 0/1 | only the fields the program actually changed are written back. Everything else stays the same, byte for byte. | 1/1 |
| Your mainframe’s number settings | one fixed way of storing numbers, whatever your shop’s settings are. | 0/1 | number handling follows your own compiler settings. | 1/1 |
| Special “empty” bytes | mainframe LOW-VALUES are turned into a Unicode placeholder character, in the code and in the database. | 0/1 | the real bytes are kept, and stored as raw bytes only in the columns where they actually appear. | 1/1 |
| Characters and languages | one hand-built character table for US EBCDIC. | 0/1 | the character set comes from your settings, including the euro sign and Japanese double-byte text. | 1/1 |
| Fields that share space | the program reads the same two bytes in two ways to show the VSAM error code. That trick was dropped. | 0/1 | every shared-space layout (REDEFINES) is kept as two views of the same bytes. | 1/1 |
| Unused space in records | it assumes unused space starts as blanks. Its laptop test setup makes the same assumption, so the test agreed with itself. | 0/1 | taken from your run-time settings and checked against your real records. | 1/1 |
| Size limits on numbers | counters that wrap around on the mainframe never wrap in Java. | 0/1 | every number field keeps the mainframe’s size limits and arithmetic rules. | 1/1 |
| Loading bad data | one bad byte in one record stops the whole data load, without saying which record. | 0/1 | every record is checked first. Bad values are stored safely, nothing is skipped, and every problem is reported by field. | 1/1 |
| Sort order | the database sorts keys like a dictionary, not like the mainframe. | 0/1 | keys sort exactly as they did on the mainframe, so programs that read in key order get the same order. | 1/1 |
| Requirement | Anthropic's COBOL plugin | Anthropic score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Restarting after a failure | transactions are written one at a time, with no database transaction around them and no restart. A rerun posts again what was already posted. | 0/1 | work is saved in batches together with a restart point, so a rerun never posts anything twice. | 1/1 |
| File generations | the database has no idea of generations. | 0/1 | every table keeps generations, so today’s and yesterday’s versions live side by side. | 1/1 |
| Every kind of VSAM file | only one kind (keyed files). | 0/1 | all kinds: keyed, sequential (ESDS), numbered-slot (RRDS), variable-length, and alternate indexes. | 1/1 |
| Which file a program uses | the JCL’s file assignments become one database and two file paths. Pointing a run at test or production data is one typo away. | 0/1 | the JCL is compiled, so every file assignment works the way it did on the mainframe. | 1/1 |
| Generations of output | the rejects file, which the mainframe numbers automatically and keeps five copies of, becomes a file name the operator types in. | 0/1 | generations are numbered, kept and rolled off automatically, as on the mainframe. | 1/1 |
| Two runs at once | nothing stops the same posting job running twice at the same time. Its own reviewer said so. | 0/1 | files are locked the way the mainframe locks them, and the same job can’t run twice. | 1/1 |
| Steps in a job | the job becomes one Java application, so rules like “skip this step if the last one failed” are gone. | 0/1 | every step is kept, its return code decides what runs next, and utility steps (IDCAMS, SORT) run too. | 1/1 |
| What happens to files after a step | the program renames its own output file. | 0/1 | files are kept, cataloged or deleted at the end of each step, by the job, as on the mainframe. | 1/1 |
| Errors on files | when 3 of the files fail to open or 4 fail to close, there’s no handling at all, and one close is never checked. | 0/1 | every file action returns the same IBM status code the COBOL checks. | 1/1 |
| Messages for the night operator | 14 of the 26 messages are gone, including the ones that say which error happened and that the program is stopping. | 0/1 | every message is kept with its exact text. A missing one stops the build. | 1/1 |
| How a failure looks to the scheduler | the mainframe’s abend code (U0999) becomes exit code 16, which the scheduler’s rules don’t recognise. | 0/1 | the scheduler sees the same abend codes it always has. | 1/1 |
| Requirement | Anthropic's COBOL plugin | Anthropic score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Made-up behaviour | it added a crash the COBOL can’t cause, a check the COBOL never makes, a warning the COBOL never prints, and new message texts. | 0/1 | anything in the Java that has no source in the COBOL is flagged automatically. Claude doesn’t check for this at all. | 1/1 |
| The odd behaviour, kept | where the COBOL carries on after certain errors, the Java stops. | 0/1 | the same behaviour, odd parts included, because that’s what “the same” means. | 1/1 |
| Compiler settings | it never asks how your programs were compiled. None of its 5 setup questions covers it, and it assumes IBM’s defaults. | 0/1 | we read your compile listings and run-time settings (16 of them), because they change how the same code behaves. | 1/1 |
| Everything outside the code | it noticed files that come from outside the system, then didn’t ask about them. | 0/1 | we account for 125 things that change a program from outside its source: the file catalog, the data, the site setup, the scheduler and mainframe services. | 1/1 |
| The hardest program | one program changes its own control flow while it runs (ALTER) and reads the mainframe’s job tables directly. Claude said it “can’t be translated” and planned to rewrite it from the rules instead. | 0/1 | we convert it. The jumps are restructured, and the job information it reads is supplied from the actual run. | 1/1 |
| Programs its test can’t run | 4 of the 15 programs don’t compile on its laptop COBOL, so they can’t be compared at all. | 0/1 | every program is handled, with mainframe behaviour taken from IBM’s rules, not from a laptop compiler. | 1/1 |
| Requirement | Anthropic's COBOL plugin | Anthropic score | Codegiant | Codegiant score |
|---|---|---|---|---|
| What gets compared | the output files and 3 lines of the job log. The full log was saved and never compared. | 0/1 | every output file, return code, file status and log line. | 1/1 |
| What it’s compared against | an open-source COBOL on a laptop, with a stand-in for the mainframe’s abend routine. | 0/1 | real recorded runs from your mainframe. | 1/1 |
| The test data | sample data that couldn’t show its biggest bug, plus test cases it picked itself. Its bad-data test only hit records that were rejected before the bug could happen. | 0/1 | test data built from every field’s limits and from your real data, plus missing and locked files. | 1/1 |
| Who checks the work | its own report says the check ran in the same session that wrote the code. | 0/1 | the proof is completely separate from the conversion. | 1/1 |
| The verdict | it first reported “20 of 20 identical”. With new inputs, its own verdict became NOT PROVEN. | 0/1 | one verdict over every input, and nothing ships unproven. | 1/1 |
| Money and test strength | it breaks the code on purpose only twice to see whether the tests notice. | 0/1 | counts and money totals per file and per account, and every condition is tested by deliberately breaking it. | 1/1 |
| Who approved the changes | changes from the original are marked “approved” with no name. One schema choice was made by the AI against its own plan. | 0/1 | every change has a named approver, a date and the evidence. | 1/1 |
| Requirement | Anthropic's COBOL plugin | Anthropic score | Codegiant | Codegiant score |
|---|---|---|---|---|
| A real bank | by its own numbers, a large bank would need about 155,000 to 215,000 AI agents across 150+ separate runs, merged by hand. | 0/1 | no AI until the code is split into small pieces, and no limit on size. | 1/1 |
| Same answer twice | its own README says running the same command again can give different results. | 0/1 | every run gives exactly the same result. | 1/1 |
| Requirement | Anthropic's COBOL plugin | Anthropic score | Codegiant | Codegiant score |
|---|---|---|---|---|
| Helpers who each see one piece | the analysis was split across 97 helper agents, each reading a slice and reporting back a summary. A link between two slices survives only if a summary happened to mention it. That’s how the final map lost every copybook and miscounted who reads the transaction file. | 0/1 | links between programs come from the parser and the JCL compiler, across the whole system at once. No summary sits in between. | 1/1 |
| Programs stop knowing about each other after conversion | each program is converted on its own, with its own code for reading records and its own copy of the tables. What the next program needs to know is written as a to-do note in a text file. | 0/1 | one shared definition per file, used by every program that touches it. A program can’t read a record differently from the program that wrote it. | 1/1 |
| The hand-off between programs is never tested | each program is proven against its own COBOL, using test data it made itself. Nobody tests that what the posting program writes is exactly what the interest program reads the next night, and seven programs read the transaction file. | 0/1 | whole job chains run end to end in the proof (backup, then posting, then interest, then combine), with every hand-off compared. | 1/1 |
| Programs that call each other | called programs and the data they pass are not converted together. In this run, the statement program and the program it calls weren’t converted at all. | 0/1 | called programs, the data passed between them and their shared storage are compiled as one unit. | 1/1 |
| One brain, with a limited memory | one AI session runs the whole migration and holds the plan in its working memory. Even on this small system, the plan was written from that memory without re-reading the analysis files. A bank’s estate is thousands of times bigger, so the session has to summarise itself over and over, and later decisions are made from summaries of summaries. | 0/1 | nothing important lives in an AI’s memory. Every fact, decision and link is stored, and every stage re-reads it. | 1/1 |
| Requirement | Anthropic's COBOL plugin | Anthropic score | Codegiant | Codegiant score |
|---|---|---|---|---|
| One-click deploy | doesn’t deploy one-click anywhere. The conversion stops at Java in a folder. | 0/1 | deploys in one click to AWS, Google Cloud, Azure, Oracle Cloud, DigitalOcean, Cloudflare or Fly.io, or on-premises, or any Kubernetes infrastructure you run. | 1/1 |
| Database | your data lives in whatever database the conversion picked, and each converted program carries its own copy of the tables. | 0/1 | your tables go to your own managed database, deployed with the app. | 1/1 |
| DevSecOps | the plugin hands you Java in a folder, and you assemble the rest yourself. | 0/1 | the whole toolchain is included, nothing to assemble. | 1/1 |
| After go-live | the offering stops once the code is converted. | 0/1 | an AI operations assistant watches the system around the clock, repairs incidents and tunes cost. | 1/1 |
| Connect | MCP and connectors bring the tools you already use into the work: Google Drive, Gmail, Slack, Microsoft 365, Notion, Figma, HubSpot and more. | 1/1 | Kronos Automate, Codegiant’s own LLM, connects to 3,000+ services so you can build automations and workflows on top of your modernized transformation. Each modernized app gets its own dedicated automation tenant. | 1/1 |
| Build | none. | 0/1 | new apps, agents and workflows on the transformed code, with the app builder, workflow builder and automation engine included. | 1/1 |
| Governance | none. | 0/1 | role- and attribute-based access, audit trails and policy enforcement. | 1/1 |
| Purpose Built Models | only Claude can run it, and your source goes to the AI provider with usage reporting on by default. | 0/1 | Kronos models purpose-built for mainframe modernization and automation, which can be self-hosted inside your perimeter for added safety. | 1/1 |
Harness the power of enterprise-grade AI and thousands of connectors to build what’s next.