Manual vs Automated vs IT-Dependent Controls: Testing Each

Every control in a risk and control matrix carries a nature: manual, automated, or IT-dependent manual. The label looks administrative and is anything but, because it decides how the control can fail, how it is tested, how many items are sampled, whether general controls have to be relied on, and what evidence proves it operated. Misclassify a control and the test either proves nothing (a sample of 25 for a control the system performs identically every time) or proves the wrong thing (a test of one for a review whose quality depends on the reviewer). This guide sets out the three natures and the hybrid that causes most of the trouble, the testing implications of each, the classification decision tree, the errors that distort testing effort, and a worked reclassification at a bank’s SOX program, with a classification worksheet to copy.

In this guide

The three natures, defined by what can fail
Testing implications side by side
The classification decision tree
Testing manual controls: the sample and the reviewer
Testing automated controls: the test of one and its conditions
Testing IT-dependent manual controls: two tests, not one
The IPE dependency in detail
What the mix of natures tells you
Classification errors that distort testing effort
The control classification worksheet
Worked example: Lakeshore Bancorp reclassifies 212 key controls
Common mistakes
Related guides

The three natures, defined by what can fail

The cleanest way to define the natures is by their failure modes, because the test is designed around the failure. A manual control is performed by a person applying judgment or attention to information: a reconciliation prepared and reviewed, an invoice approved, a physical count, a contract read before signature. It fails when the person does not perform it, performs it carelessly, lacks the competence or authority to perform it, or is overridden; it can fail on any single occasion independently of the others, which is why it is sampled. An automated control is performed by a system according to its configuration without human intervention: a three-way match tolerance, a credit limit block, a mandatory field, an interface total check, a calculation. It fails when it is configured wrongly, when the configuration is changed without authorization, or when the program is altered; if it works once it works every time until something changes, which is why it is tested once and the general controls over change and access are tested for the period.

An IT-dependent manual control is a manual control whose performance depends on information a system produces: a manager reviewing an exception report, a controller analyzing a system-generated variance analysis, a reconciliation of a sub-ledger report to the general ledger, an access review performed on a user listing. It has both failure modes at once. The human part can fail like any manual control, and the system part can fail if the report is incomplete, inaccurate, run with the wrong parameters, or produced by logic nobody has verified, in which case a diligent reviewer reviews the wrong population and the control fails while appearing to operate. That double dependency is why the hybrid is treated as its own nature: it is tested as a manual control and as a piece of information produced by the entity, and skipping either half leaves the conclusion unsupported. A fourth pattern appears in practice, the automated control with manual follow-up, where the system identifies exceptions and a person resolves them; for testing purposes it is two controls, an automated identification and an IT-dependent manual resolution, and the matrix should list it as two.

Testing implications side by side

Dimension	Manual	Automated	IT-dependent manual
What can fail	Performance, care, competence, authority, override; each occurrence independent	Configuration, unauthorized change, program alteration; fails systematically once changed	Both: the human performance and the completeness, accuracy and parameters of the information used
Test of design	Walkthrough with the performer; evaluate precision and competence	Walkthrough of the configuration and the transaction flow; identify all code paths	Walkthrough of the review and of the report’s production, from source data to output
Test of operation	Attribute sample sized to frequency; inspect evidence of performance and of review	Test of one (or one per code path) plus configuration inspection; or benchmarking in later years where conditions hold	Attribute sample for the human part; report testing (logic, parameters, source, completeness and accuracy) for the IPE part
Typical sample	1 annual, 2 quarterly, 2 to 5 monthly, 5 to 15 weekly, 20 to 40 daily, 25 to 60 many times daily	One instance per relevant code path; more only if the ITGC reliance fails	Manual sample sizes as at left, plus the report tests, which are not sampled but performed on the report as used
ITGC reliance	None required, unless evidence of performance is itself system-generated	Required: change management, access to configuration, operations for the period	Required for the report writer, the source application and any customized query or spreadsheet step
Evidence of operation	Sign-off, review notes, follow-up of exceptions, reviewer’s re-performance where the control is a review	Configuration screenshots with dates, the test transaction and its result, change log for the period	Review evidence plus the report parameters, the query or logic, the reconciliation of the report to its source
Roll-forward	Fresh sample from the remaining period, sized by risk and by interim results	Inquiry and change-log review; re-test if the configuration changed	Both: fresh manual sample and confirmation the report logic did not change
Hours per control (order of magnitude)	Highest per instance; scales with frequency	Lowest once ITGCs are tested; ITGC cost shared across controls	Highest overall: manual hours plus report testing, often the largest line in a SOX budget

The sample sizes are the conventional ranges used by most programs and by the sample sizes guide on this site, which explains where they come from and when to move within the range. The automated column’s “one per code path” is the condition that most often surprises: a three-way match that treats purchase-order invoices and non-PO invoices differently has two paths, and a test of one covers one of them. The automated controls testing guide goes through the logic of the test of one, the benchmarking conditions in AS 2201 Appendix B, and what to do when the general controls the automated control depends on turn out to be deficient.

The classification decision tree

Classification is decided by asking, in order, who or what performs the control activity that actually prevents or detects the misstatement, and what the performer depends on. Five questions settle almost every case, and the matrix should record the answers rather than the conclusion alone, so that a reviewer can see why a control landed where it did.

Question 1: is the preventing or detecting action performed by a system with no human decision in the loop? If the system blocks, calculates, matches or flags and no person can alter the outcome for that transaction, the control is automated. Proceed to question 2 to check the boundary. If a person decides, proceed to question 3.
Question 2: does the system’s action complete the control, or does a person have to act on its output for the risk to be addressed? If the system’s action completes the control (a block, a forced value), it stays automated. If a person must act on what the system produces (an exception list, a flag, a report), split it: an automated identification plus an IT-dependent manual resolution.
Question 3: does the person’s decision rely on information produced by a system, in a way the person cannot independently verify? If yes, the control is IT-dependent manual, and every system-generated input is an IPE item to be tested. If the person works from source documents they inspect directly, or from information they generate themselves, it is manual.
Question 4: is a workflow or routing rule part of the control? A system-enforced approval routing (the requester cannot approve, the approver is selected by amount) is an automated control in its own right, listed separately from the manual approval judgment it enforces. Both are tested: routing by configuration and a test of one, approval by sample.
Question 5: which general controls does the classification now depend on? For automated and IT-dependent manual controls, list the applications, the report writers and the customized queries or spreadsheets involved, and confirm each is inside the ITGC scope. If one is not, the control is either brought into scope with its dependency or reclassified so the test does not rely on untested general controls.

Control as described in the matrix	Common classification	Correct classification and why
System prevents posting of a journal entry that does not balance	Automated	Automated: the system completes the action with no human decision
Controller reviews the monthly variance analysis generated from the general ledger and investigates variances over $50,000	Manual	IT-dependent manual: the review depends on a system-generated analysis whose completeness and thresholds are IPE
Three-way match with a 5 percent or $250 tolerance blocks invoice posting	Automated	Automated, with two code paths if non-PO invoices bypass the match; test one per path and the tolerance configuration
AP manager reviews the weekly report of invoices that failed the match and releases or rejects them	Automated	IT-dependent manual: the identification is automated (separate control); the release decision is a manual judgment on a system report
Quarterly user access review performed on a listing exported from the application	Manual	IT-dependent manual: the listing’s completeness is the whole control; test the export parameters and reconcile to the identity source
Warehouse manager signs the physical count sheets after supervising the count	Manual	Manual: performed on physical observation and source documents
Purchase orders over $25,000 route to the CFO for approval in the procurement system	Manual	Two controls: automated routing (configuration test) and manual CFO approval (sample)
Interface control totals between the billing system and the general ledger are compared by the system and differences logged	Automated	Automated identification plus IT-dependent manual follow-up of the logged differences, if anyone follows up; if nobody does, the second half is a missing control
Bank reconciliation prepared in a spreadsheet from a bank download and a GL export, reviewed by the assistant controller	Manual	IT-dependent manual with three IPE items: the bank download, the GL export and the spreadsheet’s own logic
Credit limit block in the order system, with overrides permitted by the credit manager	Automated	Automated block plus a manual override control; the override population is tested in full or sampled as a manual control

Testing manual controls: the sample and the reviewer

A manual control is tested by selecting instances across the period and inspecting evidence that the control was performed as designed on each. Three things separate a good manual test from a signature count. The attributes tested are the elements of the control description, not the existence of a sign-off: for a reconciliation, that the two sides were the right sources, that the reconciling items were identified, that they were resolved within the policy period, and that the reviewer looked at the items and not the total. The evidence of review is evidence of what the reviewer did, which for a review control means the questions they asked and the items they followed up, and where no such evidence exists the tester re-performs the review on the sampled instance and compares; the management review controls guide covers what review evidence has to look like and how precision is established. And the performer’s competence and authority are part of design: a control performed by someone who lacks the authority to reject, or the knowledge to notice, is deficient in design regardless of how the sample looks, which is the definition the deficiency evaluation guide works from.

Testing automated controls: the test of one and its conditions

An automated control is tested by proving the configuration is what the control description says, proving it operates on one transaction per code path, and relying on general controls to establish that nothing changed for the period. Each of those three legs has a condition. The configuration has to be inspected in the system, not in a design document, with the date, and every parameter that affects the control’s outcome recorded: tolerances, thresholds, the accounts or document types in scope, the exceptions list. The test of one has to exercise the control’s rejection or its action, which usually means a test transaction in a copy of production or a re-performance on a real transaction that met the condition, and there has to be one per path where the logic branches. And the general controls have to cover the specific application and the specific configuration: change management over the application, access to the configuration screens, and operations where the control depends on a scheduled job. Where the general controls are deficient, the test of one covers a point in time only, and the auditor extends the test to a sample across the period, as the ITGC versus application controls guide describes in its fallout rules. Benchmarking, under AS 2201 Appendix B paragraphs B28 to B33, allows the test of one to be carried across years for automated controls whose code has not changed, subject to conditions on change management and the ability to identify changes; it never applies to the manual halves of IT-dependent controls.

Testing IT-dependent manual controls: two tests, not one

The IT-dependent manual control is tested twice, and the two tests have different logic. The manual test samples the reviewer’s performance exactly as for a manual control, with the same frequency-based sizes and the same attributes. The information test is not sampled; it is performed on the report or data the reviewer used, as used, and it asks three questions. Is the report complete: does it include every item it should, which is proved by reconciling it to its source (a sub-ledger report to the sub-ledger control totals, a user listing to the identity store, an aging report to the receivables balance). Is it accurate: are the amounts, dates and classifications correct, which is proved by tracing a selection of items to source records and, for calculated fields, re-performing the calculation. And are the parameters right: the date range, the entity, the thresholds, the status filters, which is proved by inspecting the parameters on the report the reviewer actually received, not on a fresh run. Where the report is standard and produced by an application whose general controls are tested, the logic test can rely on those controls plus a validation of the standard report; where it is a custom query, a report-writer output or a spreadsheet, the logic itself is tested, because it is outside the general controls that were tested.

The PCAOB’s Staff Audit Practice Alert No. 11 of October 2013 remains the clearest statement of why the second test cannot be skipped. Its inspection observations described audits that tested a review control and its reviewer, but did not test the completeness and accuracy of the system-generated data the review used, and audits that tested the general controls over an application only to find that the report in question came from a customized query or a report writer that those general controls did not cover. The alert’s list of precision factors for review controls, the review’s objective, level of aggregation, consistency, correlation to the assertion, predictability of the expectation and the criteria for investigation, is the design test for the manual half. The IPE testing guide carries the full method for the information half; this section is the summary that fits on the classification worksheet.

The IPE dependency in detail

Each type of information produced by the entity carries a different test, and the matrix should name the type for each IT-dependent manual control so the test is planned rather than improvised.

IPE type	Examples	Completeness and accuracy test	Logic and parameter test	General control reliance
Standard application report	Sub-ledger aging, trial balance, standard exception report	Reconcile to control totals or the ledger; trace a selection to source	Validate the standard report once (vendor documentation plus a re-performance); confirm parameters on the instance used	Application change management, access and operations, tested for the period
Custom report or query	Report-writer output, SQL query, BI dashboard extract	As above, against the source tables	Inspect the query or report definition; re-perform on a sample or in full; test change control over the definition	Only if the report definition is under the same change and access controls as the application; usually it is not, so test the logic directly
Spreadsheet	Reconciliation workbook, allowance model, roll-forward schedule	Tie inputs to the source reports; foot and cross-foot	Inspect formulas and links; test version control and access; re-perform key calculations	None; spreadsheet controls (access, versioning, change log) are tested as part of the control
Interface or extract file	Bank download, payroll provider file, data warehouse extract	Reconcile record counts and totals to the source system; test rejected and suspended records	Test the extraction parameters and mapping; confirm the file version used	Interface controls and the provider’s SOC 1 report where the source is a service organization
Manually compiled information	Listing typed from documents, survey results, count sheets	Trace a sample to documents; test the compilation for omissions	Not applicable; test the compiler’s process and independence	None

Two consequences follow for planning. The first is that the number of distinct reports in a program is a better predictor of testing hours than the number of controls, because one report can serve six controls and one control can consume four reports; a report inventory built from the matrix, with each report tested once and cross-referenced, is the single largest efficiency available in a SOX program. The second is that the report inventory is the scoping input for ITGCs: the applications and report writers that produce key-control reports are the ones whose general controls must be in scope, as the SOX ITGC scoping guide sets out, and a report writer that nobody put in scope is the finding Alert 11 described.

What the mix of natures tells you, and how it moves

Once every control is classified, the distribution itself is information. A program whose key controls are mostly manual is expensive to test, fragile to turnover and slow to roll forward, and it usually reflects an environment where the systems could enforce more than they do. A program with a large IT-dependent manual share is normal for finance functions that live on reports, and its cost is dominated by the report inventory rather than by the controls. A program with a high automated share is cheap to test only if its general controls are strong, because every automated control’s assurance rests on them; when the ITGCs fail, the automated share becomes the most expensive part of the program overnight, as the fallout rules in the ITGC versus application controls guide describe. The distribution is therefore a lever management can pull, and internal audit can advise on: converting a manual approval into a workflow-routed one with a configured threshold moves the routing to the automated column and reduces the manual test to the judgment that remains; replacing a spreadsheet reconciliation with a system-matched one moves the IPE test from spreadsheet logic to a standard report; and each conversion has to be recorded as a control change, with a fresh design test, in the year it happens.

The natures also behave differently at interim and roll-forward, which is where a mis-classification is most often exposed. A manual control tested at interim needs a fresh sample from the remaining period, sized by risk and by what the interim sample found. An automated control needs an inquiry and a change-log review, with a re-test only if the configuration or the code changed, which is the efficiency that justifies the general-controls investment. An IT-dependent manual control needs both, and additionally needs confirmation that the report logic and parameters did not change between the interim instance tested and the year-end instances, because a report modified in November produces a year-end review over different information from the one tested in August. Programs that plan roll-forward by nature, rather than by a uniform “re-test everything” or “inquire on everything” rule, spend their fourth quarter where the risk is, and the classification worksheet’s roll-forward field is where that plan starts.

Classification errors that distort testing effort

Five errors account for most of the distortion, and each has a direction. Classifying an IT-dependent manual control as automated understates effort and overstates assurance: the program tests one instance of a review, never tests the report, and reports a control that was never evaluated. Classifying an automated control as manual overstates effort without adding assurance: twenty-five screenshots of a system doing the same thing prove nothing that one screenshot and the configuration did not, and the hours are taken from controls that needed them. Classifying an IT-dependent manual control as manual is the most common error and the most expensive to discover late, because the report tests are missing from the plan and surface in the external auditor’s request list in the fourth quarter. Treating a system-enforced routing and the human approval it routes as one manual control leaves the routing configuration untested, so a routing change that lets requesters approve their own orders goes unnoticed while the approval samples look clean; MidState Beverage’s route-cash audit found exactly that, a workflow routing rule that let the requester approve 1,412 of 37,400 overrides, and the segregation of duties guide covers the design side of the same failure. And classifying a monitoring control that operates at a high level of aggregation as a key preventive control over an assertion overstates the control’s precision and leaves the assertion without a control that could catch a misstatement below the review’s threshold.

The remedy is a classification review at the start of every cycle, performed by someone who did not write the control descriptions, using the decision tree and recording the answers. It takes a day or two for a program of two hundred controls, and it changes the testing budget more than any other planning step.

The control classification worksheet

One row per control in the matrix, completed at the start of the cycle and re-confirmed when a control changes. The fields are the decision tree’s answers plus the testing consequences, so that the worksheet feeds the test plan directly; the RCM template and the RCM Workbench carry the same fields.

Control classification worksheet

1. Control. Reference; description as written; process; assertion(s); frequency; performer (role) or performing system.

2. Decision tree answers. Q1 performer is a system with no human decision (Y/N); Q2 system action completes the control (Y/N) or requires human action on output (split required); Q3 human decision relies on system-generated information (Y/N, list the information); Q4 workflow or routing rule embedded (Y/N, describe); Q5 general controls the classification depends on (applications, report writers, queries, spreadsheets) and whether each is in ITGC scope.

3. Classification. Manual / automated / IT-dependent manual; split into component controls if Q2 or Q4 requires it, with new references.

4. IPE inventory. For each system-generated input: report name and type (standard, custom, spreadsheet, interface, compiled); source system; parameters; report owner; test reference (once per report across the program).

5. Testing approach. Design: walkthrough scope. Operation: sample size by frequency (manual and IT-dependent manual); code paths and configuration items (automated); benchmarking eligibility (automated, subsequent years); ITGC reliance and the ITGC test references; roll-forward approach.

6. Effort. Estimated hours for design, operation, IPE and roll-forward; change from prior year with reason.

7. Sign-off. Classifier, independent reviewer, dates; control owner acknowledgment of the description and the split.

Worked example: Lakeshore Bancorp reclassifies 212 key controls

Lakeshore Bancorp, the illustrative $9 billion regional bank used across this site, carries 212 key controls in its SOX matrix. When the SOX program office ran the decision tree over the full matrix for the first time, the prior classification stood at 40 automated, 60 IT-dependent manual and 112 manual, a distribution that had been inherited from the program’s first year and adjusted only when controls were added. The exercise took the program manager and one senior from internal audit three days, with the control owners consulted on the descriptions, and it produced the classification the program now uses: 68 automated, 91 IT-dependent manual and 53 manual. Every one of the 59 changes had a decision-tree answer recorded against it, and the three categories of change told the story of where the old matrix had been wrong.

Change	Count	Typical control	Testing consequence
Manual to IT-dependent manual	44	Reconciliations prepared from sub-ledger exports; reviews of system-generated exception and aging reports; access reviews on application listings	Report inventory grew from 31 reports to 74; 43 new IPE tests, most performed once and shared across controls; manual sample sizes unchanged
Manual to automated (or split with an automated component)	19	Workflow-routed approvals where the routing had been treated as part of the approval; interface total checks; system-enforced posting restrictions	19 configuration tests and tests of one replaced 19 samples of 25 to 40; nine controls split into an automated and a manual component
Automated to IT-dependent manual	7	Exception-flagging controls where the system identified items and a person decided; described as automated because the flag was	Seven new manual samples plus IPE tests on the flag reports; three of the seven had never had their resolution step tested
IT-dependent manual to automated	3	Calculations described as reviewed that were in fact system-enforced with no review performed	Descriptions corrected; the non-existent review removed from the matrix rather than tested
Re-split of composite controls	net +14 controls before consolidation, net 0 after	Controls that combined an automated block, a manual override and a review	Composite controls split for testing and re-consolidated for reporting so the key control count stayed at 212

The effort effect was not the one the program office expected. Management’s testing budget fell for the manual population, from 112 sampled controls to 53, and rose for the IPE work, because the 44 reclassified reconciliations and reviews had been tested for years without anyone testing the reports beneath them. The net was a reduction from roughly 2,260 planned testing hours to about 1,640 once the report inventory was consolidated (each of the 74 reports tested once and cross-referenced), with the saving concentrated in the automated reclassifications and the cost concentrated in the report tests. The more important effect was on assurance: the seven exception-flagging controls whose resolution step had never been tested included the wire platform’s daily exception queue, and the first sample of that queue found the resolution being performed by the same operations staff who initiated the wires, which became a segregation finding and, eventually, part of the year-end deficiency evaluation described in the deficiency evaluation guide. The external auditor accepted the reclassification with two adjustments, both in the direction of more testing: two custom queries the program had treated as standard reports were moved to direct logic testing.

Common mistakes

Classifying by where the control lives (in the system, therefore automated) instead of by who performs the preventing or detecting action. Treating “the system generates a report and someone reviews it” as automated because the report is. Sampling automated controls at manual sizes and calling the screenshots evidence. Testing the reviewer and not the report. Testing the report in a fresh run with clean parameters instead of the instance the reviewer used. Leaving customized queries and spreadsheets outside the general controls and assuming the application’s ITGCs covered them. Combining a routing rule and an approval into one manual control. Carrying forward last year’s classification because it was accepted last year. Building the report inventory from memory instead of from the matrix. And rating a classification error as a documentation matter when it left a key control untested for the year, which the finding severity ratings guide would rate on the assertion left uncovered, not on the paperwork. The design versus operating effectiveness guide covers the test-of-design half that this guide assumes, and the audit evidence guide the evidence standards the three natures draw on.

Comments

Leave a Reply Cancel reply

Dark Towers by David Enrich: Summary and Lessons for Auditors

Blind Spots by Bazerman and Tenbrunsel: Summary and Lessons for Auditors

Black Box Thinking by Matthew Syed: Summary and Lessons for Auditors

The Field Desk Series

Manual, Automated, and IT-Dependent Manual Controls: Testing Implications of Each