A control that operates twice a year gives the auditor two chances to see it work, and the usual sampling logic collapses. You cannot sample two. You cannot project from two. One failure is a 50 percent deviation rate, which no tolerable rate accommodates, and yet one failure in two might be an aberration or might be the whole story. The same problem appears, in milder forms, for quarterly controls (four instances), monthly controls (twelve) and the annual controls that are tested once because once is all there is: the access recertification, the budget approval, the SOC 1 review, the year-end physical count. Practitioners ask how to test these more often than almost anything else, and the public answers are either a table of numbers with no reasoning or a shrug.
This guide gives the reasoning and the method: why small populations break the sampling model, what the standards and the SOX conventions actually say, the full-population approach, design plus corroboration, using the control’s inputs and outputs as evidence, the period-coverage problem for new and changed controls, evaluating a failure in a population of two, workpaper language that survives review, three worked examples, and the mistakes that recur. It builds on the sample sizes and attribute sampling guides, which cover the large-population case, and on the management review controls guide, since most infrequent controls are reviews.
In this guide
- Why small populations break the sampling model
- What the standards and the SOX conventions say
- The full-population approach
- Design plus corroboration
- Inputs and outputs as evidence
- Event-driven controls and small transaction populations
- Automated controls: the test of one
- The period-coverage problem: new, changed and year-end controls
- Evaluating a failure in a population of two
- Workpaper language that survives review
- Three worked examples
- Mistakes that recur
- Questions about testing small populations
- Related guides
Why small populations break the sampling model
Attribute sampling answers a statistical question: given a sample with so many deviations, how confident can we be that the population’s deviation rate is below a tolerable level? The standard tables (25, 40, 60 items for the usual confidence and tolerable rates) work because the population is large enough that the sample is a small fraction of it and the binomial arithmetic applies. A population of two, four or twelve defeats this in three ways. First, any sample is most of the population, so the question is no longer inference but inspection: the auditor is examining the instances, not estimating from them. Second, the arithmetic of confidence does not improve by testing all of a tiny population; testing both instances of a semiannual control proves those two instances worked, not that the control is reliable in any statistical sense. Third, a single deviation produces a rate (50 percent in two, 25 percent in four, 8 percent in twelve) that would fail any tolerable threshold, so the deviation has to be evaluated on its facts rather than its rate. The practical consequence is that assurance over infrequent controls comes from different sources than sampling: from the quality of each instance (did the review actually happen, with evidence of what was reviewed and what was found), from the design (would this control catch the error it exists for, if performed as described), from corroboration (do the control’s inputs and outputs show it worked), and from the period’s outcomes (did the errors the control guards against occur anyway). The foundations of random sampling guide explains the statistics; this guide is about what to do when they do not help.
What the standards and the SOX conventions say
Neither the auditing standards nor the Global Internal Audit Standards prescribe sample sizes for controls by frequency; they require sufficient appropriate evidence and leave the judgment to the auditor. What exists is convention, developed in SOX practice over two decades and accepted by external auditors and regulators as a reasonable default: for a control operating annually, test the one instance; quarterly, test two of four (many firms test all four for key controls); monthly, test two to five (commonly two, three for key controls, with the additional instances where the control is complex or the population changed); weekly, five to fifteen; daily, twenty to forty; multiple times a day, twenty-five to sixty. The conventions carry two conditions that are often forgotten. They assume the control is not new and has operated throughout the period, so that the instances tested represent the control as it ran; and they assume the auditor evaluates design and inspects each tested instance for substance, not for the existence of a signature. External auditors testing internal control over financial reporting generally expect semiannual and quarterly controls to be tested at every instance (there is no sampling of four) and expect a rollforward or update procedure when the last instance tested is far from period end. The SOX 404 and SOX scoping guides describe the program these conventions live in; the table turns them into the function’s default, to be written into the methodology and applied with the adjustments below.
| Control frequency | Instances in a year | Conventional minimum test | Recommended for key or High-risk controls | Notes |
|---|---|---|---|---|
| Annual | 1 | 1 (the instance) | 1, with design evaluation and corroboration | Timing matters: test after the instance; rollforward if early |
| Semiannual | 2 | 2 (both) | 2, with corroboration | Never one; a single instance says nothing about consistency |
| Quarterly | 4 | 2 | 4 (all) | Test the quarters furthest apart if two; include the year-end quarter |
| Monthly | 12 | 2 | 3 to 5 | Spread across the period; include period-end months |
| Weekly | 52 | 5 | 10 to 15 | Random selection becomes meaningful |
| Daily | ~250 | 20 to 25 | 40 | Attribute sampling applies |
| Multiple times a day or per transaction | Thousands | 25 | 40 to 60, or full-population analytics | Standard tables; analytics often better |
| Event-driven (on hire, on change, on new vendor) | Varies | By the population’s size using the rows above | Full population if under 25 | Population must be established independently |
The full-population approach
For populations up to about twelve, the clean answer is to test every instance, and to spend the effort saved on sampling on the depth of each inspection. Testing all four quarterly reviews or all twelve monthly reconciliations removes the inference question entirely: the auditor concludes about the period’s actual operations, and a failure is a failure of a specific instance to be evaluated on its facts. The cost is usually modest, because infrequent controls are infrequent; a quarterly review takes four inspections rather than two, and the second two are often where the finding is (controls performed with care in the first quarter after an audit decay by the third). Full-population testing has one trap: it tempts the auditor to inspect lightly because there are many instances. The inspection of each instance should be the same as it would be for a single annual control: what was the control supposed to do, what evidence shows it did that (not a signature, but the reviewed data, the questions raised, the exceptions found and resolved), who performed it and did they have the competence and authority, and was it performed within the time the design requires. The walkthrough versus test of controls guide marks the difference between seeing a control once and testing its operation; for a small population, the test is a series of deep walkthroughs across the instances.
Design plus corroboration
When the instances are too few to carry a conclusion alone (one or two), the conclusion rests on three legs. Design evaluation: a documented judgment that the control, performed as described, would prevent or detect the error it exists for, with precision adequate to the risk (the control design principles guide’s tests: the right person, the right information, the right threshold, the right timing, a defined response). A control whose design would not catch the error does not become effective by operating twice. Instance inspection: each instance examined for substance, as above, with re-performance where possible (recompute the reconciliation, re-run the review on the same data and compare what the reviewer should have found with what they found). Corroboration: independent evidence that the control’s effect is visible elsewhere: exceptions the control identified and their resolution (a review that never finds anything is suspicious), the downstream outcome (no errors of the type the control guards against surfaced in the period from other sources, such as the external audit, customer complaints or later reconciliations), related controls that would have caught a failure, and the performer’s knowledge in interview (can they explain what they look for and what they have found). Together these let the workpaper conclude that the control is designed effectively and operated as designed in each instance observed, with corroborating evidence that it achieved its purpose in the period, which is as much as two instances can fairly support and more than a signature inspection pretends to. The audit evidence guide’s sufficiency and reliability standards are the test; corroboration is how a thin population is made sufficient.
Inputs and outputs as evidence
| Control type | Inputs to test | Outputs to test | Corroboration |
|---|---|---|---|
| Periodic management review (variance analysis, KPI review, reserve review) | The report reviewed: complete, accurate, the right period (the IPE testing guide); the threshold applied | Questions raised, explanations obtained, actions taken; evidence of the review’s depth (annotations, emails, follow-up) | Re-perform the review: would an attentive reviewer have found more than the reviewer did? Outcomes in the period |
| Reconciliation (monthly or quarterly) | Both sides’ sources complete; cutoff | Reconciling items identified, aged and resolved; sign-off within policy | Re-perform one reconciliation; check for items cleared without support |
| Access recertification (annual or semiannual) | The access population reviewed was complete (reconcile to the system) | Removals requested and actioned; timeliness | Compare access before and after; test a few retained accesses for appropriateness |
| Physical count (annual or semiannual) | Count instructions, cutoff, teams, independence | Count sheets, variances, adjustments booked | Observe or re-count a sample; compare variance trends |
| Vendor report review (SOC 1 annual) | The report obtained covers the right period and system; the gap to year end | Exceptions evaluated; complementary user entity controls mapped | Bridge letter obtained; related controls tested |
| Policy or model approval (annual) | The document approved was the current version; the approver had authority | Approval evidence; changes since approval | Check the policy in use matches the approved one |
| Budget or limit approval (annual, quarterly) | The proposal reviewed was complete | Approval recorded; subsequent operations within the approved limits | Compare actuals against the approved figures |
Event-driven controls and small transaction populations
Some controls have no frequency at all: they operate when something happens. A new vendor is approved; a new employee’s access is provisioned; a system change is tested and approved; a contract above a threshold is reviewed by legal; a journal entry above a limit is approved by the controller. The population is the set of events in the period, and the first task is to establish it independently, from the system that records the events rather than from the control owner’s list (the vendor master’s creation dates, the identity provider’s provisioning log, the change tool’s tickets), because an event the control missed is exactly the one the owner’s list will not show. Once the population is known, the frequency table applies by size: under 25 events, test them all; 25 to a few hundred, the conventional samples; thousands, attribute sampling or full-population analytics. Small transaction populations behave the same way: a control over the eight acquisitions in the year, the fourteen capital projects above $5 million, the six intercompany agreements, is tested on all of them, and the workpaper states the population and its source. The failure pattern here is a control that operated on every event the owner knew about and none of the events the owner did not, which only an independently established population reveals; the vendor master and IT change management guides show the independent sources for two common cases.
Automated controls: the test of one
An automated control (a system block on payments above authority, a three-way match, a validation that rejects an incomplete record, a calculation) operates identically every time by construction, so the population question inverts: one well-designed test proves the control for every instance, provided the system that performs it has not changed. The “test of one” (sometimes “benchmarking” when the result is carried across periods) rests on two conditions: a test that exercises the control’s logic, including the boundary (a transaction at the threshold, one just above, one just below, an incomplete record), in the production configuration or a verified equivalent; and effective IT general controls over change management and access, so that the configuration tested is the configuration that ran all year (the ITGC versus application controls and ITGC scoping guides). Where change management is weak, the test of one proves the control at a moment, and the auditor either tests the configuration at several points in the period (change logs showing no change to the relevant settings, or configuration snapshots) or treats the control as manual for sampling purposes. Many infrequent controls are IT-dependent rather than automated: a quarterly review of a system-generated exception report depends on the report’s completeness and accuracy, which is a test of the report’s logic and parameters (the IPE guide) performed once, plus the review instances tested as above. Separating the automated element (test once, with ITGC reliance) from the manual element (test the instances) makes the small-population problem smaller, because the part of the control that would have the largest population is the part that needs only one test.
The period-coverage problem: new, changed and year-end controls
Three timing situations complicate small populations. A new control (implemented mid-period) has fewer instances than its frequency implies, sometimes only one; the conclusion is limited to the period from implementation, the workpaper says so, and the design evaluation carries more weight. For SOX purposes, a control must have operated long enough before period end to be tested, which is why a quarterly control implemented in the fourth quarter is often not relied on for the year. A changed control (new performer, new system, new threshold) is two populations: the instances before and after the change are tested as separate controls, and a conclusion about the period requires both to be adequate or the pre-change control to be irrelevant to the conclusion. A year-end control (the annual count, the year-end reconciliation, the annual recertification) has not operated when fieldwork is performed in the autumn, so the auditor either waits, tests the prior year’s instance for design and the current year’s by rollforward (inspection after it occurs, before the report), or reports with the limitation stated. The closure evidence standards guide applies the same logic to remediated controls: a fixed quarterly control cannot be validated until it has operated at least twice, and the validation date is set by the control’s calendar, not the auditor’s.
Evaluating a failure in a population of two
One failure in two instances cannot be treated as a 50 percent deviation rate and cannot be waved away as a one-off; it has to be evaluated on its facts, and the control deficiency evaluation guide gives the structure. The questions: what exactly failed (the control was not performed; performed late; performed without the right information; performed but missed an error it should have caught; performed by someone without authority); why (the cause, and whether it would recur); what the consequence was or could have been (did an error pass, and how large could one have been); and what compensating controls exist (a later reconciliation, a second review) and whether they operated. A control not performed in one of two instances is, for that half of the year, a control that did not exist, and the auditor evaluates the period’s exposure accordingly. A control performed with a documented reason for a one-time lapse (a reviewer’s absence with a deputy who performed a lighter review) may be a deficiency in design (no adequate backup) rather than a conclusion that the control is ineffective. For SOX purposes the deficiency is evaluated for severity under the program’s framework; for an operational audit it becomes a finding under the severity ratings guide. What the auditor cannot do is extend the sample: there is no third instance to test, and “we tested one more and it passed” is not available. Where the population allows (monthly), a failure in a sample of two or three does justify extending to all twelve, which converts the question from inference to inspection and usually settles it.
Workpaper language that survives review
| Situation | Workpaper wording |
|---|---|
| Full population tested, no exceptions | “The control operated four times in the period (quarterly). We tested all four instances (100 percent of the population), inspecting [the evidence] and re-performing [the procedure] for each. No exceptions. Combined with our evaluation of design (WP ref) and corroborating evidence (WP ref: exceptions identified by the control and their resolution), we conclude the control was designed effectively and operated effectively throughout the period.” |
| Annual control, one instance | “The control operates annually and operated once in the period, on [date]. We inspected the instance for [attributes], re-performed [procedure], and evaluated design (WP ref). Corroboration: [outcome evidence]. We conclude the control operated as designed for the instance and is designed effectively; our conclusion on operating effectiveness is necessarily based on the single instance and the corroborating evidence described.” |
| Semiannual, both tested, one exception | “The control operated twice in the period. Instance 1 ([date]) operated as designed. Instance 2 ([date]) was not performed [or: was performed on [date], [n] days after the policy deadline, by [person] without the required authority]. Cause: [cause]. Consequence: [exposure, errors identified or not]. Compensating controls: [tested, WP ref]. Evaluation: [deficiency conclusion and rating]. The control cannot be concluded effective for the second half of the period.” |
| New control, partial period | “The control was implemented on [date] and operated [n] times from implementation to period end. We tested all [n] instances. Our conclusion covers the period from [date]; no control of this type operated before that date, and the exposure for the earlier period is addressed at WP ref.” |
| Year-end control tested by rollforward | “At the date of fieldwork the control had not operated for the current period. We tested the prior period’s instance ([date]) for design and operation (WP ref) and inspected the current period’s instance on [date] after it occurred, before issuance. [Result.]” |
| Sample of a monthly control, no exceptions | “The control operated twelve times in the period. We selected three instances ([months], including the period-end month) following the function’s sampling standard for monthly controls (WP ref), inspected [attributes] and re-performed [procedure]. No exceptions. Design evaluated at WP ref. We conclude the control operated effectively; the conclusion is based on the sample and design evaluation described.” |
Two habits make this language defensible. State the population and the proportion tested in numbers, so the reader knows whether the conclusion is inspection or inference. And name the evidence beyond the instances (design, corroboration) explicitly, so that a reviewer or external auditor sees the three legs rather than a conclusion resting on two signatures. The sampling memo template guide gives the memo form for the sampling decision and the workpaper example guide the surrounding file.
Three worked examples
| Control | Population and approach | What the auditor did | Result and conclusion |
|---|---|---|---|
| MidState Beverage: semiannual physical inventory count at the main distribution center, with variances investigated and booked | Two instances (June and December); both tested; design plus corroboration | Inspected count instructions, team assignments and cutoff for both counts; re-counted 40 bins at the December count by observation; traced variances to investigation and adjusting entries; compared variance rates across the two counts and the prior year; checked that counters were independent of custodians | June count complete and documented; December count’s variance investigation closed 11 variances over the threshold without documented explanation ($38,000 net). One instance operated as designed, one with a documented lapse in the investigation step; rated Medium with the cause (a new supervisor unaware of the threshold) and a compensating cycle-count control tested and found operating. Conclusion stated per instance, not as a rate |
| Lakeshore Bancorp: quarterly ALCO review of interest rate risk model assumptions (deposit betas, non-maturity deposit lives, prepayment speeds) | Four instances; all four tested (key control for a High-rated risk) | For each quarter, inspected the ALCO pack’s assumptions section, the minutes and the model documentation; tested that the assumptions presented matched those in the model run (input to output); re-performed the back-test comparison the committee was shown; checked that assumption changes were approved and effective in the next run; interviewed the ALCO chair on challenge | All four reviews held and documented; in Q3 the pack showed the prior quarter’s non-maturity deposit life while the model had been changed, so the committee reviewed stale assumptions for one quarter (the change was approved in Q4). Design effective; operation effective in three of four with a reporting lapse in one; finding on the pack’s production control rather than on ALCO. The IRR guide covers the assumptions themselves |
| Halberd Software: annual review of the payroll provider’s SOC 1 Type 2 report, with complementary user entity controls mapped and exceptions evaluated | One instance; design plus corroboration plus rollforward | Inspected the review memo for the current report (period covered, exceptions noted, CUEC mapping to Halberd’s controls); re-performed the CUEC mapping for the five controls relevant to payroll accuracy; obtained the bridge letter for the gap to year end; tested two of the mapped Halberd controls separately; evaluated design (who reviews, what they look for, what triggers action) | Review performed by the payroll manager, who mapped CUECs but did not evaluate two exceptions in the report that related to change management at the provider; corroboration found no payroll errors in the period attributable to the provider. Design adequate, operation incomplete in the single instance; Low-rated finding with the exception evaluation added to the review template. The SOC 1 review guide covers the method |
Mistakes that recur
| Mistake | Why it fails | Instead |
|---|---|---|
| Testing one of two semiannual instances | Says nothing about consistency; external auditors reject it | Test both |
| Signature inspection as testing | A signature proves someone signed | Inspect substance; re-perform; look for exceptions found |
| Applying a deviation rate to two instances | The statistics do not apply | Evaluate the failure on its facts |
| Skipping design evaluation | Two instances of a badly designed control prove nothing | Design is a leg of the conclusion |
| No corroboration | Conclusion rests on thin inspection | Exceptions found, outcomes, related controls, interviews |
| Ignoring the implementation date | A new control credited for the whole period | Conclude from implementation; address the gap |
| Testing early quarters only | Decay by Q3 missed; year-end instance untested | Include the latest instance; rollforward |
| Treating a changed control as one population | Pre- and post-change instances are different controls | Two populations, two conclusions |
| Extending the sample after a failure where no extension exists | There is no third semiannual instance | Evaluate; test compensating controls |
| Vague workpaper wording | Reviewer cannot tell inspection from inference | Population, proportion tested, legs of the conclusion, in numbers |
Questions about testing small populations
How many instances of a quarterly control should be tested?
The SOX convention is two of four; for key controls and High-rated risks, test all four, including the period-end quarter. Testing all four converts the question from inference to inspection and costs little. The instances tested should be inspected for substance and re-performed where possible, with design evaluation and corroboration documented alongside.
How do you test a control that operates once a year?
Test the one instance for substance (what was reviewed, what was found, who did it, when), re-perform the procedure where possible, evaluate the design, and corroborate with the control’s outputs and the period’s outcomes; if the instance has not yet occurred at fieldwork, test the prior year’s instance and roll forward by inspecting the current one before the report, stating the limitation.
What if one of two semiannual instances failed?
Evaluate it on its facts rather than as a 50 percent deviation rate: what failed, why, what passed through or could have, and whether compensating controls operated. The control cannot be concluded effective for the half of the period in which it did not operate; the deficiency is rated for severity under the program’s framework or the function’s rating scale.
Does sampling apply to monthly controls?
Only loosely: the convention is two to five instances spread across the period including period-end months, but with twelve instances a failure in the sample justifies testing all twelve, which is cheaper than arguing about inference. Design evaluation and corroboration still carry part of the conclusion.
What is “design plus corroboration”?
The three-legged conclusion for controls with too few instances to carry one alone: a documented evaluation that the control’s design would catch the error it exists for, inspection and re-performance of each instance for substance, and independent evidence that the control achieved its purpose (exceptions it found, outcomes in the period, related controls, the performer’s knowledge).
How should the workpaper describe the conclusion?
State the population size and the proportion tested in numbers, describe the inspection and re-performance, reference the design evaluation and the corroborating evidence, and phrase the conclusion to match what the evidence supports (per instance for failures; “based on the single instance and corroboration” for annual controls), so a reviewer can tell inspection from inference.
Related guides
- Audit sample sizes: 25, 40, 60 — the large-population conventions.
- Attribute sampling end to end — the method that stops working here.
- The foundations of random sampling — why.
- Judgmental sampling — selecting instances deliberately.
- Audit sampling memo template — documenting the sampling decision.
- Management review controls — most infrequent controls are reviews.
- IPE testing — the report the reviewer relied on.
- Control design principles — the design leg.
- Audit evidence — sufficiency and corroboration.
- Walkthrough vs test of controls — deep walkthroughs across instances.
- Control deficiency evaluation — the failure in a population of two.
- Finding severity ratings — rating the finding.
- SOX 404 — the program the conventions come from.
- SOX scoping — which controls are key.
- User access review — the annual recertification.
- SOC 1 report review — the annual vendor report control.
- How to audit account reconciliations — monthly and quarterly reconciliations.
- How to audit inventory — the physical count.
- IRR vs IRRBB vs SIRR — the ALCO assumptions in the example.
- Audit workpaper example — the surrounding file.
- Evidence standards for issue closure — the same timing logic for remediated controls.
Leave a Reply