In the back room of a credit union, during a new digital banking installation, people were stuffing checks into envelopes after getting their morning coffee. There was nothing strange about that. It's a normal task completed every morning by a single person. They take the checks from the check printer, stuff them into envelopes, and mail them out. Some checks go to members, some go to vendors, and some go elsewhere. The checks are created in the core system the day before and are assigned check numbers when they are printed on the check printer the next day. After that, they get stuffed into an envelope and mailed out.
The first day of the new install was smooth and uneventful, however the second day was a bit different. We were stationed in a small office, sharing a desk with a few people, monitoring the newly released-to-live system. There was some commotion in another office, outside of our immediate area.
There were several people cramming checks into envelopes drawn from a large stack of checks. Several hundred checks spat out on the printer and needed mailing, so the normal crew was not enough. More help was enlisted because the check-to-stuffer ratio was more than a single person could reasonably handle. Happily joining checks and envelopes in holy matrimony, preparing them for a journey through the United States Postal Service, no one stopped to ask why there were so many checks.
Walking by the check stuffing cooperative, someone from accounting finally asked what was going on. "Stuffing envelopes", someone shrugged. Concerned, they stopped the check mailing factory in its tracks. Going from a handful of printed checks a day to hundreds of printed checks and rolling out a new digital banking system the previous day couldn't be mere coincidence, the accounting person surmised. Worried, she promptly involved us in the ongoing matter.
Check printing doesn't concern us, that's a core system function. We had an open feature request to allow certified check withdrawal through digital banking, but it wasn't implemented at the time. There was no way we were sending requests to Symitar to spit out checks on the check printer. Nevertheless, we knew it was extremely likely the roll out was related to the stack of checks. We didn't know how, but these things happen sometimes.
We did some investigation and found that all the checks were payable to the members with their bill pay payees as the description. We did implement a real-time bill payment feature for our partner, Allied Payment Network. Ah-ha! There was the link that tied us to the checks. At first, the credit union thought Allied was at fault. We explained that, for the real-time payments, the debits flowed through our interface and it was our fault, actually. Why Symitar wanted to print checks for these real-time payments though, we had no idea.
Processing the real-time payments through our interface involves two steps; querying for an available balance from the core and sending a debit through that moves money from an account to a general ledger at the credit union. After that, Allied picks up the funds through ACH and everything reconciles, which makes the accounting department happy.
We didn't know, and it wasn't documented, that through SymXchange any transaction that you send through from an account to a general ledger will be made into a printable check. That is, unless you specify it as a cash transaction, which I was told by the credit union accounting person is "Like, way worse than printing checks. That messes up the cash totals." Not being an accountant, I assumed they knew what they were talking about.
We discovered that moving money from an account to a general ledger without printing a check was not supported through SymXchange. I get it, it's probably a strange thing to do for an online banking system. We do have this feature with other core systems but this was our first Symitar roll-out that enabled the Allied Real Time feature. Surprise, jokes on us I guess. We should have run check posting in the test system, but...
No one runs check posting in the test environment. Unless you have good reason to, that is. Why would they? We weren't printing checks, they weren't printing checks, nobody was printing checks. Nothing had anything to do with printing checks. We'd have to ask for the check posting to even be run, it's not something we can do ourselves, and they wouldn't just run it willy-nilly, for no good reason.
The digital banking (and core system) life lesson here is: No matter how much you prepare and test, the test system is not a 100% representation of the live system. There are some steps you can take to close that gap, but it is never foolproof. We try to test as much as possible. In this case, we did test the account to general ledger transactions quite extensively, we just never ran the check posting. I mean, why would we? We will now though!
How did we get around the check posting debacle? We had to process the transactions through a Symitar repgen (or specfile) that didn't exist at the time. No pressure though, right? Thankfully, our partner CUTEK was able to write the repgen and get the issue resolved quickly (thanks Eric!). It pays to have experts on hand, that's for sure. They saved the day for us.
In the end, everything was fixed. We process the real-time transactions for bill pay and the members don't get mailed checks. As a bonus, we unknowingly implemented the certified check withdrawal feature. (It's not a bug, it's a feature).
From now on though, we will always run the check posting before going live. Lesson learned.