ATMCash Exploits Virtualization for Security - Immutability and Reversion

This is a guest post by Ran Grushkowsky, Head of Technology at ATMCash.

Virtualization and cloud-based systems are very hype in the industry; however, most financial companies stray from those solutions. At ATMCash, we’ve approached virtualization not for the usual reason of scalability, but for the usually missed value of security.

In this article, I will introduce the concept of security added value in the utilization of virtualization and why people should consider deploying mini-clouds for those use-cases.

How do virtual machines help mitigate risk?

I am sure most of you have heard of the recent hacking in the financial sector. Financial companies are under constant hacking attempts and security is of the utmost importance. One of the bigger risks in system deployment is a breach in one of the stack components. Regular system patches and maintenance fix known exploits and issues, however, sometimes it may be too late and the component has already been breached. If the system has already been compromised in a natural environment where patches are applied to existing systems, sometimes the patch may come too late and a Trojan horse or some sort of malicious code has already been injected (as may have been seen in recent cases). Virtual machines provide a great hidden gem: immutability and reverting images.

Example of how ATMCash uses those features for security in the stack:

Our stack encompasses three major components: Front end, back end, and database.

Our boxes are running VMWare vSphere. We’ve decided to go with a setup of 4gb/ram 64gb/SSD servers for our front-end components. More robust servers with 16gb/ram for our back-end components and dedicated MySQL database servers with user id sharding and replication. Our virtual machines are running a modified version of CentOS. We learned to love CentOS with a relatively fast patching cycle and easy-to-modify configuration.

The front end and back end are static components that change with each version and the database is the only dynamic component that changes every moment. We utilize the static trait to enhance our security and eliminate the risk for infected systems that are being patched after being compromised.

We have an up to date image of each system component that is not deployed, but is constantly being patched and updated to the latest security risks. We then commit any software updates to that image and replace the running image with the newly updated and patched one. By putting a load balancer in place, we suffer no down time during the process. We replace the virtual machines one by one by taking one machine down while the load balancer directs traffic to the machines that are up and repeat the process until all VMs have been replaced. The process of swapping images usually takes less than 30 seconds each. VMWare vSphere has some powerful features that make the replacement of images on the fly rather easy. It is a relatively rapid redeployment process.

Example of how we use immutability for added security:

At ATMCash our customer service and call center are being handled internally. We also consider our customers’ security and privacy as the highest priority. In order to protect our customers further, we’ve developed our own CSR (Customer Service Representatives) system that utilizes various levels of security. One of them is the virtual machines’ immutability feature. Every CSR station is running a copy of a master VM image that is immutable (cannot be modified and is being reset after every reboot).

Deploying the stations in that fashion provides a crucial added level of security that protects our system from malicious code that may enter the station. Viruses and malicious scripts travel in various methods today. From email attachments to Word documents, they are everywhere. If you think that putting in place a policy that instructs employees not to load personal documents from a removable drive, or even restricting web access to social networks would eliminate the risk, you are proven to be wrong. You should always assume that a malicious script will find its way to your station and that your anti-virus will most likely not detect it. Using an immutable image that is rebooted every time you restart provides both an added security and the ability to apply patches and updates system-wide with only updating one image.

Why does ATMCash have their own private cloud and does not use existing infrastructure:

We get that question a lot, and the answer is very simple. We are in the money business and we need to take extra care of our customers’ information. It’s true; the cost of deploying your own mini-cloud is much higher than utilizing any existing infrastructure, but the added value for security is priceless.

Also, with the finance programs that Dell and other parts manufactures offer you can pay as little as a cloud service if you spread the payments across. For price comparison: Having 10 Linux servers with 4gb and 200gb up/down using Rackspace would cost $1804/month. Having comparable servers on your own mini-cloud using dell-servers would be $105/month in server’s hardware, additional $400/month for networking accessories, $600/month for VMWare licensing, and datacenter fees which may vary but are approximately $400/month.

On top of that you have to maintain the set up and any scalability would require purchasing and setting up more nodes. $1505/month before the fixed cost of employees to maintain it is slightly high, however, the benefits, specially on the security front should not be missed.

How do you handle reconciliation issues when downtime occurs?

Handling financial transactions, downtime is not an option. That’s why virtualization is extremely important and load-balancers are crucial for any virtualization environment. We’ve designed our system for extreme redundancy. Every piece of the stack is duplicated and designed to handle down time. We have load balancers, both on the local datacenter level, using physical load balancer devices that also perform SSL handshakes, and on the DNS level. Once one component goes down, the load balancers would automatically direct traffic to its clone node.

We perform an immense amount of data processing using batch updates where we commit crucial updates as a delta-update, small incremental updates that eliminate the need to update a whole piece of dataset; you only update the small required changes. For example, a withdrawal or a load of funds requires an immediate update throughout the system to reflect on the balance, however, the transaction history is of a lower priority. When a load occurs at the ATM level, the request is coming through our systems to ensure the transaction is valid.

Once the transaction has been approved, we would then perform a delta-update to our front-facing database to reflect on that transaction. We then would trigger different processes that alert the customer of the transaction, update their interface, and perform security velocity checks (Checks that ensure the security of our customers’ cards from being stolen).

The system handles crucial updates such as delta updates while the majority of data is being processed as batch. The nature of batch updates is a complete roll-up of massive data versus delta-updates, which include only the data that has been modified. To ensure integrity of data across the system and to allow for quicker updates we utilize delta-updates for stream-line updates and batch updates that also include more data that is not operation-crucial for data integrity checks. With a volume of millions of dollars to-date, we can safely scale up using the delta-updates without affecting stability or performance of the system since bulky batch updates would not clog the system.

To summarize:

If security is an integral part of your business, like it is of our business here at ATMCash, you should utilize virtual machine’s added security features. You will learn to love the flexibility it offers and the peace of mind you’ll have knowing that your version is always “malware-free”.

About the company: Established in 2005, ATMCash.com is a global payment platform and online money transfer service to over 150 countries. Pre-paid debit cards are used instead of the traditional third party agent-based model and bank wires. Recipients without bank accounts can withdraw the funds from over 1.5 million ATMs worldwide.  ATMCash.com is changing the way people transfer money and pay freelancers by providing more control, peace of mind, and savings for the customer.