Security is not a magic wand you can wave at the end of a project, it needs to be considered and built in from day 1. It is not a bolt-on, it is the consistent application of a range of solutions applied iteratively and reviewed regularly as part of a whole system, which is only as strong as the weakest link.

As it stands you have flagged a security concern which is a good first step. Now as a minimum you have to define:-

• What data you are trying to protect?
• Who are you trying to protect that data from?
• Who actually needs access to what (and when)?
• What is the legal/financial/business impact of that data being compromised?
• What is the legal/financial/business need for a person/group having access to the data?
• What budget is the business willing to assign to a “get secure, stay secure” strategy when it was not a business requirement previously?
• What access does the system need to the data?
• What does this process and systems this application rely on?
• What is done to secure those environments?
• Who is going to be responsible for implementing it and reviewing the whole process?

Until you know all those in detail you really don’t have anything to work with. That information will define what mitigations to those threats you can (and cannot) apply and why.

It may be that the best thing to do is recognise that you don’t have the necessary experience and that it would be best to bring in someone new with that experience. I quite often hear the response that there’s no budget – if it is considered genuinely important then the budget will be found.

You are right. If an application is capable of accessing content stored on corporate machines without the user passing extra information every time, then the programmers, maintainers, sysadmins etc. of the service provider can access that content. This is fundamentally unavoidable, and a major source of insecurity (Edward Snowden was a sysadmin, and had special privileges above “Top Secret” because there simply isn’t a way of not granting them.)

The only way of avoiding this is to require the user to provide information that never makes it to the corporate machines. This is tedious and error-prone, since no one can possibly remember enough information to form a secure access barrier, and therefore people will immediately begin to store their credentials electronically in some place, which will then become the new attack target (and probably a weaker target than the one you are maintaining). But if you want to truthfully assert “Our employees are not physically capable of accessing our users’ content”, it’s the only way.

(Note also that such a business model seems to be becoming politically untenable. Businesses have been forced out of operation by security services for trying to do exactly this. I understand that giving privacy guarantees has business value, but it can’t possibly have more business value than the fundamental goal of staying in business.)

You’re quite right; some developers will always need access to the Live data, if only to diagnose production problems. The best you can do is to limit the potential damage by reducing the number of people involved.

With great power comes great ... opportunity to really, *really* foul things up. 

Many developers won’t want that responsibility and others, just won’t be “ready” to hold it, and so shouldn’t have.

Question: Why is your Development team performing Live releases?
I would suggest you need a Release Management “team” (even it that’s just a subset of your team, plus Business representation to make any on-the-day “decisions”, like “Go/No-Go”)? This would remove much of the “need” for developers to touch anything Live.

Do you have any sort of non-disclosure/ confidentiality agreement between developers and company? It’s heavy-handed, but it might have some merit.

The problem is your developers having access to the systems. If they need production data for debugging then give them a database dump where all that sensitive information is removed. Then the developers can work with that dump.

Deploying a new version should not involve any developer – that a pure admin task, or even better – a fully automated task. Also be aware of releasing and deploying being two very different tasks. If your process isn’t aware of that then change it accordingly.

Rule #1 of security: If someone has access to information, they have access to that information

That tautology is annoying, but it is true. If you give access to an individual, they have access to the data. For users, this usually means access control, but for developers… well… they’re the ones that have to write the access control.

If this is a major issue for you (and it sounds like it is), consider building security into your software. A common pattern is to design secure software in layers. At the lowest layer, a trusted development team designs software which manages the most naked of access control. That software is validated and verified by as many people as possible. Anyone designing that code has access to everything, so trust is essential.

After that, developers can build more flexible access control on top of that core layer. This code still has to be V&Vd, but it isn’t quite as stringent because you can always rely on the core layer to cover the essentials.

The pattern extends outwards.

The hard part, indeed the art of designing these systems, is how to build each layer so that developers can continue to develop and debug while still providing your company with the security you expect. In particular, you will need to accept that debugging demands more privileges than you think it should, and attempting to lock that down will result in some very angry developers.

As a side solution, consider making “safe” databases for testing purposes where developers can rip out all of the safety mechanisms and do serious debugging.

In the end, both you and your developers need to understand a key tenet of security: All security is a balance between security and usability. You must strike your own balance as a company. The system will not be perfectly secure, and it will not be perfectly usable. That balance will probably even move as your company grows and/or demands on developers change. If you are open to this reality, you can address it.

Set up two deployments of the application which also use separated database deployments. One is the production deployment and one is the test deployment.

The test deployment should only have test data. This can either be fantasy data which got created for that purpose or a copy of the production data which was anonymized to prevent the developers from finding out the real people and entities behind the data.

In two financial firms, developers did not have access to production machines. All requests to modify production machines had to go through an approval process, with a script, and was approved by managers. The dev-ops team completed the actual deployments. I assume this team was employees only, and passed background checks. They also did not have developer knowledge so probably couldn’t snoop if they wanted to. In addition to this, you would encrypt all database entries using a secret key stored in the environment variables. Even if the databases leaked publicly no one could read it. This key can be further password protected (PBKDF) so only an executive can unlock it. Your system could require the executive password upon boot (or more likely delegated to dev-ops or a dev-ops manager). Basically the strategy is to disperse the knowledge so a critical mass of required knowledge does not exist in one person and there are checks-and-balances. This is how Coca-Cola protects its formula. Honestly, some of these answers are cop-outs.

MongoDB has limited security controls and depends on a secure environment. Binding to a specific ip and port (and ssl since 2.2), and a crude authentication, that’s what it offers. MYSQL adds GRANT o ON db.t TO… Data at rest is not encrypted, and ssl is not used by default. Create a fence. Readonly access for developers to application related log files should be enough to debug. Automate the application lifecycle.

Ansible helped us automate standard operations (deploy, upgrade, restore) over many single-tennant environments while using distinct encrypted vaults to store sensitive environment variables such as hosts, ports, and credentials. If each vault can only be decrypted by different roles, and only on a bastion host, for logged operations, then auditability provides acceptable security. If you grant SSH, then please use selinux to avoid key tampering, use a bastion host with ldap/kerberos authentication for administration, and use sudo wisely.