Originally Published 10/31/2014
There are a lot of things that go into making applications capable of achieving this level of elasticity. If you are interested in learning more, a great starting point is the The Twelve-Factor App. But a key piece is to integrate via API and not in the database. A lot of legacy applications rely on the database as an integration point. But modern applications see the database as just a place to persist data. In many ways this reduces the complexity of databases and makes it easier for contributors, other than the Database Administrator, to serve as the expert on the data storage.
Most cloud vendors have also added Database as a Service offerings to make it easier for companies without Database Administrators on staff to leverage database platforms. These services offer installation, patching, backups, monitoring and other services that would traditionally be performed by Database Administrators. These services combined with the change in how applications are being built make it easier for a project to get off the ground without the help of a Database Administrator.
I have not seen the whole sale movement of data from Relational Database Management Systems to NoSQL platforms, but I have seen a lot of new applications leverage NoSQL as the only, or at least primary, data store. As a side note, I have also seen a lot of noise lately about applications moving off of NoSQL as the needs of the application change. Like any other tool NoSQL has its role and the industry is still figuring out where it is best leveraged.
The tool that most companies are moving to for Big Data problems is Hadoop. One of the big advantages of Hadoop over traditional database platforms is that you need not define the structure of the data prior to it being stored. Instead, you can define its structure at the time of analysis. This makes the collection and storage of data much quicker from a process perspective than with a traditional data warehouse. It also is easily scaled out by adding additional nodes to a cluster. These capabilities easily solve the volume, variability, and complexity associated with Big Data. Schema design has historically been a a service most Database Administrators have been involved with. With Hadoop, these services are less in demand.
Much like NoSQL, Hadoop is a platform that does not require a Database Administrator in order for it to run. Large environments do benefit from administration however, so if a Database Administrator is not responsible someone will probably need to step in.
The Future DBA
First off, I see all the major database platforms coming out with much better scale out stories over the next several years. Developers are moving to NoSQL for the simplicity of development, but one big reason they stay is because of the ability to scale out. I see the market moving back more towards traditional database platforms once the scale out story is a better one. Current NoSQL platforms are good for certain thing, but non of them can compete with traditional database platforms for flexibility of handling diverse workloads. Here is a great article comparing some common NoSQL platforms and how they server different purposes at the Weather Co.
Secondly, no matter the future of NoSQL, Database Administrators will still be needed to address the hard to automate aspects of managing data platforms for the foreseeable future. The biggest one of these is performance. Database Management Systems are complex pieces of software and it is still easy for performance to change without any changes to application or database code. Ad systems grow in size or use the performance characteristics shift. Database Administrators will still be needed to help developers and system administrators understand and solve these issues.
I hope you have found this post interesting and I would love to hear your feedback. What services do you see the Database Administrator of the future offering? What other factors do you see shaping the role of the DBA?