I have been doing this Database Administration gig, in one form or fashion, for more years than I care to admit.  In that time I have seen a lot of change.  Change in the tools and platforms I have supported, and change in the role that the Database Administrator plays in the development and operation of software systems.  But, it has felt like the pace of change has really sped up over the last 2-3 years.  I thought it would be good to go over some of the changes I think will have the largest impact on the Database Administrator and what I think it will mean to our role.

Probably the biggest change is the adoption of the cloud.  Whether the cloud is public, private, or a hybrid cloud infrastructure is fundamentally changing the way that applications are designed and built.  A lot of people looked at the cloud early on, through the eyes of old development models, and felt like it would just be a way for small companies to compete with larger ones without having to have the financial resources to build and maintain their own data centers.  While that is one advantage of the cloud (one equally advantages to large companies), it is not the largest advantage the cloud has to offer.  The smart companies are realizing that if you rethink the way you architect and build applications you can achieve levels of elasticity in your application which allows you to meet the ever changing market without having to maintain infrastructure to meet peak demand all the time.

There are a lot of things that go into making applications capable of achieving this level of elasticity.  If you are interested in learning more, a great starting point is the The Twelve-Factor App.  But a key piece is to integrate via API and not in the database.  A lot of legacy applications rely on the database as an integration point.  But modern applications see the database as just a place to persist data.  In many ways this reduces the complexity of databases and makes it easier for contributors, other than the Database Administrator, to serve as the expert on the data storage.

Most cloud vendors have also added Database as a Service offerings to make it easier for companies without Database Administrators on staff to leverage database platforms.  These services offer installation, patching, backups, monitoring and other services that would traditionally be performed by Database Administrators.  These services combined with the change in how applications are being built make it easier for a project to get off the ground without the help of a Database Administrator.

NoSQL, or Not Only SQL, is the name for a class of data stores that do not store data in the tabular relationships associated with traditional Relational Database Management Systems (RDBMS).  Some popular examples include Redis,MongoDB, Couchbase, and Cassandra.  Each serves a specific data storage niche, and allow you to meet your data storage needs without the complexities of a full blown RDBMS.  But while you give up some of the features of a RDBMS you gain a lot in simplicity.  NoSQL platforms are typically much easier to maintain and have fewer “moving parts” than a traditional RDBMS.  This, along with their ability to easily scale horizontally, makes them very popular when building applications for the cloud.

I have not seen the whole sale movement of data from Relational Database Management Systems to NoSQL platforms, but I have seen a lot of new applications leverage NoSQL as the only, or at least primary, data store.  As a side note, I have also seen a lot of noise lately about applications moving off of NoSQL as the needs of the application change.  Like any other tool NoSQL has its role and the industry is still figuring out where it is best leveraged.

Unless you have been hiding under a rock, you have probably heard the term Big Data.  What exactly is Big Data is still open for some debate.  But you can think of Big Data as problems involving data that is either too large or complex to analyse using traditional techniques.  The data is “big” because of it’s volume, velocity, variability, or complexity.  Volume is the term used to describe the quantity of observations being analyzed.  Velocity is the speed with which new observations are being generated and need to be processed.  Variability is what is used to describe the level of inconsistency in the observations being analyzed.  And complexity is the term used to describe the overall complexity in gathering the observations and combining them to make useful information.

The tool that most companies are moving to for Big Data problems is Hadoop.  One of the big advantages of Hadoop over traditional database platforms is that you need not define the structure of the data prior to it being stored.  Instead, you can define its structure at the time of analysis.  This makes the collection and storage of data much quicker from a process perspective than with a traditional data warehouse.  It also is easily scaled out by adding additional nodes to a cluster.  These capabilities easily solve the volume, variability, and complexity associated with Big Data.  Schema design has historically been a a service most Database Administrators have been involved with.  With Hadoop, these services are less in demand.

Much like NoSQL, Hadoop is a platform that does not require a Database Administrator in order for it to run.  Large environments do benefit from administration however, so if a Database Administrator is not responsible someone will probably need to step in.

So, what does all of this mean for the Database Administrator of the future?  Well, I don’t have a crystal ball, but I think it is clear that the role will have to change.  First, many tasks that have traditionally been staples of the DBA role will either cease to be needed, or move primarily to developers or general system administrators.  Backups, monitoring, and patching will be either fully automated, or so easy that anyone will be able to “click the button” to perform the actions.  Database code deployment will no longer be the purview of the Database Administrator as the deployment process will be fully automated in order to allow for the elasticity that the cloud offers.  So, what will be left for the Database Administrator?

First off, I see all the major database platforms coming out with much better scale out stories over the next several years.  Developers are moving to NoSQL for the simplicity of development, but one big reason they stay is because of the ability to scale out.  I see the market moving back more towards traditional database platforms once the scale out story is a better one.  Current NoSQL platforms are good for certain thing, but non of them can compete with traditional database platforms for flexibility of handling diverse workloads.  Here is a great article comparing some common NoSQL platforms and how they server different purposes at the Weather Co.

Secondly, no matter the future of NoSQL, Database Administrators will still be needed to address the hard to automate aspects of managing data platforms for the foreseeable future.  The biggest one of these is performance.  Database Management Systems are complex pieces of software and it is still easy for performance to change without any changes to application or database code.  Ad systems grow in size or use the performance characteristics shift.  Database Administrators will still be needed to help developers and system administrators understand and solve these issues.

I hope you have found this post interesting and I would love to hear your feedback.  What services do you see the Database Administrator of the future offering?  What other factors do you see shaping the role of the DBA?