Our blog

design, code, business & everything in-between

The dev-ops 80-20 rule for selecting a Database Technology

Everyone in software is familiar with the 80-20 rule, or perhaps outside software less catchily known as the pareto principle - the idea that you spend about 20% of the time it takes to complete a project building out the first 80% of the features, with the remaining 80% of the duration to complete the elusive final 20%.  There are countless applications of this idea - kind of like when you’ve just learnt a new word, examples start to crop up everywhere.  The most recent example I’ve come across describes the amount of time spent working with persistence technologies from the point of view of developers and operations.  But first some background.

In the days of working on back office software, if there wasn’t a strong opinion already about how to solve a problem, developers were free to pick tools that met their basic checklist - will it work in production? Will the project cover the costs if they exist? And most importantly, HOW SHINEY IS IT?

I’m as guilty as the next developer of having picked tools to work with based primarily on how much I wanted to play with them.  Being an easy tool to code with use generally made my life as a developer easier, but it was also an easy sell to management - if it takes me, the expensive developer, less time to build a solution with, it must be good for the project.  Of course, over time, lots of counter examples have cropped up: just throw “rails performance bottleneck” or “entity framework SQL fail” into google and read to learn what happens when developer magic sauce is spread liberally and allowed to determine the architecture.  

The product I’m working on at the moment generates a reasonable amount of financial data - around 10Gb / day. It turns out that when you need to keep everything, and the product is growing daily, managing an additional 10Gb of data every day can be moderately painful.  Painful to the extent that we have an operations team, one of whose key roles is essentially to keep everything running.  Over the last year or so we’ve scaled vertically, run out of 32-bit integers, added lots of database nodes, added data centres and strategised about archiving operational data.

Turns out I had fallen victim to the most common of easy vs simple fails.  In architecting the original product, I’d focussed on tools that I knew - those that were close at hand, that I was familiar with; and therefore easy for me, the developer to use.  In this case MySql.  Now, MySql is extremely capable of scaling to massive data volumes, it’s currently powering numerous huge databases like Google Adwords and Facebook albeit in a much customised form.  But scaling traditional RDBMs does come with a significant ops overhead.  For this particular product, the data architecture at the code level is now pretty mature, we almost never add new data fields.  But as a business, we spend a lot of time working with the data at the persistence level - just to keep the lights on and the replicas fresh.

On reflection, it seems I chose a persistence technology ignorant of the devops 80-20 rule.  I’d estimate that to date, of all the many, many man-months of time the business has spent building and maintaining this product on MySql, about 20% of that time (if not less) was spent by developers on building the original product against that persistence mechanism, during which we benefitted from the wealth of developer tools and documentation at our disposal - it was easy to do.  At least 80% of the time has been spent by the operations team, keeping things going. 

With my Simple made Easy hat on, and my 20/20 hindsight goggles engaged, I see now that it was short sighted to select the easy, comfortable technology, the one that I had experience with, without understanding how the total cost of ownership (TCO) for my decision would be met largely by maintenance down the line, rather than the minimal build effort.

So what else can we do? I think the main learning point has been to select technology with a stronger consideration for TCO.  A less smooth developer experience may well be preferable to more complex operational maintenance strategy. The persistence marketplace these days is awash with distributed database solutions with significantly improved cluster management and auto-healing capabilities, offering different types of storage and varying levels of consistency.  Some of them even support ACID.

Share your thoughts with us on twitter

abutcher

14th July 2014

Work Experience

Over the past year, members of the pebble{code} and pebble.it teams have been mentoring 6th form students from The Lilian Bayliss Technology school. Through regular meetings and discussions we’re hoping to help these promising young students with guidance and advice in their studies, further education choices and careers paths.

A couple of these students joined us for a week of work experience during the half term holidays. We placed them within our UX/UI team and put them to work on the new pebble{code} website, as well as helping them with one of their pet projects. Here’s what Mohammed had to say about the experience:

"Over the half term holidays I was privileged to work for one of the most prestigious software development companies in London. I sat with their web developers for a couple of days discovering and solving problems. I learnt the employable skills required put together with my web development knowledge to create the perfect software for clients. I thoroughly enjoyed the experience and finally answered the question I have always been asking myself "how do web developers work behind a desk?""

Both teams at pebble{code} and pebble.it were mightily impressed by these students’ abilities to get stuck in and actively contribute to real-world projects. We’re convinced that the students we’ve met from LBTS have a bright future ahead of themselves.

Share your thoughts with us on twitter

thatsinthebook

24th April 2014

pebblecode.com gets a makeover | part one design

Recently here at pebble HQ we’ve been working hard on an update to pebblecode.com. Now that it’s been released we thought it would be great to talk about the design process as well as explaining the tech involved.

One of the key motivating factors for the redesign was to bring the pebble {code} and pebble.it brands closer together. We’ll be releasing an updated site for our sister company int he future so stay glued to pebbleit.com for updates.

Read More

Share your thoughts with us on twitter

markdurrant

17th March 2014

pebblecode.com gets a makeover | part two development

pebblecode.com recently got a makeover. This post is going to be about the development and the tech used. If you’d like to learn about the design process you can see part one of the story here.

Design lead

When the design team was given the brief for the new pebble sites, we were told that the project should be ‘design lead’ and we took this to heart straight away. When designing for the web it’s possible to let development concerns limit your creativity and this was something we were weary off from the start.

Read More

Share your thoughts with us on twitter

markdurrant

17th March 2014

pebble {code} at Lloyds Innovation jam

Last week, we at pebble {code} had the exciting opportunity to take part in Lloyds bank Innovation Jam. Lloyds is a highly respected banking institution in the UK, and has been a constant on our high streets for 140 years, with over 16 million retail and small business customers.

Read More

Share your thoughts with us on twitter

johnmildinhall

10th March 2014

Innovation != technology

Businesses are increasingly looking to technology to solve a requirement to innovate. Hack days are now the domain of corporations looking to generate ideas and creativity quickly. But for businesses to really embrace innovation cultural change is far more important than technology.

Offer the freedom to play

Within corporations fostering a creative environment can be highly challenging. Typically corporations have processes around everything from installing software to how to enter a building. Creativity cannot flourish when overly constrained by process and regulation. If an employee thinks that an idea will never see the light of day why would they even express it?

For innovation to be given a fighting chance a baseline is that employees are offered the freedom to express themselves, experiment and play. With the concept of experimentation a business should accept the idea that it is ok to fail. Clearly it is not acceptable for innovation to permanently fail but the word experimentation means that it should be expected that not everything will succeed.

Ideas come from anywhere

Too often in corporations a top down culture prevails around innovation. The idea that a senior management layer somehow directs innovation will seldom work. More likely to succeed is the idea that anyone can innovate and that each person within a business will bring something different. If a culture exists where individuals can express themselves freely innovation can flourish. For the senior management layer this means learning to let go of the reins if only in specific areas.

Remove barriers to creativity

As individuals hit barriers their propensity to innovate reduces proportionally. Within the realm of software the time to first commit for a developer is crucial. If someone has an idea they want to start working on it straight away. Developers do not want to go through a lengthy process for installing software or need to fill out forms to get access to a database. The ability to share ideas quickly and the time to get an idea in front of a colleague is also very important. How can innovation happen if it takes two weeks to provision a server to allow a web page to be shared?

For a developer the ability to code and share code instantly has been solved by being able to use PaaS products like Heroku. A developer can instantly push code and get a URL to share with someone else. Tools like Github and npm developers instantly.

For a corporation there may be security considerations but within a corporate firewall there should be no reason that experimentation cannot occur.

Conclusion

Innovation and creativity are not something that you can turn on and off in an organisation. If you really want to embrace creativity the culture and organisation of a business must be carefully considered. An innovative business will be a place where people can freely express themselves and are encouraged to do so. It should remove as many barriers as possible and make it trivial to share ideas. It should accept that experimentation can mean failure and that a top-down approach rarely succeeds. Then and only then real innovation can happen.

Share your thoughts with us on twitter

shapeshed

7th March 2014

pebble {code} @ AstraZeneca Tech Fair

The other week pebble {code} was lucky enough to be invited to the AstraZeneca Tech Fair, in Cambridge.

image

It was a great to be chosen as one of the most exciting and innovative companies in the tech space, sharing the room with great companies like Google, Samsung, Intel, Microsoft and Huddle. There was lots of great tech on display, and we were delighted to have an opportunity to fly the pebble {code} drone!

Read More

Share your thoughts with us on twitter

johnmildinhall

4th March 2014

pebble {code} @ Big DiP 2014

Big data is the buzzword of the times. Yesterday, we were privileged to attend the Big Data in Pharma conference in London. It was a fascinating insight into how the conservative and risk-averse pharmaceutical industry is using the cloud, analytics and a wider range of data sources to speed up the laborious process of getting drugs to market.

Read More

Share your thoughts with us on twitter

johnmildinhall

20th February 2014

Innovation Opportunities in Retail Banking

pebble {code} is a place awash with ideas. We look at traditional industries and see a wealth of opportunities for disruption from technology. We have written a short report on areas within Retail Banking where we see potential for technology to offer consumers better customer experiences.

We see four major areas that will shape the direction of customer experiences in Retail Banking:

The report Innovation Opportunities in Retail Banking is available for free.

Share your thoughts with us on twitter

shapeshed

7th February 2014

Encryption climate shows the value of Open Source

The Snowden revelations have thrown the surveillance climate wide open and some have suggested that the behaviour of the NSA has broken the web’s security model for everyone.

At the 30th Chaos Communication Congress [30c3] Nadia Heninger, djb and Tanja Lange delivered a brilliant talk about The Year in Crypto. Although there are some very technical sections I highly recommend that you watch it.

They cover a lot here.

Where open source wins

In this talk it is clear that Commercial encryption software has become generally less secure in the current climate. The strength of open source encryption is that because it is open source many developers can review the code and find any backdoors that anyone is trying to add.

Better Encryption

The bettercrypto.org site has a paper outlining practical ways to improve the way you use crypto both personally and in the software you create. It has recommendations and configuration for major web servers and best practice for using software.

If you are developing software for the web or value your privacy you should read it.

Share your thoughts with us on twitter

shapeshed

10th January 2014
W214 Westminster Business Square 1-45 Durham Street London SE11 5JH
+44 (0) 20 3327 3940 hello@pebblecode.com