BlogBest practices for scalability from an economical perspective.

In Xentime, over many years we have accumulated experience that I would like to share in series of articles.
First of all, you must be aware of pitfalls in the clouds, which are popular these days. See article:
dangers of the clouds.

If you took the right decision and you are building your own platform, then you start from analysing all the data storage options, as that is most important part.

In simplistic terms the issue look trivial: you add HDDs with growth of load and you replicate everything.

But how to measure stability and efficiency of such system and how to predict expenses on scalability ?
Turns out that HDDs were not meant for long term data storage. Engineers from Seagate, IBM and other service centers, who are in charge of data recovery know that very well.

According to Dirichlet principle, probability of the absence of failure approaches to zero with growth of HDDs number. By the way, take a look at statistics from Google: «Failure Trends in a Large Disk Drive Population». They say, that after the first scan error, IDE PATA drives are 39 times more likely to fail within 60 days than drives without scan errors. After the first reallocation, drives are over 14 times more likely to fail within 60 days than drives without reallocation counts.

I would use IDE PATA drives, as there's no statistics of failure rates on large population of SCSI drives at all. Also they consume more power: 12 watt ( IDE drive consume 7 watts ).

Conclusion: you need cheap IDE PATA hard drives.

According to Amdahl's law, theoretical maximum speedup of system, using multiple processors
calculated with the help of this formula:

where S -- the number of processors
and P -- the result of improvement in percents.

So, with 4 CPU's we have:
    1/(0.5+((1-0.5)/4)) = 1.6
with 8 CPU's: 2.9
That will give you an understanding that the number of servers will grow exponentialy. Reason: when you add another processor to server you get small improvement of performance.

Power ConsumptionCPU consume 50 watt approximately
DDR3 consume about 8 watts per DIMM
IDE PATA drive – ~7 watts. 50 + 8*2 + 7*2 = 80 watt

17 July, 2009