Space … the final frontier … or is it?!

So, over the last year, I have been working more and more on video editing and producing 4K content as part of my in depth exploration of my childhood dream to become a film maker (insert mid life crisis jokes here!) 🙂

 

One of the biggest problems I’ve had to date has been storage space, finding enough space to keep all the video I have been creating, the B-Roll, the content libraries and more.

Having bought a Promise2 R8 Raid array with 8 x 3TB drives and Thunderbolt 2, this quest for storage has been satiated for quite some time, however as the 18TB (usable) space is being eaten up rapidly (now I am filming in 4K and 6K ProRes RAW) and I am creating more and more content on an almost daily basis, I needed something bigger …. and FASTER.

 

Promise2 RAID via Thunderbolt 2

 

A new problem has arisen, one which I had previously not anticipated, and that is that I need storage which is also fast enough to be able to edit 4K/6K footage on.  The project files are generally too large to work on my local 1TB m.2 SSD in the iBin (Mac Pro Late 2013) as that only ever seems to have 200GB-300GB of space free, and that can be the size of the cache for a single project these days.  The Promise RAID solution has been good, but I’m only really seeing 180MB/sec out of the array, which is proving not to be enough as I start to render complex projects with multiple layers and effects.  I’m also sometimes working on two computers simultaneously (my MacBook Pro 2017 with discrete GPU is now faster than my desktop, so sometimes I move to work on this) … I have been syncing the project file between the Promise RAID and an m.2 SSD drive, which is giving me nearly 500Mb/sec over USB-C to the MacBook … but it is only 1TB … so only really good enough for a single project at a time, and I don’t have access to the library of B-Roll I’m building … so I need to copy that from the library, which means duplicate files everywhere eating more disk space.

m.2 SSD Internal drive

 

In my dayjob we’ve been using 10GB networks and wide area storage arrays (ceph) for years.  They’re fast, efficient, infinitely scalable and relatively “cheap” compared to other SAN solutions on the market … we have 200TB+ of storage and we can grow that daily just by adding more drives / chassis into the network.  This however is overkill for a domestic / SoHo solution (with 80+ drives and 20 servers and counting, this is definitely a “carrier grade” solution!

 

 

So I thought it was now time to merge my expertise in Enterprise storage and networking with my hobby and need for something which is “better” all round.

 

Historically, the secret to faster storage has always been “more spindles“.  The more disks you have in your array, the faster the data access is.  This is still true, to a degree, but you’re still going to hit bottle necks with the storage, namely the 6GB/sec (now 12GB/sec) speeds of the SATA/SAS interface, 7200RPM speeds of the disks (yes you can get 15K RPM drives, but they’re either ludicrously expensive, or small, or both).  

 

SSDs were always a “nice” option, but they were small and still suffered from the 6GB/sec bottle neck of the SATA interface.  Add to that reliability issues of MLC storage and the costs of SLC storage (article: SLC vs MLC) which made NAND flash storage devices impractical.  I have had many SSDs fail, some after just a few days of use, some after many months.  Spending $500 on something which might only last you 2 weeks is not good business sense).

 

Today, we have a new generation of V-NAND and NVMe hybrid flash drives which have up to seven (7) times the speed and much higher levels of reliability that interface directly to the PCIe interface and bypass previous bottle necks like the SAS/SATA interface.  And they’re (relatively) affordable and come in much larger capacities (up to 2TB at the time of writing, although I’m told “petabyte” sizes are just around the corner).

So, the question now is how do I put all of this knowledge together to deliver a faster overall solution?

 

From the networking perspective, I started off looking at 10 Gig capable switches.  I found a few options on eBay including 24 port Juniper EX2500 switches for £600 each (now end of life, but they’ll do the job) however I ended up choosing a brand new Ubiquiti EdgeSwitch 16-XG for £450, which has a mix of 10GBase-T and 10G Base-X interfaces (SFP+ and RJ45) so that I could connect a mix of devices regardless of whether they were via copper or fibre.

 

Ubiquiti Edge Switch 16 XG

 

For the MacBook Pro, I bought a Sonnet Solo 10 Gig (Thunderbolt 3 interface) for £185, and for the MacPro (iBin) I bought a Sonnet Dual 10 Gig Thunderbolt 2 interface for £385.

Sonnet 10G Solo Thunderbolt 3

 

 

 

 

 

Connecting the devices together with CAT7 cables bought on Amazon for £15 and 10GTek Direct Attach cables to link the SFP+ devices (see below) to the switch.  In my dayjob we have been using Mellanox DirectAttach cables, however my UK suppliers seem have had a falling out with Mellanox as despite trying to buy supplies of these for work through both Hammer and Boston (both of whom have promised faithfully to always carry stock of essential items such as these) have been unable to supply any to me despite my attempts to order them repeatedly over the previous 6 months.  The 10Gtek ones work, and come in at about the same price … and ordering is a lot less painless than having to raise purchase orders and deal with wholesalers on the phone.  Plus, I wanted to try and do this using only items I could buy today as a “consumer”.

 

Next, I looked at off the shelf NAS solutions .. the two lead contenders in the space appear to be Synology and QNAP.  I placed orders for a number of different units, not all turned up, some are (still) on back order with the suppliers, and at least one supplier (Ingram Micro) cancelled my order and told me to re-apply for an account as they’d changed systems and I hadn’t ordered anything in their new system yet – despite having just ordered something in their new system .. Go figure! 🙁

My original plan had been to compare Thunderbolt 3 networked devices to 10 Gig networked devices, however as QNAP are the only manufacturer (currently) to have a TB3 equipped unit, and as Ingram failed to supply the device (and nowhere else had stock) I have yet to complete that test.

As far as drives go, despite their bad rep, we’ve had fairly positive results with Seagate drives at Fido, so I opted for a batch of the ST12000NE0007 IronWolf Pro 12TB drives at £340 each

The chassis ordered for testing 

 

Synology DS2415+ (10 Gig an optional expansion card)

Synology 1817 (10 Gig built in)

QNAP TS-932X (10 Gig built in)

QNAP TS-1282T3 (10 Gig built in and Thunderbolt 3)

 

On paper, the 1282 T3 looks like the winner (if only I could get hold of one!).  The TS-932X looks like it might be ok, but the CPU worries me.

The Synology 1817 has the same CPU as the TS-932X, QNAP has QTier as well as SSD caching

 

Another way to say thank you …

Ok, bit of an experiment here ….

Over the years I’ve helped a lot of people out and never really asked for anything in return, many have said “you’ve saved our business, thank you! – we’ll send you a bottle of something to say thanks” .. others have said “times are hard, but you’ve really saved our skin, we’ll remember you in the future”, etc … and with the exception of a couple of people (including one person I helped today) who’s actually said “thank you” and “sent a gift”, the rest have really just either forgotten or received thousands of pounds worth of free consultancy and kept their businesses going as a result.

I’m not bitter, I don’t really mind, I’d rather help a friend and see them succeed than sit back knowing that I could have done something to help them, but didn’t ….

However, today I’ve put together a little Amazon Gift list .. items ranging from the cheap to the eye wateringly expensive (but all things I want and would buy if I weren’t currently so financially challenged myself).

So, anyone feeling generous? 

Failing that, here’s a list of tech that’s quite cool and you might want to consider buying for yourself if not for me 

So here’s the link to my Gift List on Amazon … feel free to contribute or donate .. or just send cold hard cash via PayPal if you prefer 🙂

Amazon: https://amzn.to/2OI61zS

PayPal: https://paypal.me/jmorby

 

 

Reflections on my first year as a born again student!

So the first year of study has completed on the BTEC Higher National Diploma in Creative Moving Media Production I’m studying with the Raindance Film Trust.

 

At the end of my first year, I have mixed emotions.  By all accounts, it looks as though I’ve completed the year with a Distinction, it has taken the tutors longer than anticipated to mark all the assignments, more than five of them had been completed before the first was returned – which made it difficult to know if all the right boxes were being ticked in respect to the format and layout, writing style and so on.  With a few small adjustments and exceptions, it seems that the style was acceptable (thankfully!).

The first year of study has been a mixed bag.  I’ve had to learn some humility and to learn that there is more than one way to do things.  For the last 20 years I have pretty much been the one everyone came to for the answers and I had to know, or at least seem to know, the answers – often having to invent a solution as we went along putting out fires or coming up with a bit of code to overcome whatever the issues were.  No matter what the problem, the buck stopped with me – and if the business was to continue and the customers were to be happy, I could not just say “I don’t know” or “ask someone else”.

Filmmaking is much more of a collaborative affair, there are many ways to deliver the finished project and there is rarely a “wrong” way, everything is much more subjective – so long as you get the take and you have a finished project at the end.  There are often times when you think (or say) “we could have done that better” or “if only we had more time / xyz resources / etc” .. However, each experience is a chance to learn and to find a better way of doing the same thing next time (and/or not to repeat the same mistake twice!)

We are working out who has strengths and weaknesses in various areas, and unlike “the real world” where we cut the chaff from the wheat or focus on the strengths, being a learning environment we get to focus much more on the weaknesses too, making them stronger (hopefully).

From my perspective, I feel as though a certain part of the “magic” of film has died, I am now pretty much unable to look at a scene in a TV programme or film without thinking “what was the Director’s instruction to the cast” or “their motivation” … I look at a scene and think “why on earth did they light it like that” … or “I can see the camera reflected in that window ” .. or similar.

Sometimes I despair at what I’m watching, mainly because the film cost $5m to make, grossed $20m and is still a mediocre piece of work (in my humble opinion) .. that does, however, fire me up to think that I could certainly do better.

I confess I was disappointed that my first narrative short film didn’t make the selection grade for screening at RDFF26, however, I take responsibility for that; in not taking along anyone from the cast/crew/family/friends/etc (well apart from Miky) so there was little chance of winning the popular vote.  That said, the competition was tough and the quality of the films that were made was incredibly high, and I did find myself voting for other student’s work as well as my own, and I’m sure other students and visitors ended up doing the same.  

Only 12 of the 23 short films were selected in the end, I would have liked to think that my film was somewhere in the top 10 however this ended up not being the case.  On the plus side, everyone in the room laughed at the right places and seemed to get the joke.  There were other films that I thought were just as good (if not better) than mine which were also not selected.  This however just fires me up to go out and produce a better film, and I have already started work on a horror genre script, as well as an idea for another comedy.  

Juggling a full-time job as well as a full-time course has been taxing at times, however as they often say, if you want something doing quickly then give it to a busy person, and on the whole that has worked.  I have been able to adequately segment my time between Fido and HND assignments and only had to request an extension on a couple of tasks due to conflicts with pesky things such as VAT returns and such like.

On the whole, I am looking forward to year 2.  The first year covered the theory of film in much more detail than I would have been able to do on my own, giving pointers towards topics for additional research as well as helping me better understand concepts such as mise en scènechiaroscuro and French new wave.  Next year we have more of the same, plus documentary filmmaking and sound design.  So lots to look forward to!

 

 

Raindance HND – Prologue

Not long now until the start of the Raindance HND.  I suspect I will be the oldest person on the course by far, certainly in my group (Group B) although I have now met some of Group A and they seem like some really nice guys (especially Simon and Orlando!)

People are asking me why I’m doing this, as well as what is going to happen to my business(es) while I’m working on the course.  Well the answers to that are somewhat multi dimensional, but I will endeavour to clear up any potential misunderstandings or just dampen the FUD 🙂

The future of websites and the internet is most certainly video.  YouTube is already the world’s second largest search engine, only eclipsed by Google – who oddly enough now own YouTube.

I think it is wise to invest in the future of the internet and future technologies early, and as such I am working to re-skill and re-tool before the flood gates open.  Now of course this is slightly selfish as video and film has been a passion of mine for many years (I think I made my first short movie when I was 8 years old on super 16mm film) … So a tad indulgent maybe, but I still maintain that it will be beneficial to the business(es) and our customers as a whole going forwards.

As for what is going to happen to the business(es) I run now (Fido/etc) – they will continue to operate just as they have done for the last 16+ years.  I am not leaving the planet, I will still be in contact daily, however I am going to be in pre-arranged “meetings” Wednesday and Thursday for most weeks until 2020.  This isn’t much different to how things have operated in the past.  I’ve operated the business whilst being in South Africa, America, France, Germany, Spain and many other Countries.  I have holidayed and the walls haven’t come tumbling down. In fact, on the whole, I’ve found the business has thrived and has hardly missed me on a day to day basis. 

Staff well being is key to a successful business.  I have been struggling to find things to keep me happy for a while now (especially since the divorce), and refocussing my energies on an HND and MA will (I think) be good for me, and as a result will be good for the company as a whole – which also means good news for customers.

I will be in London a lot more, if anything I will be closer to a number of our customers and potentially able to make meetings or perform out of hours maintenance more easily.

In summary I see this as good for Fido, good for my own mental well being and good for the future of our endeavours as just when everyone starts to say “do you think we should be doing something with video on our websites, we seem to be slipping behind the competition because they have lots of video explainers / they have a slick marketing campaign with video / etc” – I will be able to say “Don’t worry, we’ve got your back!”

So don’t worry!  The Future is Bright.  The Future is still Fido! 

(and yes, I have formed a production company called “JFDI” … and yes, I do hope that it will do exactly what it says on the tin!)