Apr
03

Woosh, the deadline passed! Or, how everything that can go wrong does…

Posted in Production by Ton

lastday2.jpgMomento; no worries! The master film+sound disks are ready, packed, sealed and were picked up by the courier. (All will be OK, airplane personnel strikes, bad weather alarms and Norway sliding into the Atlantic notwithstanding).

For your pleasure and probably to learn from (do we ever!) here’s the disaster list of all that went wrong last night, causing Andy + Sacha + Campbell + Brecht to skip sleep…

  • Sun grid crash! Usually when a node goes down it ends our session, but it can take 3-5 hours to have it all running again.
  • The credits came back from the renderfarm with the default font, and adding the desired font didn’t work anyway since we compiled blender without truetype font support. Converting it to vectors messed up a few character, so we ended up rendering the credits (2000+ frames with character animation) on our own systems.
  • So let’s use our own studio computers to render last minute changes. But… the installation differs for systems, giving problems.
  • Then a bug in the new Approximate Ambient Occlusion appeared, but luckily fixedlastday1.jpg
  • And then the grass looked completely different compared to the Sun grid images… which appeared to be a different implementation of qsort() in Solaris!
  • Blender Sequencer was used to convert float OpenEXR images to 10 bits DPX. Unfortunately in some cases, on effect strips, the strip ignored the float images…
  • DPX converting started with image names saved with 4 digits. We have 14315 images to save…
  • Andy made a great advanced setup for last minute color grading in Compositor. It was far too slow to be useful, even though a promised patch was supposed to fix that. He could only do a couple of shots.
  • Our provider had nameserver issues, so we couldn’t use svn.blender.org to update code.
  • Ooh, let’s not forget the start title!
  • When we testing things yesterday, we forgot to write to the needed USB disk… which was saving very slow. Takes 4-5 hours to write everything, which we didn’t have anymore.
  • Using our own backup disk, where all files were, was an option. Unfortunately it was formatted to be Linux-readable only…
  • Luckily at Filmmore – where the files were converted to log dpx for the film printer – they had Linux, but reading it all in would take about 2.5 hours. Just within the limits.
  • After it all was ready, we nearly missed (thanks Sacha!) the error of 40 black frames in the movie… caused by a bad symlink in our finals directory structure.
  • Dang, one shot is off with three frames… luckily there’s no sfx in it, so hopefully nobody sees it.

And… after spending almost an hour on this blog post, I lost all text when I tried to save it to blender.org, who just decided to go into a “500 Internal server error”.

Off we go!

-Ton-

(Image 1: how I found the four victim’s this morning. Image 2: final check in Filmmore of the movie…)

49 Responses

  1. Pitel Writes:

    Murphy’s laws seems to work.

  2. Timm Writes:

    Oh no, wish you more luck for the mastering of the dvd

  3. LoMac Writes:

    This blog post would make a good movie in itself.

  4. Nathan Letwory Writes:

    That must’ve been generating the wrong kind of adrenaline :)

  5. Gianmichele Writes:

    go and get some rest before the premiere, and then back on work for the dvd…you really deserve it guys

  6. horace Writes:

    sounds like fun. :) but what does this mean now? will the dvd contain a version with correct color grading and so on? :p

  7. Pablo Lizardo Writes:

    hehehe so many thing happen there! but is a great new to know that the movie is finished on time!
    Congratulations!, you demostrate that good planning is very important in this kind of projects!
    Regards! When you will ship the DVDs???

  8. Mal Writes:

    That definitely sounds like a rough night – I’m sure you guys will sleep well for the next few nights!

  9. jazzroy Writes:

    so nice to hear you’re still human :)

    have a nice rest and prepare for the premiere!

  10. Vincent Writes:

    Oh my God, I’ve never seen Murphy’s law apply so much… Best of luck! ;)

  11. Socceroos Writes:

    Uuhh, sorry if I’m wrong. But it sounds like you guys are basically saying ‘we’ve had to cut corners, we apologise if it looks a bit crappy in some parts of the movie.’. Is that right?

    …..agh! I’d rather you released late than released ‘on time’ and it wasn’t finished properly!

    On another note, I know how it feels when everything starts going wrong and you’re on a deadline. Get some sleep!

  12. cannontrodder Writes:

    Socceroos, they can’t be late as this is for the premiere!

  13. Sam Allen Writes:

    Hey, drama! Reminds me of how we used to leave assignments to the last date and do an all-nighter to only submit it later the next day.
    Ofcourse this is a bit of a different situation since things actually went wrong and Peach didn’t actually /procrastinate/ in any sense…
    …did they?

  14. Sensei Writes:

    “This is nothing. …piece of cake. Y’know, producing is being a Samurai Warrior. They pay you, day in, day out, for Years, so that, ONE DAY, when called upon, you can respond, your training At Its Peak, and save the day.” :-D

  15. Alexander Ewering Writes:

    I know that “CMS goes into error when hitting SAVE” situation very well – I also had spent so much time on a blog post that the PHP session would have expired when I hit Save :-) That’s why I tend to CTRL-A CTRL-C CTRL-V the text somewhere before I click Save.

    Anyway, the story was an exciting read, one could feel the adrenaline :)

  16. ethana2 Writes:

    Is qsort() defined by POSIX?
    Did solaris crash?

    *sigh*

    …I’m glad they had the sense to use linux at that place you went to.

  17. ethana2 Writes:

    Sensei: Blender is FOSS, making them ninjas. Samurai are proprietary devs.

    …and ninjas are better ^_^

  18. Deak Writes:

    “I love deadlines. I like the whooshing sound they make as they fly by.”
    – D. Adams

  19. Superkoop Writes:

    Oh don’t you hate it when you get finish writing an awesome blog post, and you lose it! I have had that happen to me on multiple occasions, before I started copying before submitting. =P

  20. Anonymous Writes:

    Hey, I like the back of the DVD cover.Big Buck Bunny Blah blah blah blah blah blah hilah blah blah blah…;)

  21. Alexander Ewering Writes:

    Anonymous: Heh, well spotted.

    BTW, this Blog software is broken.. it always fills the last contents of the Website field into the Email field..

  22. Alexander Ewering Writes:

    ethana2: Not wanting to interrupt you in your open source prayers, but it seems from the commit logs that the problem was a bug in Blender, which relied on an ‘undefined’ sort order being undefined in a specific way (heh). So, Solaris just had a different understanding of ‘undefined’ than Linux, which is not really Solaris’ fault. ;-)

  23. anony Writes:

    i swear, this was all andys fault

  24. LOGAN Writes:

    Some of the best productions had some of the worst things go wrong. I guess this would make another incredible DVD documentary. You guys are committed!

  25. campbell Writes:

    Alexander is right, this is not a bug in Solaris. I don’t think its really a bug in Blender either, in some cases systems give different results, we cant always help that. (there are known cases with PPC vs x86, and 64 vs 32 bit)

    To be 100% sure that qsort would always work the same way, we’d just need to include our own qsort with blender.

  26. jmk Writes:

    campbell: Not a bug? Of course it is – Blender is supposed to generate the same files on any architecture. If it doesn’t and can’t be fixed, it needs to be documented. It’d still be a bug, though.
    But in this case, it looks like it can be fixed. So it should. If reimplementing qsort() is what it takes, so be it.

  27. shul Writes:

    Hang on!! don’t let the tension ruin it for you!!

    And most important – enjoy the moment :-)

  28. campbell Writes:

    jmk, CPU’s floating point calculations are not always the same precission, some cases this results in visible differences in a rendered image.

    This bug report is an example…
    http://projects.blender.org/tracker/index.php?func=detail&aid=7977

    Blender can make reasonable measures to ensure results aren’t too noticeable, but in some cases results will be different and it can be “fixed” without adding our own floating point math which would be really slow.

  29. paul. Writes:

    Excuses, excuses . . . Just kidding guys! Good luck and I can’t wait to see it!

  30. DramaKing Writes:

    And just when you thought there would be no worries…. Poor guys.

  31. Lapshere Writes:

    Phew, sounds like a tough night huh? Good luck guys.I will be waiting for the dvd to ship :)

  32. terence Writes:

    The back cover is good with the blah blah blah. I also love the dvd image itself. No words, and the nose of BBB in the hole. Perfect. Send it out just like that

  33. Olaf Writes:

    Get some shuteye and use an eye mask to make sure you’re COMPLETELY rested!

  34. jin choung Writes:

    heheh, boy, i’ve been in this situation before! 8pm the night before and you’re thinking, hey this is gonna be good, we’ve got time. but as the hours wind into midnite, panic sets in as all kinds of things start going wrong and converging for some reason. all of a sudden, the seemingly plenty of time you thought you had before ship time in the morning seems too soon and any thought of sleep vanishes…. good times… good times…. glad you guys are done, congrats and get rest. you rock. i think this hell that you guys have gone through that happened so that it won’t happen to others… all the stuff about openexr and floating point color space is a bug that probably would not have been found otherwise. hellish but not in vain fellows so take heart. p.s. stuff with things looking different on different platforms – i WOULD document that but imo, it cannot possibly count as a bug. when it comes to floating point rounding and in certain cases how random numbers get evaluated for noise, it’s pretty common knowledge that platforms can end up generating different results.

  35. Olaf Writes:

    I have an important question for you guys: when will the DVD’s be shipped?!

  36. joesixpack Writes:

    Keep up the good work!

    Do you know who’s using a vodoo doll of you all?

  37. sNovak Writes:

    Ahhh, the joys of last minute cramming.. Way to pull it off guys!!!

  38. UglyMike Writes:

    3 more days to go guys!! I hope everything is going as well as it can!! BTW nice write-up in 3DWorld!!

  39. Gianmichele Writes:

    only one day to go….good luck guys!

  40. Haunt_House Writes:

    Now that list really comforts me, when I think of the next few years of my life.

    Well, at least it’s guaranteed not to get boring.

    cheers

    haunt

  41. randomnut Writes:

    Try streflop! http://nicolas.brodu.free.fr/en/programmation/streflop/index.html

  42. Vojtas_CZ Writes:

    Enjoy your D day tomorrow guys. It’s yours day.

  43. z0man Writes:

    Sad to hear how everything read wrong :( “Sods law” alright. One day that dreaded ‘500 internal error’ will be a thing of the past (me hopes).

  44. gumanoed Writes:

    :) Hope all bad lucks will pass away.

    When are you going to reliase Big Buck Bunny here. I would like make a presentation of the movie in a local movie fiature and i have orded the time. Can i order the next saturday the 19 of apr?

  45. Joeri Writes:

    I see that some things will never change.

  46. TxRx Writes:

    Welcome to my world… lol. That’s all happened to us before here at work. I hate USB drives, how the hell do they say that USB2.0 solves all transfer issues. Doesn’t help when the machine your using doesn’t have USB 2.0! lol. You wouldn’t believe the number of last minute things you can pick up on. With Peach being a (kind of) one off Blender gang isn’t blessed by experiencing these issues on a regular enough basis to squash them. Not a crit at all, more like, keep making shorts & films and the path to deadlines like this will be old hat and a lot smoother!

    Great work guys!!

  47. Tynach Writes:

    Darn, Deak beat me to my comment.

  48. cresshead Writes:

    well done ..looking forward to the dvd arriving in the mail on fire!
    [i hope not!]