Woosh, the deadline passed! Or, how everything that can go wrong does…
Posted in Production by Ton
Momento; no worries! The master film+sound disks are ready, packed, sealed and were picked up by the courier. (All will be OK, airplane personnel strikes, bad weather alarms and Norway sliding into the Atlantic notwithstanding).
For your pleasure and probably to learn from (do we ever!) here’s the disaster list of all that went wrong last night, causing Andy + Sacha + Campbell + Brecht to skip sleep…
- Sun grid crash! Usually when a node goes down it ends our session, but it can take 3-5 hours to have it all running again.
- The credits came back from the renderfarm with the default font, and adding the desired font didn’t work anyway since we compiled blender without truetype font support. Converting it to vectors messed up a few character, so we ended up rendering the credits (2000+ frames with character animation) on our own systems.
- So let’s use our own studio computers to render last minute changes. But… the installation differs for systems, giving problems.
- Then a bug in the new Approximate Ambient Occlusion appeared, but luckily fixed
- And then the grass looked completely different compared to the Sun grid images… which appeared to be a different implementation of qsort() in Solaris!
- Blender Sequencer was used to convert float OpenEXR images to 10 bits DPX. Unfortunately in some cases, on effect strips, the strip ignored the float images…
- DPX converting started with image names saved with 4 digits. We have 14315 images to save…
- Andy made a great advanced setup for last minute color grading in Compositor. It was far too slow to be useful, even though a promised patch was supposed to fix that. He could only do a couple of shots.
- Our provider had nameserver issues, so we couldn’t use svn.blender.org to update code.
- Ooh, let’s not forget the start title!
- When we testing things yesterday, we forgot to write to the needed USB disk… which was saving very slow. Takes 4-5 hours to write everything, which we didn’t have anymore.
- Using our own backup disk, where all files were, was an option. Unfortunately it was formatted to be Linux-readable only…
- Luckily at Filmmore – where the files were converted to log dpx for the film printer – they had Linux, but reading it all in would take about 2.5 hours. Just within the limits.
- After it all was ready, we nearly missed (thanks Sacha!) the error of 40 black frames in the movie… caused by a bad symlink in our finals directory structure.
- Dang, one shot is off with three frames… luckily there’s no sfx in it, so hopefully nobody sees it.
And… after spending almost an hour on this blog post, I lost all text when I tried to save it to blender.org, who just decided to go into a “500 Internal server error”.
Off we go!
-Ton-
(Image 1: how I found the four victim’s this morning. Image 2: final check in Filmmore of the movie…)
April 3rd, 2008 at 7:13 pm
Murphy’s laws seems to work.
April 3rd, 2008 at 7:16 pm
Oh no, wish you more luck for the mastering of the dvd
April 3rd, 2008 at 8:01 pm
This blog post would make a good movie in itself.
April 3rd, 2008 at 9:17 pm
That must’ve been generating the wrong kind of adrenaline :)
April 3rd, 2008 at 9:19 pm
go and get some rest before the premiere, and then back on work for the dvd…you really deserve it guys
April 3rd, 2008 at 9:24 pm
sounds like fun. :) but what does this mean now? will the dvd contain a version with correct color grading and so on? :p
April 3rd, 2008 at 9:42 pm
hehehe so many thing happen there! but is a great new to know that the movie is finished on time!
Congratulations!, you demostrate that good planning is very important in this kind of projects!
Regards! When you will ship the DVDs???
April 3rd, 2008 at 10:12 pm
That definitely sounds like a rough night – I’m sure you guys will sleep well for the next few nights!
April 3rd, 2008 at 10:57 pm
so nice to hear you’re still human :)
have a nice rest and prepare for the premiere!
April 3rd, 2008 at 11:12 pm
Oh my God, I’ve never seen Murphy’s law apply so much… Best of luck! ;)
April 4th, 2008 at 1:11 am
Uuhh, sorry if I’m wrong. But it sounds like you guys are basically saying ‘we’ve had to cut corners, we apologise if it looks a bit crappy in some parts of the movie.’. Is that right?
…..agh! I’d rather you released late than released ‘on time’ and it wasn’t finished properly!
On another note, I know how it feels when everything starts going wrong and you’re on a deadline. Get some sleep!
April 4th, 2008 at 2:06 am
Socceroos, they can’t be late as this is for the premiere!
April 4th, 2008 at 2:43 am
Hey, drama! Reminds me of how we used to leave assignments to the last date and do an all-nighter to only submit it later the next day.
Ofcourse this is a bit of a different situation since things actually went wrong and Peach didn’t actually /procrastinate/ in any sense…
…did they?
April 4th, 2008 at 2:53 am
“This is nothing. …piece of cake. Y’know, producing is being a Samurai Warrior. They pay you, day in, day out, for Years, so that, ONE DAY, when called upon, you can respond, your training At Its Peak, and save the day.” :-D
April 4th, 2008 at 3:07 am
I know that “CMS goes into error when hitting SAVE” situation very well – I also had spent so much time on a blog post that the PHP session would have expired when I hit Save :-) That’s why I tend to CTRL-A CTRL-C CTRL-V the text somewhere before I click Save.
Anyway, the story was an exciting read, one could feel the adrenaline :)
April 4th, 2008 at 4:40 am
Is qsort() defined by POSIX?
Did solaris crash?
*sigh*
…I’m glad they had the sense to use linux at that place you went to.
April 4th, 2008 at 4:42 am
Sensei: Blender is FOSS, making them ninjas. Samurai are proprietary devs.
…and ninjas are better ^_^
April 4th, 2008 at 4:47 am
“I love deadlines. I like the whooshing sound they make as they fly by.”
– D. Adams
April 4th, 2008 at 5:07 am
Oh don’t you hate it when you get finish writing an awesome blog post, and you lose it! I have had that happen to me on multiple occasions, before I started copying before submitting. =P
April 4th, 2008 at 6:08 am
Hey, I like the back of the DVD cover.Big Buck Bunny Blah blah blah blah blah blah hilah blah blah blah…;)
April 4th, 2008 at 6:48 am
Anonymous: Heh, well spotted.
BTW, this Blog software is broken.. it always fills the last contents of the Website field into the Email field..
April 4th, 2008 at 6:58 am
ethana2: Not wanting to interrupt you in your open source prayers, but it seems from the commit logs that the problem was a bug in Blender, which relied on an ‘undefined’ sort order being undefined in a specific way (heh). So, Solaris just had a different understanding of ‘undefined’ than Linux, which is not really Solaris’ fault. ;-)
April 4th, 2008 at 7:56 am
i swear, this was all andys fault
April 4th, 2008 at 10:07 am
Some of the best productions had some of the worst things go wrong. I guess this would make another incredible DVD documentary. You guys are committed!
April 4th, 2008 at 10:43 am
Alexander is right, this is not a bug in Solaris. I don’t think its really a bug in Blender either, in some cases systems give different results, we cant always help that. (there are known cases with PPC vs x86, and 64 vs 32 bit)
To be 100% sure that qsort would always work the same way, we’d just need to include our own qsort with blender.
April 4th, 2008 at 11:03 am
campbell: Not a bug? Of course it is – Blender is supposed to generate the same files on any architecture. If it doesn’t and can’t be fixed, it needs to be documented. It’d still be a bug, though.
But in this case, it looks like it can be fixed. So it should. If reimplementing qsort() is what it takes, so be it.
April 4th, 2008 at 11:46 am
Hang on!! don’t let the tension ruin it for you!!
And most important – enjoy the moment :-)
April 4th, 2008 at 11:55 am
jmk, CPU’s floating point calculations are not always the same precission, some cases this results in visible differences in a rendered image.
This bug report is an example…
http://projects.blender.org/tracker/index.php?func=detail&aid=7977
Blender can make reasonable measures to ensure results aren’t too noticeable, but in some cases results will be different and it can be “fixed” without adding our own floating point math which would be really slow.
April 4th, 2008 at 3:30 pm
Excuses, excuses . . . Just kidding guys! Good luck and I can’t wait to see it!
April 4th, 2008 at 4:38 pm
And just when you thought there would be no worries…. Poor guys.
April 4th, 2008 at 7:00 pm
Phew, sounds like a tough night huh? Good luck guys.I will be waiting for the dvd to ship :)
April 5th, 2008 at 6:17 am
The back cover is good with the blah blah blah. I also love the dvd image itself. No words, and the nose of BBB in the hole. Perfect. Send it out just like that
April 6th, 2008 at 12:49 am
Get some shuteye and use an eye mask to make sure you’re COMPLETELY rested!
April 7th, 2008 at 7:54 am
heheh, boy, i’ve been in this situation before! 8pm the night before and you’re thinking, hey this is gonna be good, we’ve got time. but as the hours wind into midnite, panic sets in as all kinds of things start going wrong and converging for some reason. all of a sudden, the seemingly plenty of time you thought you had before ship time in the morning seems too soon and any thought of sleep vanishes…. good times… good times…. glad you guys are done, congrats and get rest. you rock. i think this hell that you guys have gone through that happened so that it won’t happen to others… all the stuff about openexr and floating point color space is a bug that probably would not have been found otherwise. hellish but not in vain fellows so take heart. p.s. stuff with things looking different on different platforms – i WOULD document that but imo, it cannot possibly count as a bug. when it comes to floating point rounding and in certain cases how random numbers get evaluated for noise, it’s pretty common knowledge that platforms can end up generating different results.
April 7th, 2008 at 12:18 pm
I have an important question for you guys: when will the DVD’s be shipped?!
April 8th, 2008 at 7:58 am
Keep up the good work!
Do you know who’s using a vodoo doll of you all?
April 8th, 2008 at 3:19 pm
Ahhh, the joys of last minute cramming.. Way to pull it off guys!!!
April 8th, 2008 at 4:11 pm
3 more days to go guys!! I hope everything is going as well as it can!! BTW nice write-up in 3DWorld!!
April 9th, 2008 at 12:53 pm
only one day to go….good luck guys!
April 9th, 2008 at 1:53 pm
Now that list really comforts me, when I think of the next few years of my life.
Well, at least it’s guaranteed not to get boring.
cheers
haunt
April 9th, 2008 at 7:23 pm
Try streflop! http://nicolas.brodu.free.fr/en/programmation/streflop/index.html
April 9th, 2008 at 8:33 pm
Enjoy your D day tomorrow guys. It’s yours day.
April 10th, 2008 at 3:30 am
Sad to hear how everything read wrong :( “Sods law” alright. One day that dreaded ‘500 internal error’ will be a thing of the past (me hopes).
April 10th, 2008 at 7:52 am
:) Hope all bad lucks will pass away.
When are you going to reliase Big Buck Bunny here. I would like make a presentation of the movie in a local movie fiature and i have orded the time. Can i order the next saturday the 19 of apr?
April 10th, 2008 at 1:17 pm
I see that some things will never change.
April 11th, 2008 at 12:00 pm
Welcome to my world… lol. That’s all happened to us before here at work. I hate USB drives, how the hell do they say that USB2.0 solves all transfer issues. Doesn’t help when the machine your using doesn’t have USB 2.0! lol. You wouldn’t believe the number of last minute things you can pick up on. With Peach being a (kind of) one off Blender gang isn’t blessed by experiencing these issues on a regular enough basis to squash them. Not a crit at all, more like, keep making shorts & films and the path to deadlines like this will be old hat and a lot smoother!
Great work guys!!
April 11th, 2008 at 4:41 pm
Darn, Deak beat me to my comment.
April 11th, 2008 at 10:41 pm
well done ..looking forward to the dvd arriving in the mail on fire!
[i hope not!]