Last Day

After the exciting and eventful last week, this week has been very quiet and full of repetitive work. I have spent the week gathering all documents and files relating to the Next Generation Cinema Technology project and created records for them in an inventory spreadsheet. There ended up being just over one hundred records (including the large inventories of specific NGCT test shoot images as two separate individual records) specified in the inventory and heading to Norma for ingestion into the Science and Technology Council Archive.

Also this week, Mio has helped me transfer the actual NGCT image files from the server onto three hard drives. He wrote a shell script to carry out the transferring and check sums, which made the process quick and smooth. All 15 TB of the processed files took three days to transfer to the drives. We decided to make a copy of the raw files onto LTO-6 tapes which have a shelf life of 100 years. This was partly for ease of transition and partly to check that the Academy’s technology is working correctly.

Finally, I created two “readme” documents for the Science and Technology Council Archive and USC.The Sci-Tech readme explains the process I went through during my internship, the milestones, helpful points, and people I worked with. The USC document describes the file structures on each drive, the material on the LTO-6 tapes, and the other helpful documents I included with the drives.

With that, my internship is complete! Every day this summer I thought how lucky I’ve been to receive such an extraordinary opportunity. The project was perfect for my interests and future career, and even sparked more passion for film and film archiving than I had previously. I got to see the inner workings of an industry organization, rather than an academic organization, and found that I enjoy working for a non-profit organization. The Academy staff is fantastic, the building is beautiful, and Los Angeles has treated me well. I’m forever grateful for this internship and everything I will take with me!

An Eventful Wednesday

Yesterday, two phenomenal events happened at my internship: 1) Tobin Bell had lunch with us and 2) I attended and spoke at an American Society of Cinematographers (ASC) technical meeting.

Tobin Bell is an american actor perhaps known best for playing Jigsaw, the villain of the Saw movies, but also appeared in Mississippi Burning, Goodfellas, and the Quick and the Dead, to name a few. Since none of the interns are actors or want to work in film production, we threw him for a tiny loop. Instead of giving acting advice as normal, he gave career advice in general. He showed us 53 photographs and said a small comment about each one. He said he chose 53 photographs because he had 53 jobs before he landed his first role in which his acting career took off.

Tobin also asked about us ourselves, where we hope to be, and our thoughts about the academy. Overall, it was a fantastic lunch hour and he was the first famous person I saw in LA!

Last night, I had the invaluable experience attending and speaking at the ASC meeting. The technical meeting was held at the ASC Clubhouse and the main room was filled with maybe fifty to sixty people. However, these people were not just seat-fillers, they are some of the most brilliant film technologists and artists in the world. Sitting next to me at our table was the couple responsible for the “biggest, brightest” movie being played on an IMAX screen right now, in the world.

Across the room was a man who was a Sci-Tech intern at the Academy like me, and is now at the top tier of Netflix. And scattered were people who had set the standards, even two decades ago, that film production and distribution still uses today. It was extremely inspiring and even more humbling to speak with this crowd, however briefly, about my experience at the academy and my future digitally archiving arts and humanities material.

Migration

While putting the Next Generation Cinema Technology materials together last week, I thought about how we are going to actually transfer 15 TB of data off of the server in the Academy’s Imaging Lab to the digital repository at USC. After asking Sadie this very question in an email, she responded that USC will accept hard drives with the materials and will return those drives once finished transferring the materials to their repository.

I emailed my supervisor asking if we had extra hard drives or if we would have to order new. He suggested working with Mio, here in the imaging lab, to order new drives. Once Mio was on the case, he wanted to check with Dan to see what would work. Dan, in turn, pulled up a chair to discuss transferring materials off of a certain kind of server onto certain kinds of drives, to be transferred once more into a certain repository, and how all of these kinds of hardware must be able to speak to each other.

This sparked an entire conversation about how hard drives work to write and save data to their hardware. Dan also gave me a sort of history lesson in how the drives we have today came to be and the differences between Windows, Mac, and Linux systems with hard drives. A lot of it was over my head, but I think it will be extremely helpful if I ever have to deal with large systems and hard drives again in the future (probably will).

The conversation resulted in me assuming that, since our system can handle any of the various electrical filesystems and interfaces, we should do whatever USC needs in order to properly ingest the materials. This was confirmed and we decided to go with NTFS and USB 3. Now Mio will order the correct drives, we can transfer the materials, and I will hand them over to Sadie/ Digital Management Services with any appropriate metadata files, who will give them to USC!

The Bit Rate Task Force

Since I am completely new to everything concerning film, whether that is lighting, color, cameras, preserving film, I’m lucky enough to be in a place where I feel as though I’m learning everything I didn’t know through osmosis. Striking up a conversation with the house cinematographer leads to more knowledge about how a camera works than I ever thought possible. Speaking with my supervisor leads to an hour long discussion of film formats for filmmakers versus cinemas. I have always had an interest in film, but this internship and the people at the Academy are making that interest exponentially stronger.

Michael, my supervisor, invited me to a lunch with the Academy’s Bit Rate Task Force, a working group linked closely with the Next Generation Cinema Technology group (which produced the material I’m archiving). Aside from the distinct personalities each member brought to the table, I attempted to follow the volley of speech around the table, learning as much as possible. We mostly discussed bit rates, but the main purpose of the meeting was to form ideas for a shoot concerned with bit rates. I found each members knowledge about the technical details of film and filmmaking the most intriguing part of the entire meeting. If one member suggested an aspect, guaranteed another member would pick it apart and alter it, improving some detail.

The two parts I remember most were panning and tricking audiences. They discussed how panning a large crowd at a basketball game (for example) fast would have a lower bit rate because there is more blur. However, panning a large crowd slowly produces more detail, therefore producing a higher bit rate. In order to test out the videos once they are shot, one of the members proposed utilizing psychophysics in building a post-survey where the main questions in which the Task Force is interested were hidden within other, more explicit questions. The production of film is constantly occurring, so it makes sense that filmmakers and producers want to know how the audience feels, and must do it on the sly to get the honest feel, especially for comparison.

Overall, an interesting lunch. I hope to continue learning about film and filmmaking when I leave the Academy in August.

Archiving for Sci-Tech

After speaking with Sadie in the Academy’s Digital Management Services, I had a meeting with Norma today. Norma is the archivist for the Science and Technology Council. While Sci-Tech has an archivist and archived materials, it does not have its own “archive” space and infrastructure. Since DMS now exists to funnel materials into USC, the Academy has a place for the materials, and their records can be kept by Norma within Sci-Tech.

During the meeting with Norma, I learned about how she creates spreadsheet records of materials. We discussed the appropriate metadata to include in the spreadsheets and what not to include. A lot of Norma’s job is weeding records that are either duplicate or frankly unnecessary. It sounds like filmmakers or film technologists “keep everything” when they don’t necessarily need to. One of the biggest questions I’ve seen here with digital film archiving is what to keep. We want the final version, yes, but we also want outtakes, drafts, screenplays, edited screenplays, and so forth. I personally find it interesting to have a record of the film process, throughout its lifetime.

Norma said a valuable thing to me about purging and keeping, she asks: if we want to recreate this [thing] someday, what are the vital records we need to keep now in order to do that? Archives are irrelevant if the things in them are never used, looked at, built upon. So keeping the materials that are vital to the project seems like such a common sense answer, but is so useful when face-to-face with the actual documents themselves, making the decision.

DMS Meeting

Today, I met with Sadie Menchen Schwartz, Digital Archivist for the Academy’s Digital Management Services (DMS) department, to ask about next steps in archiving the 17 terabytes of NGCT materials. After explaining the file structures, file types, sizes, and purpose of the NGCT shoot/materials, she shrugged and said I seemed to have a good handle on the materials. She proposed migrating them to the University of Southern California (USC) digital film repository, as the next major step.

Sadie went on to explain the partnership between the Academy and USC through the Digital Management Services department. The Academy created the department after seeing how each department in the organization carried out digital archive/preservation practices on their own, without a central standards system or procedure. The purpose of DMS was to bridge and integrate all of these practices and efficiently archive/preserve the necessary materials under one umbrella. Luckily, they have USC as a partner, with materials being kept in a large digital repository and maintained with very high professional quality, without putting a heavier burden on the Academy.

My next steps after meeting with Sadie are to confirm all files are together and accessible. I also mentioned I would create a general document about the project explaining it and the files we are giving to USC. Much of the metadata files and peripheral materials concerning the project will be archived here in the Science & Technology Council so that Sci-Tech has a record and can easily retrieve what they need (I’ll speak with the Sci-Tech Archivist, Norma Vega, about the necessary materials).

The BIG [DATA] Question

I need you to engineer me a hard drive that can contain trillions of hours of film footage at the highest quality possible and lasts forever. Now.

This want/need of archivists is the essence of The Digital Dilemma and The Digital Dilemma 2, two studies done by the Academy. The Digital Dilemma discusses a need for infrastructure and metadata standards to preserve movies on film and born-digital materials for guaranteed access decades into the future. Hint: there is no solution yet. The Digital Dilemma 2 takes this a bit further, applying the preservation of big data problem to independent filmmakers, documentarians, and non-profit archives.

Currently, YCM Separation and Print, the most robust system for film preservation, estimated at 100 years, would take about an hour to archive the NGCT material and cost more than $150,000. Hard drives, while admittedly advancing in storage space and lowering in cost, only stay accessible for 3-5 years, by which the content must be migrated to a new hard drive. The Digital Dilemma addresses this as a relevant issue that all industries handling big data must come together to solve.

Less than 30 percent of all films produced and screened in a year belong to major Hollywood studios. The rest consist of indie films and documentaries, which in turn forms much of the cultural record. However, with little financial stability, the independent filmmakers and documentarians cannot afford the necessary infrastructure, tools, and time to successfully preserve their items as part of the cultural record. Even if those films end up in a nonprofit archive, the institution itself faces a similar struggle with resources and funding. Even then, if the film ends up in an institution that specializes in preservation, the standards may not be the same as ones utilized in a similar institution across the country.

I, like many of the independent filmmakers, believed that digital storage was more reliable than film or other types of storage, but that is simply false. Even manuscript digital reproductions are typically high-resolution image files. While there aren’t as many files for a manuscript as contained in a film, the space issue for large files is still present.

So how do we, 1) figure out archival standards for film across all industries and entities, 2) provide the infrastructure to securely preserve these made-digital and born-digital items centuries into the future, and 3) make this infrastructure affordable and accessible to organizations and individuals without brand name backing?

The two studies discuss collaboration across entities which I think is the key to solving the first one. And not necessarily just collaboration, but collaborators forming one entity that decides the standards each organization implements so that there aren’t several proprietary metadata systems, but one universal system.

The other two questions are harder to answer, themselves the digital dilemma. Perhaps there is another way to use a material or chemical process which can store information in a new way. Perhaps there is a way to leverage proprietary server systems or cloud storage. I’m not satisfied in believing that migrating the data from hard drive to hard drive every 5 years is the best answer we can think of, nor that it will not damage or lose the data over time.

Reflections to be continued as I research more…

Where are the raw files?

Today, I met with Michael to go over my Master Asset List draft and ask a few questions. In order to create the Master Asset List, I pulled from the files on the Baselight server. I compared these directories to the deliverables matrix from the NGCT test shoot and a document created by the digital preservation intern last summer, but they didn’t seem to match up correctly.

Michael confirmed my thoughts and explained what the files on the baselight server were for: renders for SMPTE and the other three categories sent to various organizations for further research. So where are the original, raw files from the camera? Michael thought this was a great question and wasn’t sure himself, which was a bit shocking! My next task became to find the raw files!

After leaving Michael’s office, I went to speak with Joe, a cinematographer who knows everything working near the imaging lab. He then talked to Scott, and ended up finding a hard drive in a random cardboard box in Dan’s office with a sticky note on stop warning “KEEP SAFE!” (shivers) and an “NGCT RAW” label. Joe took me through a lesson about shutter angle/speed and explained the different conditions under which the NGCT test images were shot so that I had context for archiving. My task for next week is to analyze what is on the drive and decide how to archive it. Hopefully next week or early the week after, I will meet with someone in the Academy Archives department to learn their best practices.

Master Asset List

In order to archive anything, physical or digital, it is in the archivist’s best interest to do a scan of the collection and basically see what is there. Michael, my supervisor, suggested doing just that. My first task, other than reading and learning about the NGCT test shoot, was to create a Master Asset List of all of the Academy material relating to NGCT.

During the first week, Mio taught me how to access the Baselight server on which the images are kept. However, there are more than 499,000 files in various folders on the server relating to NGCT. The folder structures are similar and predictable and their names relate to the different variables under which the scene was shot. In order to see all of the folders and somehow create a list in a spreadsheet, I got to use the command line (thank goodness I’m working on a Mac)!

Mio first thought I could use a Ruby or Python script to process the file and folder names, but the TREE command with a SED statement after a pipe was suitable! I used the following command once I was in the NGCT folder on the server to display the directory tree (TREE) and change the resulting connector lines into commas to form a .csv file which could be opened in Excel:

tree | sed ‘s/\([0-9]{4}[.][0-9]{4}\)/,,\1/g; s/\([A-Z].*\)/,\1/g; s/\([0-9]*.tiff\)/,,,\1/g; s/\([0-9]*\.exr\)/,,,\1/g; s/│//g; s/─//g; s/├//g; s/└//g;’ > /Users/oscar/Desktop/directories_trans.csv

The TREE command was very easy to use and has a lot of different flags to change the output, such as -d if you want only directories (as shown in the image below), selection of only a certain directory, and -i to take away the connector lines. I am loving the flexibility of this internship, and getting to creatively come up with solutions using my skill set!

Example of the TREE command only outputting directories (folders).

Orientation Day 2

The second day of orientation was entirely a field trip visiting three different locations. The first was the Fairbanks Center for Motion Picture Study, which is basically the Academy’s library. As a specialized library, their core collection is mainly published books about filmmaking, Hollywood, etc. however, the building also houses various special collections departments: special collections, the photograph archive, and the graphics department (these are the ones we saw, but I believe there are others, as well). Special Collections holds several personal collections from people involved in the American film industry. Some records relate to their films specifically, and others are about their lives. Nearly every collection or item is donated by the subject, their family, or head of their estate. We saw the original screenplay page of the Psycho shower scene, a scrapbook from the Wizard of Oz set, a letter from James Dean’s high school about his academic performance, and the original lion wig from the Wizard of Oz.

The photograph archive at the Fairbanks library holds any type of photograph imaginable, from actors on set, the movie-making process, or even photographs shot for movie advertisements. The graphics department holds material such as original movie posters and original costume/set drawings. I absolutely loved every part of the Fairbanks library and can’t think of any other material that would be more exciting to work with on a daily basis.

Just before lunch, we headed to the beautiful DreamWorks campus with Spanish architecture, a creek, and so many trees. There we saw a preview of a film soon to come and a short video about the animated film production pipeline (process), then headed to lunch. At DreamWorks, breakfast and lunch are free to employees and any of their guests (AWESOME!) so we got in line, school-style, and picked out whatever we wanted. After lunch, we filed into a room where an animator was sitting in front of four computer monitors. He explained the process of animating each character’s facial and body movements, sometimes filming himself first to get the motions human-like and believable. This was absolutely fascinating because each animator on the animation team working on the film has to be detailed in every single scene for the entire movie.

We all reluctantly left DreamWorks to arrive at our final orientation destination: Paramount Pictures. But first, we were ushered into Technicolor and spoke with a representative about their processes. We also got to sit in with two sound mixers and ask questions about their jobs. However, the most breathtaking/shocking/phenomenal part of this day and orientation in general (in my opinion) was getting to see a foley sound stage. When a scene is shot, there is dialogue, movement, and other sounds naturally recorded, but they aren’t at good enough quality. The foley artist’s job is to recreate that sound for the final film. The foley sound stage (room) basically looked like a shed with different types of flooring and junk acquired over years. Within this room, the foley artist can recreate nearly any sound in a movie, not necessarily using the same materials as the actor in the film. Just amazing!

After technicolor, we received the grand tour of Paramount Pictures. This includes the “parking lot” they fill with water when they need any kind of water scene, the city street facades, and the classic Paramount gate. To wrap up orientation and send us off, back to our internships, we had a lovely dinner at a restaurant called Off Vine, near the Pickford Center. We got to speak with previous interns and eat a delicious meal!