Raw Format Workflow, Issues & Information Updated June 24, 2004
Raw Format Workflow Outline:
A. Introduction
D. Raw Format Conversion Tools
E. Workflow
F. Archival and Storage Issues
G. Color Management (New)
This page was first created more than 3 years ago, in connection with the introduction of the Canon D30, which in my opinion was a ground-breaking digital SLR. One of its advantages was the ability to shoot in "Raw Format" in order to obtain the highest quality images this camera was able to produce. But the disadvantage was that, especially in the 'early days,' there were very few software tools to work with these Raw Files. Much has changed since then, and I have finally found and taken the time to update this page to give a more current view on my personal workflow, and the much more sophisticated tools that have become available.
There is no one workflow that will meet all the needs of all photographers, so each must assess their own goals, preferences and needs, survey the available tools, and determine their own optimal way of dealing with digital images and files. My goal here is to provide a basic understanding of the nature and advantages of using Raw Files, the tools generally used for working with these files (reasonably complete, but certainly not comprehensive), and using my current workflow, to demonstrate one way of putting it all together.
While all of my personal experience is with Canon Raw files, most of the discussion and concepts presented here will apply to the Raw Format files produced by cameras from other manufacturers.
Finally, Phase One's introduction of Capture One DSLR, in my opinion, has completely changed the landscape of Raw Format workflow and conversion, and is the single biggest reason my workflow has been changed and improved.
Regardless of what camera you own, you need to choose which format your camera will use to store the images you take. There are 3 basic options here:
1. Jpeg:
When Jpegs are chosen, the original data from the electronic sensor (whether CCD, CMOS, Foveon, etc.) goes through considerable processing in the camera (adjusting of white balance, contrast, sharpness, color saturation, etc.), and is then stored using a "lossy" compressed format called a jpeg. Using the example of a 3 megapixel camera, what starts out as 9 megabytes worth of data ends up as a jpeg of let's say 900 kilobytes, a tenfold reduction in file size. A jpeg is a useful format for general purposes because the file size is so much smaller: more images can be stored on the memory card, they can be emailed or uploaded to the internet quicker, etc.
But that reduced file size is gained at the expense of image quality. Basically, some data is thrown away, resulting in what is usually referred to as jpeg "artifacts." These can be seen when the image is carefully examined, especially in areas like a clear, blue sky, but are visible elsewhere in the image as well. The more the image is enlarged, the more obvious they become. And a further disadvantage of the jpeg format is that these artifacts tend to get more severe and more noticeable if the image is adjusted in a graphics editor (Adobe Photoshop, Corel PhotoPaint 9, JASC PaintShopPro, etc.). In general, the photographer does have some degree of control here, in that most cameras offer different user-selectable levels of jpeg compression, but regardless of which level is chosen, some data is being thrown out. Also, what started out as 12-bits of data per color is now being stored as 8-bits of data per color, so there is data being thrown away here as well. More on this below.
2. TIF:
This is a standard graphics format, which can be read by virtually all graphics programs. As with the Jpeg, the image created here still contains data produced after all the in-camera processing discussed above. But the advangtage (and disadvantage) here is that there is no compression, and therefore no data loss associated with this format. The price paid for this higher quality is the larger file size. Typically a 3 megapixel camera will create a TIF file approximately 9 megabytes in size. This large size typically results in very slow writing times in the camera, and for these reasons, TIF is not in common usage for initial file creation in-camera.
3. Raw:
The image file created by a camera using Raw Format contains the full spectrum of the data captured by the camera's electronic sensor, prior to all of the processing for White Balance, Contrast, Sharpness, Color Saturation, etc. It contains the full 12-bits of data per color, as opposed to the 8-bits/color in Jpegs. And at least in the case of Canon's Raw Format, the data is stored in a "lossless" compressed format, meaning that no data is thrown away. This results in relatively smaller file sizes, but without the data loss that occurs in Jpegs. The downside here is that each camera's Raw Format is created using proprietary technology, and so a conversion to a standardized graphics format is a necessary step.
Ultimately, the decision on whether or not to work with Raw files is a personal one - weighing whether or not the perceived improvement gained is worth the additional time and hassle. In my case, I concluded a long time ago that shooting in Raw Format would enable me to extract the best possible image that I could get, with whatever camera I was using, and so I shoot Raw virtually 100% of the time. And with the current crop of software tools, the 'hassle' associated with using Raw Format has been markedly reduced, as will be seen below. Some of the major areas that benefit from shooting Raw are:
1. 12-bit vs 8-bit Color Depth and 16-bit Editing:
The electronic sensor in cameras capable of recording in Raw format are usually capable of "seeing" 12-bit depth for each color. The bit-depth of an image refers to how many 'levels' per color can be captured in the file: in 8-bit files (used in a Jpeg):, 256 levels are recorded, while in 12-bit files (as in a Raw file), there are 4096 levels for each color. So the Jpeg is once again throwing away data, while the Raw file contains all the information seen by the sensor. Since color in the 'real world' is continuously variable, the greater the bit-depth, the better approximation to what we actually see with our eyes.
And perhaps even more significant than this is that editing can then be done utilizing the full 12-bits, rather than 8-bits, of data, allowing more precision and smoother tonality in the resulting images. And for clarification - where do the "16-bits" come from? - for technical reasons, graphics editors like Photoshop are unable to see and use 12-bit files, and so require files with 16-bits. The 'extra' 4-bits are added without actual data, but the original 12-bits are fully retained. So conversion programs give the user the option to convert to 16-bit TIF's to maintain all the original sensor data. This is now particulary important when using Photoshop CS, which is finally capable of performing the majority of its operations on 16-bit files.
How significant is this bit-depth difference in the real world? Let me defer to the opinion of Bruce Fraser, a very high-respected and knowledgeable Photoshop 'guru.' In an article he wrote a while back, he explains and demonstrates that this is not just a theoretical advantage, but one with real differences in the final images. This article can be read here. Others with similar background and experience have drawn the same conclusion.
2. Dynamic Range:
In addition to containing 12 bits of data per color, most reviewers have concluded that the Raw files contains superior Dynamic Range. Unlike the human visual system, which can encompass an incredibly broad range from dark to light, digital (and film) cameras are much more restricted, and so either shadow or highlight detail will be lost if the dynamic range of the scene being photographed is too wide. Shooting in Raw Format enables you to maximize the range that your camera can capture, giving the photographer more latitude in their exposure. If an image is inadvertently overexposed, some highlights that would ordinarily be 'blown' and lost with a Jpeg, can be recovered in the process of Raw conversion. In addition, you can also convert the same Raw file using different parameters, one with a higher EV Compensation, to bring out the shadow detail, and the other with lower EV Compensation to maximize retention of detail in the highlight areas. These can then be combined in Photoshop to maximize the dynamic range captured in the final image.
3. White Balance:
As indicated above, the Raw file contains the 'pure' sensor data prior to adjustments made for White Balance. Getting accurate colors can be a fairly difficult process, and if a Jpeg was inadvertently shot with the wrong White Balance selected, it can be very difficult to adequately color correct the photo. With Raw format, you have much more powerful options to select and control the White Balance of the converted file, and so color accuracy is much more easily achieved.
4. No Data Loss:
Since there is no 'lossy' compression, the Raw file contains the full data presented at the camera's sensor: there are no Jpeg or other compression artifacts.
5. Processing Latitude:
What I refer to here is that different images may benefit from different processing in terms of brightness, contrast, color sharpness, sharpening, etc., and in fact different photographers will differ in their personal vision of what any given image needs. When using Raw Format, the photographer has complete control over each individual image, so they can adjust the final image according to their personal vision, and doesn't have their options limited by choices already made during in-camera processing.
I think it's important in this context to acknowledge that the benefits obtained by using Raw Format, as outlined above, are going to be relatively subtle in many instances, rather than 'night and day' improvements, and will depend a lot on the nature of the initial image. For example, if an image is taken at relatively low ISO, using a Custom White Balance with a good Grey Card, the exposure is dead-on, the scene photographed has a limited dynamic range, and little to no editing in Photoshop is required, then there is relatively little to be gained with a Raw file vs a high-quality/low-compression in-camera Jpeg. On the other hand, if the opposite is the case (high ISO, lack of opportunity to do an accurate Custom White Balance, difficult lighting, high dynamic range, unintended poor exposure, etc.) then having a Raw file will enable the photographer to end up with a more acceptable image than if it were shot as a Jpeg (in my opinion). Since I know I can't take the "perfect" shot every single time, and that lighting and other conditions will typically further limit my ability to get it right the first time, I shoot Raw format essentially all the time, to maximize the chances that I can get the best quality shot possible.
Approached from a completely different point of view, I would rather take steps that even if only for theoretical reasons allow for the best possible results, with the least compromises on quality, and to me that means shooting in Raw Format.
D. Raw Format Conversion Tools:
Since the Raw Files are in a proprietary format, with no processing applied, they can't be used in that form, and software of some type is needed to process and convert the Raw file to a more standard format that other people and programs can easily utilize. These converters can be put into 3 categories:
1. Manufacturer Supplied Applications:
My comments here will be restricted to Canon, as I've had no experience with the software provided by other manufacturers. ZoomBrowser was the first software Canon provided to work with the Raw files for their digital SLR's. While it 'got the job done' most of us were dissatisfied with its speed, user interface and other limitations. In the years since the D30, ZoomBrowser updates became available, and various other utilities and applications have been provided by Canon, with various acronyms and degrees of improvement, but it seemed that most of the photographers I had contact with still pursued and preferred third party applications. Many have expressed the sentiment that while Canon makes incredible cameras, their software offerings have left much to be desired. Their newest offering, Digital Photo Professional, to be included with the Canon 1D Mark II, is said to offer significant improvements from previous applications. As of the date of this writing, it hasn't yet been released, and no formal reviews have been posted on the web.
2. Third Party Applications based on the Manufacturer's SDK:
The two most prominent applications (at least in the Canon world) are BreezeBrowser and YarcPlus. These are basically 3rd party shells build around the core conversion engine provided by the Canon SDK, with substantial enhancements in features and capabilities.
BreezeBrowser - Chris Breeze is the developer of this fine application, which has expanded dramtically in its depth and breadth in the years since the Canon D30 was introduced. It now supports a wide variety of camera formats, has noise reduction capability, maximizes dynamic range with its "combined raw conversion', has a very versatile HTML page generator, and has many more features. Chris provides first-rate support and both he and his products are well-respected in the digital photography community. More information can be found at the BreezeBrowser website.
Yarc and YarcPlus - the original Yarc was a DOS-based, command-line utility developed by Bruce Henderson, to convert Canon D30 and Canon G1 Raw files. It was oriented toward high volume, production type work, using batch conversion. As time went on, he joined forces with Michael Tapes and YarcPlus was the result. Still based on the Canon SDK for the basic conversion, their collaboration added "ARF" - Artifact Removal Filter - to the mix, and in my opinion generated much clearner images than could be achieved with the Canon SDK by itself. Along with many other features and enhancements, it rose quickly in popularity, and was the foundation of my personal workflow for several years. As careers, tools and technologies have evolved and changed, Bruce and Michael have announced that YarcPlus would no longer be kept current, and while still quite useable with previous Canon cameras, no future support or features will be available.
3. Third Party Applications with their own 'Conversion Engines':
The approach here has been to create a conversion process completely separate from what Canon provided in their SDK. The main 'players' here are:
Bibble - Eric Hyman is the talented creator of Bibble, one of the first 3rd party conversion applications to be available. I played with it a little in the early D30 days, but haven't had opportunity to look at it in the last several years. It seems to be primarily oriented toward Nikon and Fuji cameras, with some support of Kodak, Canon (D30 only) and Olympus. More information can be found at the BibbleLabs website.
Adobe Raw Converter (ARC) - Adobe Photoshop has long been the industry standard in graphics, and has recently developed tools to work directly with the Raw Format files from a variety of cameras. First available as a 'plug-in', this capability is now fully integrated into its latest upgrade - Photoshop CS. As with any other application, it has its share of staunch advocates, and others who find various faults, but it certainly is a force to be reckoned with. I have not had time to evaluate it, and will leave formal comparisons to others.
Capture One DSLR (C1) - In my opinion, C1 has revolutionized the entire field of Raw Format Workflow and conversion. One of its biggest attractions is the ability to work with Raw files in "real-time" in order to see the effects of adjustments made in WhiteBalance, Exposure, Contrast, Saturation, Sharpness, etc., and to do so with a large, high resolution preview image. This makes working with Raw files essentially painless, and removes any serious obstacles to their routine use. When it was introduced, I was quite content with my prior YarcPlus-based workflow, supplemented by various BreezeBrowser features, but felt I really needed to have a look at this new application. The more I used it, the more I liked it. And when I made direct comparisons in terms of basic image quality between my prior Canon-SDK based conversions and those based on the C1 engine, I had to admit that the C1 conversions were clearly superior. They seemed more 'natural' with much better shadow detail. The ability to make adjustments in real-time put it in a completely different category.
To be complete, I have to acknowledge that there are several shortcomings and limitations to the C1 basic user interface that can be very frustrating. But the advantages gained with C1 dramatically outweigh the minuses, which is why I'm happily using it now. Put in more positive terms, if they can make some specific modifications in how tools are accessed and adjustments made, it will be in a category completely of its own, with no serious competition. Improvements have been made, and I am convinced that substantial enhancements in this regard will be forthcoming. A significant upgrade is due out May 15th, and I'll be anxious to see what changes and improvements are made. Michael Tapes (of YarcPlus and Archive Creator fame) is now working with them, and his support and ideas add immeasureably to the C1 'experience.' Additional information on C1 can be found here. (Disclaimer: I have no financial interest in C1, or in any other products or applications discussed here or elsewhere.)
There are probably as many workflows as there are photographers. What I'm presenting here isn't the only way, or the best way, for all photographers. I'm just presenting here the series of steps I usually follow, as it seems to meet my personal needs and goals the best. I continue to re-examine and refine these steps as my needs or goals change, new tools become available, new ideas occur to me, etc.
1. Transfer - most digital cameras come equipped with some type of connectivity (USB 1.1, USB 2.0, Firewire) to get the images from the camera to your computer. But personally, I think this is a very cumbersome way to accomplish this task, so I use a separate Card Reader. Currently I'm using Microtech's Firewire Card Reader. I just take the CF card from my camera, insert it in the Reader, and then run Chris Breeze's Downloader (which I have configured to automatically copy all files to a designated temporary file on my hard drive, then delete the files from the card, and then it shuts itself down).
If you're using a Laptop (as I do when traveling), you can use a standard PCMCIA adapter for file transfers (still using Downloader), or better yet, a 32-bit CardBus PCMCIA adapter like the Delkin unit I now own, which allows for much faster transfers than the standard adapters.
2. Cull Images - with the freedom that digital photography brings, I tend to be pretty liberal with my shutter finger, experiment with different techniques, etc., so there's a lot of deleting to be done. The first thing I do is to run Capture One DSLR (C1). If there's a large number of images in that batch, C1 may need a few minutes to generate its preview images, but it's still pretty quick. I usually make one pass through, deleting shots that are obviously unacceptable, and then go through a few more times, deleting as I go. What I really like about C1 is that with a few simple keystrokes, I can very quickly apply EV Compensation (and other adjustments) to get a real-time view of what the converted image will be. This enables me to make accurate assessments on whether a poorly exposed or otherwise borderline shot can be 'saved.' Having the 'split-screen' option also really comes in handy in comparing similar shots of a given subject. At this stage in my photographic career, I delete pretty heavily, keeping only the best and/or uniquely interesting shots.
3. Rename/Renumber Images - this is a personal choice, as I like to have my images in a continuous numerical sequence, with the prefix indicating what camera was used. So for example, I'm now up to image 1D_02750. Still using C1, I select all the remaining images in that batch, and rename them according to this system. One bug in C1 currently is that it doesn't "remember" the last used number, so I have to manually enter the starting digit. It then handles the rest.
4. Convert to 16-bit TIF - in preparing for this step, I make whatever adjustments I need to in terms of White Balance, EV Compensation, and Saturation, for each image. C1 does have the ability to adjust curves and levels, but at this point, I just use EV Compensation to correct the overall exposure, and will make further adjustments in Photoshop. C1 also has some noise reduction capabilities, but unfortunately this is currently selected on a global, rather than a per-image basis (I'm hoping this gets fixed in the May 15th release!). This can add a little time to the preparation process if the images in that particular batch have differing needs, but is still pretty quick. Images can be added to the conversion 'queue' individually, or as a batch of selected images. I then let C1 do its thing, and do something else (on the computer, or elsewhere), until the conversions are done. Once it's finished, I close up C1.
5. Edit 16-bit TIF Images - I open up ThumbsPlus (which I use to organize and categorize my images), go to the folder containing the converted images, and just drag and drop them into my editor one at a time for processing. I am currently using Photoshop CS for my image editing, which typically includes Cropping (which I sometimes do in C1 before conversion), adjustment of Curves and Levels, Cloning and Noise Reduction as needed, Unsharp Mask, etc. I have been particularly impressed with the new Shadow and Highlight Tool in the CS version, as it enables me to very quickly and precisely adjust the overall tonality of the image. Books are written on proper editing of photographic images, and I refer the reader to any of those for details, tutorials and advice on good editing techniques.
6. Save as High Quality Jpeg (PS level 10) - at this stage in the workflow, some photographers choose to archive all of their images as Photoshop files, or 16 or 8 bit TIF's, rather than Jpegs. Because the PS or TIF files can be huge, this has enormous consequences in terms of storage needs. My rationale is that for most practical purposes (posting on the web, at relatively low resolution, emailing friends, viewing on my computer monitor, small prints), a high quality Jpeg, created after all the conversion and editing has been accomplished, will be functionally identical to the much larger PS or TIF files. So I choose to save each image after editing as a high quality Jpeg, requiring substantially less storage space. This is my routine treatment of all of my images. For the much fewer 'special' images, which will be printed large, or used for publication, I go back to the original Raw file, re-convert, re-edit, really taking my time to make each of these images looks its best, upsample as needed, and then save these as TIF files. Since there aren't nearly as many of these, I don't mind devoting some extra storage space to make sure the quality of these special images is not compromised in any way.
7. Transfer EXIF data to Jpegs - this is another personal preference, and isn't a necessity in the overall workflow. I like to be able to see all the original EXIF data in these Jpeg files, and so I use BreezeBrowser's EXIF Copy function to transfer all the EXIF data from the original Raw files to the newly created Jpegs (which have the identical name other than the file extension). When I view my images in ThumbsPlus, the EXIF data is right there, which can be both helpful and educational.
8. Housekeeping - this addresses some of the loose ends and also the storage and archiving strategies I'm currently using:
F. Archival and Storage Issues:
One advantage of shooting with film or slides is that the physical medium containing the image, while subject to ageing and damage, is not dependent on any particular tool or technology, to be viewed. Just shine light through it, and the image can be viewed. This is not the case with digital files, in particular with Raw Format files. As indicated above, the data contained in these files is coded with a proprietary graphics format, and so is dependent on having the correct tools and technologies to be viewed or accessed. There is thus the potential that down the road these files would become unreadable (consider a collection of 8-track tapes, or even vinyl LP's in the not too distant future).
This concern, while valid, doesn't lead me to abandon Raw format, but it does mean that storage and archiving strategies need to be revisited every few years or so. You can't just make a decision once, and forget about it. If one's assessment of the state of digital imaging technology is such that the tools and/or operating systems necessary to convert or view Raw files are on the verge of extinction, then steps will have to be taken to convert the data (or the media used to store the data) into a format that will preserve the ability to access this data for the foreseeable future (and/or transferred to whatever new storage media then available). This does require time and effort, but then again, film and slides also need special care to avoid or minimize the destructive effects of the passing of the years and chemical changes in the medium itself.
Finally, as has been said many times, there are 2 types of hard drives: ones that have failed, and ones that will fail. For this reason, you can't rely on any single storage method for long-term archiving. My current strategy uses a 3-fold approach. I have all of my images (Raw and edited Jpegs) on the primary internal hard drive in my desktop computer, as well as on an external firewire hard drive. When traveling, I take the external hard drive to a physically different location than where my desktop is, just in case of fire or theft. And in addition, I keep all my images backed up on high quality CD-R's as well, and these are stored in a small fire-proof safe in my house. Having a duplicate set of CD-R's in a second location also makes sense, but I haven't implemented this yet. And while I do have DVD recording capacity, I have yet to decide to rely on this as an integral part of my backup strategy, as the format differences and compatibility issues still haven't really settled out to my satisfaction.
I am by no means an expert in this area, which can be very confusing (often increasingly so as you dig deeper and deeper). But the basics are relatively simple: in order to get consistent and accurate colors between what you see with your eyes, and your camera, monitor, printer, etc., you need to implement a Color Management strategy. This consists of obtaining Color Profiles for each device, so they talk the same "color language." Without doing this, there will be variations from one device to another, and from what things look like in "real life." There are many different Color Management strategies, and what I am currently using has no claim to be "the best." It is a practical solution for me, and is producing high quality results. As with all my other endeavors, it is continually subject to scrutiny and modification as time goes on. One excellent site for people wanting to learn more about this subject is Ian Lyon's Computer Darkroom.
1. Monitor
I am using Gretag MacBeth's Eye One Display to create the profile for my (LCD) monitor, and have been very pleased with the results. This can be purchased through many vendors, and has recently been added to Michael Tapes offerings as part of a total digital workflow system - see his site at RawWorkflow. While I have no personal experience with it, many seem to be successfully using the ColorVision Spyder.
2. Camera
I'm using Magne Nilsen's excellent profiles. I used them with my 1D, and he has recently come out with 1D Mark II profiles as well. One advantage of using Capture One for raw file conversion is that it allows the use of 3rd party Camera Profiles like Magne's. Just open the Color Management section in C1, and select the Profile that you want to use for your camera. Magne's profiles can be purchased at his Etcetera site.
3. Working Space
AdobeRGB seems to be the best general space for working with digital images, and that's what I'm currently doing. Working in C1, go to the Color Management section, and select AdobeRGB in the Destination Workspace section. There has been some recent discussion where using PhotoRGB color space can be helpful in certain images, and I'll probably play with this more as the need arises. Finally it should be pointed out that in general, images should be converted to sRGB for posting on the web, since most browsers are not color managed, and so don't recognize or implement color profiles.
4. Printer
This has been a tougher area to find an ideal solution. Printer Manufacturers often supply color profiles for their prints, and while it might be reasonable to expect these would produce high quality results, this is rarely the case, forcing the discriminating photographer to seek other solutions. I tried some of Cathy's Profiles and while I was very impressed with her knowledge and professional behavior, in my particular case, I wasn't very happy with the results. This is in contrast to the experience of several others I know, who have been very pleased with their results using her profiles.
I have also spent quite a lot of time with Mike Chaney's Profile Prism software. Mike is also the author of QImage, which I use for all my printing, and is a very bright and helpful person. I have had a love/hate relationship with Profile Prism, and it has taken a fair amount of time and effort to get the best results with it. But after buying a different (but inexpensive) scanner to use with it, and careful and persistent efforts, I was finally able to get very good results.
Michael Tapes is also gearing up to create Printer Profiles with professional grade equipment, and the profile for Epson Heavyweight Matte paper which I obtained from him, has given me the best results so far, being slightly better than those with Profile Prism.