Adobe Camera Raw, Adobe Photoshop, Adobe Photoshop Lightroom

My Adobe DNG chat with Eric Chan

For the past several years I’ve used Adobe’s DNG file format to store the raw image data captured by my digital cameras. I convert my Canon CR2 raw files to DNG early in my workflow and don’t keep the original raw captures. I’ve had great success with a DNG workflow and since DNG files contains the original raw image data, I’ve seen no need to retain the native files.

In every class and workshop I teach, the subject of DNG inevitably comes up. There’s a lot of confusion and uncertainty about DNG. So when researching subject matter for my next book, I thought I’d polish up my knowledge of this essential image file format.

In doing so, I reached out to one of today’s leading imaging software developers, Eric Chan, Senior Computer Scientist at Adobe. Following is a [very minimally edited] transcript of our email conversation.

(NC – me, Nat Coalson EC – Eric Chan)

###

“NC: Can you give me a concise summary of the DNG conversion process? Specifically, what happens with the raw data, and what does the resulting DNG file contain? I’ve read the stuff on the official Adobe DNG pages, etc. But I saw recently where you stated that DNG is basically a TIF format and this inspired me to get a more useful/practical description from you.

EC: DNG is indeed very much like TIFF.  It’s actually a set of TIFF extensions, with tags to describe things like white balance, color profiles, and various calibration data needed for processing raw images (as opposed to already-rendered images like regular jpegs and tiffs).

A raw DNG typically stores 12-bit, 14-bit, or (rarely) 16-bit Bayer mosaic data, usually in lossless compressed format.  (Uncompressed is also supported, but rarely used because the files are so big!)  The lossless compression is part of the JPEG specification and is simply called Lossless JPEG.  It is a pretty common lossless compression scheme used by many vendors (i.e., both DNG as well as non-DNG raw files).

Aside from the image data itself, it’s important to have the metadata about the image, for image processing purposes.  This includes things like the so-called “as shot” white balance, the white point, the black point, etc.  Some of these tags are required, and others are optional.  The nice thing is that there isn’t much required to put together a perfectly valid DNG file, so a handful of vendors are now doing so in the camera (Casio, Leica, Pentax, Ricoh).

NC: Thanks so much, I think at this point the one biggest question I want to put to rest is this: When comparing the pure, raw, unmodified mosaic data in the original raw capture, is there any difference at all between that data and the image data contained in the DNG? A while back someone was trying to convince me they don’t like DNG because the raw data is being modified from the native capture. I’ve never believed this to be the case. Can you describe to me the nature of the image data in the DNG as it compares to the data in the original native file?

EC: This is a tough question to answer. It is a bit of a gray area. The shortest and most honest answer is: it depends!

Usually the reason for the difference has to do with calibration data. Many cameras perform internal calibration (eg, flat fielding, per-row equalization, etc) before writing out the raw file to the memory card, but others do not (eg, because of limited processing power in the camera). The ones that do not instead usually write a set of tags describing this calibration data to the private metadata of the raw file. It is then expected that the raw processing software (such as Lightroom) would apply the calibration data upon reading in the raw file. The image is practically useless without the calibration data, so it is always desirable to apply it. In other words, the camera vendor would have preferred to apply it directly in the camera, but couldn’t.

The reason I mention this is because when we build DNG images for such cameras, we will usually bake in the calibration data into the resulting DNG mosaic image. Thus, the resulting mosaic data is not identical to the original non-DNG mosaic data because the calibration data has already been applied. This has the advantage of making the image data much more usable to third-party DNG reading software (which usually doesn’t handle the vendor-specific calibration data), and it also makes reading the DNG faster (because the software can skip the calibration step).

There are some other cases where the converted DNG doesn’t have bit-identical pixel data. For example, usually the mosaic data contains extra pixels around the border to help with edge operations (like demosaicing), as well as black-level estimation.  In these cases, we will preserve this edge pixel data when producing the output DNG. However, in some cases the camera will write bogus values to these edge pixels (basically random values –uninitialized memory). These aren’t useful to preserve, and can confuse other DNG readers, so we usually omit these when building the output DNG.

So, we generally preserve the mosaic data exactly, but there are some exceptions.

NC: Thanks. So with all this in mind, is it still accurate to say that the image data in _all_ DNG files is _raw_ data?

EC: No, not necessarily. The reason is that the DNG file format can also store non-raw data.  For example, you can feed a JPEG to the DNG Converter and have it spit out a DNG.  However, that DNG isn’t raw, of course (you can’t take a rendered file like a JPEG and go “backwards” to make it raw again … it would be as miraculous as uncooking a steak!).

At its most essential level, DNG is just a container of image data and its associated metadata.  It can hold raw image data (a.k.a., scene-referred data), and it can also hold rendered image data (a.k.a., output-referred data).  For raw data, the data can be in the mosaic form, or it can be demosaiced (so-called “linear DNG”).  For example, Foveon cameras and BetterLight scanning backs store scene-referred linear data, but they aren’t in mosaic form.

NC: Great, got it. Just one final question, if you don’t mind: if I convert a raw file from a _typical_ DSLR (eg my Canon 5D Mark II) is the scene-referred data contained in that DNG any different than the original scene-referred data in the native raw file?

EC: No, they should be identical in the typical case (e.g., Nikon SLR, Canon SLR).  An exception in the case of Canon SLR is the S-RAW and M-RAW modes.  Those are not really mosaic files; they are stored in a quasi-JPEG like YCbCr space (luminance and color components), which DNG does not natively support.  As a result, if you convert a S-RAW or M-RAW Canon file to DNG, it will get mapped to a RGB color space before getting stored into the DNG container (and hence will not be the same as the original).

NC: Thanks a million for taking the time to explain all this to me. It’s a huge help! I’ve been a big fan of DNG for years and I’m glad my understanding wasn’t so far off that I got myself into trouble 😉

I know that this kind of support isn’t technically part of your job and it means a lot to me and many of my colleagues that you are so willing to share your expertise.”

###

If the above all seems too technical, the main point you really need to understand is that if you shoot with a DSLR and convert your raw files to DNG you are not losing any data or image quality (assuming you’re capturing raw not sRaw). And the advantages of DNG are significant; see the links below for more about this.

As I mentioned in the introduction to this post, I believe in the practical, flexible, long term usefulness of DNG and will continue to use them as my raw file format. I welcome your comments and questions!

You can get more info about DNG at the links below:

Adobe DNG resource pages
Wikipedia

14 Comments

  1. Thanks for posting that Nat – really interesting. Eric’s full of gems like that!

    Reply
  2. Great post, Nat! Thanks!

    I am also a big fan of DNG. One question I didn’t see addressed (and I get this one frequently) is how the conversion to DNG impacts backup strageties. For example with a raw and sidecar XMP, changes made in Lightroom impact the XMP and only that is backed up on the next iteration. However, those same changes would be stored directly in a DNG file and thus trigger a backup of a much larger file the next time around. Is this just a flaw of the format or is there some way around this issue. While it isn’t a large problem on a local backup, cloud solutions will creat a bottleneck when you edit a large number of DNG based images.

    Reply
  3. > A while back someone was trying to convince me they don’t like DNG because the raw data is being modified from the native capture. I’ve never believed this to be the case.

    so I guess you know now that you were wrong and that in fact was the case w/ some cameras :-)… and for some (Sigma) still is.

    Reply
  4. @Victora – yeah, Eric is simply brilliant and we’re very fortunate to have him on our side!! 😉

    @Gene – cloud storage/backup presents some challenges with all types of large files and photography certainly is no exception. One option: for the initial backup, send a hard drive to the service provider so they can copy everything to the servers directly. Then, going forward, the backups will take much less time, since only recently changed stuff is getting copied. IMO the benefit of having one portable file (DNG) outweighs the niceties of standalone XMP sidecars. Hopefully there will someday be a way to only update metadata within a DNG file; Lightroom already does this when using the Save to Metadata command, so maybe it could be implemented over remote storage systems as well.

    @DP – yes, I was definitely wrong! My initial understanding of the DNG conversion process was flawed. I thought that in all cases the raw data was just copied directly to the new file and then wrapped in metadata. I was very interested to learn from Eric that the DNG conversion process can potentially be much more complicated and with many variations depending on the camera. But these cases are the exception; I was happy (and relieved!) to get confirmation that with the major-brand DSRLs, the original raw data remains intact and unmodified in the DNG. (… but I also hope that in the future more cameras will write directly to DNG in the camera.) I think that most people using unconventional gear (like Sigma cameras) get used to the fact that “standard” workflows won’t always apply due to the variation in technology.

    Reply
  5. > I was happy (and relieved!) to get confirmation that with the major-brand DSRLs, the original raw data remains intact and unmodified in the DNG.

    nope… Eric was careful not to mention that it was not always the case either… there were versions of Adobe DNG Converter before (production versions) that were removing some data (for example so called masked from light areas of the sensor, which nevertheless might be used by specialized code/software to fight some forms of banding for some images, like high ISO images) that was read off the sensor and that camera’s firmware was writing in native raw files (Pentax, Nikon)… not a big deal ? may be not a big deal for people who are using only Adobe tools, however that was silently corrected in more recent Adobe DNG Converter versions…

    Reply
  6. so the point is – while a non-native (= not written by a camera) DNG is a perfectly good tool in your workflow, as a intermediate format like for example a TIFF , it is not a reliable archival format… because you never know what is going on :-)… and if you were careless to do a conversion w/o saving the original raw files (either native DNG or non DNG) then you actually even do not know what was lost…

    Reply
  7. DP: all good points and really useful information, thanks for your input!

    Reply
  8. UPDATE: As a result of all I’ve recently learned about DNG I’ve decided to start archiving (to archival gold Blu-Ray) my original, native camera raw files. I will still use DNG as my working raw file format, but retain the original raw captures as well…. just in case 😉

    If you’re using DNG in your workflow I would love to hear your thoughts, comments and ways you handle your raw and working files.

    Thanks for visiting!

    Reply
  9. Nat, thanks very much for sharing this interview with Eric! These were great insights into how the DNG format works. My question is this:
    1. Now that you’ve had a few years of experience working with DNG, are you still using it in your workflow instead of the native RAW formats for each camera?

    2. Why did you actually begin archiving the native RAW files again? Something Eric said must have made you uncomfortable enough to cause the change. Do you continue to archive these native RAW files?

    Thanks again for this great story and the feedback. 🙂

    Reply
    • Ulysses, thanks for your comment. I am converting my raw files to DNG and I do not archive the original raw captures.

      Why? In a nutshell, my conversation with Eric revealed one key fact: when a raw file is converted to DNG, the raw data is demosaiced using the Adobe algorithm. There’s no going back from this conversion (which is why Eric was so reluctant to state the data is not altered ‘in any way’.)

      Although it’s possible that a better demosaicing algorithm exists in competing software, and/or Adobe’s algorithms might improve over time, I believe Adobe’s imaging software represents the best all-around solutions for digital photography. Furthermore, I expect to continue using Adobe (or Adobe supported) software long into the future, perhaps until the day I die. So I am OK with the demosaicing issue. Similarly, I like working with DNGs, even for JPEG captures, because I can get the quality I want from those files with a simplified workflow.

      I don’t want to complicate my file management by keeping multiple versions of a capture in different formats. So even if — from a very esoteric, technical standpoint — there might be a marginally better raw processor for my native raw files from different cameras, I can’t afford to spend the time and effort on a range of different methods for processing my images. For me, DNG represents the best all-around solution.

      So in the end, DNG is still the way to go for my workflow, and I am perfectly OK with deleting my original raw captures after the conversion. As has been shown, this is a very personal decision that might not work as well for everybody.

      I hope this helps. Thanks for reading and commenting!

      Reply
      • Nat wrote “when a raw file is converted to DNG, the raw data is demosaiced using the Adobe algorithm”.
        My impression is that “Raw DNG” contains mosaic data, not demosaicized data and that Eric implied this: “So, we generally preserve the mosaic data exactly, but there are some exceptions.”
        Linear DNG is demosaicized, but I doubt that is what you are converting to early in your workflow.
        So, my understanding is that with your old raw DNG files you can always apply the latest and greatest demosaicing.

        Reply
        • Thanks for your comments David. It’s been my understanding that when the raw data is converted to go into a DNG it is demosaiced. If so, this process can’t be re-done later unless the original raw data is also embedded within the DNG file, effectively doubling the file size.

          But I might be wrong about this. I’m still a little confused about exactly what processing is being done during DNG conversion and Adobe don’t seem overly keen to reveal everything. Eric saying they ‘preserve mosaic data’ doesn’t address this directly. Rather, maybe all the data is still in there, only rearranged.

          Anyway, for my workflow, demosaiced image data in the DNG is perfectly fine as I don’t plan to reconvert any raw files in the future using a different method. I’m able to get the results I want with the current solution. So I don’t retain any of the original raw data (I position I realize is controversial).

          Reply
  10. After reading Martin’s excellent article I now have a much clearer understanding of what happens during the conversion to DNG.

    The original raw data is NOT demosaiced. It simply undergoes a different type of lossless compression. It is still raw mosaic data, which is the crucial point.

    I’m more confident than ever that DNG is the right file format for my photography.

    Reply

Submit a Comment

Your email address will not be published. Required fields are marked *

Get Photography tips, techniques and
tutorials FREE to your inbox

We value your privacy and will never share your details with anyone.

Almost there... Check your inbox to confirm.

Pin It on Pinterest

Share This