Big file size

[Debs]

Just been out to grab a coffee and fire off a few test shots on the new A7CR.

Doesn't a 61 megapixel sensor create some blood ridiculous raw file sizes

 

[Uncle Kevriano]

356mb in 16bit on Photoshop.

 

[Debs]

356mb in 16bit on Photoshop.

I know.....it's crazy

 

[Brownie]

You could shoot in Lossless Compressed RAW and save 40%-ish

 

[Debs]

You could shoot in Lossless Compressed RAW and save 40%-ish

Your dead right. But there will always be a tiny thought at the back of my brain that says I'm losing something by applying any form of compression. Nothing to base it on, just a feeling!

 

[Brownie]

Your dead right. But there will always be a tiny thought at the back of my brain that says I'm losing something by applying any form of compression. Nothing to base it on, just a feeling!

It's been proven, but I know exactly how you feel. There's always that nagging doubt.

 

[iamdlewis]

You can always test it out on your own.

 

[Deleted Member 5045]

It's pretty astonishing but computer memory is pretty much being given away today.

 

[AlphaWorld]

Your dead right. But there will always be a tiny thought at the back of my brain that says I'm losing something by applying any form of compression. Nothing to base it on, just a feeling!

If you shoot the original Sony "Compressed RAW", then yes, you are losing some data. We could talk about how it works, and the situations where it can affect your files, but many people find it completely acceptable. I have too much experience with data compression to be happy with it, even though I understand it.

Starting with the A1, Sony introduced "Lossless Compressed RAW" - this is a genuinely lossless compression, and you lose no data using it. Think of it as being like ZIP, where you compress a file, then decompress it, and the content of the decompressed file is identical to the file before compression. You will lose NOTHING using Lossless Compressed RAW. I can explain why it is able to compress the data without loss, but I'll refrain unless you really want to know. Suffice it to say that you will not lose anything by using lossless compressed files, except that some bodies won't shoot at their maximum frame rate using lossless (all the older bodies, and many of the newer ones, only shoot their the maximum frame rate using lossy compressed).

Interesting to note that Sony omitted the "Uncompressed RAW" option on the A6700. I think we will see Sony dropping the uncompressed format in newer bodies. There is no need for uncompressed now that lossless compressed is supported by most of the RAW processing software.

Canon has been using a lossless compressed RAW format since at least 2007 (and probably earlier), and has not offered uncompressed for at least that long.

 

[Brownie]

If you shoot the original Sony "Compressed RAW", then yes, you are losing some data. We could talk about how it works, and the situations where it can affect your files, but many people find it completely acceptable. I have too much experience with data compression to be happy with it, even though I understand it.

Starting with the A1, Sony introduced "Lossless Compressed RAW" - this is a genuinely lossless compression, and you lose no data using it. Think of it as being like ZIP, where you compress a file, then decompress it, and the content of the decompressed file is identical to the file before compression. You will lose NOTHING using Lossless Compressed RAW. I can explain why it is able to compress the data without loss, but I'll refrain unless you really want to know. Suffice it to say that you will not lose anything by using lossless compressed files, except that some bodies won't shoot at their maximum frame rate using lossless (all the older bodies, and many of the newer ones, only shoot their the maximum frame rate using lossy compressed).

Interesting to note that Sony omitted the "Uncompressed RAW" option on the A6700. I think we will see Sony dropping the uncompressed format in newer bodies. There is no need for uncompressed now that lossless compressed is supported by most of the RAW processing software.

Canon has been using a lossless compressed RAW format since at least 2007 (and probably earlier), and has not offered uncompressed for at least that long.

Panasonic's RAW is lossless compressed, there is no other option. They just call it RAW.

 

[AlphaWorld]

Panasonic's RAW is lossless compressed, there is no other option. They just call it RAW.

That's the simple way to avoid arguments over the "best" option

Canon went that way. Nikon and Sony didn't.

About the only option that might arguably be better would be lossless compressed DNG (which would mean not waiting for support of each new camera...).

 

[Debs]

It's been interesting to read all your feedback on this subject and I am going to try the lossless RAW format.

It will be intriguing to see how it works out.

 

[Thad E Ginathom]

It is a case of reading the word lossless carefully: there is no loss. We have been trusting all sorts of data to lossless compression for decades. What comes out is what went in, and vice versa.

So long as the developers/manufacturers are not lying! So long as they are using the word in it's 100% literal and technical sense. I'm not saying it's impossible, but I think it highly unlikely that Sony (or other major camera makers) would do something else.

I use lossless raw, just as I use FLAC for audiofiles, with confidence that I really am not loosing anything.

 

[pointreyes]

About the only unique thing I know about Sony lossless is that lossless compressed is really a compressed TIFF; however, Sony did not follow the TIFF standard.

 

[Brownie]

About the only unique thing I know about Sony lossless is that lossless compressed is really a compressed TIFF; however, Sony did not follow the TIFF standard.

Do you have a cite for that? Not saying you're wrong, but I'd need to read that from an informed source before I believe it. TIFF files can be viewed by any computer like a jpeg, RAW files have to be decoded. Below are couple of articles from Sony. One discusses the differences between RAW and TIFF, the second describes the different types of Sony files and suggests when each is appropriate to use.

https://www.sony.com/electronics/support/articles/00028729

https://www.sony.com/electronics/support/articles/00257081

 

[pointreyes]

Do you have a cite for that? Not saying you're wrong, but I'd need to read that from an informed source before I believe it. TIFF files can be viewed by any computer like a jpeg, RAW files have to be decoded. Below are couple of articles from Sony. One discusses the differences between RAW and TIFF, the second describes the different types of Sony files and suggests when each is appropriate to use.

https://www.sony.com/electronics/support/articles/00028729

https://www.sony.com/electronics/support/articles/00257081

I'm surprised to see Sony state TIFF is not compressed. Maybe since that is from 2019, they were stating it from the view point of them not using lossless compression at that time. TIFF can be lossless compressed.

TIFF - Image File Format

Learn about TIFF file format and APIs that can create and open TIFF files.

docs.fileformat.com

 

[Brownie]

Well, I'm at a loss (get it?). Two parts of this don't make any sense.

1) If the file doesn't follow a TIFF standard, how can it be a called a TIFF?
2) It sounds like the program they're using converts the Lossless Compressed file to a full-sized TIFF and leaves behind a file that's twice as big, but maintains an ARW file extension?

 

[pointreyes]

A TIFF standard varies based on purpose I think. The TIFF/EP standard is probably the one being used with the vendor deciding the JEITA EXIF standard differences that they might want to use. At least that is my best guess on it. On top of demosaicing the raw file and then decompressing the tiff (if in that order, I don't know), I can see why the photo editor will be working the computer harder.

TIFF/EP - Wikipedia

en.wikipedia.org

 

[AlphaWorld]

A TIFF standard varies based on purpose I think. The TIFF/EP standard is probably the one being used with the vendor deciding the JEITA EXIF standard differences that they might want to use. At least that is my best guess on it. On top of demosaicing the raw file and then decompressing the tiff (if in that order, I don't know), I can see why the photo editor will be working the computer harder.

TIFF/EP - Wikipedia

en.wikipedia.org

Not in that order, no. You have to decompress the data first. Think about it. Almost all the data in a RAW file is the samples from the Bayer mosaic - millions of numbers, each representing the intensity of light received by one cell on the sensor - that's the data which the compression must compress to make any real difference to the size of the file. The Bayer de-mosaic process has to access that data, and it cannot do it in compressed form (OK, in theory it could, but the performance would be appalling). So it decompresses the data first, but it does so in memory (not on disk). Your 60-80MB lossless compressed ARW file becomes ~120MB of uncompressed data in memory, then the de-mosaic process turns that into almost 3 times as many 16 bit values (you get a value for each of red, green, and blue); if you save the processed RAW data in uncompressed form it's around 360MB = 60Mp x 3 colours x 16 bits (2 bytes).

TIFF is a general purpose file format, and it's convenient for people to use parts of the format as something familiar - I suspect that a lot of RAW formats resemble TIFF under the covers. If nothing else, it makes it easier for third party software to support new cameras as they arrive. But some parts of the TIFF standard may require a company to reveal things they do not want to reveal, so maybe the company "neglects" to fill in a "mandatory" field or two. or something like that. Or maybe their coders get a little lazy, or don't read every page of the standard! I've implemented standards in the past, and it can be a pain to handle everything.

I suspect that the Time Fix people could have been supplied with object code to decompress Sony's lossless compressed files (it would be a smart move on Sony's part to get third parties to support their files more quickly). If that's the case, I can imagine Sony only supplying decompression code (after all, the companies are decompressing it to do things like run de-mosaic processing, right? They don't need the compression code!). Supplying just decompression code would keep Sony's options open - they could tweak the compression process in the future, if need be, and still decompress it using the same code. That would explain why Time Fix cannot re-compress the file. And you may get various "explanations" as to why that's the case (I've been on both sides of those conversations in the past)

 

[Brownie]

Not in that order, no. You have to decompress the data first. Think about it. Almost all the data in a RAW file is the samples from the Bayer mosaic - millions of numbers, each representing the intensity of light received by one cell on the sensor - that's the data which the compression must compress to make any real difference to the size of the file. The Bayer de-mosaic process has to access that data, and it cannot do it in compressed form (OK, in theory it could, but the performance would be appalling). So it decompresses the data first, but it does so in memory (not on disk). Your 60-80MB lossless compressed ARW file becomes ~120MB of uncompressed data in memory, then the de-mosaic process turns that into almost 3 times as many 16 bit values (you get a value for each of red, green, and blue); if you save the processed RAW data in uncompressed form it's around 360MB = 60Mp x 3 colours x 16 bits (2 bytes).

TIFF is a general purpose file format, and it's convenient for people to use parts of the format as something familiar - I suspect that a lot of RAW formats resemble TIFF under the covers. If nothing else, it makes it easier for third party software to support new cameras as they arrive. But some parts of the TIFF standard may require a company to reveal things they do not want to reveal, so maybe the company "neglects" to fill in a "mandatory" field or two. or something like that. Or maybe their coders get a little lazy, or don't read every page of the standard! I've implemented standards in the past, and it can be a pain to handle everything.

I suspect that the Time Fix people could have been supplied with object code to decompress Sony's lossless compressed files (it would be a smart move on Sony's part to get third parties to support their files more quickly). If that's the case, I can imagine Sony only supplying decompression code (after all, the companies are decompressing it to do things like run de-mosaic processing, right? They don't need the compression code!). Supplying just decompression code would keep Sony's options open - they could tweak the compression process in the future, if need be, and still decompress it using the same code. That would explain why Time Fix cannot re-compress the file. And you may get various "explanations" as to why that's the case (I've been on both sides of those conversations in the past)

Was hoping you'd chime in.

So, given your experience, it sounds like the email pointreyes posted is legitimate?

 

[Thad E Ginathom]

it sounds like the email pointreyes posted is legitimate?

Sounds like it is legitimate, but not in the way many people, including me, would immediately understand it. My misunderstanding would be that TIFF is necessarily an image, whilst everything I read about RAW files always stresses that they are not images until they have been through at least the rudimentary first steps of processing.

 

[Brownie]

Sounds like it is legitimate, but not in the way many people, including me, would immediately understand it. My misunderstanding would be that TIFF is necessarily an image, whilst everything I read about RAW files always stresses that they are not images until they have been through at least the rudimentary first steps of processing.

That's pretty much what's happening when you open an image in a RAW processing software though. Demosaic, for example, is automatically applied along with some basic things in order to have an image that you can see.

 

[GT12]

On my A7RV I shoot in lossless RAW Large. While travelling I was running out of storage so dropped to lossless Medium. Turned out to be a mistake as a lot of shooting was in low light/high ISO and I wanted to use Lightroom Classic’s noise reduction feature but found that would only work on Large not Medium lossless files. That said, the file size grows when using that noise reduction feature, so no escaping needing more storage at some stage of the process.

 

[Brownie]

That said, the file size grows when using that noise reduction feature, so no escaping needing more storage at some stage of the process.

I think they all do. DxO Deep Prime creates a huge DNG file after applying denoise, but you can delete it after you finish processing.

 

[pointreyes]

I think they all do. DxO Deep Prime creates a huge DNG file after applying denoise, but you can delete it after you finish processing.

I found out from DxO, that I can only use large for processing raw files.
To use denoising tech with the mRAW and sRAW files, I have to use the post processed results with Topaz.

https://support.dxo.com/hc/en-us/articles/6940709955229-Are-additional-RAW-formats-such-as-sRAW-mRAW-RAW-S-HE-and-HE-etc-always-supported-if-a-camera-s-standard-RAW-images-are-compatible-

This was also discussed at: https://www.alphashooters.com/community/threads/what-settings-for-raw.8108/#post-51564

 

[AlphaWorld]

Was hoping you'd chime in.

So, given your experience, it sounds like the email pointreyes posted is legitimate?

Yes, that sounds entirely credible.

It’s a shame that changing part of the EXIF metadata means they have to leave the file uncompressed - I’d have tried to avoid that, but I can picture the project manager stamping their foot over any suggestions of spending more time over trying to retain the compression.

 

[AlphaWorld]

I found out from DxO, that I can only use large for processing raw files.
To use denoising tech with the mRAW and sRAW files, I have to use the post processed results with Topaz.

https://support.dxo.com/hc/en-us/articles/6940709955229-Are-additional-RAW-formats-such-as-sRAW-mRAW-RAW-S-HE-and-HE-etc-always-supported-if-a-camera-s-standard-RAW-images-are-compatible-

This was also discussed at: https://www.alphashooters.com/community/threads/what-settings-for-raw.8108/#post-51564

I suspect you can use APS-C M-RAW, too - it is a true RAW file, like RAW-L. The others are something else.

I would be rather curious to hear if DxO works on any other company’s M-RAW or S-RAW.

 

[pointreyes]

I suspect you can use APS-C M-RAW, too - it is a true RAW file, like RAW-L. The others are something else.

I would be rather curious to hear if DxO works on any other company’s M-RAW or S-RAW.

Yes, that is correct. The APSC mRAW is a true RAW file. With the A1, I have been comfortable using mRAW for events and noticed that the images I took in APSC mode could be fed through DxO denoising.

 

[Jeff A]

On my A7RV I shoot in lossless RAW Large. While travelling I was running out of storage so dropped to lossless Medium. Turned out to be a mistake as a lot of shooting was in low light/high ISO and I wanted to use Lightroom Classic’s noise reduction feature but found that would only work on Large not Medium lossless files. That said, the file size grows when using that noise reduction feature, so no escaping needing more storage at some stage of the process.

As a new A7RV owner, your comment about LR's de noise not working with medium lossless was news to me. I have worked little with the R5 but I was planning on some testing to see the difference in photos that I took using different levels of compression. You have just saved me a lot of time. I'll do testing AFTER this coming week, as a Surgeon is going to have his way with me next Thursday.

 

[AlphaWorld]

As a new A7RV owner, your comment about LR's de noise not working with medium lossless was news to me. I have worked little with the R5 but I was planning on some testing to see the difference in photos that I took using different levels of compression. You have just saved me a lot of time. I'll do testing AFTER this coming week, as a Surgeon is going to have his way with me next Thursday.

Best of luck with the surgery!

Using the Adobe Camera RAW de-noise you will want to try out the new AI de-noise; the older algorithm did a horrible job, basically wiping out fine detail - looked like one of those programs that used to be advertised heavily on photo websites, which turned people into plastic dolls by removing all trace of pores.

To use the new AI de-noise you'll want to be using uncompressed RAWs, or using L-RAW at FF, or M-RAW at APS-C - those are the two lossless compressed formats that are true RAW files.