Sony Compression Discussion

[DrJohn]

Does anyone have any experience or opinions on Sony's raw lossless compression vs raw regular compression vs raw?

John

 

[AlphaWorld]

I have stopped using uncompressed on the A1, because lossless compressed contains the same data in less space, Not “visually“ the same, or ”almost“ the same, but actually the same. If you have a camera that supports lossless compressed, the only reason to use uncompressed is if you are taking the files to a piece of software which supports uncompressed, but not lossless compressed - most of the main packages support lossless compressed ARW files now.

As an aside, Canon RAW files have been lossless compressed for a very long time - the original 5D supported them in 2007 (albeit in 12 bit, because the 5D was a 12 bit camera), and even earlier. Not sure when they started, but it was before 2006. Nikon was later, and offered choices for quite a while (Canon didn’t). Sony was very late to the party on lossless compression.

As for lossy compression, I have no problem with it for output formats like JPEG - it allows JPEG files yo be much smaller than they would otherwise be. But I don’t like it for RAW (I want my RAW files to be the complete data from the sensor).

I have looked into Sony’s lossy compression (it is documented!) and it is quite clever. It only loses precision where there is a wide range of values within a fairly small block horizontally - vertical edges between black and white for example - and even there it only loses some low bits. It can introduce artefacts, which is why I avoid it, but I really do believe the people who say they have never seen any artefacts.

I am sure Sony introduced their lossy compression to allow the relatively low-powered processors in their cameras to compress the files quickly - I think they include hardware specifically tailored to this compression so they could produce smaller files very quickly, which is why the fastest shooting modes require the lossy compression. Now that Sony is using faster processors and bigger frame buffers, maybe they will lift that restriction (we can hope!)

So for my own use (your choices may be different), I will use lossless compression on cameras that support it. I won’t use lossy compression, and I accept the hit that imposes on my frame rate. And I’ll only use uncompressed until my new camera gets lossless support in my RAW processing tool.

 

[AlphaWorld]

Well, I'm back to uncompressed. Not because the IQ is an issue, but Affinity Photo reads it as a 36.7MP files (A74 is 33MP) and puts a black band on two sides. It requires me to crop every photo to get rid of the bands. Tried Darktable too, it doesn't recognize the file at all and won't open it.

Also, I made over 500 photos yesterday and uploading was no faster, so the heck with it.

that would annoy me immensely - can’t blame you for switching back.

I’m sure it’s no comfort to hear that Photoshop and LightRoom have no problems with lossless compressed files, nor does DXO PhotoLab. I would have thought most software understood lossless compression by now, given it appeared on the A1 early last year, so they’ve had 18 months to get it working.

Surprised you saw no speed increase loading the files, though. I load using a card reader to an SSD, and the time taken is consistently related to the number of bytes, so compressed files do load faster. Are you loading from the camera?

 

[pointreyes]

that would annoy me immensely - can’t blame you for switching back.

I’m sure it’s no comfort to hear that Photoshop and LightRoom have no problems with lossless compressed files, nor does DXO PhotoLab. I would have thought most software understood lossless compression by now, given it appeared on the A1 early last year, so they’ve had 18 months to get it working.

Surprised you saw no speed increase loading the files, though. I load using a card reader to an SSD, and the time taken is consistently related to the number of bytes, so compressed files do load faster. Are you loading from the camera?

Much to my surprise DXO PhotoLab 6 latest version does not support M lossless compressed raw from my A1. Suspect the same for S lossless compressed raw file.

 

[AlphaWorld]

Much to my surprise DXO PhotoLab 6 latest version does not support M lossless compressed raw from my A1. Suspect the same for S lossless compressed raw file.

Oh, I hadn't tested that. I only use full-size images, but I assumed it was supporting all the sizes.

Then again, Sony had full-size lossless compressed files at release time, but they added M and S sizes in a firmware update.

 

[Deleted Member 777]

Much to my surprise DXO PhotoLab 6 latest version does not support M lossless compressed raw from my A1. Suspect the same for S lossless compressed raw file.

Oh, I hadn't tested that. I only use full-size images, but I assumed it was supporting all the sizes.

Then again, Sony had full-size lossless compressed files at release time, but they added M and S sizes in a firmware update.

I remember reading somewhere that someone else had the same issue. Please explain to me again what the S and M do exactly? Are they just crops? What is the expected use?

I'd love to see this work like a quad pixel technology, where say, 60MP become 15, but the entire sensor is still used. That'd be cool. I don't see the use of simply cropping, but then again this isn't atypical for me and I will occasionally have an epiphany!

 

[Uncle Kevriano]

I have used uncompressed as an experiment on the A7RIV, but could find no noticeable advantage in IQ over compressed, and I couldn't live with the reduced FPS rate when shooting wildlife (it drops to 7fps). The file sizes were enormous too. I suspect you'd only notice a difference if you were blowing you're photos up to gigantic sizes.

 

[pointreyes]

I remember reading somewhere that someone else had the same issue. Please explain to me again what the S and M do exactly? Are they just crops? What is the expected use?

I'd love to see this work like a quad pixel technology, where say, 60MP become 15, but the entire sensor is still used. That'd be cool. I don't see the use of simply cropping, but then again this isn't atypical for me and I will occasionally have an epiphany!

Not crops. In camera down-sampled files. However, it looks like you must use full size raw with DXO. I found the answer:

https://support.dxo.com/hc/en-us/articles/4435675787537?input_string=camera+support+for+sony+s%2Cm+lossless+compressed+raw+files

The quad bayer tech you are talking about is what the A7Siii uses (along with many cell phones).

Quad Bayer sensors: what they are and what they are not

Marketing departments are trying to sell you on the wrong thing - resolution isn't why these sensors are good, it's the better image processing.

www.gsmarena.com

 

[pointreyes]

I have used uncompressed as an experiment on the A7RIV, but could find no noticeable advantage in IQ over compressed, and I couldn't live with the reduced FPS rate when shooting wildlife (it drops to 7fps). The file sizes were enormous too. I suspect you'd only notice a difference if you were blowing you're photos up to gigantic sizes.

Lucky you. I have been burned with lossy compressed raw. The artifacts on a shot of the moon completely destroyed an image I took with the A7Riii a few years ago. I still use lossy compressed raw on my A7Siii simply because so far event photography has not given me a problem. But I'm still ticked off with what happened to a shot of the moon I took. And yes, gigantic crops should be avoided. The link below gives an example of a shot with the moon with exactly what happened to me

Sony Raw Compression Artifacts: Real Examples

One of the curses of shooting with Sony cameras is their RAW file format. Currently they only offer two methods for saving RAW files: (1) a RAW file with lossy compression, and (2) an uncompressed RAW file. Did you notice what’s missing? Sony doesn’t offer a lossless compressed RAW file unlike...

stephenbayphotography.com

 

[AlphaWorld]

I remember reading somewhere that someone else had the same issue. Please explain to me again what the S and M do exactly? Are they just crops? What is the expected use?

I'd love to see this work like a quad pixel technology, where say, 60MP become 15, but the entire sensor is still used. That'd be cool. I don't see the use of simply cropping, but then again this isn't atypical for me and I will occasionally have an epiphany!

No, they are not crops. Well, some are! Let me re-state that.

L size is an uncropped 60Mp image.
M size can be EITHER an APS-C cropped 26Mp image OR a full frame image scaled to 26Mp
S size is EITHER an APS-C cropped image OR a full frame image scaled to 15Mp.

There's been confusion because of the two versions of M and S sizes - you can see why. The key takeaway is that a full frame L, M, or S image has the same field of view as full frame, and an APS-C M or S image has the same field of view as APS-C.

Someone pointed out that the S size is one quarter the pixel count of the L, so half the number of pixels horizontally and vertically. But the M size isn't half the pixel count of L (close, but not exactly).

There's not a lot of information about how Sony are making the scaled versions of the files. I haven't read anything where people have taken the files apart to work out what is in them - I'm curious exactly what is going on inside them. Are they truly RAW files that can be processed to set colour temperature? Or are they some semi-digested format with less flexibility than true RAW?

Some people have speculated (without giving me confidence that they know what they are talking about!) that these files are processed, scaled, then turned into something that's not really RAW.

I've wondered if it's something as simple as line skipping + column skipping? But the quality of the images is reported to be better than something that simple.

 

[pointreyes]

I did some reading on it between my question and your answer. It's tough to find anything related directly to Sony, but it sounds like what they do (pretty much across manufacturers) is downscale by combining pixels after the shot. It's not a true RAW file in that some processing has to take place for this to happen, and it affects image quality. One article said Canon crops the image to a 4/3 ratio first, then applies the downscaling.

According to what I could find out, sRAW was developed by Kodak to increase storage/processing etc. back when things were slower and smaller. It still retained more information than a jpeg and was closer to a HEIF, allowing for some processing.

It sounds like the main use historically has been for event photography, or commercial photography where the client wants their images fast. I don't think it's anything I'd find useful, everything I read ended up urging the reader to stay with RAW.

It'll be interesting to see if anyone ever comes up with how Sony approaches it.

The main thing it taught me is to not bother with anything other than L raw (crop-mode on the L raw thankfully still keeps the raw file intact for DXO). I'm glad I tested the M raw during a rehearsal and not a production shoot.

What I have learned is the following:
1) For small files, use the A7Siii. I will definitely continue to hang onto this wonderful camera for my photography.
2) For the A1 and A7Rv, glad I have greater than 128 GB cards in the cameras.

 

[AlphaWorld]

Using lossless compressed RAW, I'm happy enough with 128GB cards. Until recently, 128GB was the largest you could get in the fast G series Tough cards (300MB/s read/299MB/s write) - they just released a 256GB G, but I can't find them anywhere.

Using uncompressed RAW, the 128GB cards do feel a bit cramped

 

[AlphaWorld]

Using lossless compressed RAW, I'm happy enough with 128GB cards. Until recently, 128GB was the largest you could get in the fast G series Tough cards (300MB/s read/299MB/s write) - they just released a 256GB G, but I can't find them anywhere.

Using uncompressed RAW, the 128GB cards do feel a bit cramped

if anyone is interested, the 256GB Sony Tough G (fast!) SD cards appear to be in stock at last - unfortunately, they are pricy! I had pre-ordered one and just received it - price about the same as a 160GB CFeA Tough. Yes, it holds 94GB mor, but that is still pricy!