Upload Image: MD5Sum problem

[flyingdutchie]

When I upload an image, using Kallasoft Smugmug API, i provide the byte-array and its MD5 sum.

First i tried to upload it without the MD5Sum. That works fine.
Then i retrieved the image-info (getInfo) from smugmug and noted the md5sum.
Then i used the java messagedigest for MD5 and calculated the md5sum. The result is exactly the same as in point (2.).
However, when i provide this same calculated MD5 sum during the upload, i get an error message from smugmug that the MD5 sum is incorrect... :scratchThis is the code called just before uploading imgData.

final ByteArrayOutputStream bos = new ByteArrayOutputStream();
final BufferedInputStream bufIS = new BufferedInputStream(mData.mIS);
final BufferedOutputStream bufOS = new BufferedOutputStream(bos);
synchronized(smBuffer) {
int didRead = bufIS.read(smBuffer);
while (didRead > 0) {
bufOS.write(smBuffer, 0, didRead);
totalWritten += didRead;

didRead = bufIS.read(smBuffer);
}
bufOS.flush();
imgData = bos.toByteArray();

try {
final MessageDigest md5 = MessageDigest.getInstance("MD5");
final byte[] digest = md5.digest(imgData);
for (byte bt : digest) {
int intBt = ((int)bt) & (0x000000ff);
md5Sum += Integer.toHexString(intBt);
}
}
catch (NoSuchAlgorithmException e) {
md5Sum = "";
}
}





The MD5 sum for image http://www.streetsofboston.com/gallery/3827114/1#221257930 is
Content-Length = 56342
Content-MD5 = 8d57d127da7259e01ef9b92544b93c0

These are the values assigned to the HTTP Put upload as headers.

What am i doing wrong?

This is the error message:
wrong format (ByteCount given: , received. MD5Sum given: , actual.)

(why are the actual values not given in the error-message?)

[devbobo]

Dutchie,

can u post the raw http put request ?

Cheers,

David

[flyingdutchie]

Dutchie,

can u post the raw http put request ?

Cheers,

David

Wow! Talk about a quick reply! Thanks!

Here is the raw data (PUT and response):

PUT /hack/json/1.2.0//C%3A%5Cme.jpg HTTP/1.1
X-Smug-FileName: C:\me.jpg
X-Smug-ResponseType: JSON
Content-Length: 56342
Content-MD5: 8d57d127da7259e01ef9b92544b93c0
X-Smug-SessionID: 83524a08e7203afd4f28f32eea85d4a6
X-Smug-Version: 1.2.0
X-Smug-AlbumID: 3827114
X-Smug-Caption: Test Image
User-Agent: Jakarta Commons-HttpClient/3.1-rc1
Host: upload.smugmug.com
ÿØÿà JFIF

H H ÿá ÌExif MM *  
 


 
b
 
j
( 

1  

...
...

@aíR6Hms 2Œ €gä’”ë
[É•
Ï ;‰íÉ…' àAî¢tÚ<4ü
HTTP/1.1 200 OK
Date: Sat, 17 Nov 2007 22:00:58 GMT
Server: Apache
X-Powered-By: smugmug/1.2.0
Set-Cookie: SMSESS=83524a08e7203afd4f28f32eea85d4a6; path=/; domain=.smugmug.com
Cache-Control: private, max-age=1, must-revalidate
Pragma:
Content-Length: 107
X-Cnection: close
Content-Type: text/html; charset=utf-8
{"stat":"fail","code":4,"message":"wrong format (ByteCount given: , received. MD5Sum given: , actual.)"}

(i removed all the gibberish binary data).

And if Content-MD5: 8d57d127da7259e01ef9b92544b93c0 is not sent, all goes fine.

[luke_church]

Wow! Talk about a quick reply! Thanks!
And if Content-MD5: 8d57d127da7259e01ef9b92544b93c0 is not sent, all goes fine.

I fought with something similar to this a year or so ago. Forgive me if my memory is flaky, I might be barking up the wrong tree.

If I remember correctly there are issues with making sure that the padding on hex strings is done correctly, otherwise they get rejected.

In C# the chunk of code that I have used is (straight out of my source code repository, shout if it doesn't make any sense)

int byteCount = System.Convert.ToInt32(file.Length); //Get byte count for image
string md5sum = null;
//Compute the MD5 Hash for the image
using (FileStream fs = new FileStream(file.FullName, FileMode.Open, FileAccess.Read))
{
//Compute the hash
MD5 md5 = new MD5CryptoServiceProvider();
byte[] hash = md5.ComputeHash(fs);
//Convert it to the format that smugmug want (0 padded bytes, lower case)
StringBuilder buff = new StringBuilder();
foreach (byte hashByte in hash)
{
buff.Append(String.Format("{0:x2}", hashByte));
}
md5sum = buff.ToString();
} //Dispose of the file stream


The Java string formatting APIs will have something similar.

I remember that it wasn't exactly fun finding out what the issue was. Hopefully this will speed you on your way.

Luke

[flyingdutchie]

I fought with something similar to this a year or so ago. Forgive me if my memory is flaky, I might be barking up the wrong tree.

If I remember correctly there are issues with making sure that the padding on hex strings is done correctly, otherwise they get rejected.

In C# the chunk of code that I have used is (straight out of my source code repository, shout if it doesn't make any sense)

int byteCount = System.Convert.ToInt32(file.Length); //Get byte count for image
string md5sum = null;
//Compute the MD5 Hash for the image
using (FileStream fs = new FileStream(file.FullName, FileMode.Open, FileAccess.Read))
{
//Compute the hash
MD5 md5 = new MD5CryptoServiceProvider();
byte[] hash = md5.ComputeHash(fs);
//Convert it to the format that smugmug want (0 padded bytes, lower case)
StringBuilder buff = new StringBuilder();
foreach (byte hashByte in hash)
{
buff.Append(String.Format("{0:x2}", hashByte));
}
md5sum = buff.ToString();
} //Dispose of the file stream


The Java string formatting APIs will have something similar.

I remember that it wasn't exactly fun finding out what the issue was. Hopefully this will speed you on your way.

Luke

Thank you Luke.

Comparing your code to mine, i'm doing the same thing. I get 16 bytes (signed bytes), i make them unsigned and convert each of them in to 2-character hex-digits. The final result is exactly the same as the value returned by Smugmug when i do a 'images.getInfo' on the image.

-- Anton

[devbobo]

Anton,

Try this (http://hacks.introversion.com.au/SmugUploadTester) out, I have a FF extension for uploading testing.

It generates the MD5 hash, so you can compared to Riyad's java api.

Let me know how you get on.

David

[flyingdutchie]

Anton,

Try this (http://hacks.introversion.com.au/SmugUploadTester) out, I have a FF extension for uploading testing.

It generates the MD5 hash, so you can compared to Riyad's java api.

Let me know how you get on.

David

I tried it, but when i run it, i get this error:


XML Parsing Error: not well-formed
Location: chrome://smuguploadtester/content/uploader.xul
Line Number 78, Column 62: <radio label="api.smugmug.com" value="api.smugmug.com"selected="true" />
---------------------------------------------------------------------------------------^

[devbobo]

urgh sorry...try again :thumb

[flyingdutchie]

urgh sorry...try again :thumb

It runs now, but the page is entirely transparent... i can only see the borders of the fields and the field-groups. All the backgrounds are transparent...

I'll try anyway and see if i can get it to work.

[flyingdutchie]

It runs now, but the page of the plugin is entirely transparent... i can only see the borders of the fields and the field-groups. All the backgrounds are transparent...

I'll try anyway and see if i can get it to work.

I got it to work... luckily i could click on the input-fields and type in stuff.
However, all the transparent areas are click-through. Instead of clicking on the upload form, my desktop gets focus. Therefore i can not copy (to the clipboard) from the plug-in page.

However, the 'Request' part on the right side shows me the MD5-Sum. It is the exactly the same as i got: 8d57 etc etc... 03c0

The raw request data looks different, though:

POST instead of PUT
xmlrawadd.mg instead of /hack/jason/1.2.0
filename is me.jpg instead of C:\me.jpg
Host is api.smugmug.com instead of upload.smugmug.comBut Content-Length and Content-MD5 are exactly the same.

[devbobo]

PUT /hack/json/1.2.0//C%3A%5Cme.jpg HTTP/1.1
X-Smug-FileName: C:\me.jpg
....



Anton,

this could be the problem...both the PUT location and FileName, should only have the file not that path...

so it should be...

[code]
PUT /me.jpg HTTP/1.1
X-Smug-FileName: me.jpg
....
[/quote]

Give that a go.

Cheers,

David

[flyingdutchie]

Anton,

this could be the problem...both the PUT location and FileName, should only have the file not that path...

so it should be...

[code]
PUT /me.jpg HTTP/1.1
X-Smug-FileName: me.jpg
....


Give that a go.

Cheers,

David[/quote]

Alas, that did not work. I removed the "C:\" in front of the file-name and still it did not work.

BTW: Could the error-message be changed and actual specify what MD5-sum is expected?

Thanks!

[devbobo]

Anton,

please email me the file you're trying to upload to david at [smugmug].

also, grab the latest version of the upload tester, i have fixed the transparent window issue.

David

[flyingdutchie]

Anton,

please email me the file you're trying to upload to david at [smugmug].

also, grab the latest version of the upload tester, i have fixed the transparent window issue.

David

Hi David,

I ran the upload tester and here is a screen shot of the result. The md5-sum is the same. If the md5-sum is the same, what then is different? :D

PS: I'd like to attach a screen-shot, but DGrin is reporting that i already have "uploaded the maximum of 0 files" :scratch

[mjohnsonperl]

This is the error message:
wrong format (ByteCount given: , received. MD5Sum given: , actual.)
You can get this exact response if the content-length isn't calculated correctly. If the content-length is less then what it should be the HTTP PUT will finish early, and the HTTP call with think everything is fine, but when SmugMug's servers try and analyze an incomplete file this kind of response is given.

Here is my REST response I got when testing this.

<rsp stat="fail">
<err code="4" msg="wrong format (ByteCount given: , received. MD5Sum given: , actual.)"/>
</rsp>

My HTTP Request should have been this:

'_request' => bless( {
'_content' => sub { "DUMMY" },
'_uri' => bless( do{\(my $o = 'http://upload.smugmug.com/IMG_8589_color%20corrected.jpg')}, 'URI::http' ),
'_headers' => bless( {
'user-agent' => 'SinkMug/0.1 libwww-perl/5.806',
'x-smug-version' => '1.2.0',
'x-smug-responsetype' => 'REST',
'x-smug-sessionid' => 'b7127dad5252d85001a8b5e5237bbe76',
'content-length' => 2725682,
'x-smug-filename' => 'IMG_8589_color corrected.jpg',
'x-smug-albumid' => '3859440',
'content-md5' => '7c01e55d7a9f25d0389e864a471eda0f'
}, 'HTTP::Headers' ),
'_method' => 'PUT'
}, 'HTTP::Request' )
The above request resulted in a successful upload.


I modified my HTTP Request to this:

'_request' => bless( {
'_content' => sub { "DUMMY" },
'_uri' => bless( do{\(my $o = 'http://upload.smugmug.com/IMG_8589_color%20corrected.jpg')}, 'URI::http' ),
'_headers' => bless( {
'user-agent' => 'SinkMug/0.1 libwww-perl/5.806',
'x-smug-version' => '1.2.0',
'x-smug-responsetype' => 'REST',
'x-smug-sessionid' => '4954d1bac20bb0db16c16442ec00d0e2',
'content-length' => '2725687',
'x-smug-filename' => 'IMG_8589_color corrected.jpg',
'x-smug-albumid' => '3859440',
'content-md5' => '7c01e55d7a9f25d0389e864a471eda0f'
}, 'HTTP::Headers' ),
'_method' => 'PUT'
}, 'HTTP::Request' )
The only thing I modified was subtracting 5 from the content-length.

I tried adding 5 to the content-length but my script died with an error because the file it was reading to send wasn't big enough or something.

I know I beat my head on this one for a bit last month, but in my case I was opening the image in ASCII mode instead of BINARY mode to calculate the file size to small.

[flyingdutchie]

This means that you should always subtract 5 from the byte-array?

Content-Length = byte_array.length - 5 ?

[rkalla]

This means that you should always subtract 5 from the byte-array?

Content-Length = byte_array.length - 5 ?

Guys, sorry for the delay in responding, I finally got around to testing this and it worked fine using straight API call into the kallasoft SmugMug Java API images.UploadHTTPPut method.

The result, is a very suave-looking Anton ;)
http://kalla.smugmug.com/gallery/3403336#223617633

[flyingdutchie]

Guys, sorry for the delay in responding, I finally got around to testing this and it worked fine using straight API call into the kallasoft SmugMug Java API images.UploadHTTPPut method.

The result, is a very suave-looking Anton ;)
http://kalla.smugmug.com/gallery/3403336#223617633

I'm using the images.UploadHTTPPut method and it works great :thumb .... except that the MD5Sum is not accepted. Riyad, how did you get it to work with the MD5Sum?

PS. That image is from a b-day party of a friend of mine where i dressed up as a bouncer... it was pretty effective, since i almost turned a few people away... hehehehe

[rkalla]

I'm using the images.UploadHTTPPut method and it works great :thumb .... except that the MD5Sum is not accepted. Riyad, how did you get it to work with the MD5Sum?

I updated the implementation of UploadHTTPPut to now offer a slew of convenience methods that auto-fill out the HTTP header values for you, so you don't need to pass that stuff anymore. So for example passing in a session ID, album ID and File for an image, the API will load the image, get it's content length, calculate the MD5 hash for it, and set all the remaining headers then do the upload for you.

Additionally you can set all the geo-coding values too if you want. Overall it's just a lot friendly. BUT that being said, I didn't change any of the MD5 logic or anything like that... it just worked. Maybe the whole MD5 issue was a red herring before? Not sure. Either way it should be smoother with the new class.

I'll do another pre-M2 build for you and get it to you later.

[flyingdutchie]

Thanks Riyad!

I sent you an IM a little while ago with the text below. I don't know if you received it or not on your IM:

In SmugFig, you can upload an images as follows:
Image image;
java.io.InputStream myInputStream = new FileInputStream("C:\\somefile.jpg");
...
image.read(myInputStream); // sets the MD5Sum, Size and ImageData
image.setAlbumID(albumID);
image.setFilename("somefile.jpg");
image.setCaption("New Image");
image.update();

image.getMD5Sum() is used to uniquely identify the raw image-data. If the md5sum changed, the image-data has changed.
With your API changes, I hope i still can pass a byte-array and have access to the MD5sum of that byte-array. :)

[rkalla]

Thanks Riyad!

I sent you an IM a little while ago with the text below. I don't know if you received it or not on your IM:

Ahhh... so you were computing the MD5 independently in your API? I think this might explain why we were seeing a discrepency in the behavior.

In that case the changes I made won't help you, they are automated use-and-throw-away changes done with additional convenience methods... if your API still needs the MD5 (as a hash for example) then you'll still need to compute it yourself.

If it helps, I'm using the Apache, Commons-Codec project's DigesterUtils class, and the md5Hex method:
md5Sum = DigestUtils.md5Hex(imageData);

Maybe we could use the same method call to computer the MD5 so the APIs match up?

[flyingdutchie]

Ahhh... so you were computing the MD5 independently in your API? I think this might explain why we were seeing a discrepency in the behavior.

In that case the changes I made won't help you, they are automated use-and-throw-away changes done with additional convenience methods... if your API still needs the MD5 (as a hash for example) then you'll still need to compute it yourself.

If it helps, I'm using the Apache, Commons-Codec project's DigesterUtils class, and the md5Hex method:
md5Sum = DigestUtils.md5Hex(imageData);

Maybe we could use the same method call to computer the MD5 so the APIs match up?

Hi Riyad,
I'll try that one (DigestUtils) when i get home. I have doubts it would help, because the MD5Sum that i calculate using my current code (MessageDigest) is the same one that is returned by 'images.getInfo' and the same one that is calculated by David's utility (http://hacks.introversion.com.au/SmugUploadTester)

David, did you figure out why my upload fails when i specify an MD5-sum?

[devbobo]

David, did you figure out why my upload fails when i specify an MD5-sum?

Anton,

I forwarded a copy of that image onto Riyad, so that he could verify whether or not he was seeing the same result.

I haven't heard back from him.

David

[rkalla]

Anton,

I forwarded a copy of that image onto Riyad, so that he could verify whether or not he was seeing the same result.

I haven't heard back from him.

David

Per David's request, here's my raw HTTP Put data:
PUT /services/api/json/1.2.1//221257930-O.jpg HTTP/1.1
Content-Length: 56342
Content-MD5: 8d57d127da7259e01ef9b92544b903c0
X-Smug-SessionID: 59c590f38a9ba4c4e636490fdbaba59b
X-Smug-Version: 1.2.1
X-Smug-ResponseType: JSON
X-Smug-AlbumID: 3403336
X-Smug-FileName: 221257930-O.jpg
X-Smug-Caption: [UploadedImage] Caption 1195677657343
X-Smug-Keywords: [UploadedImage] Keyword1 Keyword2 Keyword3
X-Smug-Latitude: 110.56
X-Smug-Longitude: 32.7
X-Smug-Altitude: 787.6
User-Agent: Jakarta Commons-HttpClient/3.1-rc1
Host: upload.smugmug.com

[flyingdutchie]

Per David's request, here's my raw HTTP Put data:
PUT /services/api/json/1.2.1//221257930-O.jpg HTTP/1.1
Content-Length: 56342
Content-MD5: 8d57d127da7259e01ef9b92544b903c0
X-Smug-SessionID: 59c590f38a9ba4c4e636490fdbaba59b
X-Smug-Version: 1.2.1
X-Smug-ResponseType: JSON
X-Smug-AlbumID: 3403336
X-Smug-FileName: 221257930-O.jpg
X-Smug-Caption: [UploadedImage] Caption 1195677657343
X-Smug-Keywords: [UploadedImage] Keyword1 Keyword2 Keyword3
X-Smug-Latitude: 110.56
X-Smug-Longitude: 32.7
X-Smug-Altitude: 787.6
User-Agent: Jakarta Commons-HttpClient/3.1-rc1
Host: upload.smugmug.com



Thanks!

"8d57d127da7259e01ef9b92544b903c0" is the same MD5 my code is sending. Content-Length is the same too "56342".

The main difference is that i uploaded my image using "PUT /services/api/json/1.2.0" and not through "PUT /services/api/json/1.2.1" and X-Smug-Version is set to "1.2.0" instead of "1.2.1".

[devbobo]

Thanks!

"8d57d127da7259e01ef9b92544b903c0" is the same MD5 my code is sending. Content-Length is the same too "56342".

The main difference is that i uploaded my image using "PUT /services/api/json/1.2.0" and not through "PUT /services/api/json/1.2.1" and X-Smug-Version is set to "1.2.0" instead of "1.2.1".

Anton,

putting to /services/api/json/1.2.0 is not required, /filename.jpg is fine. And I don't believe there are any code differences between 1.2.0 and 1.2.1

[flyingdutchie]

Anton,

putting to /services/api/json/1.2.0 is not required, /filename.jpg is fine. And I don't believe there are any code differences between 1.2.0 and 1.2.1

Thanks

Then i don't know what the difference is. The MD5Sum that my code sends is correct, so why is the Smugmug server responding that the MD5 sum is incorrect? (see the raw request-data in one of my earlier messages)

[devbobo]

Thanks

Then i don't know what the difference is. The MD5Sum that my code sends is correct, so why is the Smugmug server responding that the MD5 sum is incorrect? (see the raw request-data in one of my earlier messages)

Anton,

Are you able to provide me with your app so I can do some testing ?

And if need be, do some debugging on internal servers ?

[flyingdutchie]

Anton,

Are you able to provide me with your app so I can do some testing ?

And if need be, do some debugging on internal servers ?

I sent you an e-mail with all the code.

[flyingdutchie]

All the time it was my fault!!:rolleyes

David, Riyad and everyone else: So sorry to have taken up your time. Thank you very much for all your help.

My routine that translates a byte-array into a hex-string, skipped leading 0s.

My MD5Sum
8d57d127da7259e01ef9b92544b93c0

Actual MD5Sum
8d57d127da7259e01ef9b92544b903c0

Again, thanks!

[rkalla]

All the time it was my fault!!:rolleyes

David, Riyad and everyone else: So sorry to have taken up your time. Thank you very much for all your help.

My routine that translates a byte-array into a hex-string, skipped leading 0s.

My MD5Sum
8d57d127da7259e01ef9b92544b93c0

Actual MD5Sum
8d57d127da7259e01ef9b92544b903c0

Again, thanks!

Anton no worries, I'm glad it's working. And I think this thread will be helpful to anyone else that wanders by running into the same problem, atleast they will know where to look right away.

[flevine]

I came across this thread after having the same MD5 issues that initially started the thread. Since I'm working in Windows / C#, I figured I would provide my C# 3.0 Extension Method that *appears* to create a valid MD5 hash (ie, I haven't had any problems with it yet)


public static string CreateMD5Hash(this byte[] contentsToHash)
{
System.Security.Cryptography.MD5 hasher = System.Security.Cryptography.MD5.Create();
byte[] hashBytes = hasher.ComputeHash(contentsToHash);
StringWriter writer = new StringWriter();
foreach (byte b in hashBytes)
{
writer.Write("{0:x2}", b);
}
return writer.ToString();
}


Does this look right to you smugmug experts out there? If so, what about adding a little note in the documentation that clearly shows some examples in C#, Java, PHP, Perl, etc on how to create MD5 hashes that sugmug understands. I don't think that the the user should need to to surf the dgrin forums to figure out how to MD5 hash their image.