IPTC keyword bug?

BenV · June 27, 2005

I use iView MediaPro for cataloging my photos and ran into what I think is a bug in uploading. In iMP, I can enter multi-word keywords and save and retrieve them from the original JPEG just fine using various programs. But when the photos are uploaded to smugmug, the single multi-word keywords get seperated at the spaces into multiple keywords.

I realize that when using the smugmug keywording interface there are limitation and rules with regard to the use of quotes. But when you are importing keywords from IPTC data, I should like to think that the keywords ought to be imported properly.

BenV

BenV · June 29, 2005

Bueller... Bueller... Bueller

onethumb · June 29, 2005

BenV wrote:

I use iView MediaPro for cataloging my photos and ran into what I think is a bug in uploading. In iMP, I can enter multi-word keywords and save and retrieve them from the original JPEG just fine using various programs. But when the photos are uploaded to smugmug, the single multi-word keywords get seperated at the spaces into multiple keywords.

I realize that when using the smugmug keywording interface there are limitation and rules with regard to the use of quotes. But when you are importing keywords from IPTC data, I should like to think that the keywords ought to be imported properly.

BenV

If you put your multi-word keywords in quotes, they should be preserved.

Since, AFAIK, there's no industry-standard definition of how keyword fields are layed out (it's just free-form text), we went with most of the herd and used any non-alphanumeric as a spacer. You can get around this by using quotes.

Don

BenV · June 30, 2005

Yep, it's a bug.

onethumb wrote:

If you put your multi-word keywords in quotes, they should be preserved.

Since, AFAIK, there's no industry-standard definition of how keyword fields are layed out (it's just free-form text), we went with most of the herd and used any non-alphanumeric as a spacer. You can get around this by using quotes.

Don

But Don, the rest of the herd is grazing a different pasture...

SM is a can-do company, not a "we don't do it right, but here's a hack" company. An Apple, not a Sun. A Google, not an AOL.

RANT {
So I have to go and modify my catalogs of 10,000 photos because SM can't handle importing the keywords correctly as every other program does. Not to mention that having all the pointless quotes is, well, just ugly. Leave the quotes to book titles, not keywords. Put one of your guys on it and he could probably fix it in under an hour. Heck, send ME a block of code and I'll fix it for you!
}

In the case of iView MediaPro, it uses semi-colons as delimiters. Can't you test for the existance of these (as a branched test case) and then split on that? This is the text of the keywords field out of iMP (3 keywords):

"""DELETED Test"""; This should work too; Boo

According to here:
http://www.controlledvocabulary.com/metalogging/keywording.html
and here:
http://www.controlledvocabulary.com/metalogging/ck_guidelines.html

In order to copy and paste into the keyword field in Photoshop CS, use a comma and then a space between each term. For example: xxxx, xxx xxxx, xxxxx, term, word phrase, two word phrase, etc. You can can also separate terms using a semi-colon or even a hard carriage return.

While this site isn't "authoritative", it is in wide regard.

However, this source is authoritative: http://www.iptc.org/IIM/

(It also also happens to reference controlledvocabulary.com above as an authoritative documentation source.)

Specifically:
Version 4 of the IPTC Headers spec:
http://www.iptc.org/download/download.php?fn=IIMV4.1.pdf
Version 3 of the IPTC Headers spec:
http://www.iptc.org/download/download.php?fn=IIMV3.PDF

Both of which say (emphasis mine):

Keywords -- Repeatable, maximum 64 octets, consisting of graphic characters plus spaces.

It is a bug to split keywords on spaces.

Thanks for reading.

BenV
-- breathe... breathe...

onethumb · June 30, 2005

I'm afraid you just proved my point.

When I built the tool, I couldn't find any consensus, and it looks like you haven't either.

Some places use semicolons. Some use commas. Some use whitespace. Some use tabs.

So instead of trying to autodetect (Excel can't auto-detect delimiters, either, so how am I supposed to tell that you WANTED that semicolon or comma in there?), I went to the source: our users' photos.

Of those photos that had IPTC keywords, most of them were space delimited. Not all, true, but a clear majority were.

They'd have things like: Paris cheese bread cafe

Obviously, that's not a phrase. Also, obviously, there's no comma or semicolon delimiter.

We *are* a "can-do" company, and to me, "can do" equals making things "just work" for most of our customers.

I am truly sorry that our methodology doesn't fit with yours. If I could easily autodetect between spaces, commans, semicolons, and the like, I would. If you've got a magic way, I'm all ears.

Don

mbrady · June 30, 2005

onethumb wrote:

Some places use semicolons. Some use commas. Some use whitespace. Some use tabs.

So instead of trying to autodetect (Excel can't auto-detect delimiters, either, so how am I supposed to tell that you WANTED that semicolon or comma in there?), I went to the source: our users photos.

Of those photos that had IPTC keywords, most of them were space delimited. Not all, true, but a clear majority were.

They'd have things like: Paris cheese bread cafe

Obviously, that's not a phrase. Also, obviously, there's no comma or semicolon delimiter.

We *are* a "can-do" company, and to me, "can do" equals making things "just work" for most of our customers.

I am truly sorry that our methodology doesn't fit with yours. If I could easily autodetect between spaces, commans, semicolons, and the like, I would. If you've got a magic way, I'm all ears.

While certainly not magic and foolproof (and this is right off the top of my head, so there may be a huge gaping whole in this), you could take the IPTC keyword string and do a count for various special characters (spaces, semicolors, tabs, commas) and whichever character came out on top is probably being used as the delimiter.

It could get tricky if say, you are using semicolons and you had several multi-word phrases as a keyword, spaces could still come out on top. And actually even when semicolons are used, there is probably still a space after the semicolon (otherwise we'd end up with keywords like "trip;holiday;ocean"). Maybe try doing a count of semicolor-space pairs instead.

That's just a start and is just stream of through, it would take some massaging to get the theory into a more usable shape to see if there other holes. Obviously no solution would be completely foolproof, but perhaps there is a solution that results in fewer false phrase splits.

Matt

jfriend · June 30, 2005

An algorithm suggestion

onethumb wrote:

I am truly sorry that our methodology doesn't fit with yours. If I could easily autodetect between spaces, commans, semicolons, and the like, I would. If you've got a magic way, I'm all ears.

Since there's not a whole lot of use for putting commas or semicolons in a keyword, why not use this algorithm:

1) If you find quotes, treat the stuff inside the quotes as a single keyword.

2) If you see semicolons or commas in the keyword string, use that as the delimiter for that keyword string. The only keywords you would guess wrong on would be the ones that wanted a semicolon or comma as part of the keyword. That seems like a reasonable compromise since that's pretty unlikely. If a customer really wants this in the keyword, the above quotes are the escape mechanism for it.

3) If you don't see quotes or semicolons or commas in the string, then use space as the delimiter like you do today.

Wouldn't this "just work" more often that what you have today?

--John

onethumb · June 30, 2005

jfriend wrote:

Since there's not a whole lot of use for putting commas or semicolons in a keyword, why not use this algorithm:

1) If you find quotes, treat the stuff inside the quotes as a single keyword.

2) If you see semicolons or commas in the keyword string, use that as the delimiter for that keyword string. The only keywords you would guess wrong on would be the ones that wanted a semicolon or comma as part of the keyword. That seems like a reasonable compromise since that's pretty unlikely. If a customer really wants this in the keyword, the above quotes are the escape mechanism for it.

3) If you don't see quotes or semicolons or commas in the string, then use space as the delimiter like you do today.

Wouldn't this "just work" more often that what you have today?

--John

It might.

I'll have to run some test-cases and see.

Bear in mind that while I see the issues here, this is the first complaint of this nature I can remember seeing, so it's not as high-priority as some of the other things people are clamoring for. No promises on timeframe.

Don

jfriend · June 30, 2005

Thanks for listening...

onethumb wrote:

It might.

I'll have to run some test-cases and see.

Bear in mind that while I see the issues here, this is the first complaint of this nature I can remember seeing, so it's not as high-priority as some of the other things people are clamoring for. No promises on timeframe.

Thanks for listening. I understand on the priority issue - just thought I'd make a design suggestion that came to mind while reading here. Heck, I didn't even know keywords could contain spaces until this thread <grin>.

But, then I mostly can't use keywords in Smugmug because that functionality isn't supported with password protected galleries which many of my photo events require, so doing something about that is obviously higher on my list.

--John

onethumb · June 30, 2005

jfriend wrote:

Thanks for listening. I understand on the priority issue - just thought I'd make a design suggestion that came to mind while reading here. Heck, I didn't even know keywords could contain spaces until this thread <grin>.

But, then I mostly can't use keywords in Smugmug because that functionality isn't supported with password protected galleries which many of my photo events require, so doing something about that is obviously higher on my list.

--John

Sorry to be the bearer of bad news, but I'm 99.9% certain keywords will never work with passworded or private galleries. It just violates the reason for having keywords, passwords, and private galleries on almost every level.

I realize some people really want this to happen, but it just adds too much complexity for our users to understand, and they're *very* sensitive about their private and passworded galleries. We take that very seriously.

It's not on our todo list. Ditto for searching private/passworded galleries.

Don

jfriend · June 30, 2005

Another explanation of virtual galleries based on keywords

onethumb wrote:

Sorry to be the bearer of bad news, but I'm 99.9% certain keywords will never work with passworded or private galleries. It just violates the reason for having keywords, passwords, and private galleries on almost every level.

I realize some people really want this to happen, but it just adds too much complexity for our users to understand, and they're *very* sensitive about their private and passworded galleries. We take that very seriously.

It's not on our todo list. Ditto for searching private/passworded galleries.

Don

I wasn't sure whether anyone from Smugmug read my post here where I actually took a fair amount of time to try to describe some really awesome keyword functionality (since nobody from Smugmug ever responded). I have to use passwords to protect galleries of kid's sports teams and school events. That's the only way the parents feel comfortable with the images being on the web. At the same time, my viewers could benefit tremendously from keyword functionality. I don't actually want keyword functionality the way you've implemented it today where it's global across my whole account. I only truly want it scoped to a particular password protected gallery. The notion of a virtual gallery that is defined to contain the set of images from a parent gallery that are satisfied by a keyword query could very much be made to be compatible with passwords while not violating any sense of protection or privacy.

In the absence of keyword queries to make these virtual galleries, the only work-around I've found is to upload duplicate copies of the images organized into galleries a different way. In my most recent case, I uploaded one set of images organized into galleries by event and a second copy of images organized by person. This is inefficient for both of us. If I could have just uploaded a master chronological set of images and then defined virtual galleries for viewing by event and viewing by person, all three of us (myself, Smugmug and my viewers) would have come out ahead. I would have saved upload time and prep time. You would save storage space (2x in this case) and the viewer would get this type of expanded functionality a lot more often.

So, please keep in mind that I am NOT asking for your current keyword functionality on a user's homepage to somehow magically incorporate password protected data. That WOULD be a mess. I'm asking for a new use of keywords (virtual galleries) that could easily be (optionally) scoped to one or more password protected galleries. Even further, this would enable great functionality for the account holder, but the end-user wouldn't have to learn anything new because they'd just see the familiar gallery interface (some of the galleries would just be virtual based on keyword queries instead of populated with duplicate images).

--John

flyingpylon · July 1, 2005

Why couldn't you just make the choice of delimiter an account setting?

jfriend · July 1, 2005

"Just works" is better

flyingpylon wrote:

Why couldn't you just make the choice of delimiter an account setting?

This could be an advanced and optional setting, but it's nearly always better to have an algorithm/service that "just works" rather than one that requires proper understanding and configuration before it works properly.

--John

BenV · July 1, 2005

I just typed a big long response and lost it. But suffice to say that:

onethumb wrote:

I am truly sorry that our methodology doesn't fit with yours.

It's not "my" methodology Don, it's the spec.

-- At least admit that you implement the spec incorrectly. Even if you did it as a willfull design decision, it is, in fact, incorrect. Just admit it.

-- What other systems implement spaces as keyword delimiters?

-- Smugmug could at least support the official spec as a first course of action. Then, you fall back on your current implementation as a last resort. That would be "can-do" to me. Right now, an incorret implementation of the spec is your first course of action.

PSEUDO CODE:
Convert recuring double-quotes to double-quotes;
Convert recuring single-quotes to single-quotes;
if (keywords contains ![A-Za-z0-9\s] that are not within quotes)
{split on what you find}
else
{fall back to your current implementation}

What will that break?

It doesn't conform to the spec completely but at least it gets you 99% there. A photo with a single multi-word keyword would still parse wrong but at least anything with commas or semi-colons as delimiters will parse according to spec by parsing phrases correctly.

Sorry to be a bother, I won't belabor(sp?) the subject...

BenV

BenV · July 1, 2005

Don:

I apologize for sounding so angry/annoyed/whatever, I just happen to feel rather strongly about the subject of standards and compliance when they do exist in a given area of endeavor.

I wouldn't be with SM if I didn't think you guys were the best. I'm just trying to help you guys buff out the little scratches and buffing sometimes requires a little friction.

BenV

Gary Glass · October 1, 2005

status?
Just wondering what the status of this delimiter issue is? I'm usin Photoshop. It wants to delimit keywords with semicolons. Seems like smugmug isn't supporting this, because when I upload my files, all the keywords are gone.

pat.kane · October 1, 2005

I'm curious as well. Jfriend's post #7 lays out a reasonable approach that treats keywords closer to the IPTC standard than what is being done now.

I'm using another web service at work that is similar to smugmug and it properly recognizes multi-word keywords that are delimited with commas using a professional IPTC editing program.

I've having to go back and edit all my keywords prior to uploading. It's a pain that I'd rather avoid.

Mike Lane · October 1, 2005

Gary Glass wrote:

Just wondering what the status of this delimiter issue is? I'm usin Photoshop. It wants to delimit keywords with semicolons. Seems like smugmug isn't supporting this, because when I upload my files, all the keywords are gone.

All of my keywords get entered in Smugmug when I input them in the IPTC using semicolons in Photoshop.

colourbox · October 1, 2005

flyingpylon wrote:

Why couldn't you just make the choice of delimiter an account setting?

Yeah, either that or an import setting. An example was used earlier that Excel can't auto-detect delimiters. No, it does better than that. If I import a raw text file, I get the import wizard where I can tell Excel what the delimiter should be: tabs, spaces, commas, other...

onethumb wrote:

Bear in mind that while I see the issues here, this is the first complaint of this nature I can remember seeing

Then I'll toss in a vote about compound keywords being broken up - the worst for are first name+lastname keywords that become disassociated in smugmug.

And I'll also post a related problem. I use hyphens in my filenames to make my personal image naming convention more readable. For example, I might add a "-det" to indicate that a version that's a detail view with the same base filename. Unfortunately, smugmug (as documented) thinks the hyphen indicates keywords, so when I import filenames where hyphens were used, I have to go clean up the keyword field using the bulk keyword edit tool (glad it's there!).

What I probably should do currently is add a pre-processing step that deals with hyphens in filenames of images headed for smugmug. But if I could just tell the smugmug importer which character is supposed to be the delimiter, just like in Excel and others...but since I use keyword-embedding image organizers like iView and Bridge, I would actually like to be able to select "No Delimiter" for uploaded filenames.

Gary Glass · October 1, 2005

larger issue?
I seem to be having a problem with getting keywords to work at all, and I've been corresponding with the smugmug gods about it. I'm just curious if anyone else has been having the same or similar issues. No matter how I delimit my keywords, they aren't getting parsed when I upload the image. Anybody having this problem?

BenV · October 3, 2005

Mike Lane wrote:

All of my keywords get entered in Smugmug when I input them in the IPTC using semicolons in Photoshop.

I think you missed the original point -- it was about SM not supporting multi-word keywords (i.e. phrases) correctly.

BenV

asd · October 3, 2005

colourbox wrote:

What I probably should do currently is add a pre-processing step that deals with hyphens in filenames of images headed for smugmug.

I use Irene for my batch file renaming. It helps when I come back and want to consolidate several cards into one folder -- you could use it to clean up your filenames right before uploading (copy the files first..).

BenV · October 4, 2005

colourbox wrote:

Then I'll toss in a vote about compound keywords being broken up - the worst for are first name+lastname keywords that become disassociated in smugmug.

And I'll also post a related problem. I use hyphens in my filenames to make my personal image naming convention more readable. For example, I might add a "-det" to indicate that a version that's a detail view with the same base filename. Unfortunately, smugmug (as documented) thinks the hyphen indicates keywords, so when I import filenames where hyphens were used, I have to go clean up the keyword field using the bulk keyword edit tool (glad it's there!).

What I probably should do currently is add a pre-processing step that deals with hyphens in filenames of images headed for smugmug. But if I could just tell the smugmug importer which character is supposed to be the delimiter, just like in Excel and others...but since I use keyword-embedding image organizers like iView and Bridge, I would actually like to be able to select "No Delimiter" for uploaded filenames.

Unfortunately, while this bug does introduce a significant processing step for you and I, onethumb made it quite clear that this isn't going to get fixed nor admit that it even needs to be fixed.

SM served me well for an event I did that made up for any outlay I incurred in setting up a pro account. But I have realized that SM really doesn't intend to serve the pro market like PrintRoom or PhotoReflect or ExposureManager.

For future stuff I'll probably be using www.ExposureManager.com -- cheaper and they have most of the pro features people have been clamouring for here on dgrin...

www.ExposureManager.com is clearly serving the pro market while SM seems to be caught in this in-between land of "flickr wannabe but kinda sorta might wanna let you sell your stuff on your, I mean our, site too."

Having said all that, it's just one man's opinion. YMMV.

BenV

Mike Lane · October 4, 2005

BenV wrote:

Unfortunately, while this bug does introduce a significant processing step for you and I, onethumb made it quite clear that this isn't going to get fixed nor admit that it even needs to be fixed.

SM served me well for an event I did that made up for any outlay I incurred in setting up a pro account. But I have realized that SM really doesn't intend to serve the pro market like PrintRoom or PhotoReflect or ExposureManager.

For future stuff I'll probably be using www.ExposureManager.com -- cheaper and they have most of the pro features people have been clamouring for here on dgrin...

www.ExposureManager.com is clearly serving the pro market while SM seems to be caught in this in-between land of "flickr wannabe but kinda sorta might wanna let you sell your stuff on your, I mean our, site too."

Having said all that, it's just one man's opinion. YMMV.

BenV

You have any examples of the customization of sites on exposure manager?

jfriend · October 4, 2005

Hmmm - exposuremanager.com looks very interesting

BenV wrote:

Unfortunately, while this bug does introduce a significant processing step for you and I, onethumb made it quite clear that this isn't going to get fixed nor admit that it even needs to be fixed.

SM served me well for an event I did that made up for any outlay I incurred in setting up a pro account. But I have realized that SM really doesn't intend to serve the pro market like PrintRoom or PhotoReflect or ExposureManager.

For future stuff I'll probably be using www.ExposureManager.com -- cheaper and they have most of the pro features people have been clamouring for here on dgrin...

www.ExposureManager.com is clearly serving the pro market while SM seems to be caught in this in-between land of "flickr wannabe but kinda sorta might wanna let you sell your stuff on your, I mean our, site too."

Having said all that, it's just one man's opinion. YMMV.

BenV

It is starting to feel like smugmug is caught in the middle and it's hard to tell if smugmug is serious about serving the pros or not. I have only read the web-site, but I was very impressed with the feature list on www.exposuremanager.com - self fulfillment (you print and fullfill, they just handle the money), pro lab fulfillment (like smugmug does), upload on-demand (the pro gets notified what's been ordered and can then upload the actual originals they want printed for that order so you only have to optimize the prints that are actually ordered), print-ready originals for each size (the pro gets total control of the cropping for each size), you control the display image for each size, including sharpening and JPEG quality, unlimited storage, full template control over the display of your work, full control over watermarking, etc... These are ALL things that have been asked for here.

--John

pat.kane · October 4, 2005

Mike Lane wrote:

You have any examples of the customization of sites on exposure manager?

I did a Google search and found a few:
http://photobybruce.exposuremanager.com/
http://richmichaels.exposuremanager.com/
http://www.mikebrice.exposuremanager.com
http://emiphoto.exposuremanager.com/

Hard to tell what type of customization is allowed from this limited sample.

Here's a discussion on Sports Shooter about their service.
http://www.sportsshooter.com/message_display.html?tid=17386

jfriend · October 4, 2005

Full page control?

Mike Lane wrote:

You have any examples of the customization of sites on exposure manager?

The bottom of this page: http://www.exposuremanager.com/scripts/expman.pl?rm=knowledge_base&type=stylesheets, makes it sound like you get full HTML control over the template used to display your page, but to know for sure, we'd probably either need someone to try it or at least have an email exchange with them.

--John

Mike Lane · October 4, 2005

jfriend wrote:

The bottom of this page: http://www.exposuremanager.com/scripts/expman.pl?rm=knowledge_base&type=stylesheets, makes it sound like you get full HTML control over the template used to display your page, but to know for sure, we'd probably either need someone to try it or at least have an email exchange with them.

--John

Every site looks exactly the same except you can change the colors. All the layouts are built with tables and there are hardly any classes or id's. Their customization options are much worse than the previous smugmug customization options. There is no way that you could get a site to look differently from all the rest. So on the customization side of the house (which is important to me and many others, but maybe not everyone) Smugmug has them beat hands down.

Gary Glass · October 4, 2005

keywords
It seems to me several reasonable and practical suggestions have been made for how to improve the keywords implementation on SM. Seems like this is an issue that could be addressed with relatively little effort by SM, with big benefits for users.

Mike Lane · October 4, 2005

Gary Glass wrote:

It seems to me several reasonable and practical suggestions have been made for how to improve the keywords implementation on SM. Seems like this is an issue that could be addressed with relatively little effort by SM, with big benefits for users.

And I guarantee you that Smugmug takes these suggestions very seriously. That said, Rome wasn't built in a day so to speak. These things take time to implement even after they are decided upon.

jfriend · October 4, 2005

Exposure Manager customizations

Mike Lane wrote:

Every site looks exactly the same except you can change the colors. All the layouts are built with tables and there are hardly any classes or id's. Their customization options are much worse than the previous smugmug customization options. There is no way that you could get a site to look differently from all the rest. So on the customization side of the house (which is important to me and many others, but maybe not everyone) Smugmug has them beat hands down.

I contacted the Exposure Manager folks to ask them about their customization abilities. They do, indeed, have full template customizability. This is what they said:

We allow for different sorts of customizations:

The first level is just using stylesheets, which doesn't require any
knowledge of stylesheets (which Smugmug requires) but has an easy to use Stylesheet selector where you can set the colors, borders and fonts.

The second level is using the custom header and footer using HTML. Because it is a header and footer, you can wrap your site in any custom html that you want. For examples, see:
http://gallery.tourphotographer.com/ (custom header, footer, domain mapped) http://eliteimages.exposuremanager.com/g/portrait_portfolio
footer, stylesheets)

On the third level you can customize the entire HTML. We give you the template, and you can adjust anything you want (it doesn't get more custom like this). As a good example:
http://www.lightexposures.com/

On top of that we allow a lot of customization without needing to customize templates. You can change the sizes of your images, have an image watermark, define the location of the watermark per, etc.

SmugMug and ExposureManager are very different services and have different approaches on customization, as well as almost everything else. It often comes down to personal preferences which service works for you. From what we hear ExposureManager is better suited for event photography and SmugMug does well for portfolio websites. Again, that is a matter of personal preference, and since we are not experts on SmugMug, we can only comment on what we offer.

Please feel free to throw any other questions our way, and we'll get back to you as soon as possible.

Best regards,
Kerry Duopnt
support@exposuremanager.com

IPTC keyword bug?

Comments