Problems repeatedly grabbing Smugmug data

Lint · November 8, 2010

I have a php page - using phpmug wrapper - that pulls all albums, categories and sub-categories from my smugmug account and displays the hierarchy it gets. I'm using oauth but I had much the same problem with email/password login. The first couple of times I refresh the page I get the data fine but then I get what appears to be a php timeout after 10-15 seconds - blank page, no errors.

If I force a limit on the number of sub-categories (currently 60, mostly empty) then the page comes back but just what limit works depends very much on how fast I refresh. 10 usually works, 30 often but not always fails.

I'm a bit of an oauth newbie so I'm probably missing something very obvious. Or is there a limit on how many times or how fast I can make API calls to Smugmug? Or is it a session thing maybe?

I'm basically doing this

$options = get_option( 'smuglymugly');
        $f = new phpSmug("APIKey=" . $options['apikey'], "AppName=" . $options['apistring'], "OAuthSecret=" . $options['secret'], "APIVer=1.2.0" );
        $token = $options['token'];
        $f->setToken("id={$token['Token']['id']}", "Secret={$token['Token']['Secret']}");

before grabbing everything I need.

(siteground php timeout is set at 10 seconds, I can't change it).

SamirD · November 9, 2010

If you're php time limit is causing it to not get all the data in time, you might have to break up the process, calling the function multiple times and having it pick up where it left off each time.

But 10 seconds sounds like a long time. I have a function that gets all the urls for the over 1000 albums in a category and write it to a file and it's done almost instantaneously.

Lint · November 10, 2010

SamirD wrote: »

But 10 seconds sounds like a long time.

That was my thinking. The real pain, of course, is that nothing's coming back and I'm not getting debug statements. I'm going to have to do some file logging, I think.

thanks for the input.

Lint · November 10, 2010

OK, I've built and a simplified version of my code. The problem is definitely intermittent; It'll work a couple of times then fail with a very quiet, blank page, then come back again for a couple of times. No php errors in the log file either.

This is my code. Basically, I'm grabbing albums, then categories then looping through each category to get subcategories under the category. I have around 60 categories. If I limit it to 10-15 I get many fewer blank pages in a sequence but still occasionally:

$f = new phpSmug("APIKey=$apikey", "AppName=$apistring", "OAuthSecret=$secret" );
    $f->setToken("id={$token['Token']['id']}", "Secret={$token['Token']['Secret']}");
        
    $albums = $f->albums_get( "Heavy=1" );
    $cats = $f->categories_get();
    
    foreach ( $cats as $cat )
    {
        echo "Category: " . $cat['Name'] . "<br>";
        $subcats = $f->subcategories_get( "CategoryID=" . $cat['id']);
    }

SamirD · November 10, 2010

Hmm...what I'd do is write the categories to a file, then have a wait for about 15 seconds, then have it grab the sub-cats, wait again, then the albums underneath one sub-cat, then wait, etc.

You can keep breaking it down until it's basically just getting one album at a time if need be. And you'll probably have to use some sort of marker in the file since the php script needs to finish in under 10 seconds. This way, it can quickly read the file and pick up where it left off.

Lint · November 10, 2010

SamirD wrote: »

Hmm...what I'd do is write the categories to a file, then have a wait for about 15 seconds, then have it grab the sub-cats, wait again, then the albums underneath one sub-cat, then wait, etc.

That's an awfully long time to wait for a page load. I'd have to go and make a cup of tea. Or 3 ;-)

A better way would be to load all the albums, all the categories, all the subs and build the tree from that. Assuming it's the number of rapidly sequential calls that's causing the problem, of course.

That's really what I was trying to find out, I guess; whether making a bunch of API calls like that in rapid succession is a no-no. If it is, I may have to have a bit of a re-think.

SamirD · November 10, 2010

I highly doubt making all those calls to the API is an issue. Let me check the code for one of my API apps...

SamirD · November 10, 2010

Ahhh...I think you're right.

But I think there's a much quicker and more efficient way to do what you want. The category and sub-category information is already there in $albums once you get that. You can just parse the information from it using standard php operations and display the results. No need to grab it again from the API.

Lint · November 10, 2010

Yeah, I'd thought about that but I'd need to do the whole create if it doesn't exist thing. It just seemed easier to do a categore->sub-category->album thing. But I'm still keen to find out if this is a limitation or bug or if it's something going on at my end. I need to know what the limitations are.

SamirD · November 10, 2010

It would actually be much, much faster to parse $albums than to fetch that info using the API. And the likelihood that the info would be different from both methods would next to nil since there isn't much time to change anything.

There's a lot of different methods to implement too. You can use php array_count_values to just count the categories/sub-categories. The resulting array will give you all of your categories/sub-categories. There may be a way to use array_unique, or compact, or count, or uasort. And these are just from some simple looking around on php.net. A google search for the exact thing you want--output all unique values in an array--will probably lead you right to the code.

Lint · November 10, 2010

I know. But the other way is simpler. And I like simple. :-)

Lint · November 11, 2010

So, in closing, and by way of putting some back; I now use getAll() to get all my subcategories and use the category field to populate categories, leaving 3 API calls in place of the 65 and odd I'd been making before. It's a whole bunch quicker and I'm not getting the timeouts any more. So it looks like rapid and longish sequences of API calls is probably a bad idea.

This will ultimately, for anyone interested, be a Wordpress plugin for Smugmug. It's currently driving images and slide shows on the first couple of posts on my website - see sig - and I'll be making it publicly available once I'm comfortable that it's not going to explode the Internet.

Thanks to everyone who input.

K.

SamirD · November 11, 2010

Personally I think it's simpler to parse the information once it's in php versus grabbing it again, but we're all different in our approaches.

I didn't know about getAll for the subcats. Good find.

So I'm curious, what the functionality of your plugin will do exactly? I only know about retrieving albums, categories, and subcats from the development of a little random picture module for my main website which allowed someone to click on the image and be taken to that image in its source gallery. I checked out your wordpress site, but didn't really notice where the plugin was operating.

I guess that's the sign of a good design.

Lint · November 11, 2010

At the moment it's quite simple. The shots you see in the first two posts are shortcodes that just lift smugmug shots based on ID. There's a couple of links in the post to more shots which point to another page that has a carousel of the rest of the gallery, again picked up from Smugmug. I'll add some search functionality so you can grab based on keywords, titles, ratings, etc.

I'll also be adding a couple of widgets with a gallery tree and something that displays a number of random thumbnails and behind the scenes in the admin pages I'll be adding a treeview of the gallery that lets you grab off the shortcodes rather than digging around in albums. I'd also quite like a d&d into the tree allowing you to organise your smugmug gallery more easily.

I'm still thinking about the whole sell, print and email thing but I'd like to figure a way of making that easy from the site too.

I'm also open to suggestions but what I'm really after is a way to blog and show off work on my own site but only have to maintain one archive of images.

SamirD · November 11, 2010

Interesting. Sounds like you're attempting some heavy-duty wordpress-SM integration. I've worked on vbulletin-SM integration for years now. It's still a bunch of pieces that have to be used in conjunction, like a javascript applet that generates html code that has to be cut and pasted somewhere else, but I'd like to progress towards tighter integration such as shared user logins for photo commenting and such. I think the achievement I'm most proud of though is that random image module. To pull from over 100,000 images and display one instantly with a link to its source gallery was a real accomplishment to me considering my very limited coding skills.

Hey I just found something that might be even better for building your tree: smugmug.users.getTree "Retrieve a hierarchical album tree for a user."

SamirD · November 15, 2010

So I'm writing out the logic for a script that will use the API to collect photos since the UI is so slow, lol.

The problem that I may run into is the same one Lint mentioned here in the thread. I was going to retrieve an album's image list via smugmug.images.get and then use a foreach on each of the images in the resulting array, have it pass each of the image id and keys to smugmug.images.collect to collect each image. But the problem is, a loop like this would execute in milliseconds and hammer the API server with possibly hundreds of request in a second. Will it be able to handle it or will I run into the same problem Lint did?

If I will run into problems, what's the spacing need to be like? 5 seconds, 1 second, 1/2 second?

Lint · November 17, 2010

If the problem I've outlined exists - and I think it does - then it's not going to go away, even with get_tree because there doesn't seem to be a get_all method for images, even in get_tree, meaning if you want to get a bunch of images you have to do them per album. At the moment that's not a problem for me, I'm only picking up half a dozen or so albums but if I increase that - from past issues to around 30 or 40 - then I think I'm going to run into problems.

I should say though, that I'm still not sure if this is a problem caused by my not storing something in the session. Although I'm only getting a new phpSmug object the once on a page, during the testing process I'm calling it multiple times in the session. I'm wondering if this might be wrong.

If nobody minds, I'm going to move this over to support to see if I can't get some thoughts from the smugmug end.

SamirD · November 17, 2010

Lint wrote: »

If the problem I've outlined exists - and I think it does - then it's not going to go away, even with get_tree because there doesn't seem to be a get_all method for images, even in get_tree, meaning if you want to get a bunch of images you have to do them per album. At the moment that's not a problem for me, I'm only picking up half a dozen or so albums but if I increase that - from past issues to around 30 or 40 - then I think I'm going to run into problems.

Oh...I see what you're trying to do now. Yeah, you and I are going to run into the same problem as we need to request information from the API repeatedly in a loop.

If you could have support chime in on the discussion here, that would help everyone.

devbobo · November 17, 2010

If you are having issues with timeouts on your end, I suggest that using the Heavy option probably isn't a good idea.

If you aren't using all the info that is returned from that response, I recommend that you build your own custom response using the Extras parameter by including any parameters that you want kinda like this...

$albums = $f->albums_get( "Extras=Description,Public,URL" );

Cheers,

David

SamirD · November 17, 2010

What about the issue of rapidly repeating requests to the API? In Lint's case, using images.get on each album, and in my case of using images.collect on all the images in an album?

Problems repeatedly grabbing Smugmug data

Comments