new *BETA* API release - May 10, 2005

[onethumb]

Hey everyone...

The threatened revamped REST release is out, and with it, a bunch of new things for both REST and XML-RPC. Highlights include:

- new BETA endpoint to test your code against before it rolls out to the production endpoints. See REST docs (http://www.smugmug.com/hack/rest-beta) and XML-RPC docs (http://www.smugmug.com/hack/xmlrpc-beta) on the subject.

- REST is now quite a bit more verbose. All ID tags should now use the requested format, and all responses should now reside in a root container in addition to the RSP container. This is what I need feedback on more than anything else.

- new 'Heavy' flags for a few methods allow you to get heavier results, when desired, and get more data back. Methods include smugmug.albums.get, smugmug.images.get, and smugmug.users.getTree.

- smugmug.albums.getStats automatically uses a Heavy-like mode already, so you'll get images.getStats in the same call.

- new methods, including: smugmug.users.getTransferStats, smugmug.users.getTree

- renamed smugmug.accounts.getType to smugmug.users.getType. old method will, of course, continue to work but will be removed from the docs once this is out of beta.

- both interfaces now support gzip encoding (use standard HTTP header 'Accept-Encoding'). Use it, love it. I've seen responses go from 500K to 13K.

- both interfaces now support standard HTTP "If-Not-Modified" headers using ETags. Save even that 13K if you'd like if things haven't changed.

- doc bugs fixed, including smugmug.images.get, smugmug.albums.changeSettings, smugmug.albums.getInfo, and more.

- REST now properly responds in UTF-8, so foreign chars should propogate properly. Be interesting to see what happens to UTF-8 submissions. Anyone care to test a bit more?

- fixed bug in logout, should work properly now

- fixed bug in smugmug.images.delete, should work properly now

UPDATE 11am PST: XML-RPC now accepts a struct with named elements for all calls. No more worrying about the order of the elements, just make it a struct and name each one appropriately. I imagine this will probably become the standard way of doing things (no, the old way won't go away, it will just cease to be recommended or documented).

Whew. There might be more, but luckily, it's a BETA. Some of the docs have intentionally been left a little vague while we test and refine any responses to the methods. Suggestions welcome, but anything tagged with BETA in the docs is clearly not complete.

Enjoy!

Don

[devbobo]

Don,

Looks like some great stuff here, but...
smugmug.users.getTree ??

I am going to cry
Took to hours to write and debug my code to build that exact same tree

David

[devbobo]

Don,

With smugmug.users.getTree if Heavy is set, does that return all the images in an album ?

Thanks,

David

[onethumb]

Don,

With smugmug.users.getTree if Heavy is set, does that return all the images in an album ?

Thanks,

David

No, it doesn't. It did for a little while in early testing, but since images aren't really part of the organization, I figured they didn't belong.

Additionally, if the call gets too heavy, it could fail to finish - I have checks for runaway calls. Some people have 60,000 photos (in who knows how many galleries), so it could get pretty enormous.

Combined with the other calls, though, this should dramatically reduce the # of calls required to accomplish almost anything.

I *might* be open to putting the images in there, but I'd have to hear a pretty compelling reason why you'd want all that stuff in a single call.

Don

[devbobo]

No, i didn't really want it in there. I was actually worried about how large the response my be, that was the original intent of the message.

Cheers,

David

[devbobo]

Don,

ok, a couple of questions regarding XMLRPC...

1. can't seem to retrieve the public flag, from any methods....smugmug.albums.getInfo, smug.albums.get with Heavy and smugmug.albumtemplates.get. Any ideas ?

2. using smugmug.images.get with the heavy flag, I can't retrieve ImageID ? Or is it called something else ?

Thanks,

David

[Nikolai]

Great news, Don, thank a bunch!
Sorry being too late with my response, was postprocessing:-)
It looks like many of my personal wishes came to reality:-)
I'm looking forward to get my hands dirty again:-)

Cheers!:1drink

[Nikolai]

- both interfaces now support gzip encoding. Use it, love it. I've seen responses go from 500K to 13K.
It's seems like a truly great feature, but I don't seem to be able to find it in the RPC-XML docs. What do I need to do to activate it?

Thanks!

[onethumb]

It's seems like a truly great feature, but I don't seem to be able to find it in the RPC-XML docs. What do I need to do to activate it?

Thanks!

It's a web-standard via HTTP headers, really has nothing to do with the API itself, and thus, isn't documented on smugmug.

Google probably knows all about it, though. Look for HTTP specs. 'Accept-Encoding' is the header in question, I believe. ETag & If-Not-Modified headers would be in those specs, too.

Don

[onethumb]

Don,

ok, a couple of questions regarding XMLRPC...

1. can't seem to retrieve the public flag, from any methods....smugmug.albums.getInfo, smug.albums.get with Heavy and smugmug.albumtemplates.get. Any ideas ?

Hmm, I see Public in a few of these just fine. I'll keep looking, though. Can you post the XML-RPC response?


2. using smugmug.images.get with the heavy flag, I can't retrieve ImageID ? Or is it called something else ?

Hmm, yeah, that would help. :)

Fixed, I think. Let me know.

Don

[devbobo]

Hmm, I see Public in a few of these just fine. I'll keep looking, though. Can you post the XML-RPC response?


Hmm, yeah, that would help. :)

Fixed, I think. Let me know.

Don
ok...seems that the type of Public (and the other boolean flags) has changed back to INTs again....ggrrrrr

Can't seem to access the ImageID though...any thoughts ?

Thanks,

David

[Nikolai]

It's a web-standard via HTTP headers, really has nothing to do with the API itself, and thus, isn't documented on smugmug.

Google probably knows all about it, though. Look for HTTP specs. 'Accept-Encoding' is the header in question, I believe. ETag & If-Not-Modified headers would be in those specs, too.

Don
More grinder time:-)
Cheers!:thumb

[onethumb]

ok...seems that the type of Public (and the other boolean flags) has changed back to INTs again....ggrrrrr

Can't seem to access the ImageID though...any thoughts ?

Thanks,

David

Woops, try now. Both.

Don

[dsdee]

getTree looks pretty slick; can't wait to get home from work and play with it.

Hey Don, any chance (still) of getting an interface into the print pricing into the API ??

--David

[onethumb]

getTree looks pretty slick; can't wait to get home from work and play with it.

Hey Don, any chance (still) of getting an interface into the print pricing into the API ??

--David

Just so we're clear, the eventual goal with the API is to expose everything that A) is exposed in the web interface, and B) doesn't require SSL-grade security (ie, shopping cart, signing up for an account).

So, yes, pro pricing is on the TODO list. Who knows which year it'll get done, but it's coming. :)

Don

[Nikolai]

Just so we're clear, the eventual goal with the API is to expose everything that A) is exposed in the web interface, and B) doesn't require SSL-grade security (ie, shopping cart, signing up for an account).

So, yes, pro pricing is on the TODO list. Who knows which year it'll get done, but it's coming. :)

Don
Pricing API would be soooo damn cool!:thumb

[onethumb]

As noted in the first post on this thread, it's been updated with the following info:

XML-RPC now accepts a struct with named elements for all calls. No more worrying about the order of the elements, just make it a struct and name each one appropriately. I imagine this will probably become the standard way of doing things (no, the old way won't go away, it will just cease to be recommended or documented).

Don

[devbobo]

Woops, try now. Both.

Don
Don,

Thanks...both are working http://dgrin.com/images/smilies/thumb.gif


David

[Nikolai]

As noted in the first post on this thread, it's been updated with the following info:

XML-RPC now accepts a struct with named elements for all calls. No more worrying about the order of the elements, just make it a struct and name each one appropriately. I imagine this will probably become the standard way of doing things (no, the old way won't go away, it will just cease to be recommended or documented).

Don
This is great news, I think it's a much more robust way of dealing with all those calls!
Thanks again!:thumb

[studiosells]

I'm working on a Perl library for the REST API and some small clients. I hadn't read the BETA API stuff so missed out several new features (getTree, heavy requests, ...) until today. My current implementation (that doesn't use the BETA API features) uses parallel HTTP GET requests over separate connections to do a series of albums.getInfo and images.getInfo.

Without knowing about the BETA API, I was thinking that I wanted a smugmug.albums.getAllInfo that's like smugmug.subcategories.getAll and will return info for all albums for a given user. This would save clients having to enumerate each album and doing a smugmug.albums.getInfo on each one. And similarly, when getting image information for a particular album, a client has to enumerate the ImageIDs returned by smugmug.images.get and do a smugmug.images.getInfo for each one. Similarly, I had wanted a smugmug.images.getAllInfo. These both seem to be covered by the BETA API through the Heavy parameter - well done!

The reason I post, however, is to let others know that the production REST API supports persistent HTTP connections which seem like a good alternative to doing parallel connections/requests. For those not framiliar with persistent HTTP:

RFC2068, describes an improvement to HTTP which maintains a continuous connection to a HTTP server for multiple requests, P-HTTP. This removes the inefficiency of continually reconnecting to a web server to download multiple images from the same page. The constant connection and reconnection results in a lot of unnecessary overhead.

Some advantages over the original HTTP protocol are:
1. By opening and closing fewer TCP connections, CPU time and memory are saved.
2. Multiple HTTP requests and responses can be sent without waiting for a response which would otherwise be necessary when opening and closing multiple connections.
3. Network congestion is reduced since there are fewer packets.


With the BETA API covering my above requests, I suspect it's still a performance enhancement to use persistent HTTP. Has anybody tested the performance of persistent connections vs. parallel ones? I've fiddled with both, but only a little. I suspect the SM admins would be happier if everything went through a persistent connection.

Beware!! I'll also note that responses from the BETA REST API aren't the same as the production one. I think this has happened with adding support for Heavy requests. For example, smugmug.albums.get returns just a series of Album entries whereas the beta one returns those entries inside an Albums entry.

-m

[onethumb]

I'm working on a Perl library for the REST API and some small clients. I hadn't read the BETA API stuff so missed out several new features (getTree, heavy requests, ...) until today. My current implementation (that doesn't use the BETA API features) uses parallel HTTP GET requests over separate connections to do a series of albums.getInfo and images.getInfo.

Without knowing about the BETA API, I was thinking that I wanted a smugmug.albums.getAllInfo that's like smugmug.subcategories.getAll and will return info for all albums for a given user. This would save clients having to enumerate each album and doing a smugmug.albums.getInfo on each one. And similarly, when getting image information for a particular album, a client has to enumerate the ImageIDs returned by smugmug.images.get and do a smugmug.images.getInfo for each one. Similarly, I had wanted a smugmug.images.getAllInfo. These both seem to be covered by the BETA API through the Heavy parameter - well done!

The reason I post, however, is to let others know that the production REST API supports persistent HTTP connections which seem like a good alternative to doing parallel connections/requests. For those not framiliar with persistent HTTP:

RFC2068, describes an improvement to HTTP which maintains a continuous connection to a HTTP server for multiple requests, P-HTTP. This removes the inefficiency of continually reconnecting to a web server to download multiple images from the same page. The constant connection and reconnection results in a lot of unnecessary overhead.

Some advantages over the original HTTP protocol are:
1. By opening and closing fewer TCP connections, CPU time and memory are saved.
2. Multiple HTTP requests and responses can be sent without waiting for a response which would otherwise be necessary when opening and closing multiple connections.
3. Network congestion is reduced since there are fewer packets.


With the BETA API covering my above requests, I suspect it's still a performance enhancement to use persistent HTTP. Has anybody tested the performance of persistent connections vs. parallel ones? I've fiddled with both, but only a little. I suspect the SM admins would be happier if everything went through a persistent connection.

Beware!! I'll also note that responses from the BETA REST API aren't the same as the production one. I think this has happened with adding support for Heavy requests. For example, smugmug.albums.get returns just a series of Album entries whereas the beta one returns those entries inside an Albums entry.

-m


I've always assumed everyone is using persistent connections, but your post makes me realize that maybe they aren't. Thanks for bringing this up.

If you're opening and closing the port each and every time, you're wasting valuable time. We allow keep-alive requests and persistent connections. If you're idle too long, the connection will close, so you'll have to check your state, but that's fairly trivial.

I think the best way to do API connections is a mixture of persistant and parallel connections. For example, doing parallel uploads is very handy - while we're checking to make sure the file you've uploaded is a valid file, there's a pause. During this pause, there's no reason you can't be uploading the next photo on a different socket and thread. Go for it - it'll save a lot of time.

Likewise, getting info for many photos/galleries/etc can be spread across multiple persistent connections.

Don

[devbobo]

Don,

Just a quick note to let you know how cool some of the new features are. I really love the new heavy flag. It's made my work some much easier.

Love ur work http://dgrin.com/images/smilies/thumb.gif

David

[studiosells]

http://api.smugmug.com/hack/rest/beta.mg?APIKey=FOO&method=smugmug.users.getTree&Version=1.1.0&SessionID=BAR&Heavy=true&NickName=studiosells

Doesn't seem to return Album entries.

I also tried it with Heavy=1

-m

[onethumb]

http://api.smugmug.com/hack/rest/beta.mg?APIKey=FOO&method=smugmug.users.getTree&Version=1.1.0&SessionID=BAR&Heavy=true&NickName=studiosells

Doesn't seem to return Album entries.

I also tried it with Heavy=1

-m

First of all, the correct call would be:

http://api.smugmug.com/hack/rest/beta.mg?method=smugmug.users.getTree&Heavy=1&NickName=studiosells&SessionID=SESSIONID

Remember, Version, APIKey, etc, are all set during the login phase, not on subsequent calls.

Secondly, I get an Album there just fine. In the Motorcycles Category (id=14). You don't see it?

Don

[studiosells]

Gotcha on the version/APIkey/version not being needed for each request.

From an anonymous session it seems to never return the Albums. If I login as studiosells then getTree will return albums for that one user but not any others. Did you use some super-admin-session to be able to list the albums for studiosells ?

With the old API I noticed that albums.get would return private entries for which the albums.getInfo call would subsequently fail. Perhaps this is related somehow?

-m

[studiosells]

Unless I'm doing something silly, accounts.getType for an anonymous session returns AccountType Power /AccountType.

I also noticed that the help page for smugmug.login.anonymously has

Latest Version is currently String "1.1.0".

So perhaps my problem is that the BETA APIs aren't enabled properly for anonymous connections, even if I specify a version of 1.1.1.

-m

[rutt]

I'm trying the gzip feature. I add the header: "accept-encoding: gzip". I get back "encoding: gzip", which makes sense to me. But the python patches I dug up to support this are expecting "x-gzip". Which is right?

[rutt]

Maybe I missed this in the thread, but it seems that the results for xmlrpc calls are pretty different. I used to get something like this from login:

{'SessionID': 'a8fa6666eca05179ccb6d4618', 'PasswordHash': '$1$sTTB3bPG$sRMm3o0hXqR/', 'UserID': 666}

But now I get something like:

{'Login': {'values': {'SessionID': 'f50d0d556519d9d72192', 'PasswordHash': '$1$ojiCMwYz.nqNSVZ.', 'UserID': 666}, 'parameters': {'arrayName': 'Login'}}}

Is this intentional? Perhaps my python xmlrpc parser has been fooled, though it does seem to have parsed correctly.

Anyway, I didn't see this documented.

[dsdee]

Is this intentional? Perhaps my python xmlrpc parser has been fooled, though it does seem to have parsed correctly.

Anyway, I didn't see this documented. Nope, don't think it' python as i saw similar with Perl's xml-rpc.

it also seems that an extra layer (or two?) is put into the structure returned

I used to have:

smAPI.pm: $smSessionID = $resp->{'SessionID'}->value;

but with the beta must have:

smAPIBeta.pm: $smSessionID = $resp->{'Login'}->{'values'}->{'SessionID'}->value;


I just figured that the docs for the call hadn't been updated yet...

[onethumb]

Maybe I missed this in the thread, but it seems that the results for xmlrpc calls are pretty different. I used to get something like this from login:

{'SessionID': 'a8fa6666eca05179ccb6d4618', 'PasswordHash': '$1$sTTB3bPG$sRMm3o0hXqR/', 'UserID': 666}

But now I get something like:

{'Login': {'values': {'SessionID': 'f50d0d556519d9d72192', 'PasswordHash': '$1$ojiCMwYz.nqNSVZ.', 'UserID': 666}, 'parameters': {'arrayName': 'Login'}}}

Is this intentional? Perhaps my python xmlrpc parser has been fooled, though it does seem to have parsed correctly.

Anyway, I didn't see this documented.

Strange. No, the REST results are now wrapped in 'Login', but XML-RPC shouldn't be.

Are you seeing this with other calls, too?

Don

[rutt]

Strange. No, the REST results are now wrapped in 'Login', but XML-RPC shouldn't be.

Are you seeing this with other calls, too?

Don

This seems to have become fixed since the last time I tried it.

[studiosells]

I'm trying the gzip feature. I add the header: "accept-encoding: gzip". I get back "encoding: gzip", which makes sense to me. But the python patches I dug up to support this are expecting "x-gzip". Which is right?

Both. :D According to http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html

Use of program names for the identification of encoding formats
is not desirable and is discouraged for future encodings. Their
use here is representative of historical practice, not good
design. For compatibility with previous implementations of HTTP,
applications SHOULD consider "x-gzip" and "x-compress" to be
equivalent to "gzip" and "compress" respectively.

Not sure what HTTP library you're using from Python, but I've had great success in other Python projects with libcurl (http://curl.haxx.se/libcurl/) using the pycurl interface (http://curl.haxx.se/libcurl/python/). It's thread safe and supports every imaginable option for every protocol. You might have to rewrite some of your code to use curl but it will give you huge flexability in the future since you're nearly guaranteed that there's support for whatever wierd request you want to make. Fiddle with the "curl" binary which is the command line version and you'll see all the magic it supports (--compressed in particular). From Python, you're interested in CURLOPT_ENCODING.

Sadly, Perl support for libcurl seems miles behind python - no multi support!.

-m

[rutt]

This seems to have become fixed since the last time I tried it.

I take it back. It seems to be unfixed. Don, are you hacking ont this right now?

I coded around the first problem to answer your questions. The result of getCagetories seems unchanged from the 1.1.0 version. The result of getTree doesn't seem right. I have a lot of subcategories and albums in my 'Other' and they are not showing up (the are not public, perhaps that's why?)

Python uses expat to parse xmlrpc results. It parses the result of getTree as an array. But it's first member has an unnamed element which is a single subcategory:

[{'': {'SubCategoryID': 77964, 'Title': '1998'}, 'CategoryID': 0, 'Title': 'Other'},


Trying to parse the returned XML is enough to convert me to rest. If I can find a python interface, I'm going there. Meanwhile, I will try to parse this well enough to finish the bug report.

But Don, are you hacking?

[rutt]

I'm still mystified by the incredibly verbose XML returned from the xmlrpc call to getTree, but now I'm sure it must be wrong. I have plenty of categories and subcategories, and lots of albums in both categories and subcategories, but the XML has only one <array>...</array> tag pair, which I don't think can be what's documented. Further, I think the documentation mug be garbled. What is the name of the field of a category that holds the array of it's subcategories? What is the name of the album array fields.

[studiosells]

Trying to parse the returned XML is enough to convert me to rest. If I can find a python interface, I'm going there. Meanwhile, I will try to parse this well enough to finish the bug report.

You're using http://docs.python.org/lib/module-xml.parsers.expat.html right? AFAIK you can parse the responses from the REST API with any XML parser - or at least that's what I'm doing for now. I'm currently using REST since it was easier for me to build my requests as simple URLs. I'll probably switch to the XML API since I suspect the REST API is less formal and could be subject to more wild changes than the XML one. Don: Any comment on which API is lower impact on the SM servers or what direction the future of each is?

Some hints for people playing with the new beta APIs and parsing in general:

1. If you're on a unixy box try using 'xmllint -shell some_xml_file.xml' which will give you a shell to navigate around in the XML file. You can type 'help' for a list of commands but the ones I find most usefull are: cd, dir, ls, cat, pwd. I'm not enough of an expert to figure out what the leading fields on an ls are but I do know that to cd into arrays you use the syntax 'cd array[4]' to get to a particular element. See http://marty.feebleandfrail.org/macosxhints/webscrape/ for a tutorial I wrote on using xmllint and XML::LibXML to scrape web pages.

2. Run the XML response from smugmug through a trivial parser like http://docs.python.org/lib/expat-example.html. If you're traversing the XML structure, adding calls to something like Perl's Data::Dumper or the Python module pprint to your code is a huge help.

3. Open the XML file in your browser! Compare the output of (1) and (2) as well as your debug messages.

4. Some APIs to XML parsers have annoying built in smartness where they automagically change how the data is returned. These can do wierd things to your parsed results! One of the Perl XML parsers, XML::Simple, for example, has a bunch of flags such as ForceArray (http://search.cpan.org/~grantm/XML-Simple-2.12/lib/XML/Simple.pm#ForceArray_=%3E_1_#_in_-_important) and KeyAttr that control how the library will roll single nested elements up into attributes and when it will return arrays vs. hashes. In particular I found that you have to be very aware how XML::Simple behaves when there's one element (a user with 1 album, say) vs. multiple elements.

-m

[studiosells]

I'm still mystified by the incredibly verbose XML returned from the xmlrpc call to getTree, but now I'm sure it must be wrong. I have plenty of categories and subcategories, and lots of albums in both categories and subcategories, but the XML has only one <array>...</array> tag pair, which I don't think can be what's documented. Further, I think the documentation mug be garbled. What is the name of the field of a category that holds the array of it's subcategories? What is the name of the album array fields.

Yeah - I get different responses from the REST API vs. the XML one. Sample responses are at http://marty.feebleandfrail.org/smug/ You'll notice that the XML one has no mention of, say, the sub category ID 77964 (title 1998).

-m

[onethumb]

I take it back. It seems to be unfixed. Don, are you hacking ont this right now?

I coded around the first problem to answer your questions. The result of getCagetories seems unchanged from the 1.1.0 version. The result of getTree doesn't seem right. I have a lot of subcategories and albums in my 'Other' and they are not showing up (the are not public, perhaps that's why?)

Python uses expat to parse xmlrpc results. It parses the result of getTree as an array. But it's first member has an unnamed element which is a single subcategory:

[{'': {'SubCategoryID': 77964, 'Title': '1998'}, 'CategoryID': 0, 'Title': 'Other'},


Trying to parse the returned XML is enough to convert me to rest. If I can find a python interface, I'm going there. Meanwhile, I will try to parse this well enough to finish the bug report.

But Don, are you hacking?

Looks like a bug. I'll track it down. getTree is, as you can imagine, a fairly complex bit of code compared to the other methods, so there were bound to be some landmines. I'm a big REST advocate now, having used both side-by-side, so I mostly test myself using REST. It makes for much more rapid development and testing on my part.

I'll post back when I've had a chance to drill into what's going wrong with the XML-RPC response. It almost looks like it's getting truncated somewhere on my side.

Don

[onethumb]

You're using http://docs.python.org/lib/module-xml.parsers.expat.html right? AFAIK you can parse the responses from the REST API with any XML parser - or at least that's what I'm doing for now. I'm currently using REST since it was easier for me to build my requests as simple URLs. I'll probably switch to the XML API since I suspect the REST API is less formal and could be subject to more wild changes than the XML one. Don: Any comment on which API is lower impact on the SM servers or what direction the future of each is?


They'll both be maintained in parallel and lockstep. One won't fall behind the other, and they're both fairly lightweight on our servers, AFAICT.

If Amazon, Flickr, and some of the other web services companies are to be trusted, REST quickly outpaces both SOAP and XML-RPC for # of users and # of calls made with them. I didn't fully understand why that was until a few weeks ago, but I get it: REST is simply easier to use and *far* easier to debug. Even the 'eyeball' method of debugging works great. :)

Don

[studiosells]

Good to hear that REST will be fully supported! I'm also working on a super light weight REST client for debugging that's just a set of bash (the unix shell) functions. I'll post it when I get it a little more polished.

Add to your 2do why getTree heavy as an anonymous user via REST doesn't return albums. In fact, even for a session logged in via username/password it will only return albums for the user that the session is logged in as.

For those playing with compression support, it's a very big improvement. A smugmug.images.get with Heavy=1 without compression takes 4 seconds and transfers 39,382 bytes. With compression it's 1 second and 3,655 bytes. I did my best when doing my tests to do them from separate sessions so that any caching on the SM side wouldn't be a factor. With calls that return small (<1k?) amounts of data the difference seems negligible - the compressed size is around 400 bytes vs. 900 and both take about .4 of a second.

-m

[onethumb]

Add to your 2do why getTree heavy as an anonymous user via REST doesn't return albums. In fact, even for a session logged in via username/password it will only return albums for the user that the session is logged in as.

Try now?

Don

[onethumb]

I take it back. It seems to be unfixed. Don, are you hacking ont this right now?

Fixed, I think?

Don

[studiosells]

Try now?

Don

Fixed! I tested it from an anon session but I trust that it's also fixed for a normal user session doing getTree on users other than that user.

-m

[studiosells]

http://marty.feebleandfrail.org/smug/smsh.txt has some functions I wrote for testig the REST API from a bash shell. Here's what's there now, but I may update the web version at some point:


# smsh - bash shell routines for the smugmug REST API
#
# Requires curl, perl and xmllint. If you dont have xmllint change
# the sm_response routine to just use echo
#
# Source this file into your shell. then try something like:
#
# sm_anon
# sm_request users.getTree 'NickName=onethumb&Heavy=1'
# sm_request albums.get 'NickName=onethumb&Heavy=0'
# sm_request albums.get 'NickName=onethumb&Heavy=1'
#
# You can view or set your SessionID with 'sm_sid' which is handy if
# you're testing sessions other than the anon one.
#
#

# You will have to put a real API key in here...

export SM_APIKEY='PUT-YOUR-API-KEY-HERE'

export SM_URL='http://api.smugmug.com/hack/rest/beta.mg'
export SM_VER='1.1.1'

##
# These control what's displayed for each request
# Set them to '' to disable them or anything else
# to enable them.
#

export SM_show_gets='1'
export SM_show_response='1'

##
# Handy accessors for the above options as well as our session ID
#
function sm_sid {
if [ $# -eq 1 ] ; then export SM_sid="$1" ; fi
echo "SM_sid is $SM_sid"
}
function sm_showgets {
if [ $# -eq 1 ] ; then export SM_show_gets="$1" ; fi
echo "SM_show_gets is $SM_show_gets"
}
function sm_showresponse {
if [ $# -eq 1 ] ; then export SM_show_response="$1" ; fi
echo "SM_show_response is $SM_show_response"
}

# Display routines
#
function sm_gets {
if [ "$SM_show_gets" != "" ]; then echo $* ; fi
}

function sm_response {
if [ "$SM_show_response" != "" ]; then

# If you don't have xmllint then just use this:
# echo $*
#
echo $* > /tmp/sm-$$.xml
xmllint --format /tmp/sm-$$.xml
rm -f /tmp/sm-$$.xml
fi
}

# Anon login that sets our session ID
#

function sm_anon {
URL="${SM_URL}?method=smugmug.login.anonymously&Version=${SM_VER}&APIKey=${SM_APIKEY}"
sm_gets "Getting $URL"
export RESP=`curl -isk "${URL}"`
export SM_sid=`echo $RESP | perl -ne 'print "$1\n" if (m#<SessionID>([^<]+)</SessionID>#);'`
sm_response "$RESP"
echo "Session is $SM_sid"
}

function sm_request {
METHOD=$1
ARGS=$2
URL="${SM_URL}?method=smugmug.${METHOD}&SessionID={$SM_sid}&${ARGS}"
sm_gets "Getting $URL"
export RESP=`curl -s "${URL}"`
sm_response "$RESP"
}


-m

[studiosells]

Spotted a typo already. In sm_request, this: URL="${SM_URL}?method=smugmug.${METHOD}&SessionID={$SM_sid}&${ARGS}"
should have the {}s around only SM_sid, not $SM_sid, so it should read: URL="${SM_URL}?method=smugmug.${METHOD}&SessionID=${SM_sid}&${ARGS}" It seems to work OK even with the typo.

For those without curl (fools!), you can also change sm_request to use netcat (nc), assuming you have a version of echo that supports the "-e" option. Instead of export RESP=`curl -s "${URL}"`
use:


export RESP=`echo -e "GET /hack/rest/beta.mg?method=smugmug.${METHOD}&SessionID=${SM_sid}&${ARGS} HTTP/1.0\n\n" | nc api.smugmug.com 80 | perl -ne 'print if ($show); $show=1 if (m#^\r$#);'`

Yes, that's all one huge ugly line... You'll have to change sm_anon in a similar way.

For those with a late enough version (one that properly works with stdin/stdout in client mode) of stunnel [http://www.stunnel.org] that don't have curl (wierd fools!) that want to make calls over HTTPS you can use "stunnel -c -r api.smugmug.com 443" in place of the nc call.

For those without xmllint, try this in sm_resposne which should give a slightly nicer output, but still not as good as xmllint:

# If you don't have xmllint then just use this:
echo $* | sed -e "s/> *</>\n</g"


-m

[onethumb]

It looks like the responses from the REST API have more nicely formatted XML than what I've seen in the past which were all glumped together on a single line. Don: Has this changed?

Yes, it has, and I expect it to be permanent, but you never know. XML doesn't care one way or the other, so single line, clumpy, or human-readable should all parse the same.

Currently, it indents one space for each new element in the heirarchy, and I think it's quite readable. Far more readable than XML-RPC, for example. :)

Don

[studiosells]

http://marty.feebleandfrail.org/smug/smsh.txt should now auto detect if you have xmllint and curl and use the fallbacks I mentioned above. I changed the names of sm_anon and sm_request to simply smrequest and smanon as well as adding intelligent completion to the smrequest command so you don't have to remember the exact method names. Don: Feel free to link it (or mirror it, but it may get updated) on the Hacks page.

Is there any programatic way to get information about the API in terms of what methods it supports and the parameters they want and the data types they return? My reason is two fold:

1. I'd like to be able to offer auto-completion of paramters for the various methods.

2. If I write a tool that does, say a getTree, I'd like to know what all the possible return fields will be without having to copy/paste them from the API web pages or having the client figure them all out itself by walking the response [walking the first element of a given type isn't sufficient since other elements may have additional fields!]. Hard coding them seems bad since they may change in the future and writing the code to detect them also seems silly.

If, say, a text file or XML document was created this would also probably help in keeping the API documentation up to date since it could be auto-generated from the description file.

-m

[rutt]

I used studiosells' excellent shell debugging stuff to take a look at the REST result from getTree. I had to make a loginWithPassword function, but that was pretty easy. Even after I had done this, my private galleries don't show up in getTree.

The rest output is really nice! Might be time to convert.

[rutt]

Fixed, I think?

Don

This does seem to be fixed.

[rutt]

There does still seem to be a problem with the format of the result from xmlrpc getTree (beyond the problem with private albums.)

In short, thie problem is that each category has each of its subcategories as an unnamed member. I think that each category which has any subcategories should have a named member which is an array of subcategories. It's no mystery why the python xmlrpc parser is confused by the way things are now. Each category ends up with a single member named "" containing the last subcategory listed.

In case that wasn't clear, here it is in excrutiating detail.

I have:


Cagegory - Other
Subcategory - 1998
Subcategory - 1999
...
Subcategory 2005


The REST version of getTree returns gives this for this branch of the tree:

<Categories>
<Category id="0">
<Title>Other</Title>
<SubCategories>
<SubCategory id="77964">
<Title>1998</Title>
<SubCategory id="77954">
<Title>1999</Title>
</SubCategory>
<SubCategory id="77955">
<Title>2000</Title>
</SubCategory>
<SubCategory id="77956">
<Title>2001</Title>
</SubCategory>
<SubCategory id="77957">
<Title>2002</Title>
</SubCategory>
<SubCategory id="77958">
<Title>2003</Title>
</SubCategory>
<SubCategory id="77959">
<Title>2004</Title>
</SubCategory>
<SubCategory id="77960">
<Title>2005</Title>
</SubCategory>
</Category>


Fine. But the xmlrpc result looks like this:


junk elided
<struct>
<member>
<name>CategoryID</name>
<value>
<int>0</int>
</value>
</member>
<member>
<name>Title</name>
<value>
<string>Other</string>
</value>
</member>
<member>
<name/>
<value>
<struct>
<member>
<name>SubCategoryID</name>
<value>
<int>77954</int>
</value>
</member>
<member>
<name>Title</name>
<value>
<string>1999</string>
</value>
</member>
</struct>
</value>
</member>
<member>
<name/>
<value>
<struct>
<member>
<name>SubCategoryID</name>
<value>
<int>77955</int>
</value>
</member>
<member>
<name>Title</name>
<value>
<string>2000</string>
</value>
</member>
</struct>
</value>
</member>
more junk elided



Notice that the category "Other" has an unnamed member (look for "name/") for each subcategory. How is this supposed to work?

[jberd126]

Hey everyone...
- new methods, including: smugmug.users.getTransferStats, smugmug.users.getTree
Don,
There seems to be inconsistencies with the returned data structure and/or formats using REST (I'm not sure about XML-RPC).

1) getTree and Heavy=1
IDs for Category, SubCategory, Community, and Highlight in Albums are not consistent whether the album is or is not in a subcategory.

If the album is NOT in a Subcategory, these IDs are attributes of empty elements.
< Highlight id="14457386"/>
< Category id="45"/>
< SubCategory id="0"/>
< Community id="0"/>If the album is in a Subcategory, these IDs are values in the same-named element.
< HighlightID>14501698< /HighlightID></HIGHLIGHTID>
< CategoryID>33< /CategoryID></CATEGORYID>
< SubCategoryID>60460< /SubCategoryID></SUBCATEGORYID>
< CommunityID>0< /CommunityID></COMMUNITYID><HIGHLIGHTID>(Sorry about the spaces in the XML but I'm not sure how to post it otherwise without them getting blown away when I submit)

2) images.get with Heavy=1 (or images.getInfo)
The 'LastUpdated' item format does not contain any non-numerical characters, < LastUpdated><LASTUPDATED>20050323025337< /LastUpdated></LASTUPDATED>while 'Date' does
< Date><DATE>2005-02-07 23:18:23< /Date></DATE>The 'Date' format is used consistently (at least from what I've seen) throughout.

-J

[rutt]

There are two outstanding bugs that I know of with getTree.

Private albums don't appear, even after login
XML-RPC has structs with unnamed members. See: here. (http://www.dgrin.com/showpost.php?p=104362&postcount=49)


I know I'm not always terse enough, so I just wanted to summarize to make it easier for you, Don.

[Kyhi]

First, I must say that I really like the new REST response format. Makes parsing a whole lot easier.

I've only played with a handful of methods so far but there a a few things I have found.

smugmug.users.getTree does not return album information. I think this has been brought up previously.

smugmug.accounts.getType returns an &lt;Accounts&gt; element. Should it be &lt;Account&gt; since it only returns one item and is referring to one account? Accounts suggest multiple account items being returned. This is for naming consistency with the other elements.

I like what I see with the new methods and format. I hope to have more time to play around with the other methods.

Bill

[rutt]

A successful call the subcategories.rename actually does manage to rename, but the result is empty xml causing a fault in python.

[studiosells]

I put a new version of smsh at http://marty.feebleandfrail.org/smug/smsh-098.txt

Changes:
1. smcontrol should cover all the environment variables. This is a show all/show one/set one interface.

2. smrequest now has auto-completion. . This especially helps when you can't rememebr a method name. I also added basic argument checking and a feeble help message.

3. New functions smalbumsize which takes an albumid and smuseralbums that takes a nickname. These are suitable prototypes for writing your own shell fucntions that do something beyond the raw exposed API through smrequest. One returns a single line and the other a series of lines. I'll probably find a slightly slicker (generic) way of writing these but they're OK for nowl. No auto-completion on these but they do check arguments and provide a help message.

I based these two examples off some fancy command lines that I was playing with. Here's an example of how to use the easier functions:


$ smanon ; for i in `smuseralbums onethumb`; do echo "Album $i is " `smalbumsize $i` ; done
Album 399848 is bytes.
Album 395683 is 2,731,127 bytes.
Album 79151 is 407,941 bytes.
Album 171345 is 1,586,598 bytes.
...

4. Shortened some functions to one-liners. This seems fine for bash but if you're using some other shell it may cuase problems. Added SM_CURLOPT which is a good place to put any proxy parameters.

5. Added smfields which is a [I]super cool function you should try. For example:

for i in `smuseralbums onethumb`; do echo "Album $i < br > < br >" ; smfields images.get "Heavy=1&AlbumID=$i" ThumbURL | perl -ne 'chop; print qq#< img src =" $_ ">\n#;' ; done

or to download all the TinyURL pictures:

$ smfields images.get "AlbumID=443&Heavy=1" TinyURL | xargs -n 10 curl -OOOOOOOOOO
[I have yet to find a good way to post raw command lines - I added some whitespace in the above command around the br and img tags. What's the easiest way to get no special treatment other than escaping HTM entiries like less-than, greather-than??]

This could be the basis of some fairly nice HTML rendering of something like getTree or some other unixy parsing of the SM responses. See (6) below to know that the impact of multiple smfields calls won't be as heavy as 1:1.

6. Added caching! This is based on the md5 of the arguments to the call so it's not smart enough to realize if a "Heavy=1" call contained the same data as another call. It's good enough to save you for multiple requests if you're just greping some data.

The caching is pretty simple - it uses /tmp/smcache as a working directory and caches responses based of the MD5 of the method and argument list. It pays no attentino to in-modified so if there's a file that matches a request then that's what's used. There's a log file inside the cache directory that will tell you what each file coresponds to and when it was created/used as a cache hit.

Once I get confident that the HTTP if-modified headers for the REST API work I'll add something more complex than the basic -f check.

7. YOUR IDEA HERE. So far shsh has only dealt with read requests. I've played a little with write requests and am particularly interested in doing uploads. Can someone give me a curl command line that does an upload via REST? smsh seems good enough that with some cute shell / perl code you could get read access to most things.

-m

[onethumb]

Once I get confident that the HTTP if-modified headers for the REST API work I'll add something more complex than the basic -f check.


They work. :)

The entire smugmug site uses the same code, and has for more than two years.

Don

[jberd126]

I tried using an old SessionID for smugmug.users.getTree and it returned:
<RSP stat="fail">
<ERR msg="invalid API key" code="18" />
</RSP>
< rsp stat="fail">
< err code="18" msg="invalid API key"/>
< /rsp>
I'm assuming that it's supposed to return code 4 - "invalid user"

-J

[snadboy]

What is the status of the v1.1.1 BETA (XmlRpc)? It seems to have been kicking around for some time - is it getting close to release?

Thanks,
snadboy

[Darryl Morgan]

Onethumb there seems to be a problem with "smugmug.users.getTransferStats". Everytime I call it, it returns the same information as "smugmug.users.getType". It's not a coding error on my behalf as I've tried using different API functions in the same code.