I'm trying the gzip feature. I add the header: "accept-encoding: gzip". I get back "encoding: gzip", which makes sense to me. But the python patches I dug up to support this are expecting "x-gzip". Which is right?
Use of program names for the identification of encoding formats
is not desirable and is discouraged for future encodings. Their
use here is representative of historical practice, not good
design. For compatibility with previous implementations of HTTP,
applications SHOULD consider "x-gzip" and "x-compress" to be
equivalent to "gzip" and "compress" respectively.
Not sure what HTTP library you're using from Python, but I've had great success in other Python projects with libcurl (http://curl.haxx.se/libcurl/) using the pycurl interface (http://curl.haxx.se/libcurl/python/). It's thread safe and supports every imaginable option for every protocol. You might have to rewrite some of your code to use curl but it will give you huge flexability in the future since you're nearly guaranteed that there's support for whatever wierd request you want to make. Fiddle with the "curl" binary which is the command line version and you'll see all the magic it supports (--compressed in particular). From Python, you're interested in CURLOPT_ENCODING.
Sadly, Perl support for libcurl seems miles behind python - no multi support!.
This seems to have become fixed since the last time I tried it.
I take it back. It seems to be unfixed. Don, are you hacking ont this right now?
I coded around the first problem to answer your questions. The result of getCagetories seems unchanged from the 1.1.0 version. The result of getTree doesn't seem right. I have a lot of subcategories and albums in my 'Other' and they are not showing up (the are not public, perhaps that's why?)
Python uses expat to parse xmlrpc results. It parses the result of getTree as an array. But it's first member has an unnamed element which is a single subcategory:
Trying to parse the returned XML is enough to convert me to rest. If I can find a python interface, I'm going there. Meanwhile, I will try to parse this well enough to finish the bug report.
I'm still mystified by the incredibly verbose XML returned from the xmlrpc call to getTree, but now I'm sure it must be wrong. I have plenty of categories and subcategories, and lots of albums in both categories and subcategories, but the XML has only one ... tag pair, which I don't think can be what's documented. Further, I think the documentation mug be garbled. What is the name of the field of a category that holds the array of it's subcategories? What is the name of the album array fields.
Trying to parse the returned XML is enough to convert me to rest. If I can find a python interface, I'm going there. Meanwhile, I will try to parse this well enough to finish the bug report.
You're using http://docs.python.org/lib/module-xml.parsers.expat.html right? AFAIK you can parse the responses from the REST API with any XML parser - or at least that's what I'm doing for now. I'm currently using REST since it was easier for me to build my requests as simple URLs. I'll probably switch to the XML API since I suspect the REST API is less formal and could be subject to more wild changes than the XML one. Don: Any comment on which API is lower impact on the SM servers or what direction the future of each is?
Some hints for people playing with the new beta APIs and parsing in general:
1. If you're on a unixy box try using 'xmllint -shell some_xml_file.xml' which will give you a shell to navigate around in the XML file. You can type 'help' for a list of commands but the ones I find most usefull are: cd, dir, ls, cat, pwd. I'm not enough of an expert to figure out what the leading fields on an ls are but I do know that to cd into arrays you use the syntax 'cd array[4]' to get to a particular element. See http://marty.feebleandfrail.org/macosxhints/webscrape/ for a tutorial I wrote on using xmllint and XML::LibXML to scrape web pages.
2. Run the XML response from smugmug through a trivial parser like http://docs.python.org/lib/expat-example.html. If you're traversing the XML structure, adding calls to something like Perl's Data::Dumper or the Python module pprint to your code is a huge help.
3. Open the XML file in your browser! Compare the output of (1) and (2) as well as your debug messages.
4. Some APIs to XML parsers have annoying built in smartness where they automagically change how the data is returned. These can do wierd things to your parsed results! One of the Perl XML parsers, XML::Simple, for example, has a bunch of flags such as ForceArray and KeyAttr that control how the library will roll single nested elements up into attributes and when it will return arrays vs. hashes. In particular I found that you have to be very aware how XML::Simple behaves when there's one element (a user with 1 album, say) vs. multiple elements.
I'm still mystified by the incredibly verbose XML returned from the xmlrpc call to getTree, but now I'm sure it must be wrong. I have plenty of categories and subcategories, and lots of albums in both categories and subcategories, but the XML has only one <array>...</array> tag pair, which I don't think can be what's documented. Further, I think the documentation mug be garbled. What is the name of the field of a category that holds the array of it's subcategories? What is the name of the album array fields.
Yeah - I get different responses from the REST API vs. the XML one. Sample responses are at http://marty.feebleandfrail.org/smug/ You'll notice that the XML one has no mention of, say, the sub category ID 77964 (title 1998).
I take it back. It seems to be unfixed. Don, are you hacking ont this right now?
I coded around the first problem to answer your questions. The result of getCagetories seems unchanged from the 1.1.0 version. The result of getTree doesn't seem right. I have a lot of subcategories and albums in my 'Other' and they are not showing up (the are not public, perhaps that's why?)
Python uses expat to parse xmlrpc results. It parses the result of getTree as an array. But it's first member has an unnamed element which is a single subcategory:
Trying to parse the returned XML is enough to convert me to rest. If I can find a python interface, I'm going there. Meanwhile, I will try to parse this well enough to finish the bug report.
But Don, are you hacking?
Looks like a bug. I'll track it down. getTree is, as you can imagine, a fairly complex bit of code compared to the other methods, so there were bound to be some landmines. I'm a big REST advocate now, having used both side-by-side, so I mostly test myself using REST. It makes for much more rapid development and testing on my part.
I'll post back when I've had a chance to drill into what's going wrong with the XML-RPC response. It almost looks like it's getting truncated somewhere on my side.
You're using http://docs.python.org/lib/module-xml.parsers.expat.html right? AFAIK you can parse the responses from the REST API with any XML parser - or at least that's what I'm doing for now. I'm currently using REST since it was easier for me to build my requests as simple URLs. I'll probably switch to the XML API since I suspect the REST API is less formal and could be subject to more wild changes than the XML one. Don: Any comment on which API is lower impact on the SM servers or what direction the future of each is?
They'll both be maintained in parallel and lockstep. One won't fall behind the other, and they're both fairly lightweight on our servers, AFAICT.
If Amazon, Flickr, and some of the other web services companies are to be trusted, REST quickly outpaces both SOAP and XML-RPC for # of users and # of calls made with them. I didn't fully understand why that was until a few weeks ago, but I get it: REST is simply easier to use and *far* easier to debug. Even the 'eyeball' method of debugging works great.
Good to hear that REST will be fully supported! I'm also working on a super light weight REST client for debugging that's just a set of bash (the unix shell) functions. I'll post it when I get it a little more polished.
Add to your 2do why getTree heavy as an anonymous user via REST doesn't return albums. In fact, even for a session logged in via username/password it will only return albums for the user that the session is logged in as.
For those playing with compression support, it's a very big improvement. A smugmug.images.get with Heavy=1 without compression takes 4 seconds and transfers 39,382 bytes. With compression it's 1 second and 3,655 bytes. I did my best when doing my tests to do them from separate sessions so that any caching on the SM side wouldn't be a factor. With calls that return small (<1k?) amounts of data the difference seems negligible - the compressed size is around 400 bytes vs. 900 and both take about .4 of a second.
Add to your 2do why getTree heavy as an anonymous user via REST doesn't return albums. In fact, even for a session logged in via username/password it will only return albums for the user that the session is logged in as.
smsh - Test tools for use in bash http://marty.feebleandfrail.org/smug/smsh.txt has some functions I wrote for testig the REST API from a bash shell. Here's what's there now, but I may update the web version at some point:
# smsh - bash shell routines for the smugmug REST API
#
# Requires curl, perl and xmllint. If you dont have xmllint change
# the sm_response routine to just use echo
#
# Source this file into your shell. then try something like:
#
# sm_anon
# sm_request users.getTree 'NickName=onethumb&Heavy=1'
# sm_request albums.get 'NickName=onethumb&Heavy=0'
# sm_request albums.get 'NickName=onethumb&Heavy=1'
#
# You can view or set your SessionID with 'sm_sid' which is handy if
# you're testing sessions other than the anon one.
#
#
# You will have to put a real API key in here...
export SM_APIKEY='PUT-YOUR-API-KEY-HERE'
export SM_URL='http://api.smugmug.com/hack/rest/beta.mg'
export SM_VER='1.1.1'
##
# These control what's displayed for each request
# Set them to '' to disable them or anything else
# to enable them.
#
export SM_show_gets='1'
export SM_show_response='1'
##
# Handy accessors for the above options as well as our session ID
#
function sm_sid {
if [ $# -eq 1 ] ; then export SM_sid="$1" ; fi
echo "SM_sid is $SM_sid"
}
function sm_showgets {
if [ $# -eq 1 ] ; then export SM_show_gets="$1" ; fi
echo "SM_show_gets is $SM_show_gets"
}
function sm_showresponse {
if [ $# -eq 1 ] ; then export SM_show_response="$1" ; fi
echo "SM_show_response is $SM_show_response"
}
# Display routines
#
function sm_gets {
if [ "$SM_show_gets" != "" ]; then echo $* ; fi
}
function sm_response {
if [ "$SM_show_response" != "" ]; then
# If you don't have xmllint then just use this:
# echo $*
#
echo $* > /tmp/sm-$$.xml
xmllint --format /tmp/sm-$$.xml
rm -f /tmp/sm-$$.xml
fi
}
# Anon login that sets our session ID
#
function sm_anon {
URL="${SM_URL}?method=smugmug.login.anonymously&Version=${SM_VER}&APIKey=${SM_APIKEY}"
sm_gets "Getting $URL"
export RESP=`curl -isk "${URL}"`
export SM_sid=`echo $RESP | perl -ne 'print "$1\n" if (m#<SessionID>([^<]+)</SessionID>#);'`
sm_response "$RESP"
echo "Session is $SM_sid"
}
function sm_request {
METHOD=$1
ARGS=$2
URL="${SM_URL}?method=smugmug.${METHOD}&SessionID={$SM_sid}&${ARGS}"
sm_gets "Getting $URL"
export RESP=`curl -s "${URL}"`
sm_response "$RESP"
}
Spotted a typo already. In sm_request, this: URL="${SM_URL}?method=smugmug.${METHOD}&SessionID={$SM_sid}&${ARGS}"
should have the {}s around only SM_sid, not $SM_sid, so it should read: URL="${SM_URL}?method=smugmug.${METHOD}&SessionID=${SM_sid}&${ARGS}" It seems to work OK even with the typo.
For those without curl (fools!), you can also change sm_request to use netcat (nc), assuming you have a version of echo that supports the "-e" option. Instead of export RESP=`curl -s "${URL}"`
use:
export RESP=`echo -e "GET /hack/rest/beta.mg?method=smugmug.${METHOD}&SessionID=${SM_sid}&${ARGS} HTTP/1.0\n\n" | nc api.smugmug.com 80 | perl -ne 'print if ($show); $show=1 if (m#^\r$#);'`
Yes, that's all one huge ugly line... You'll have to change sm_anon in a similar way.
For those with a late enough version (one that properly works with stdin/stdout in client mode) of stunnel [http://www.stunnel.org] that don't have curl (wierd fools!) that want to make calls over HTTPS you can use "stunnel -c -r api.smugmug.com 443" in place of the nc call.
For those without xmllint, try this in sm_resposne which should give a slightly nicer output, but still not as good as xmllint:
# If you don't have xmllint then just use this:
echo $* | sed -e "s/> *</>\n</g"
It looks like the responses from the REST API have more nicely formatted XML than what I've seen in the past which were all glumped together on a single line. Don: Has this changed?
Yes, it has, and I expect it to be permanent, but you never know. XML doesn't care one way or the other, so single line, clumpy, or human-readable should all parse the same.
Currently, it indents one space for each new element in the heirarchy, and I think it's quite readable. Far more readable than XML-RPC, for example.
http://marty.feebleandfrail.org/smug/smsh.txt should now auto detect if you have xmllint and curl and use the fallbacks I mentioned above. I changed the names of sm_anon and sm_request to simply smrequest and smanon as well as adding intelligent completion to the smrequest command so you don't have to remember the exact method names. Don: Feel free to link it (or mirror it, but it may get updated) on the Hacks page.
Is there any programatic way to get information about the API in terms of what methods it supports and the parameters they want and the data types they return? My reason is two fold:
1. I'd like to be able to offer auto-completion of paramters for the various methods.
2. If I write a tool that does, say a getTree, I'd like to know what all the possible return fields will be without having to copy/paste them from the API web pages or having the client figure them all out itself by walking the response [walking the first element of a given type isn't sufficient since other elements may have additional fields!]. Hard coding them seems bad since they may change in the future and writing the code to detect them also seems silly.
If, say, a text file or XML document was created this would also probably help in keeping the API documentation up to date since it could be auto-generated from the description file.
getTree only public galleries
I used studiosells' excellent shell debugging stuff to take a look at the REST result from getTree. I had to make a loginWithPassword function, but that was pretty easy. Even after I had done this, my private galleries don't show up in getTree.
The rest output is really nice! Might be time to convert.
There does still seem to be a problem with the format of the result from xmlrpc getTree (beyond the problem with private albums.)
In short, thie problem is that each category has each of its subcategories as an unnamed member. I think that each category which has any subcategories should have a named member which is an array of subcategories. It's no mystery why the python xmlrpc parser is confused by the way things are now. Each category ends up with a single member named "" containing the last subcategory listed.
In case that wasn't clear, here it is in excrutiating detail.
Hey everyone...
- new methods, including: smugmug.users.getTransferStats, smugmug.users.getTree
Don,
There seems to be inconsistencies with the returned data structure and/or formats using REST (I'm not sure about XML-RPC).
1) getTree and Heavy=1
IDs for Category, SubCategory, Community, and Highlight in Albums are not consistent whether the album is or is not in a subcategory.
If the album is NOT in a Subcategory, these IDs are attributes of empty elements.
First, I must say that I really like the new REST response format. Makes parsing a whole lot easier.
I've only played with a handful of methods so far but there a a few things I have found.
smugmug.users.getTree does not return album information. I think this has been brought up previously.
smugmug.accounts.getType returns an <Accounts> element. Should it be <Account> since it only returns one item and is referring to one account? Accounts suggest multiple account items being returned. This is for naming consistency with the other elements.
I like what I see with the new methods and format. I hope to have more time to play around with the other methods.
BUG: subcategories.rename -- xmlrpc version returns nothing
A successful call the subcategories.rename actually does manage to rename, but the result is empty xml causing a fault in python.
Changes:
1. smcontrol should cover all the environment variables. This is a show all/show one/set one interface.
2. smrequest now has auto-completion. [I did post that the _ in the function names went away, right?]. This especially helps when you can't rememebr a method name. I also added basic argument checking and a feeble help message.
3. New functions smalbumsize which takes an albumid and smuseralbums that takes a nickname. These are suitable prototypes for writing your own shell fucntions that do something beyond the raw exposed API through smrequest. One returns a single line and the other a series of lines. I'll probably find a slightly slicker (generic) way of writing these but they're OK for nowl. No auto-completion on these but they do check arguments and provide a help message.
I based these two examples off some fancy command lines that I was playing with. Here's an example of how to use the easier functions:
$ smanon ; for i in `smuseralbums onethumb`; do echo "Album $i is " `smalbumsize $i` ; done
Album 399848 is bytes.
Album 395683 is 2,731,127 bytes.
Album 79151 is 407,941 bytes.
Album 171345 is 1,586,598 bytes.
...
4. Shortened some functions to one-liners. This seems fine for bash but if you're using some other shell it may cuase problems. Added SM_CURLOPT which is a good place to put any proxy parameters.
5. Added smfields which is a super cool function you should try. For example:
[I have yet to find a good way to post raw command lines - I added some whitespace in the above command around the br and img tags. What's the easiest way to get no special treatment other than escaping HTM entiries like less-than, greather-than??]
This could be the basis of some fairly nice HTML rendering of something like getTree or some other unixy parsing of the SM responses. See (6) below to know that the impact of multiple smfields calls won't be as heavy as 1:1.
6. Added caching! This is based on the md5 of the arguments to the call so it's not smart enough to realize if a "Heavy=1" call contained the same data as another call. It's good enough to save you for multiple requests if you're just greping some data.
The caching is pretty simple - it uses /tmp/smcache as a working directory and caches responses based of the MD5 of the method and argument list. It pays no attentino to in-modified so if there's a file that matches a request then that's what's used. There's a log file inside the cache directory that will tell you what each file coresponds to and when it was created/used as a cache hit.
Once I get confident that the HTTP if-modified headers for the REST API work I'll add something more complex than the basic -f check.
7. YOUR IDEA HERE. So far shsh has only dealt with read requests. I've played a little with write requests and am particularly interested in doing uploads. Can someone give me a curl command line that does an upload via REST? smsh seems good enough that with some cute shell / perl code you could get read access to most things.
BUG: Old Session ID error is Invalid API Key
I tried using an old SessionID for smugmug.users.getTree and it returned:
<RSP stat="fail">
<ERR msg="invalid API key" code="18" />
</RSP>
Status of BETA 1.1.1 ?
What is the status of the v1.1.1 BETA (XmlRpc)? It seems to have been kicking around for some time - is it getting close to release?
Onethumb there seems to be a problem with "smugmug.users.getTransferStats". Everytime I call it, it returns the same information as "smugmug.users.getType". It's not a coding error on my behalf as I've tried using different API functions in the same code.
Comments
Both.
Use of program names for the identification of encoding formats
is not desirable and is discouraged for future encodings. Their
use here is representative of historical practice, not good
design. For compatibility with previous implementations of HTTP,
applications SHOULD consider "x-gzip" and "x-compress" to be
equivalent to "gzip" and "compress" respectively.
Not sure what HTTP library you're using from Python, but I've had great success in other Python projects with libcurl (http://curl.haxx.se/libcurl/) using the pycurl interface (http://curl.haxx.se/libcurl/python/). It's thread safe and supports every imaginable option for every protocol. You might have to rewrite some of your code to use curl but it will give you huge flexability in the future since you're nearly guaranteed that there's support for whatever wierd request you want to make. Fiddle with the "curl" binary which is the command line version and you'll see all the magic it supports (--compressed in particular). From Python, you're interested in CURLOPT_ENCODING.
Sadly, Perl support for libcurl seems miles behind python - no multi support!.
-m
I take it back. It seems to be unfixed. Don, are you hacking ont this right now?
I coded around the first problem to answer your questions. The result of getCagetories seems unchanged from the 1.1.0 version. The result of getTree doesn't seem right. I have a lot of subcategories and albums in my 'Other' and they are not showing up (the are not public, perhaps that's why?)
Python uses expat to parse xmlrpc results. It parses the result of getTree as an array. But it's first member has an unnamed element which is a single subcategory:
Trying to parse the returned XML is enough to convert me to rest. If I can find a python interface, I'm going there. Meanwhile, I will try to parse this well enough to finish the bug report.
But Don, are you hacking?
Some hints for people playing with the new beta APIs and parsing in general:
1. If you're on a unixy box try using 'xmllint -shell some_xml_file.xml' which will give you a shell to navigate around in the XML file. You can type 'help' for a list of commands but the ones I find most usefull are: cd, dir, ls, cat, pwd. I'm not enough of an expert to figure out what the leading fields on an ls are but I do know that to cd into arrays you use the syntax 'cd array[4]' to get to a particular element. See http://marty.feebleandfrail.org/macosxhints/webscrape/ for a tutorial I wrote on using xmllint and XML::LibXML to scrape web pages.
2. Run the XML response from smugmug through a trivial parser like http://docs.python.org/lib/expat-example.html. If you're traversing the XML structure, adding calls to something like Perl's Data::Dumper or the Python module pprint to your code is a huge help.
3. Open the XML file in your browser! Compare the output of (1) and (2) as well as your debug messages.
4. Some APIs to XML parsers have annoying built in smartness where they automagically change how the data is returned. These can do wierd things to your parsed results! One of the Perl XML parsers, XML::Simple, for example, has a bunch of flags such as ForceArray and KeyAttr that control how the library will roll single nested elements up into attributes and when it will return arrays vs. hashes. In particular I found that you have to be very aware how XML::Simple behaves when there's one element (a user with 1 album, say) vs. multiple elements.
-m
Yeah - I get different responses from the REST API vs. the XML one. Sample responses are at http://marty.feebleandfrail.org/smug/ You'll notice that the XML one has no mention of, say, the sub category ID 77964 (title 1998).
-m
Looks like a bug. I'll track it down. getTree is, as you can imagine, a fairly complex bit of code compared to the other methods, so there were bound to be some landmines. I'm a big REST advocate now, having used both side-by-side, so I mostly test myself using REST. It makes for much more rapid development and testing on my part.
I'll post back when I've had a chance to drill into what's going wrong with the XML-RPC response. It almost looks like it's getting truncated somewhere on my side.
Don
They'll both be maintained in parallel and lockstep. One won't fall behind the other, and they're both fairly lightweight on our servers, AFAICT.
If Amazon, Flickr, and some of the other web services companies are to be trusted, REST quickly outpaces both SOAP and XML-RPC for # of users and # of calls made with them. I didn't fully understand why that was until a few weeks ago, but I get it: REST is simply easier to use and *far* easier to debug. Even the 'eyeball' method of debugging works great.
Don
Add to your 2do why getTree heavy as an anonymous user via REST doesn't return albums. In fact, even for a session logged in via username/password it will only return albums for the user that the session is logged in as.
For those playing with compression support, it's a very big improvement. A smugmug.images.get with Heavy=1 without compression takes 4 seconds and transfers 39,382 bytes. With compression it's 1 second and 3,655 bytes. I did my best when doing my tests to do them from separate sessions so that any caching on the SM side wouldn't be a factor. With calls that return small (<1k?) amounts of data the difference seems negligible - the compressed size is around 400 bytes vs. 900 and both take about .4 of a second.
-m
Try now?
Don
Fixed, I think?
Don
Fixed! I tested it from an anon session but I trust that it's also fixed for a normal user session doing getTree on users other than that user.
-m
http://marty.feebleandfrail.org/smug/smsh.txt has some functions I wrote for testig the REST API from a bash shell. Here's what's there now, but I may update the web version at some point:
# smsh - bash shell routines for the smugmug REST API # # Requires curl, perl and xmllint. If you dont have xmllint change # the sm_response routine to just use echo # # Source this file into your shell. then try something like: # # sm_anon # sm_request users.getTree 'NickName=onethumb&Heavy=1' # sm_request albums.get 'NickName=onethumb&Heavy=0' # sm_request albums.get 'NickName=onethumb&Heavy=1' # # You can view or set your SessionID with 'sm_sid' which is handy if # you're testing sessions other than the anon one. # # # You will have to put a real API key in here... export SM_APIKEY='PUT-YOUR-API-KEY-HERE' export SM_URL='http://api.smugmug.com/hack/rest/beta.mg' export SM_VER='1.1.1' ## # These control what's displayed for each request # Set them to '' to disable them or anything else # to enable them. # export SM_show_gets='1' export SM_show_response='1' ## # Handy accessors for the above options as well as our session ID # function sm_sid { if [ $# -eq 1 ] ; then export SM_sid="$1" ; fi echo "SM_sid is $SM_sid" } function sm_showgets { if [ $# -eq 1 ] ; then export SM_show_gets="$1" ; fi echo "SM_show_gets is $SM_show_gets" } function sm_showresponse { if [ $# -eq 1 ] ; then export SM_show_response="$1" ; fi echo "SM_show_response is $SM_show_response" } # Display routines # function sm_gets { if [ "$SM_show_gets" != "" ]; then echo $* ; fi } function sm_response { if [ "$SM_show_response" != "" ]; then # If you don't have xmllint then just use this: # echo $* # echo $* > /tmp/sm-$$.xml xmllint --format /tmp/sm-$$.xml rm -f /tmp/sm-$$.xml fi } # Anon login that sets our session ID # function sm_anon { URL="${SM_URL}?method=smugmug.login.anonymously&Version=${SM_VER}&APIKey=${SM_APIKEY}" sm_gets "Getting $URL" export RESP=`curl -isk "${URL}"` export SM_sid=`echo $RESP | perl -ne 'print "$1\n" if (m#<SessionID>([^<]+)</SessionID>#);'` sm_response "$RESP" echo "Session is $SM_sid" } function sm_request { METHOD=$1 ARGS=$2 URL="${SM_URL}?method=smugmug.${METHOD}&SessionID={$SM_sid}&${ARGS}" sm_gets "Getting $URL" export RESP=`curl -s "${URL}"` sm_response "$RESP" }-m
should have the {}s around only SM_sid, not $SM_sid, so it should read: URL="${SM_URL}?method=smugmug.${METHOD}&SessionID=${SM_sid}&${ARGS}" It seems to work OK even with the typo.
For those without curl (fools!), you can also change sm_request to use netcat (nc), assuming you have a version of echo that supports the "-e" option. Instead of export RESP=`curl -s "${URL}"`
use:
export RESP=`echo -e "GET /hack/rest/beta.mg?method=smugmug.${METHOD}&SessionID=${SM_sid}&${ARGS} HTTP/1.0\n\n" | nc api.smugmug.com 80 | perl -ne 'print if ($show); $show=1 if (m#^\r$#);'`Yes, that's all one huge ugly line... You'll have to change sm_anon in a similar way.For those with a late enough version (one that properly works with stdin/stdout in client mode) of stunnel [http://www.stunnel.org] that don't have curl (wierd fools!) that want to make calls over HTTPS you can use "stunnel -c -r api.smugmug.com 443" in place of the nc call.
For those without xmllint, try this in sm_resposne which should give a slightly nicer output, but still not as good as xmllint:
# If you don't have xmllint then just use this: echo $* | sed -e "s/> *</>\n</g"-m
Yes, it has, and I expect it to be permanent, but you never know. XML doesn't care one way or the other, so single line, clumpy, or human-readable should all parse the same.
Currently, it indents one space for each new element in the heirarchy, and I think it's quite readable. Far more readable than XML-RPC, for example.
Don
Is there any programatic way to get information about the API in terms of what methods it supports and the parameters they want and the data types they return? My reason is two fold:
1. I'd like to be able to offer auto-completion of paramters for the various methods.
2. If I write a tool that does, say a getTree, I'd like to know what all the possible return fields will be without having to copy/paste them from the API web pages or having the client figure them all out itself by walking the response [walking the first element of a given type isn't sufficient since other elements may have additional fields!]. Hard coding them seems bad since they may change in the future and writing the code to detect them also seems silly.
If, say, a text file or XML document was created this would also probably help in keeping the API documentation up to date since it could be auto-generated from the description file.
-m
I used studiosells' excellent shell debugging stuff to take a look at the REST result from getTree. I had to make a loginWithPassword function, but that was pretty easy. Even after I had done this, my private galleries don't show up in getTree.
The rest output is really nice! Might be time to convert.
This does seem to be fixed.
In short, thie problem is that each category has each of its subcategories as an unnamed member. I think that each category which has any subcategories should have a named member which is an array of subcategories. It's no mystery why the python xmlrpc parser is confused by the way things are now. Each category ends up with a single member named "" containing the last subcategory listed.
In case that wasn't clear, here it is in excrutiating detail.
I have:
The REST version of getTree returns gives this for this branch of the tree:
<Categories> <Category id="0"> <Title>Other</Title> <SubCategories> <SubCategory id="77964"> <Title>1998</Title> <SubCategory id="77954"> <Title>1999</Title> </SubCategory> <SubCategory id="77955"> <Title>2000</Title> </SubCategory> <SubCategory id="77956"> <Title>2001</Title> </SubCategory> <SubCategory id="77957"> <Title>2002</Title> </SubCategory> <SubCategory id="77958"> <Title>2003</Title> </SubCategory> <SubCategory id="77959"> <Title>2004</Title> </SubCategory> <SubCategory id="77960"> <Title>2005</Title> </SubCategory> </Category>Fine. But the xmlrpc result looks like this:
[i] junk elided [/i] <struct> <member> <name>CategoryID</name> <value> <int>0</int> </value> </member> <member> <name>Title</name> <value> <string>Other</string> </value> </member> <member> <name/> <value> <struct> <member> <name>SubCategoryID</name> <value> <int>77954</int> </value> </member> <member> <name>Title</name> <value> <string>1999</string> </value> </member> </struct> </value> </member> <member> <name/> <value> <struct> <member> <name>SubCategoryID</name> <value> <int>77955</int> </value> </member> <member> <name>Title</name> <value> <string>2000</string> </value> </member> </struct> </value> </member> [i] more junk elided[/i]Notice that the category "Other" has an unnamed member (look for "name/") for each subcategory. How is this supposed to work?
There seems to be inconsistencies with the returned data structure and/or formats using REST (I'm not sure about XML-RPC).
1) getTree and Heavy=1
IDs for Category, SubCategory, Community, and Highlight in Albums are not consistent whether the album is or is not in a subcategory.
If the album is NOT in a Subcategory, these IDs are attributes of empty elements. If the album is in a Subcategory, these IDs are values in the same-named element. (Sorry about the spaces in the XML but I'm not sure how to post it otherwise without them getting blown away when I submit)
2) images.get with Heavy=1 (or images.getInfo)
The 'LastUpdated' item format does not contain any non-numerical characters, while 'Date' does The 'Date' format is used consistently (at least from what I've seen) throughout.
-J
I know I'm not always terse enough, so I just wanted to summarize to make it easier for you, Don.
I've only played with a handful of methods so far but there a a few things I have found.
smugmug.users.getTree does not return album information. I think this has been brought up previously.
smugmug.accounts.getType returns an <Accounts> element. Should it be <Account> since it only returns one item and is referring to one account? Accounts suggest multiple account items being returned. This is for naming consistency with the other elements.
I like what I see with the new methods and format. I hope to have more time to play around with the other methods.
Bill
A successful call the subcategories.rename actually does manage to rename, but the result is empty xml causing a fault in python.
I put a new version of smsh at http://marty.feebleandfrail.org/smug/smsh-098.txt
Changes:
1. smcontrol should cover all the environment variables. This is a show all/show one/set one interface.
2. smrequest now has auto-completion. [I did post that the _ in the function names went away, right?]. This especially helps when you can't rememebr a method name. I also added basic argument checking and a feeble help message.
3. New functions smalbumsize which takes an albumid and smuseralbums that takes a nickname. These are suitable prototypes for writing your own shell fucntions that do something beyond the raw exposed API through smrequest. One returns a single line and the other a series of lines. I'll probably find a slightly slicker (generic) way of writing these but they're OK for nowl. No auto-completion on these but they do check arguments and provide a help message.
I based these two examples off some fancy command lines that I was playing with. Here's an example of how to use the easier functions:
4. Shortened some functions to one-liners. This seems fine for bash but if you're using some other shell it may cuase problems. Added SM_CURLOPT which is a good place to put any proxy parameters.
5. Added smfields which is a super cool function you should try. For example:
or to download all the TinyURL pictures:
[I have yet to find a good way to post raw command lines - I added some whitespace in the above command around the br and img tags. What's the easiest way to get no special treatment other than escaping HTM entiries like less-than, greather-than??]
This could be the basis of some fairly nice HTML rendering of something like getTree or some other unixy parsing of the SM responses. See (6) below to know that the impact of multiple smfields calls won't be as heavy as 1:1.
6. Added caching! This is based on the md5 of the arguments to the call so it's not smart enough to realize if a "Heavy=1" call contained the same data as another call. It's good enough to save you for multiple requests if you're just greping some data.
The caching is pretty simple - it uses /tmp/smcache as a working directory and caches responses based of the MD5 of the method and argument list. It pays no attentino to in-modified so if there's a file that matches a request then that's what's used. There's a log file inside the cache directory that will tell you what each file coresponds to and when it was created/used as a cache hit.
Once I get confident that the HTTP if-modified headers for the REST API work I'll add something more complex than the basic -f check.
7. YOUR IDEA HERE. So far shsh has only dealt with read requests. I've played a little with write requests and am particularly interested in doing uploads. Can someone give me a curl command line that does an upload via REST? smsh seems good enough that with some cute shell / perl code you could get read access to most things.
-m
They work.
The entire smugmug site uses the same code, and has for more than two years.
Don
I tried using an old SessionID for smugmug.users.getTree and it returned:
<RSP stat="fail">
<ERR msg="invalid API key" code="18" />
</RSP> I'm assuming that it's supposed to return code 4 - "invalid user"
-J
What is the status of the v1.1.1 BETA (XmlRpc)? It seems to have been kicking around for some time - is it getting close to release?
Thanks,
snadboy