APIKeys & You!

onethumb · April 1, 2005

So, it turns out that people are doing some crazy things with the API. Including a few attempts to basically resell sub-accounts from smugmug and things like that.

We've wanted some nice way of tracking API usage for awhile, and combined with the beginnings of some abuse issues, we thought it was time to start authorizing API usage.

For the most part, we'll probably just authorize you quickly and easily if you're doing non-commercial stuff. If you're planning on making money via your API work, though, it becomes a gray area that we'll have to take on a case-by-case basis.

Regardless, post any questions, comments, complaints, whatever here for us to discuss and we'll see how it goes.

Don

devbobo · April 1, 2005

Don,

I do have a concern but I will email you privately about it.

BTW, great job...the expanded API is looking great

Cheers,

David

onethumb wrote:

So, it turns out that people are doing some crazy things with the API. Including a few attempts to basically resell sub-accounts from smugmug and things like that.

We've wanted some nice way of tracking API usage for awhile, and combined with the beginnings of some abuse issues, we thought it was time to start authorizing API usage.

For the most part, we'll probably just authorize you quickly and easily if you're doing non-commercial stuff. If you're planning on making money via your API work, though, it becomes a gray area that we'll have to take on a case-by-case basis.

Regardless, post any questions, comments, complaints, whatever here for us to discuss and we'll see how it goes.

Don

onethumb · April 1, 2005

Should have mentioned a few more things:

- If you're building more than one app, sign up for more than one Key.

- You can see the status of your current Key applications on your control panel.

Don

devbobo · April 1, 2005

onethumb wrote:

Should have mentioned a few more things:

- If you're building more than one app, sign up for more than one Key.

- You can see the status of your current Key applications on your control panel.

Don

Does this mean for apps like Smugmug Explorer, each end-user must get a key ??

onethumb · April 1, 2005

devbobo wrote:

Does this mean for apps like Smugmug Explorer, each end-user must get a key ??

Nope, it means that if Nik wants to do a 2nd, different application, say "Smugmug Destroyer", he needs two keys.

Each application, service, plugin, whatever needs a key. But not each binary or anything crazy like that.

Don

devbobo · April 1, 2005

onethumb wrote:

Nope, it means that if Nik wants to do a 2nd, different application, say "Smugmug Destroyer", he needs two keys.

Each application, service, plugin, whatever needs a key. But not each binary or anything crazy like that.

Don

ok...what's happens in the situation where you code being distributed is in script form not compiled. anyone can easily get access to an application's APIKey.

David

onethumb · April 1, 2005

devbobo wrote:

ok...what's happens in the situation where you code being distributed is in script form not compiled. anyone can easily get access to an application's APIKey.

David

Hmm, good question. Code obfuscation tools might help, but obviously they're not impervious.

Anyone else wanna chime in here with any suggestions as to how we deal with this sort of situation?

Worst case, we shut that key down and give you a new one in case of abuse. People running your script will need to get the new copy. Temporary fix, and a pain in the butt, but I don't know what else to suggest.

APIKeys aren't designed to be Fort Knox or anything, just enough of a deterrent that we can track and squash abuse. I think it'll make the abusers, and especially profiteers, leery if their app can stop working at any time.

Don

rutt · April 1, 2005

Ouch! This is potentially really bad for things like smugmug.py. Here's an alternative approach. Why not register and validate smugmug users to use any API tool instead of the tools? A powerful enough tool would facilitate all kinds of abuse in the wrong hands, but might be just the nuts for the majority of users. So:

Users need to register in order to use any API tool.
An API exists to facilitate this registration.
Turn off users who abuse.

Maybe this dosen't exactly address the issue? Maybe you are looking for a bigger flyswatter?

Here is another idea. Suppose you keep a set of <user,tool> pairs. When the tool logs in it tells you it's name, which has to be registered. Do you have a way to datamine for the kind of misuse you have in mind? Anyway if you knew the name of the tool I suppose you'd have some additional info.

Can we be more concrete about exactly what you are trying to prevent? Maybe we need to take this to an email discussion for security reasons?

flyingpylon · April 1, 2005

I understand the need for this. I think that right now you've got a model that is similar to Amazon's. I know they require an ID (developer token) that identifies the developer of the application so that they can track usage. Or at least that's the way it was in AWS 3.0, which is now ECS or whatever they're calling it today. My point is that the developer tokens were exposed to the public in scripts, etc. and I don't recall there being a big problem with it. But depending on how Smugmug intends to use these IDs, I can see how there could be issues.

Cameron · April 1, 2005

Well, I think that there's NO way they could make it "bulletproof". There are always going to be those that will abuse or attempt to circumvent the system. However, if they can cut out 90% of the API usage that they don't approve of, the other 10% will be easier to sort through I imagine. I'm new to smugmug and have only brushed the surface of the API but I think it's wonderful. Keep up the great work!

onethumb · April 1, 2005

CSwinton wrote:

Well, I think that there's NO way they could make it "bulletproof". There are always going to be those that will abuse or attempt to circumvent the system. However, if they can cut out 90% of the API usage that they don't approve of, the other 10% will be easier to sort through I imagine. I'm new to smugmug and have only brushed the surface of the API but I think it's wonderful. Keep up the great work!

This is exactly the intent.

Thanks for the kind words.

Don

rutt · April 1, 2005

OK, got my key. Can you tell us what the drop dead date is for starting to use it?

One possibility for scripts: a fairly large set of valid keys. The script writer could be responsible for use a different one for each download. Then you can disable a specific one of the keys and not effect the majority of users, only those who have inheritied the script from a wrongdoer.

Nikolai · April 1, 2005

That's a swell idea

onethumb wrote:

Nope, it means that if Nik wants to do a 2nd, different application, say "Smugmug Destroyer", he needs two keys.

Don

thanks guys, I'm onto it

On a business side, having just one key is never gonna be a fool proof. Even in my own case, while it would be pretty tought to get the key from my executable (it's pretty heavily protected), it's very easy to set up a local web server, relay all the smugmug requests to it - and recieve unencrypted codes..

So unless Don is gonna use some some sort of asymmetrical encryption, this hole thing can be only used for a statistical purposes, and I would not bet my life on the results.

With public/private key, however, it becomes much more secure - while not necessarily much tougher to implement.
Yes, it does not help much script guys, since their IDs will be wide open, and any additional dll/so/exe can also be hacked.. But hey, at least a step in a right direction.

I say - let's do something and see how it goes.

onethumb · April 1, 2005

rutt wrote:

OK, got my key. Can you tell us what the drop dead date is for starting to use it?

For anything "new" (REST, stats calls, category calls, subcategory calls, image and album deletion), that drop-dead date is today. I'll probably even make the new method names require it later today, too.

For anything "old" (all the old methods via XML-RPC) I don't have a planned horizon. Possibly never, but certainly a long time down the road.

Don

Nikolai · April 1, 2005

Got my key! REST, here I come!
Don, one more question:
can session IDs be shared/mixed for xmpl-prc call and REST calls?

Scenario:
I log in via REST (so my api key is confirmed)
* do smth via REST
* do smth via RPC
* logout via RPC
* login with hash via RPC

would this work? or it's unknown and I need to try..

By biggest thing here: as you know, I already have a whole bunch of code kinda field-tested to work with RPC.

I would love to add new REST-based features, and I don't mind to replace few login/logout methods, but if sessions are incompatible, this means I'd have to redo the whole app before I can use any of the new stuff...

waddayasay?

devbobo · April 1, 2005

Nikolai wrote:

With public/private key, however, it becomes much more secure - while not necessarily much tougher to implement.

Nik, but you end up in the same situation...but now you have two keys to manage and you can't keep either private.

David

Nikolai · April 1, 2005

David,

devbobo wrote:

Nik, but you end up in the same situation...but now you have two keys to manage and you can't keep either private.

David

The point of that ecryption is to avoid sending AppKey into the the open.

Here's how I thought of it:
* SM publishes public key(s), keeps private key(s)
* I'm using (a) public key to encrypt all the requests
* SM decrypts them using pertinent private key
* since my appkey is only known to SM and me (and, trust me, a hacker would have a pretty hard time to extract that appid from my binary file), they'd know that app is claiming to be me *is* me.

I'm not saying this is unbreakable - it is.
However, it would proably take a few weeks to break it. Once break is noticed, it would take SM a few minutes to change the keys, and maybe an hour for me to change the internal protection.

If someone wants to go through that hassle only to get a hold of $15..$50 worth of software intended to upload pictures to the account you'd have to pay for anyway - he can be my guest...:-)

Cheers!

devbobo · April 1, 2005

Nikolai wrote:

The point of that ecryption is to avoid sending AppKey into the the open.

Here's how I thought of it:
* SM publishes public key(s), keeps private key(s)
* I'm using (a) public key to encrypt all the requests
* SM decrypts them using pertinent private key
* since my appkey is only known to SM and me (and, trust me, a hacker would have a pretty hard time to extract that appid from my binary file), they'd know that app is claiming to be me *is* me.

I'm not saying this is unbreakable - it is.
However, it would proably take a few weeks to break it. Once break is noticed, it would take SM a few minutes to change the keys, and maybe an hour for me to change the internal protection.

If someone wants to go through that hassle only to get a hold of $15..$50 worth of software intended to upload pictures to the account you'd have to pay for anyway - he can be my guest...:-)

Cheers!

Nik,

I understand encryption...i am currently doing a Masters in Information Security.

All I meant, is for the guys doing scripting...whether it be symmetric or asymmetric encryption makes no different as its difficult to obfascate the keys. In this case (IMHO), encryption is a wasted overhead.

Cheers,

David

rutt · April 1, 2005

So, was my idea of branding the scripts at download time with different keys a dumb idea? That would allow SM the flexibility to disable one or many of the keys and not turn off legitamate users who never shared their keys with bad abusers. It seems simple enough.

Nikolai · April 1, 2005

Scripts are kinda tough..
I agree.. As I mentioned before, this additional asymmetric keys approach would only work for (protected/encrypted) binaries only.

I believe the idea behind this is simply to put a gate to API usage.
However, with the scripting languages all those keys would be lying around wide open, so nothing would prevent a "bad guy" to simpley download yours or rutt's script, get the key and they use it at his discretion.

I don't know how to protect those apikeys without introducing at least some sort of local encryption.
I can come out with windows dll/activex/... (i.e. heavily secured/encypted/protected binary) which would carry such an apikey for you, but then you'd need to distibute your own license key associated with particular user account...

We're talking of pretty big can of worms here.
Yet again, only for scripted solutions I don't see any easy way, properly protected binaries do not have this issue..

onethumb · April 1, 2005

Nikolai wrote:

Don, one more question:
can session IDs be shared/mixed for xmpl-prc call and REST calls?

Scenario:
I log in via REST (so my api key is confirmed)
* do smth via REST
* do smth via RPC
* logout via RPC
* login with hash via RPC

would this work? or it's unknown and I need to try..

By biggest thing here: as you know, I already have a whole bunch of code kinda field-tested to work with RPC.

I would love to add new REST-based features, and I don't mind to replace few login/logout methods, but if sessions are incompatible, this means I'd have to redo the whole app before I can use any of the new stuff...

waddayasay?

Should work fine. Try it.

Note that there aren't any "REST-based features" ... everything REST can do, XML-RPC can do as well.

Don

rutt · April 1, 2005

Is the different-key-per-download idea for scripts just too stupid to deserve a comment? If so, please explain why to me. I'm not getting it.

onethumb · April 1, 2005

rutt wrote:

So, was my idea of branding the scripts at download time with different keys a dumb idea? That would allow SM the flexibility to disable one or many of the keys and not turn off legitamate users who never shared their keys with bad abusers. It seems simple enough.

I wouldn't say it's dumb, but it's just more work than we want to do.

The APIKeys are purely a minor deterrent, and they have other uses than preventing abuse.

If it keeps 90% of people from gaming the system, I'll consider it a runaway success.

Don

rutt · April 1, 2005

Please can you explain what the bad things are that people are doing or that you expect that they might do. Maybe this isn't a good thing to do in public, but I like my script and being able to script the smugmug api and I really don't want that to stop working, so I'm trying to help make keep it from doing stuff you don't like. But I don't really know what I'm trying to avoid.

Should I just not worry my little head about this? Or can/should I help?

Nikolai · April 1, 2005

Thank you!

onethumb wrote:

Should work fine. Try it.
Don

I will!

onethumb wrote:

Note that there aren't any "REST-based features" ... everything REST can do, XML-RPC can do as well.
Don

That's even better..

It means I don't have to change my existing stuff! Shweeeet!

Cheers!

Nikolai · April 1, 2005

subcategories.getAll - not compatible!!!
I modified SE code to get the new settings a whurl.
Most of the "old code" works - with one huge exception:
smugmug.subcatgories.getAll (aka getAllSubcategories) returns result which is stucturakky different from the previos version!!

The good news - existing version (the one that uses version 1.0 and no API Key) works.

The bad news - new one does not..:-(

Here's the log from the SE version 106 (latest deployed):

[PHP]
Post to http://api.smugmug.com/xmlrpc/

<?xml version="1.0"?>
<methodCall>
<methodName>getAllSubCategories</methodName>
<params>
<param><value>... session id ...</value></param>
</params>
</methodCall>
<?xml version="1.0" encoding="iso-8859-1"?>

<methodResponse>
<params>
<param>
<value>
<array>
<data>
<value>
<struct>
<member>
<name>CategoryID</name>
<value>
<int>0</int>
</value>
</member>
<member>
<name>SubCategoryID</name>
<value>
<int>53110</int>
</value>
</member>
<member>
<name>Title</name>
<value>
<string>Empty</string>
</value>
</member>
</struct>
</value>
[/PHP]

and here's the identical log from the updated version:
[PHP]
Post to http://api.smugmug.com/hack/xmlrpc/

<?xml version="1.0"?>
<methodCall>
<methodName>smugmug.subcategories.getAll</methodName>
<params>
<param><value>.. another, longer session id ....</value></param>
</params>
</methodCall>

<?xml version="1.0" encoding="iso-8859-1"?>
<methodResponse>
<params>
<param>
<value>
<struct>
<member>
<name>SubCategory0</name>
<value>
<struct>
<member>
<name>CategoryID</name>
<value>
<int>0</int>
</value>
</member>
<member>
<name>SubCategoryID</name>
<value>
<int>53110</int>
</value>
</member>
<member>
<name>Title</name>
<value>
<string>Empty</string>
</value>
</member>
</struct>
</value>
[/PHP]

As it is easy to see there are two differences
1) session IDs have different length - not that I care, but it may be a hint
2) "new code" has additional level of nesting (see Subcategory0) - of course, existing parsing code "chokes" on it..

Don, is this a new behaviour or just a bug?

Other new rpcxml methods that I tested and found working:

login with password
login with hash
logout
get albums
create album
upload via post

Nikolai · April 1, 2005

APIKeys & You!

Comments