Dropbox fuckery

Dropbox is vile and I want no part of it in my life. However, one particular tool I work with has no way of getting its data to me except via Dropbox, and I can't fix that.

I have one of those anonymous "share" links for the folder in which my data materializes (the "dropbox.com/sh/XXXXXXXXXXXXXXX/YYYYYYYYYYYYYY_ZZZZZZZZZZ" kind) and I have a script that has been just scraping the HTML and mirroring the files directly. It's slow and wasteful and irritating but it had been working for 5+ years... except recently the Dropbox web pages have begun shitting the bed, and half the time when I load and parse them, they return different, unparsable data. Maybe these are intermittent backend errors, or maybe they've changed their Javascript that be even more obscure and only some proxies have caught up to the new, more useless way. I dunno, I can't tell.

So I tried doing it the "official" way and creating and "app" and generating an OAuth token and using the official API, but after successfully authenticating, all it ever says is "missing_scope, required_scope: files.metadata.read". (That checkbox is among the many that are checked on my app's config page.)

Have any of you ever successfully used this API to list a folder and read a file?

Note: I am not asking for your anecdote about how you installed the Dropbox GUI on your machine. And even if it is possible to install a Dropbox file-system-impersonating rootkit daemon on a headless CentOS 7 server, I have no interest in doing that. I just want to mirror files out of this one URL, by polling from cron.

Previously, previously, previously, previously.

Tags: , , ,

33 Responses:

  1. so, I'm woefully not very knowledgeable in this stuff, but my wife wants to mirror some stuff to cloud storage so I had a look at rclone. Maybe it would make this all a lot less painful?

  2. Ingo says:

    I'll go on a limb here and ask a very stupid question. Have you tried using rclone, https://rclone.org/dropbox/?

    • jwz says:

      No, because it sounded like it was just an over-complicated rsync clone, that required me to set up a Dropbox app anyway?

      Have you successfully used rclone to mirror arbitrary Dropbox folders? Because the comment below suggest that the API just doesn't support that.

      • Ingo says:

        I won't know about arbitrary folders, but works for my personal account. rclone offers "
        --dropbox-shared-folders", which might work on your use case.

  3. Why are Dropbox vile? Facebook/Twitter vile or technically vile?

  4. isntitvacant says:

    Don't take this as gospel because I only futzed around with this enough to get this to work for a personal blog thing (& I'm downloading a zip of the entire folder, not listing files), but I _believe_ any given app is limited to a single "application" folder without general access to the toplevel dropbox folder. E.g., I have code here [1] to download a "/blog" directory. However, that's mounted at `~/Dropbox/Apps/<my app name>/blog`.

    Hopefully this helps / commiserations!

    [1]: https://github.com/chrisdickinson/neversaw.us/blob/latest/bin/download_folder.sh

  5. John Wilson says:

    I just want to commiserate: dropbox is indeed vile and they have not done a single fucking thing in the past decade to make any aspect of any experience they offer better in any dimension

  6. jghaines says:

    how about one of the third party clients? I don’t know about Linux, but there are decent Mac ones.

  7. sys)~> says:

    I feel sorry for you

  8. Jeroen Baert says:

    I have used rclone in the past to grab folders.

    The official dropbox headless (!) linux binary also does the job, and it allows you to exclude folders, so you could narrow it down to hold just a local copy of the specific folder you want.

    No package install needed, it's a binary that runs from your homedir and is controlled by a python script.

    If you're anxious about telemetry, you can launch it, check for the "sync" status, then shut it down again when it's done.

    • Jeroen Baert says:

      check the last paragraph on this page: https://www.dropbox.com/install-linux

      Don't use the DEB or RPM packages, just the daemon. You can check the install script yourself, it just puts it in .dropboxd

      It is what I'm currently using to make a backup of my family's shared dropbox content. I rsync the shared folder periodically to a safe, versioned location.

      • jwz says:

        Everyone suggesting "here's a 3rd party reimplementation of the Dropbox daemon and GUI that populates ~/Dropbox/" is missing the fact that Dropbox share links, the entire point of this exercise, don't show up there at all.

        • Jeroen Baert says:

          sorry, I mistakenly thought it was from your own account. A possible workaround: create a dummy Dropbox account and force the other party to share with you? Probably also not feasible, since it sounds like the tool is not in your control :(

          Other idea is using Selenium to manipulate a spun up headless browser to trigger the zip download

  9. maestral was pretty inoffensive when I last used it https://www.maestral.app/ could be less work than doing a cutdown client?

  10. Paul Roub says:

    Yes, and I initially hit the same error you're seeing. I updated the permissions, same error.

    Then I re-created the Token and all was well. I imagine you've already tried this, but since you didn't explicitly mention the order of permissions-vs-token-creation, it was at least worth mentioning.

  11. Kyzer says:

    Disclaimer. I have never used Dropbox. However, I am keenly aware of Oauth2 fuckery. It's a mess.

    As every API implements a different subset of OAuth, you quickly get into a situation where you are forced to read their long pages of OAuth docs in detail

    You don't actually mention what scopes you specified in the authorize call, soooo.... when calling /oauth/authorize to get the token, did you include this "files.metadata.read" scope in the scope parameter? If not, the token it gives you won't let the API know you're allowed. You will get a valid token. It will not just give you all the permissions your app is allowed to request. It will only give you permissions you're allowed and you ask for. Permissions for what you're allowed to do are signed right into the token, and if you don't ask, you dont get.

    "I am this app, I would like a token to do NOTHING, please" -> "OK, here you go buddy! Success!"

    "Hello API, I have this token..." -> "FUCK OFF"

    You need to say "I am this app, I would like a token that lets me do THING_I_WANT, please". Add a scope.

    • jwz says:

      I think you're right, I needed to re-issue my session token after clicking the boxes.

      After doing that I'm able to get metadata on things in my own dropbox account with e.g. {"path":"/thing.png"} but I still can't figure out how to refer to files that exist outside of my own folder like the /sh/ link of interest.

      • jwz says:

        While {"path":"id:XXXXXXXXX"} works on my own file, I was able to find the "file_id" of a file within the /sh/ tree, and I cannot access that file by id.

        Continuing to suspect that the Dropbox API only lets you touch your own copies of files, not files shared with you from someone else's account.

        • Kyzer says:

          I have now looked at the dropbox API site, which, again, is the most I have ever used dropbox.


          >  You can use a shared link to retrieve metadata by calling /sharing/get_shared_link_metadata

          In that example the shared link being queried is of the format "/s/". In the overall API document, the only links with "/sh/" are seen in the output of /create_shared_link_with_settings, /get_shared_link_file, /get_shared_link_metadata and /modify_shared_link_settings, so I'm hoping that "/s/" and "/sh/" are just different versions of the same conceptual thing, a shared link, and not entirely different concepts and somehow /get_shared_link_metadata won't work with your "/sh/" URL

          • jwz says:

            Thanks, I'm making progress on this. It's even more horrible than OAuth things usually are. (E.g. shared folders have both "folder IDs" and "folder IDs" and they are not the same thing.)

            Anyway, my current struggle is, how do I either get a long-lived access token, or refresh the one that I have? The API seems to want a token before I can refresh it.

            Poking around, it seems that this will have been the fourth time in my life that I have had to implement OAuth token refreshing, and each of those sites does it a completely different way.

            • Kyzer says:

              Ask for the authorization code  with token_access_type=offline (the incantaion needed to get one of these varies between providers because the standard doesn't make them do it uniformly, this appears to be the dropbox incantation), then call  /oauth2/token?grant_type=authorization_code&code=..., and it will give you _two_ codes in the response.

              One is  access_token, short lived, you can use now.

              The other is refresh_token, one-time-use, but you don't have to use for a long time (for some providers, you can go a year or more without using it and it'll still be valid. They can change it at any time. They're meant to say in their documentation. The API itself will never say, it only says how long the access token lasts).

              When the access_token expires, call /oauth2/token?grant_type=refresh_token&refresh_token=... with the refresh token, and it'll issue you a new access_token and refresh_token. When that access token expires, use the new refresh token, and so on.

              • jwz says:

                I still can't tell how I'm supposed to do that. Loading /oauth2/authorize?client_id=APP_ID&token_access_type=offline&response_type=code in a browser makes me click authorize and then says "enter this code in the app" and returns a 43 character code that does not work as an access token. The access tokens that worked were 143 characters.

                Posting that to "/oauth2/token grant_type=refresh_token refresh_token=...that... client_id=... client_secret=..." says "refresh token is malformed".

                • Kyzer says:

                  You call /oauth2/authorize like that. The code is your authorization code, let's say it's 12345. I'm suprised it lets you call without a callback url, but hooray that it does.

                  You then must POST /oauth2/token grant_type=authorization_code code=12345 client_id=... client_secret=... Because you asked for offline in the authorization call, you should get a response with at least access_token and refresh_token. eg. {"access_token":"HUGALUGA", "refresh_token":HERPDERP"}

                  You now use the access_token on all the other API calls. Authorization: Bearer HUGALUGA

                  When that expires, you need another access_token. POST /oauth2/token grant_type=refresh_token refresh_token=HERPDERP client_id=... client_secret=... and it should issue another access_token. Keep doing that.

            • Kyzer says:

              Me: and you'll get a new refresh token with each refresh

              Dropbox: This request won't return a new refresh token since refresh tokens don't expire automatically and can be reused repeatedly

              Well so much for goddamn standards. It looks like you get to reuse the same refresh token indefinitely each time you need a new access token (until they feel like expiring the refresh token, at which point you'll need to login again with authorize)

            • cdavies says:

              You have to pass token_access_type=offline to the /oath/authorize request to get a refresh token from the /oauth/token request. Otherwise you have to go through the user interactive flow every time.

  12. Daniel Toman says:

    I've been quite enjoying Cyberduck as an FTP-style GUI for Dropbox and Google Drive. There's also Mountain Duck if you want to interact with them as mounted network volumes.

  13. astonishingriverboat says:

    I have also been forced to use Dropbox for one specific use case of mine, for some time now. And I'm pretty happy using https://maestral.app/.

    • Jonathan says:

      I also thought about this, since they have a robust command line interface... maybe that could abstract away the whole authentication flow mess, but it sounds like jwz is already pretty deep into it now...

      • jwz says:

        Nobody has provided any evidence that some other app's supposed command line utilities would make any of this easier. Or even that they work at all with shared links. Plus, all of them require me to do all of this OAuth crap first. So, this is the "I googled a thing, maybe you should burn it all down and start from scratch" answer.

  14. Grey Hodge says:

    Just out of curiosity, what don't you like about DB?

    • jwz says:
      • It's a fucking rootkit.
      • It has a history of security problems.
      • It's a perpetual fucking spam vector.
      • It insinuates itself into the Finder GUI in ways that are impossible to disable.
      • It provides nothing of value to me in any way, but every now and then I encounter some other thing that has decided to have Dropbox as a hard dependency.

Leave a Reply

Your email address will not be published. But if you provide a fake email address, I will likely assume that you are a troll, and not publish your comment.

Starting a new top-level thread.

  • Previously